2024-06-27T20:51:25.707 INFO:root:teuthology version: 0.0.1.dev329+g544fecb 2024-06-27T20:51:25.707 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} --name yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508 --verbose -- /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/orig.config.yaml 2024-06-27T20:51:25.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-27T20:51:25.840 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508 branch: wip-yuri12-testing-2024-06-26-0904 description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7775508' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri12-testing-2024-06-26-0904 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: f51acc18cc950586c82c58d666661d4391fc1122 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f51acc18cc950586c82c58d666661d4391fc1122 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: min_in: 4 workunit: branch: wip-yuri12-testing-2024-06-26-0904 sha1: f51acc18cc950586c82c58d666661d4391fc1122 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 9950 sha1: f51acc18cc950586c82c58d666661d4391fc1122 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri12-testing-2024-06-26-0904 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f51acc18cc950586c82c58d666661d4391fc1122 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4faC3FrEFGjS4zSpNSmYRjBOXGoOT2wHxGw7Q+kmVNtsS8B5XmKmlC8mdSbFTjh1bp2zI/H0y0W2vk7yium+g= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFuRh3bDR5JfHsF1FhGUa9O7oi2w76/itKJrXEP8cH5QKSJmSDLFyZob+l6GHzdlTiiDlpVQDsgDnyitnvQQo4k= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN25x+hWpZrHlv+4VtGDE/zBi05VeygWOm7WUug3TSRYgvkRJsqSh3rRaUg8h6Ao+j6h8Vkh6JE8jOX8mtWEaYM= tasks: - install: branch: squid exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 544fecbcd55f3d2b6f478823823ce40cbefef1d4 timestamp: 2024-06-27_14:07:21 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698093 2024-06-27T20:51:25.840 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa; will attempt to use it 2024-06-27T20:51:25.841 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f51acc18cc950586c82c58d666661d4391fc1122/qa/tasks 2024-06-27T20:51:25.841 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-27T20:51:25.843 INFO:teuthology.task.internal:Saving configuration 2024-06-27T20:51:25.858 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-27T20:51:25.865 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-27T20:51:25.889 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-27 20:46:25.776257', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4faC3FrEFGjS4zSpNSmYRjBOXGoOT2wHxGw7Q+kmVNtsS8B5XmKmlC8mdSbFTjh1bp2zI/H0y0W2vk7yium+g='} 2024-06-27T20:51:25.913 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-27 20:46:25.777746', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFuRh3bDR5JfHsF1FhGUa9O7oi2w76/itKJrXEP8cH5QKSJmSDLFyZob+l6GHzdlTiiDlpVQDsgDnyitnvQQo4k='} 2024-06-27T20:51:25.936 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-27 20:46:25.779137', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN25x+hWpZrHlv+4VtGDE/zBi05VeygWOm7WUug3TSRYgvkRJsqSh3rRaUg8h6Ao+j6h8Vkh6JE8jOX8mtWEaYM='} 2024-06-27T20:51:25.936 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-27T20:51:25.943 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-06-27T20:51:25.943 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-06-27T20:51:25.943 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-06-27T20:51:25.943 INFO:teuthology.run_tasks:Running task console_log... 2024-06-27T20:51:26.040 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f295bb4fbe0>, signals=[15]) 2024-06-27T20:51:26.040 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-27T20:51:26.046 INFO:teuthology.task.internal:Opening connections... 2024-06-27T20:51:26.046 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-06-27T20:51:26.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:51:26.123 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-06-27T20:51:26.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:51:26.196 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-06-27T20:51:26.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:51:26.274 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-27T20:51:26.281 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-06-27T20:51:26.299 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-06-27T20:51:26.299 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-27T20:51:26.356 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-06-27T20:51:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-06-27T20:51:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-27T20:51:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-06-27T20:51:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-27T20:51:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-27T20:51:26.357 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-27T20:51:26.357 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-06-27T20:51:26.372 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-06-27T20:51:26.389 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-06-27T20:51:26.390 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-06-27T20:51:26.447 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-27T20:51:26.448 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-06-27T20:51:26.448 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-27T20:51:26.448 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-27T20:51:26.448 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-27T20:51:26.448 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-06-27T20:51:26.472 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-06-27T20:51:26.489 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-06-27T20:51:26.489 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-06-27T20:51:26.546 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-06-27T20:51:26.546 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-06-27T20:51:26.546 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-06-27T20:51:26.546 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-06-27T20:51:26.546 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-06-27T20:51:26.546 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-27T20:51:26.547 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-27T20:51:26.547 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-06-27T20:51:26.572 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-27T20:51:26.581 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-27T20:51:26.598 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-27T20:51:26.598 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-06-27T20:51:26.600 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-06-27T20:51:26.602 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-06-27T20:51:26.618 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-27T20:51:26.624 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-27T20:51:26.624 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-06-27T20:51:26.657 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-06-27T20:51:26.660 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-06-27T20:51:26.710 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-27T20:51:26.787 INFO:teuthology.run_tasks:Running task kernel... 2024-06-27T20:51:26.801 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-27T20:51:26.801 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-27T20:51:26.801 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-27T20:51:26.802 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-27T20:51:26.802 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-27T20:51:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T20:51:26.818 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-06-27T20:51:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T20:51:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T20:51:26.820 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-06-27T20:51:26.820 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-06-27T20:51:26.875 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-437.el9.x86_64 2024-06-27T20:51:26.875 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-437.el9.x86_64 2024-06-27T20:51:26.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-06-27T20:51:26.876 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-437.el9.x86_64 2024-06-27T20:51:26.876 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-437.el9.x86_64 2024-06-27T20:51:26.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-06-27T20:51:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-437.el9.x86_64 2024-06-27T20:51:26.878 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-437.el9.x86_64 2024-06-27T20:51:26.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-06-27T20:51:29.975 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.1 MB 00:01 2024-06-27T20:51:30.026 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.1 MB 00:01 2024-06-27T20:51:31.480 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.1 MB 00:02 2024-06-27T20:51:31.690 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 35 MB/s | 20 MB 00:00 2024-06-27T20:51:37.922 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 3.7 MB/s | 20 MB 00:05 2024-06-27T20:51:38.032 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.4 MB 00:02 2024-06-27T20:51:40.117 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 17 kB 00:00 2024-06-27T20:51:41.337 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-06-27T20:51:42.997 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.4 MB 00:01 2024-06-27T20:51:44.606 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 17 kB 00:00 2024-06-27T20:51:45.156 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 20 MB 00:13 2024-06-27T20:51:45.485 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-06-27T20:51:46.660 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-06-27T20:51:48.740 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-27T20:51:48.741 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-27T20:51:48.837 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T20:51:48.842 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T20:51:48.842 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-06-27T20:51:48.842 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T20:51:48.842 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T20:51:48.842 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-06-27T20:51:48.843 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T20:51:48.844 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 82 M 2024-06-27T20:51:48.844 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 125 M 2024-06-27T20:51:48.844 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T20:51:50.273 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.4 MB 00:01 2024-06-27T20:51:50.310 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 988 kB/s | 1.2 MB 00:01 2024-06-27T20:51:50.708 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-06-27T20:51:50.986 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 8.7 MB/s | 16 MB 00:01 2024-06-27T20:51:51.678 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 14 MB/s | 35 MB 00:02 2024-06-27T20:51:51.699 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 17 kB 00:00 2024-06-27T20:51:52.519 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 32 MB/s | 21 MB 00:00 2024-06-27T20:51:52.719 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-467.el9.x86_6 12 MB/s | 29 MB 00:02 2024-06-27T20:51:52.719 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T20:51:52.720 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 82 MB 00:03 2024-06-27T20:51:52.730 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-27T20:51:52.730 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-27T20:51:52.826 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T20:51:52.838 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T20:51:52.839 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T20:51:52.839 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-06-27T20:51:52.839 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T20:51:52.839 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 82 M 2024-06-27T20:51:52.839 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 125 M 2024-06-27T20:51:52.840 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T20:51:53.292 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 3.7 MB/s | 1.2 MB 00:00 2024-06-27T20:51:53.430 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T20:51:53.785 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T20:51:53.785 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T20:51:54.738 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T20:51:54.739 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T20:51:57.161 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T20:51:57.259 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 3.8 MB/s | 16 MB 00:04 2024-06-27T20:51:57.717 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-06-27T20:51:57.809 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-27T20:51:57.903 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:51:59.789 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-27T20:51:59.790 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-27T20:51:59.891 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T20:51:59.899 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T20:51:59.899 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-467.el9 baseos 1.2 M 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-467.el9 baseos 16 M 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-467.el9 baseos 35 M 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-467.el9 baseos 29 M 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-06-27T20:51:59.900 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T20:51:59.901 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 82 M 2024-06-27T20:51:59.901 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 125 M 2024-06-27T20:51:59.901 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T20:52:00.451 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-467.el9.x86_64.rpm 4.2 MB/s | 1.2 MB 00:00 2024-06-27T20:52:00.589 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:00.724 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:05.009 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-core-5.14.0-467.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-06-27T20:52:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:05.565 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:07.566 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-core-5.14.0-467.el9.x86_6 2.1 MB/s | 29 MB 00:14 2024-06-27T20:52:09.318 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-modules-core-5.14.0-467.el9.x86_6 3.3 MB/s | 29 MB 00:08 2024-06-27T20:52:10.391 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:10.407 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 2.0 MB/s | 35 MB 00:17 2024-06-27T20:52:10.408 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T20:52:10.408 INFO:teuthology.orchestra.run.smithi190.stdout:Total 4.7 MB/s | 82 MB 00:17 2024-06-27T20:52:10.576 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-5.14.0-467.el9.x86_64.rpm 3.4 MB/s | 35 MB 00:10 2024-06-27T20:52:10.576 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T20:52:10.576 INFO:teuthology.orchestra.run.smithi192.stdout:Total 7.7 MB/s | 82 MB 00:10 2024-06-27T20:52:11.123 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T20:52:11.292 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T20:52:11.487 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T20:52:11.487 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T20:52:11.632 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T20:52:11.632 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T20:52:12.473 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T20:52:12.474 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T20:52:12.613 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T20:52:12.613 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T20:52:14.899 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T20:52:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T20:52:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-27T20:52:15.674 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:15.779 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-467.el9.x86_64 1/4 2024-06-27T20:52:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:18.393 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:18.542 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:18.609 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:18.747 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:23.551 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:23.603 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:28.214 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:39.842 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:39.882 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:40.221 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:40.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-27T20:52:40.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:40.222 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:41.019 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:41.019 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T20:52:41.019 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T20:52:41.019 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:41.019 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-27T20:52:41.020 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-27T20:52:41.020 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-27T20:52:41.020 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T20:52:41.020 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T20:52:41.392 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-06-27T20:52:41.472 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:41.472 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-27T20:52:41.472 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-27T20:52:41.473 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-27T20:52:41.473 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-27T20:52:41.473 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-27T20:52:41.473 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-06-27T20:52:41.473 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-06-27T20:52:41.526 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:41.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-27T20:52:41.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-27T20:52:41.563 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T20:52:41.564 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-27T20:52:41.565 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-27T20:52:41.565 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-27T20:52:41.565 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-06-27T20:52:41.565 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T20:52:41.565 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-06-27T20:52:41.566 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-27T20:52:41.566 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-467.el9.x86_64 2024-06-27T20:52:41.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-27T20:52:42.198 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-06-27T20:52:43.609 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-27T20:52:43.651 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-06-27T20:52:43.653 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-06-27T20:52:43.668 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.S1Z2viLmZV 2024-06-27T20:52:43.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.S1Z2viLmZV 2024-06-27T20:52:43.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.S1Z2viLmZV 2024-06-27T20:52:43.873 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.S1Z2viLmZV is 6KB 2024-06-27T20:52:43.921 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.S1Z2viLmZV 2024-06-27T20:52:43.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-06-27T20:52:43.998 INFO:teuthology.orchestra.run.smithi151.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-27T20:52:43.998 INFO:teuthology.orchestra.run.smithi151.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-27T20:52:43.998 INFO:teuthology.orchestra.run.smithi151.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-27T20:52:43.998 INFO:teuthology.orchestra.run.smithi151.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-27T20:52:43.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-27T20:52:44.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-06-27T20:52:57.820 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:57.859 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:58.034 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:58.076 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:58.191 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:58.192 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-27T20:52:58.192 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:58.192 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:58.446 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:58.446 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-467.el9.x86_64 1/4 2024-06-27T20:52:58.446 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-467.el9.x86_64 2/4 2024-06-27T20:52:58.446 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-467.el9.x86_64 3/4 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-27T20:52:58.978 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T20:52:58.979 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T20:52:59.254 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-467.el9.x86_64 4/4 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T20:52:59.255 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T20:52:59.399 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-06-27T20:52:59.466 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.468 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.468 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.468 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-27T20:52:59.468 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-27T20:52:59.468 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-27T20:52:59.468 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-06-27T20:52:59.468 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-06-27T20:52:59.522 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.522 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-06-27T20:52:59.552 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-06-27T20:52:59.552 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-27T20:52:59.553 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T20:52:59.554 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-06-27T20:52:59.555 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-27T20:52:59.555 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-27T20:52:59.634 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-06-27T20:52:59.708 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.709 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.709 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.709 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-27T20:52:59.709 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-27T20:52:59.710 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-06-27T20:52:59.710 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-06-27T20:52:59.710 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-06-27T20:52:59.764 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-467.el9.x86_64 2024-06-27T20:52:59.764 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 68.el9 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8264254 2024-06-27T20:52:59.801 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-06-27T20:52:59.802 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T20:52:59.803 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-27T20:52:59.803 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-27T20:52:59.803 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-27T20:52:59.803 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-06-27T20:52:59.803 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T20:52:59.803 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-06-27T20:52:59.804 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-27T20:52:59.804 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-467.el9.x86_64 2024-06-27T20:52:59.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-27T20:53:00.240 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-06-27T20:53:00.415 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-06-27T20:53:01.677 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-27T20:53:01.708 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-06-27T20:53:01.710 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-06-27T20:53:01.765 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.Xu4B4WSSDp 2024-06-27T20:53:01.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Xu4B4WSSDp 2024-06-27T20:53:01.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.Xu4B4WSSDp 2024-06-27T20:53:01.945 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-27T20:53:01.980 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.Xu4B4WSSDp is 6KB 2024-06-27T20:53:01.982 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-06-27T20:53:01.984 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-06-27T20:53:02.030 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.Xu4B4WSSDp 2024-06-27T20:53:02.039 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.fkBvMOSejb 2024-06-27T20:53:02.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fkBvMOSejb 2024-06-27T20:53:02.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-06-27T20:53:02.105 INFO:teuthology.orchestra.run.smithi190.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-27T20:53:02.105 INFO:teuthology.orchestra.run.smithi190.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-27T20:53:02.105 INFO:teuthology.orchestra.run.smithi190.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-27T20:53:02.105 INFO:teuthology.orchestra.run.smithi190.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-27T20:53:02.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-27T20:53:02.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.fkBvMOSejb 2024-06-27T20:53:02.249 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.fkBvMOSejb is 6KB 2024-06-27T20:53:02.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-06-27T20:53:02.297 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.fkBvMOSejb 2024-06-27T20:53:02.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-06-27T20:53:02.373 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-27T20:53:02.373 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-27T20:53:02.373 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-27T20:53:02.373 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64.conf 2024-06-27T20:53:02.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-467.el9.x86_64 2024-06-27T20:53:02.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-06-27T20:53:14.170 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-27T20:53:14.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T20:53:14.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:53:32.269 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-27T20:53:32.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-06-27T20:53:32.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:53:32.531 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-06-27T20:53:32.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-27T20:53:32.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:53:48.703 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-06-27T20:53:50.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-06-27T20:53:51.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-06-27T20:53:57.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T20:53:57.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:53:59.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-06-27T20:53:59.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:54:00.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-27T20:54:00.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:54:02.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-06-27T20:54:03.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-06-27T20:54:14.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-06-27T20:54:14.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:54:15.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-27T20:54:15.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:54:16.098 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-06-27T20:54:28.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T20:54:28.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:54:31.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-06-27T20:54:46.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T20:54:46.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:54:46.609 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-06-27T20:54:46.928 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-06-27T20:54:47.043 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-06-27T20:54:47.043 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-467.el9.x86_64"... 2024-06-27T20:54:47.044 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-06-27T20:54:47.059 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-467.el9.x86_64 2024-06-27T20:54:47.060 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-27T20:54:47.060 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-27T20:54:47.060 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-27T20:54:47.304 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-27T20:54:47.304 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-467.el9.x86_64"... 2024-06-27T20:54:47.304 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-06-27T20:54:47.318 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-467.el9.x86_64 2024-06-27T20:54:47.319 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-27T20:54:47.319 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-27T20:54:47.319 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-27T20:54:48.061 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-27T20:54:48.061 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-27T20:54:48.133 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-06-27T20:54:48.177 DEBUG:teuthology.parallel:result is None 2024-06-27T20:54:48.319 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-06-27T20:54:48.320 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-27T20:54:48.404 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-06-27T20:54:48.452 DEBUG:teuthology.parallel:result is None 2024-06-27T20:54:49.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-06-27T20:55:07.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T20:55:07.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T20:55:07.602 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-06-27T20:55:08.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T20:55:08.076 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-467.el9.x86_64"... 2024-06-27T20:55:08.076 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-06-27T20:55:08.123 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-467.el9.x86_64 2024-06-27T20:55:08.123 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-467.el9.x86_64 vs 5.14.0-467.el9.x86_64 2024-06-27T20:55:08.123 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-27T20:55:08.123 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-27T20:55:09.124 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-27T20:55:09.125 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-27T20:55:09.198 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-06-27T20:55:09.247 DEBUG:teuthology.parallel:result is None 2024-06-27T20:55:09.248 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-27T20:55:09.278 INFO:teuthology.task.internal:Creating test directory... 2024-06-27T20:55:09.278 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-27T20:55:09.281 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-27T20:55:09.284 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-27T20:55:09.303 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-27T20:55:09.323 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-27T20:55:09.356 INFO:teuthology.task.internal:Creating archive directory... 2024-06-27T20:55:09.356 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-27T20:55:09.359 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-27T20:55:09.361 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-27T20:55:09.398 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-27T20:55:09.410 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-27T20:55:09.410 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-27T20:55:09.418 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-27T20:55:09.433 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-27T20:55:09.456 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T20:55:09.466 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T20:55:09.470 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T20:55:09.482 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T20:55:09.504 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T20:55:09.516 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-27T20:55:09.518 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-27T20:55:09.532 INFO:teuthology.task.internal:Configuring sudo... 2024-06-27T20:55:09.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-27T20:55:09.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-27T20:55:09.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-27T20:55:09.563 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-27T20:55:09.577 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-27T20:55:09.577 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-27T20:55:09.599 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-27T20:55:09.605 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-27T20:55:09.623 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T20:55:09.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T20:55:09.741 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T20:55:25.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T20:55:25.659 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T20:55:25.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-27T20:55:25.728 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T20:55:25.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T20:55:25.837 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T20:55:25.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T20:55:25.924 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T20:55:25.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-27T20:55:25.990 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T20:55:26.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-27T20:55:26.092 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T20:55:26.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-27T20:55:26.219 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T20:55:26.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-27T20:55:26.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-06-27T20:55:26.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-06-27T20:55:26.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-06-27T20:55:26.332 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-27T20:55:26.339 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-27T20:55:26.373 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-27T20:55:26.821 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-27T20:55:26.829 INFO:teuthology.task.internal:Starting timer... 2024-06-27T20:55:26.830 INFO:teuthology.run_tasks:Running task pcp... 2024-06-27T20:55:26.855 INFO:teuthology.run_tasks:Running task selinux... 2024-06-27T20:55:26.891 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-27T20:55:26.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-06-27T20:55:26.985 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-06-27T20:55:26.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-06-27T20:55:27.047 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-06-27T20:55:27.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-06-27T20:55:27.117 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-06-27T20:55:27.119 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-27T20:55:27.119 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-06-27T20:55:27.174 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-06-27T20:55:27.174 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-06-27T20:55:27.204 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-06-27T20:55:27.205 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-06-27T20:55:27.254 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-06-27T20:55:27.254 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-06-27T20:55:27.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-27T20:55:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T20:55:27.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-27T20:55:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T20:55:27.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-27T20:55:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T20:55:27.344 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-27T20:55:27.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-06-27T20:55:27.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-06-27T20:55:27.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-06-27T20:55:27.443 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-27T20:55:27.456 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-27T20:55:27.468 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-27T20:55:27.469 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi190.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-27T21:00:01.023 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-06-27T21:00:01.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T21:00:01.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:00:01.105 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-06-27T21:00:01.183 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-06-27T21:00:01.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-06-27T21:00:01.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:00:01.256 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-06-27T21:00:01.335 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-06-27T21:00:01.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-27T21:00:01.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-27T21:00:01.411 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-06-27T21:00:01.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-27T21:00:01.496 INFO:teuthology.run_tasks:Running task clock... 2024-06-27T21:00:01.507 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-27T21:00:01.507 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-27T21:00:01.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T21:00:01.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-27T21:00:01.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T21:00:01.512 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-27T21:00:01.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T21:00:01.538 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-27T21:00:01.541 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-27T21:00:01.552 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-27T21:00:01.555 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-27T21:00:01.567 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-27T21:00:01.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-06-27T21:00:01.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-06-27T21:00:01.582 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-27T21:00:01.588 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-06-27T21:00:01.591 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-06-27T21:00:01.603 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-27T21:00:01.606 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-27T21:00:01.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-06-27T21:00:01.617 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-27T21:00:01.619 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-06-27T21:00:01.620 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-27T21:00:01.633 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-27T21:00:01.647 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-27T21:00:01.675 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-06-27T21:00:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-27T21:00:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-06-27T21:00:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.679 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.680 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-06-27T21:00:01.684 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-27T21:00:01.684 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-06-27T21:00:01.684 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.685 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.685 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.685 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.704 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-06-27T21:00:01.707 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-27T21:00:01.707 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-06-27T21:00:01.707 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.707 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.707 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.707 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T21:00:01.708 INFO:teuthology.run_tasks:Running task install... 2024-06-27T21:00:01.716 DEBUG:teuthology.task.install:project ceph 2024-06-27T21:00:01.716 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'}} 2024-06-27T21:00:01.716 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'} 2024-06-27T21:00:01.716 INFO:teuthology.task.install:Using flavor: default 2024-06-27T21:00:01.723 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-27T21:00:01.723 INFO:teuthology.task.install:extra packages: [] 2024-06-27T21:00:01.724 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122', 'tag': None, 'wait_for_package': False} 2024-06-27T21:00:01.724 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:00:01.724 INFO:teuthology.packaging:ref: None 2024-06-27T21:00:01.724 INFO:teuthology.packaging:tag: None 2024-06-27T21:00:01.724 INFO:teuthology.packaging:branch: squid 2024-06-27T21:00:01.724 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:00:01.725 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-27T21:00:01.727 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122', 'tag': None, 'wait_for_package': False} 2024-06-27T21:00:01.727 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:00:01.727 INFO:teuthology.packaging:ref: None 2024-06-27T21:00:01.727 INFO:teuthology.packaging:tag: None 2024-06-27T21:00:01.727 INFO:teuthology.packaging:branch: squid 2024-06-27T21:00:01.727 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:00:01.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-27T21:00:01.728 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122', 'tag': None, 'wait_for_package': False} 2024-06-27T21:00:01.728 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:00:01.728 INFO:teuthology.packaging:ref: None 2024-06-27T21:00:01.728 INFO:teuthology.packaging:tag: None 2024-06-27T21:00:01.728 INFO:teuthology.packaging:branch: squid 2024-06-27T21:00:01.728 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:00:01.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-27T21:00:01.933 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/ 2024-06-27T21:00:01.934 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3315.g98c31c1a 2024-06-27T21:00:02.022 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/ 2024-06-27T21:00:02.022 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3315.g98c31c1a 2024-06-27T21:00:02.035 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/ 2024-06-27T21:00:02.036 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3315.g98c31c1a 2024-06-27T21:00:02.105 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-27T21:00:02.106 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:00:02.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-27T21:00:02.132 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-27T21:00:02.132 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:00:02.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-27T21:00:02.133 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-27T21:00:02.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:00:02.133 INFO:teuthology.packaging:ref: None 2024-06-27T21:00:02.133 INFO:teuthology.packaging:tag: None 2024-06-27T21:00:02.134 INFO:teuthology.packaging:branch: squid 2024-06-27T21:00:02.134 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:00:02.134 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-27T21:00:02.141 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/98c31c1ac4a1afb24e2179b554f89c5e40efdfb0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-27T21:00:02.141 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:00:02.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-27T21:00:02.160 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-27T21:00:02.160 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:00:02.160 INFO:teuthology.packaging:ref: None 2024-06-27T21:00:02.160 INFO:teuthology.packaging:tag: None 2024-06-27T21:00:02.160 INFO:teuthology.packaging:branch: squid 2024-06-27T21:00:02.160 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:00:02.160 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-27T21:00:02.167 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-27T21:00:02.167 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:00:02.167 INFO:teuthology.packaging:ref: None 2024-06-27T21:00:02.167 INFO:teuthology.packaging:tag: None 2024-06-27T21:00:02.167 INFO:teuthology.packaging:branch: squid 2024-06-27T21:00:02.167 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:00:02.167 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-27T21:00:02.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-27T21:00:02.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-27T21:00:02.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-27T21:00:02.274 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-27T21:00:02.303 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-27T21:00:02.306 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-27T21:00:02.339 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-06-27T21:00:02.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-06-27T21:00:02.367 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-06-27T21:00:02.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-06-27T21:00:02.371 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-06-27T21:00:02.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-06-27T21:00:02.579 INFO:teuthology.orchestra.run.smithi151.stdout:45 files removed 2024-06-27T21:00:02.595 INFO:teuthology.orchestra.run.smithi192.stdout:45 files removed 2024-06-27T21:00:02.605 INFO:teuthology.orchestra.run.smithi190.stdout:45 files removed 2024-06-27T21:00:02.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-06-27T21:00:02.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-06-27T21:00:02.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-06-27T21:00:03.274 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 222 kB/s | 83 kB 00:00 2024-06-27T21:00:03.316 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 195 kB/s | 83 kB 00:00 2024-06-27T21:00:03.375 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-06-27T21:00:03.708 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 41 kB/s | 12 kB 00:00 2024-06-27T21:00:03.716 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-06-27T21:00:03.741 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-06-27T21:00:04.109 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-06-27T21:00:04.125 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-06-27T21:00:04.134 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-06-27T21:00:04.884 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-06-27T21:00:05.603 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.1 MB 00:01 2024-06-27T21:00:05.619 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.1 MB 00:01 2024-06-27T21:00:08.485 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 20 MB 00:02 2024-06-27T21:00:09.351 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 20 MB 00:02 2024-06-27T21:00:13.902 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.4 MB 00:00 2024-06-27T21:00:14.036 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.4 MB 00:01 2024-06-27T21:00:14.801 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 20 MB 00:08 2024-06-27T21:00:15.845 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 17 kB 00:00 2024-06-27T21:00:16.344 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 17 kB 00:01 2024-06-27T21:00:16.841 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-06-27T21:00:17.382 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-06-27T21:00:19.368 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.4 MB 00:00 2024-06-27T21:00:20.994 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 17 kB 00:00 2024-06-27T21:00:22.130 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-06-27T21:00:22.199 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-06-27T21:00:22.763 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-06-27T21:00:24.605 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:00:24.619 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:00:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-06-27T21:00:24.619 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 10 M 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.4 M 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.1 M 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 5.4 M 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 21 M 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 25 k 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-27T21:00:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 749 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 504 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 5.3 M 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 45 k 2024-06-27T21:00:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 135 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 162 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 320 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 301 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 99 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-27T21:00:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout:Install 30 Packages 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 60 M 2024-06-27T21:00:24.623 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:00:25.229 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:00:25.264 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:00:25.264 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 10 M 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.4 M 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.1 M 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 5.4 M 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 21 M 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 25 k 2024-06-27T21:00:25.265 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 749 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 504 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 5.3 M 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-27T21:00:25.266 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 45 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 135 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 162 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 320 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 301 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 99 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-06-27T21:00:25.267 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout:Install 30 Packages 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:25.268 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 60 M 2024-06-27T21:00:25.269 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:00:25.881 INFO:teuthology.orchestra.run.smithi151.stdout:(1/32): ceph-radosgw-19.0.0-3315.g98c31c1a.el9. 10 MB/s | 10 MB 00:01 2024-06-27T21:00:26.014 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-base-19.0.0-3315.g98c31c1a.el9.x86 8.6 MB/s | 5.4 MB 00:00 2024-06-27T21:00:26.040 INFO:teuthology.orchestra.run.smithi151.stdout:(2/32): ceph-base-19.0.0-3315.g98c31c1a.el9.x86 4.6 MB/s | 5.4 MB 00:01 2024-06-27T21:00:26.057 INFO:teuthology.orchestra.run.smithi151.stdout:(3/32): ceph-selinux-19.0.0-3315.g98c31c1a.el9. 143 kB/s | 25 kB 00:00 2024-06-27T21:00:26.064 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-selinux-19.0.0-3315.g98c31c1a.el9. 503 kB/s | 25 kB 00:00 2024-06-27T21:00:26.215 INFO:teuthology.orchestra.run.smithi151.stdout:(4/32): libcephfs2-19.0.0-3315.g98c31c1a.el9.x8 4.2 MB/s | 749 kB 00:00 2024-06-27T21:00:26.223 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): libcephfs2-19.0.0-3315.g98c31c1a.el9.x8 4.6 MB/s | 749 kB 00:00 2024-06-27T21:00:26.249 INFO:teuthology.orchestra.run.smithi151.stdout:(5/32): libradosstriper1-19.0.0-3315.g98c31c1a. 2.6 MB/s | 504 kB 00:00 2024-06-27T21:00:26.298 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): libradosstriper1-19.0.0-3315.g98c31c1a. 6.6 MB/s | 504 kB 00:00 2024-06-27T21:00:26.516 INFO:teuthology.orchestra.run.smithi151.stdout:(6/32): python3-ceph-argparse-19.0.0-3315.g98c3 169 kB/s | 45 kB 00:00 2024-06-27T21:00:26.599 INFO:teuthology.orchestra.run.smithi151.stdout:(7/32): python3-ceph-common-19.0.0-3315.g98c31c 1.6 MB/s | 135 kB 00:00 2024-06-27T21:00:26.675 INFO:teuthology.orchestra.run.smithi151.stdout:(8/32): librgw2-19.0.0-3315.g98c31c1a.el9.x86_6 12 MB/s | 5.3 MB 00:00 2024-06-27T21:00:26.700 INFO:teuthology.orchestra.run.smithi151.stdout:(9/32): python3-cephfs-19.0.0-3315.g98c31c1a.el 1.6 MB/s | 162 kB 00:00 2024-06-27T21:00:26.782 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): librgw2-19.0.0-3315.g98c31c1a.el9.x86_6 11 MB/s | 5.3 MB 00:00 2024-06-27T21:00:26.900 INFO:teuthology.orchestra.run.smithi151.stdout:(10/32): ceph-common-19.0.0-3315.g98c31c1a.el9. 10 MB/s | 21 MB 00:02 2024-06-27T21:00:26.926 INFO:teuthology.orchestra.run.smithi151.stdout:(11/32): python3-rbd-19.0.0-3315.g98c31c1a.el9. 1.3 MB/s | 301 kB 00:00 2024-06-27T21:00:26.951 INFO:teuthology.orchestra.run.smithi151.stdout:(12/32): python3-rados-19.0.0-3315.g98c31c1a.el 1.1 MB/s | 320 kB 00:00 2024-06-27T21:00:26.990 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): ceph-common-19.0.0-3315.g98c31c1a.el9.x 13 MB/s | 21 MB 00:01 2024-06-27T21:00:27.018 INFO:teuthology.orchestra.run.smithi151.stdout:(13/32): python3-rgw-19.0.0-3315.g98c31c1a.el9. 850 kB/s | 99 kB 00:00 2024-06-27T21:00:27.032 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): python3-ceph-argparse-19.0.0-3315.g98c3 180 kB/s | 45 kB 00:00 2024-06-27T21:00:27.066 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-ceph-common-19.0.0-3315.g98c31c 1.8 MB/s | 135 kB 00:00 2024-06-27T21:00:27.091 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-cephfs-19.0.0-3315.g98c31c1a.el 2.7 MB/s | 162 kB 00:00 2024-06-27T21:00:27.125 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-rados-19.0.0-3315.g98c31c1a.el 5.4 MB/s | 320 kB 00:00 2024-06-27T21:00:27.150 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-rbd-19.0.0-3315.g98c31c1a.el9. 5.0 MB/s | 301 kB 00:00 2024-06-27T21:00:27.167 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): python3-rgw-19.0.0-3315.g98c31c1a.el9. 2.3 MB/s | 99 kB 00:00 2024-06-27T21:00:27.201 INFO:teuthology.orchestra.run.smithi151.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 388 kB/s | 107 kB 00:00 2024-06-27T21:00:27.252 INFO:teuthology.orchestra.run.smithi151.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 194 kB/s | 45 kB 00:00 2024-06-27T21:00:27.277 INFO:teuthology.orchestra.run.smithi151.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 492 kB/s | 160 kB 00:00 2024-06-27T21:00:27.369 INFO:teuthology.orchestra.run.smithi151.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 461 kB/s | 42 kB 00:00 2024-06-27T21:00:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-06-27T21:00:27.451 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-06-27T21:00:27.493 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 493 kB/s | 160 kB 00:00 2024-06-27T21:00:27.527 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-06-27T21:00:27.527 INFO:teuthology.orchestra.run.smithi151.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-06-27T21:00:27.570 INFO:teuthology.orchestra.run.smithi151.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-06-27T21:00:27.603 INFO:teuthology.orchestra.run.smithi151.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 477 kB/s | 36 kB 00:00 2024-06-27T21:00:27.653 INFO:teuthology.orchestra.run.smithi151.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 568 kB/s | 47 kB 00:00 2024-06-27T21:00:27.685 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-06-27T21:00:27.710 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-06-27T21:00:27.712 INFO:teuthology.orchestra.run.smithi151.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-06-27T21:00:27.737 INFO:teuthology.orchestra.run.smithi151.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-06-27T21:00:27.761 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 560 kB/s | 42 kB 00:00 2024-06-27T21:00:27.763 INFO:teuthology.orchestra.run.smithi151.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-27T21:00:27.786 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-06-27T21:00:27.788 INFO:teuthology.orchestra.run.smithi151.stdout:(25/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2024-06-27T21:00:27.822 INFO:teuthology.orchestra.run.smithi151.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-06-27T21:00:27.828 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 537 kB/s | 36 kB 00:00 2024-06-27T21:00:27.853 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-06-27T21:00:27.872 INFO:teuthology.orchestra.run.smithi151.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.6 MB/s | 192 kB 00:00 2024-06-27T21:00:27.953 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-06-27T21:00:27.955 INFO:teuthology.orchestra.run.smithi151.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 20 MB/s | 1.6 MB 00:00 2024-06-27T21:00:27.978 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-06-27T21:00:27.997 INFO:teuthology.orchestra.run.smithi151.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-06-27T21:00:28.004 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-27T21:00:28.029 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-27T21:00:28.064 INFO:teuthology.orchestra.run.smithi151.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-27T21:00:28.096 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-06-27T21:00:28.122 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-06-27T21:00:28.188 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-06-27T21:00:28.189 INFO:teuthology.orchestra.run.smithi151.stdout:(31/32): librados2-19.0.0-3315.g98c31c1a.el9.x8 15 MB/s | 3.4 MB 00:00 2024-06-27T21:00:28.239 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-06-27T21:00:28.489 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): librados2-19.0.0-3315.g98c31c1a.el9.x8 11 MB/s | 3.4 MB 00:00 2024-06-27T21:00:28.548 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): librbd1-19.0.0-3315.g98c31c1a.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-06-27T21:00:28.765 INFO:teuthology.orchestra.run.smithi151.stdout:(32/32): librbd1-19.0.0-3315.g98c31c1a.el9.x86_ 4.1 MB/s | 3.1 MB 00:00 2024-06-27T21:00:28.770 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:00:28.770 INFO:teuthology.orchestra.run.smithi151.stdout:Total 14 MB/s | 60 MB 00:04 2024-06-27T21:00:28.823 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): ceph-radosgw-19.0.0-3315.g98c31c1a.el9 3.0 MB/s | 10 MB 00:03 2024-06-27T21:00:28.827 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:00:28.828 INFO:teuthology.orchestra.run.smithi192.stdout:Total 17 MB/s | 60 MB 00:03 2024-06-27T21:00:28.872 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:00:28.908 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:00:28.909 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:00:28.929 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:00:28.966 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:00:28.966 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:00:29.470 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:00:29.470 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:00:29.521 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:00:29.522 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:00:29.863 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 10 M 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.4 M 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.1 M 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-06-27T21:00:29.867 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 5.4 M 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 21 M 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 25 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 749 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 504 k 2024-06-27T21:00:29.868 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 5.3 M 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 45 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 135 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 162 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 320 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 301 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 99 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-27T21:00:29.869 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout:Install 30 Packages 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-06-27T21:00:29.870 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:29.871 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 60 M 2024-06-27T21:00:29.871 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:00:30.555 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-base-19.0.0-3315.g98c31c1a.el9.x86 11 MB/s | 5.4 MB 00:00 2024-06-27T21:00:30.605 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-selinux-19.0.0-3315.g98c31c1a.el9. 506 kB/s | 25 kB 00:00 2024-06-27T21:00:30.664 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): libcephfs2-19.0.0-3315.g98c31c1a.el9.x8 13 MB/s | 749 kB 00:00 2024-06-27T21:00:30.723 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): libradosstriper1-19.0.0-3315.g98c31c1a. 8.5 MB/s | 504 kB 00:00 2024-06-27T21:00:30.891 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:00:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:00:30.982 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-27T21:00:31.007 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): librgw2-19.0.0-3315.g98c31c1a.el9.x86_6 19 MB/s | 5.3 MB 00:00 2024-06-27T21:00:31.012 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-27T21:00:31.019 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-27T21:00:31.045 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-27T21:00:31.062 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-27T21:00:31.068 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-27T21:00:31.087 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-27T21:00:31.098 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-27T21:00:31.110 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-27T21:00:31.125 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): ceph-radosgw-19.0.0-3315.g98c31c1a.el9. 9.8 MB/s | 10 MB 00:01 2024-06-27T21:00:31.133 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-27T21:00:31.150 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-ceph-argparse-19.0.0-3315.g98c3 315 kB/s | 45 kB 00:00 2024-06-27T21:00:31.201 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): python3-cephfs-19.0.0-3315.g98c31c1a.el 3.1 MB/s | 162 kB 00:00 2024-06-27T21:00:31.226 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-ceph-common-19.0.0-3315.g98c31c 1.3 MB/s | 135 kB 00:00 2024-06-27T21:00:31.251 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): python3-rados-19.0.0-3315.g98c31c1a.el 6.2 MB/s | 320 kB 00:00 2024-06-27T21:00:31.294 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): python3-rgw-19.0.0-3315.g98c31c1a.el9. 2.3 MB/s | 99 kB 00:00 2024-06-27T21:00:31.300 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-27T21:00:31.345 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-27T21:00:31.379 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-27T21:00:31.402 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-27T21:00:31.578 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 376 kB/s | 107 kB 00:00 2024-06-27T21:00:31.670 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-06-27T21:00:31.726 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-27T21:00:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-27T21:00:31.745 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-06-27T21:00:31.839 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:31.854 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): python3-rbd-19.0.0-3315.g98c31c1a.el9. 480 kB/s | 301 kB 00:00 2024-06-27T21:00:31.863 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:31.955 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:32.038 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 8/34 2024-06-27T21:00:32.042 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 8/34 2024-06-27T21:00:32.065 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 9/34 2024-06-27T21:00:32.073 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 9/34 2024-06-27T21:00:32.129 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 9/34 2024-06-27T21:00:32.153 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 9/34 2024-06-27T21:00:32.157 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-27T21:00:32.181 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-27T21:00:32.201 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9. 11/34 2024-06-27T21:00:32.225 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9. 11/34 2024-06-27T21:00:32.263 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 716 kB/s | 292 kB 00:00 2024-06-27T21:00:32.271 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 12/34 2024-06-27T21:00:32.293 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 13/34 2024-06-27T21:00:32.294 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 12/34 2024-06-27T21:00:32.317 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 13/34 2024-06-27T21:00:32.346 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 508 kB/s | 42 kB 00:00 2024-06-27T21:00:32.357 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 13/34 2024-06-27T21:00:32.381 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 13/34 2024-06-27T21:00:32.396 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-27T21:00:32.408 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-27T21:00:32.515 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-27T21:00:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-27T21:00:32.542 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-27T21:00:32.552 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-27T21:00:32.555 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 819 kB/s | 662 kB 00:00 2024-06-27T21:00:32.580 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 945 kB/s | 221 kB 00:00 2024-06-27T21:00:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-27T21:00:32.592 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-27T21:00:32.605 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-06-27T21:00:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-27T21:00:32.624 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-27T21:00:32.647 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-06-27T21:00:32.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-27T21:00:32.653 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-27T21:00:32.686 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-27T21:00:32.686 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-27T21:00:32.706 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-06-27T21:00:32.723 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.8 MB/s | 26 kB 00:00 2024-06-27T21:00:32.740 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-06-27T21:00:32.765 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-06-27T21:00:32.816 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-06-27T21:00:32.849 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-06-27T21:00:32.925 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-06-27T21:00:32.966 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-06-27T21:00:32.997 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-27T21:00:33.007 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-27T21:00:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 22/34 2024-06-27T21:00:33.049 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 22/34 2024-06-27T21:00:33.139 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 22/34 2024-06-27T21:00:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 22/34 2024-06-27T21:00:33.170 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 23/34 2024-06-27T21:00:33.170 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 23/34 2024-06-27T21:00:33.200 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): librados2-19.0.0-3315.g98c31c1a.el9.x8 12 MB/s | 3.4 MB 00:00 2024-06-27T21:00:33.211 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-27T21:00:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-27T21:00:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-27T21:00:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-27T21:00:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 26/34 2024-06-27T21:00:33.718 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 26/34 2024-06-27T21:00:33.773 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 26/34 2024-06-27T21:00:33.790 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 26/34 2024-06-27T21:00:33.841 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 27/34 2024-06-27T21:00:33.868 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 27/34 2024-06-27T21:00:33.870 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x8 28/34 2024-06-27T21:00:33.904 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x8 28/34 2024-06-27T21:00:34.659 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): ceph-common-19.0.0-3315.g98c31c1a.el9. 4.6 MB/s | 21 MB 00:04 2024-06-27T21:00:34.736 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): librbd1-19.0.0-3315.g98c31c1a.el9.x86_ 1.8 MB/s | 3.1 MB 00:01 2024-06-27T21:00:34.741 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:00:34.741 INFO:teuthology.orchestra.run.smithi190.stdout:Total 12 MB/s | 60 MB 00:04 2024-06-27T21:00:34.837 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:00:34.872 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:00:34.872 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:00:35.464 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:00:35.465 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:00:36.104 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:36.142 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:36.704 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:36.726 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:36.755 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 30/34 2024-06-27T21:00:36.764 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 30/34 2024-06-27T21:00:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 30/34 2024-06-27T21:00:36.833 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-27T21:00:36.833 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-27T21:00:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 30/34 2024-06-27T21:00:36.863 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-27T21:00:36.863 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-27T21:00:36.863 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:36.875 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:00:36.881 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:36.958 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-06-27T21:00:36.989 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-06-27T21:00:37.022 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-06-27T21:00:37.046 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-27T21:00:37.069 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-06-27T21:00:37.264 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-27T21:00:37.317 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-06-27T21:00:37.648 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-06-27T21:00:37.791 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:37.883 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:37.970 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 8/34 2024-06-27T21:00:38.009 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 9/34 2024-06-27T21:00:38.072 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 9/34 2024-06-27T21:00:38.100 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-06-27T21:00:38.144 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9. 11/34 2024-06-27T21:00:38.212 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 12/34 2024-06-27T21:00:38.236 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 13/34 2024-06-27T21:00:38.300 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 13/34 2024-06-27T21:00:38.336 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-06-27T21:00:38.458 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-06-27T21:00:38.494 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-06-27T21:00:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-06-27T21:00:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-06-27T21:00:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-06-27T21:00:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-06-27T21:00:38.940 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-06-27T21:00:38.984 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 22/34 2024-06-27T21:00:39.066 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 22/34 2024-06-27T21:00:39.097 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 23/34 2024-06-27T21:00:39.133 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-06-27T21:00:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-06-27T21:00:39.636 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 26/34 2024-06-27T21:00:39.708 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 26/34 2024-06-27T21:00:39.784 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 27/34 2024-06-27T21:00:39.814 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x8 28/34 2024-06-27T21:00:42.097 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:42.144 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:42.716 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 29/34 2024-06-27T21:00:42.756 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 30/34 2024-06-27T21:00:42.845 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 30/34 2024-06-27T21:00:42.846 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-27T21:00:42.846 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-27T21:00:42.846 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:42.865 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:46.586 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-06-27T21:00:46.587 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:46.771 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-06-27T21:00:46.772 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:47.544 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/34 2024-06-27T21:00:47.573 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/34 2024-06-27T21:00:47.573 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:00:47.573 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-27T21:00:47.573 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:00:47.574 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:00:47.574 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:47.575 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-27T21:00:47.664 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-27T21:00:47.664 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:47.725 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/34 2024-06-27T21:00:47.750 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/34 2024-06-27T21:00:47.751 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:00:47.751 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-27T21:00:47.751 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:00:47.751 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:00:47.751 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:47.752 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-27T21:00:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-27T21:00:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:48.898 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:48.898 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/34 2024-06-27T21:00:48.898 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/34 2024-06-27T21:00:48.898 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/34 2024-06-27T21:00:48.898 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/34 2024-06-27T21:00:48.899 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 5/34 2024-06-27T21:00:48.899 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 6/34 2024-06-27T21:00:48.899 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:48.899 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9. 8/34 2024-06-27T21:00:48.899 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x8 9/34 2024-06-27T21:00:48.899 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 10/34 2024-06-27T21:00:48.900 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 11/34 2024-06-27T21:00:48.900 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 12/34 2024-06-27T21:00:48.900 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 13/34 2024-06-27T21:00:48.900 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-27T21:00:48.900 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-27T21:00:48.901 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-27T21:00:48.901 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-27T21:00:48.901 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-27T21:00:48.901 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-27T21:00:48.901 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-27T21:00:48.902 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-27T21:00:48.902 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-27T21:00:48.902 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-27T21:00:48.902 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-27T21:00:48.902 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-27T21:00:48.902 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-27T21:00:48.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-27T21:00:48.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-27T21:00:48.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-27T21:00:48.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-27T21:00:48.903 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:48.904 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-27T21:00:48.904 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 5/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 6/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9. 8/34 2024-06-27T21:00:48.965 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x8 9/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 10/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 11/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 12/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 13/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-27T21:00:48.966 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-27T21:00:48.967 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/34 2024-06-27T21:00:49.726 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:49.726 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:49.726 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-06-27T21:00:49.726 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.726 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.726 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-27T21:00:49.727 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.728 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-27T21:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-27T21:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-27T21:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-3.el9.x86_64 2024-06-27T21:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-27T21:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:49.729 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:00:49.842 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:49.842 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:49.842 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-06-27T21:00:49.842 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.842 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-27T21:00:49.843 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.844 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:49.845 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:00:50.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-06-27T21:00:50.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-06-27T21:00:50.649 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:28 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:00:50.762 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:28 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:00:50.775 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:00:50.775 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:00:50.775 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 48 M 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 163 k 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:00:50.776 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-06-27T21:00:50.777 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:00:50.777 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 49 M 2024-06-27T21:00:50.777 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 207 M 2024-06-27T21:00:50.777 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:00:50.888 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:00:50.888 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:00:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-27T21:00:50.888 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 48 M 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 163 k 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-06-27T21:00:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:00:50.890 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 49 M 2024-06-27T21:00:50.890 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 207 M 2024-06-27T21:00:50.890 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:00:51.075 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-06-27T21:00:51.117 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-27T21:00:51.235 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 995 kB/s | 304 kB 00:00 2024-06-27T21:00:51.294 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-27T21:00:51.392 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): libcephsqlite-19.0.0-3315.g98c31c1a.el9. 345 kB/s | 163 kB 00:00 2024-06-27T21:00:51.427 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): libcephsqlite-19.0.0-3315.g98c31c1a.el9. 328 kB/s | 163 kB 00:00 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-06-27T21:00:52.476 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-06-27T21:00:52.477 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-06-27T21:00:52.477 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:53.126 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): ceph-test-19.0.0-3315.g98c31c1a.el9.x86_ 22 MB/s | 48 MB 00:02 2024-06-27T21:00:53.128 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:00:53.128 INFO:teuthology.orchestra.run.smithi151.stdout:Total 21 MB/s | 49 MB 00:02 2024-06-27T21:00:53.156 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:00:53.176 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:00:53.176 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:00:53.451 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/34 2024-06-27T21:00:53.476 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/34 2024-06-27T21:00:53.476 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:00:53.477 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-27T21:00:53.477 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:00:53.477 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-27T21:00:53.477 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:53.478 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-27T21:00:53.544 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:00:53.544 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:00:53.545 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-06-27T21:00:53.545 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:54.079 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:00:54.161 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-27T21:00:54.199 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-27T21:00:54.227 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/4 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 5/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_6 6/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 7/34 2024-06-27T21:00:54.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9. 8/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x8 9/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 10/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 11/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 12/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 13/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-06-27T21:00:54.668 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 31/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-06-27T21:00:54.669 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/34 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:00:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-27T21:00:55.554 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.555 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:55.556 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:00:55.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-06-27T21:00:56.302 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): ceph-test-19.0.0-3315.g98c31c1a.el9.x86_ 9.0 MB/s | 48 MB 00:05 2024-06-27T21:00:56.304 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:00:56.304 INFO:teuthology.orchestra.run.smithi192.stdout:Total 9.0 MB/s | 49 MB 00:05 2024-06-27T21:00:56.332 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:00:56.352 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:00:56.352 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:00:56.470 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:29 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:00:56.596 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 48 M 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-06-27T21:00:56.597 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 163 k 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-06-27T21:00:56.598 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:00:56.599 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 49 M 2024-06-27T21:00:56.599 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 207 M 2024-06-27T21:00:56.599 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:00:56.719 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:00:56.720 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:00:56.898 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.6 MB/s | 304 kB 00:00 2024-06-27T21:00:56.940 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-27T21:00:56.991 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): libcephsqlite-19.0.0-3315.g98c31c1a.el9. 597 kB/s | 163 kB 00:00 2024-06-27T21:00:57.243 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:00:57.320 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-27T21:00:57.359 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-27T21:00:57.387 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/4 2024-06-27T21:00:58.641 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): ceph-test-19.0.0-3315.g98c31c1a.el9.x86_ 25 MB/s | 48 MB 00:01 2024-06-27T21:00:58.643 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:00:58.643 INFO:teuthology.orchestra.run.smithi190.stdout:Total 24 MB/s | 49 MB 00:02 2024-06-27T21:00:58.671 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:00:58.691 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:00:58.691 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:00:58.798 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/4 2024-06-27T21:00:58.875 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/4 2024-06-27T21:00:59.058 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:00:59.058 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:00:59.581 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:00:59.659 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-27T21:00:59.706 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-27T21:00:59.734 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/4 2024-06-27T21:01:01.257 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/4 2024-06-27T21:01:01.257 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/4 2024-06-27T21:01:01.257 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/4 2024-06-27T21:01:01.257 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-27T21:01:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-27T21:01:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:01.542 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:01.542 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-27T21:01:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-27T21:01:01.543 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:01.543 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:01.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-06-27T21:01:01.961 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/4 2024-06-27T21:01:02.045 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/4 2024-06-27T21:01:02.396 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:40 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:02.534 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 6.6 k 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 2.4 M 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 1.5 M 2024-06-27T21:01:02.537 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3315.g98c31c1a.el9 ceph-noarch 249 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 4.7 M 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 17 M 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-27T21:01:02.538 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-27T21:01:02.539 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout:Install 36 Packages 2024-06-27T21:01:02.540 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:02.541 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-06-27T21:01:02.541 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 107 M 2024-06-27T21:01:02.541 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:03.351 INFO:teuthology.orchestra.run.smithi151.stdout:(1/36): ceph-19.0.0-3315.g98c31c1a.el9.x86_64.r 22 kB/s | 6.6 kB 00:00 2024-06-27T21:01:03.460 INFO:teuthology.orchestra.run.smithi151.stdout:(2/36): ceph-mds-19.0.0-3315.g98c31c1a.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-06-27T21:01:03.710 INFO:teuthology.orchestra.run.smithi151.stdout:(3/36): ceph-mgr-19.0.0-3315.g98c31c1a.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-06-27T21:01:04.177 INFO:teuthology.orchestra.run.smithi151.stdout:(4/36): ceph-osd-19.0.0-3315.g98c31c1a.el9.x86_ 23 MB/s | 17 MB 00:00 2024-06-27T21:01:04.202 INFO:teuthology.orchestra.run.smithi151.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3315.g98c3 506 kB/s | 249 kB 00:00 2024-06-27T21:01:04.290 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/4 2024-06-27T21:01:04.353 INFO:teuthology.orchestra.run.smithi151.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-06-27T21:01:04.367 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/4 2024-06-27T21:01:04.367 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/4 2024-06-27T21:01:04.367 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/4 2024-06-27T21:01:04.367 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-27T21:01:04.374 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/4 2024-06-27T21:01:04.428 INFO:teuthology.orchestra.run.smithi151.stdout:(7/36): ceph-mon-19.0.0-3315.g98c31c1a.el9.x86_ 4.4 MB/s | 4.7 MB 00:01 2024-06-27T21:01:04.453 INFO:teuthology.orchestra.run.smithi151.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-06-27T21:01:04.504 INFO:teuthology.orchestra.run.smithi151.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-06-27T21:01:04.537 INFO:teuthology.orchestra.run.smithi151.stdout:(10/36): python3-cryptography-36.0.1-4.el9.x86_ 3.7 MB/s | 1.2 MB 00:00 2024-06-27T21:01:04.562 INFO:teuthology.orchestra.run.smithi151.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-06-27T21:01:04.588 INFO:teuthology.orchestra.run.smithi151.stdout:(12/36): python3-pycparser-2.20-6.el9.noarch.rp 849 kB/s | 135 kB 00:00 2024-06-27T21:01:04.685 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:04.686 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:04.854 INFO:teuthology.orchestra.run.smithi151.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-06-27T21:01:05.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-06-27T21:01:05.038 INFO:teuthology.orchestra.run.smithi151.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 278 kB/s | 51 kB 00:00 2024-06-27T21:01:05.063 INFO:teuthology.orchestra.run.smithi151.stdout:(15/36): lua-5.4.4-4.el9.x86_64.rpm 358 kB/s | 188 kB 00:00 2024-06-27T21:01:05.088 INFO:teuthology.orchestra.run.smithi151.stdout:(16/36): python3-mako-1.1.4-6.el9.noarch.rpm 327 kB/s | 172 kB 00:00 2024-06-27T21:01:05.139 INFO:teuthology.orchestra.run.smithi151.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 414 kB/s | 42 kB 00:00 2024-06-27T21:01:05.239 INFO:teuthology.orchestra.run.smithi151.stdout:(18/36): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-06-27T21:01:05.265 INFO:teuthology.orchestra.run.smithi151.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 352 kB/s | 43 kB 00:00 2024-06-27T21:01:05.289 INFO:teuthology.orchestra.run.smithi151.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-06-27T21:01:05.315 INFO:teuthology.orchestra.run.smithi151.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 473 kB/s | 11 kB 00:00 2024-06-27T21:01:05.340 INFO:teuthology.orchestra.run.smithi151.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-06-27T21:01:05.374 INFO:teuthology.orchestra.run.smithi151.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-06-27T21:01:05.399 INFO:teuthology.orchestra.run.smithi151.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-06-27T21:01:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-06-27T21:01:05.450 INFO:teuthology.orchestra.run.smithi151.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-06-27T21:01:05.475 INFO:teuthology.orchestra.run.smithi151.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-06-27T21:01:05.500 INFO:teuthology.orchestra.run.smithi151.stdout:(28/36): lua-devel-5.4.4-4.el9.x86_64.rpm 51 kB/s | 22 kB 00:00 2024-06-27T21:01:05.550 INFO:teuthology.orchestra.run.smithi151.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 786 kB/s | 79 kB 00:00 2024-06-27T21:01:05.566 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:43 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:05.576 INFO:teuthology.orchestra.run.smithi151.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-06-27T21:01:05.601 INFO:teuthology.orchestra.run.smithi151.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-06-27T21:01:05.626 INFO:teuthology.orchestra.run.smithi151.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-27T21:01:05.651 INFO:teuthology.orchestra.run.smithi151.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-06-27T21:01:05.676 INFO:teuthology.orchestra.run.smithi151.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-06-27T21:01:05.694 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:05.696 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:05.696 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-27T21:01:05.696 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:05.696 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 6.6 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 2.4 M 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 1.5 M 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3315.g98c31c1a.el9 ceph-noarch 249 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 4.7 M 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 17 M 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-27T21:01:05.697 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-27T21:01:05.698 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-27T21:01:05.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout:Install 36 Packages 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:05.700 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 30 M 2024-06-27T21:01:05.701 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 107 M 2024-06-27T21:01:05.701 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:05.702 INFO:teuthology.orchestra.run.smithi151.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-06-27T21:01:05.727 INFO:teuthology.orchestra.run.smithi151.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-06-27T21:01:05.731 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:05.731 INFO:teuthology.orchestra.run.smithi151.stdout:Total 9.4 MB/s | 30 MB 00:03 2024-06-27T21:01:05.866 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:05.905 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:05.906 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:06.371 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:06.371 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:06.572 INFO:teuthology.orchestra.run.smithi192.stdout:(1/36): ceph-19.0.0-3315.g98c31c1a.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-06-27T21:01:06.624 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/4 2024-06-27T21:01:06.624 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/4 2024-06-27T21:01:06.625 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/4 2024-06-27T21:01:06.625 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-27T21:01:06.697 INFO:teuthology.orchestra.run.smithi192.stdout:(2/36): ceph-mgr-19.0.0-3315.g98c31c1a.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-06-27T21:01:06.747 INFO:teuthology.orchestra.run.smithi192.stdout:(3/36): ceph-mds-19.0.0-3315.g98c31c1a.el9.x86_ 5.8 MB/s | 2.4 MB 00:00 2024-06-27T21:01:06.823 INFO:teuthology.orchestra.run.smithi192.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3315.g98c3 3.2 MB/s | 249 kB 00:00 2024-06-27T21:01:06.919 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-27T21:01:06.919 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:06.919 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:06.919 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:06.919 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:06.919 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-27T21:01:06.920 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-27T21:01:06.920 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:06.920 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:07.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-06-27T21:01:07.189 INFO:teuthology.orchestra.run.smithi192.stdout:(5/36): ceph-osd-19.0.0-3315.g98c31c1a.el9.x86_ 34 MB/s | 17 MB 00:00 2024-06-27T21:01:07.652 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:07.721 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:40 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:07.729 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-27T21:01:07.748 INFO:teuthology.orchestra.run.smithi192.stdout:(6/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 274 kB/s | 253 kB 00:00 2024-06-27T21:01:07.780 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-27T21:01:07.815 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-27T21:01:07.853 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 6.6 k 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 2.4 M 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 1.5 M 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3315.g98c31c1a.el9 ceph-noarch 249 k 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 4.7 M 2024-06-27T21:01:07.856 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 17 M 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-27T21:01:07.857 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-27T21:01:07.858 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-27T21:01:07.859 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:07.860 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:07.860 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:07.860 INFO:teuthology.orchestra.run.smithi190.stdout:Install 36 Packages 2024-06-27T21:01:07.860 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:07.860 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 M 2024-06-27T21:01:07.860 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 107 M 2024-06-27T21:01:07.861 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:07.887 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-27T21:01:07.939 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-27T21:01:07.940 INFO:teuthology.orchestra.run.smithi192.stdout:(7/36): python3-cryptography-36.0.1-4.el9.x86_6 1.6 MB/s | 1.2 MB 00:00 2024-06-27T21:01:07.967 INFO:teuthology.orchestra.run.smithi192.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-06-27T21:01:07.972 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-27T21:01:08.057 INFO:teuthology.orchestra.run.smithi192.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-06-27T21:01:08.065 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-27T21:01:08.082 INFO:teuthology.orchestra.run.smithi192.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-06-27T21:01:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-27T21:01:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-27T21:01:08.183 INFO:teuthology.orchestra.run.smithi192.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.7 MB/s | 215 kB 00:00 2024-06-27T21:01:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-27T21:01:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-27T21:01:08.250 INFO:teuthology.orchestra.run.smithi192.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-06-27T21:01:08.283 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-27T21:01:08.292 INFO:teuthology.orchestra.run.smithi192.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 834 kB/s | 35 kB 00:00 2024-06-27T21:01:08.315 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-27T21:01:08.325 INFO:teuthology.orchestra.run.smithi192.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-27T21:01:08.351 INFO:teuthology.orchestra.run.smithi192.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 873 kB/s | 51 kB 00:00 2024-06-27T21:01:08.356 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-27T21:01:08.367 INFO:teuthology.orchestra.run.smithi192.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 996 kB/s | 42 kB 00:00 2024-06-27T21:01:08.392 INFO:teuthology.orchestra.run.smithi190.stdout:(1/36): ceph-19.0.0-3315.g98c31c1a.el9.x86_64.r 28 kB/s | 6.6 kB 00:00 2024-06-27T21:01:08.399 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-27T21:01:08.434 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-27T21:01:08.435 INFO:teuthology.orchestra.run.smithi192.stdout:(17/36): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-06-27T21:01:08.460 INFO:teuthology.orchestra.run.smithi192.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-06-27T21:01:08.472 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-27T21:01:08.485 INFO:teuthology.orchestra.run.smithi192.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 866 kB/s | 43 kB 00:00 2024-06-27T21:01:08.546 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-27T21:01:08.578 INFO:teuthology.orchestra.run.smithi192.stdout:(20/36): ceph-mon-19.0.0-3315.g98c31c1a.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-06-27T21:01:08.625 INFO:teuthology.orchestra.run.smithi190.stdout:(2/36): ceph-mgr-19.0.0-3315.g98c31c1a.el9.x86_ 3.2 MB/s | 1.5 MB 00:00 2024-06-27T21:01:08.636 INFO:teuthology.orchestra.run.smithi192.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 979 kB/s | 172 kB 00:00 2024-06-27T21:01:08.661 INFO:teuthology.orchestra.run.smithi192.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-06-27T21:01:08.686 INFO:teuthology.orchestra.run.smithi192.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 101 kB/s | 11 kB 00:00 2024-06-27T21:01:08.693 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-27T21:01:08.712 INFO:teuthology.orchestra.run.smithi192.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 241 kB/s | 18 kB 00:00 2024-06-27T21:01:08.737 INFO:teuthology.orchestra.run.smithi192.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-06-27T21:01:08.762 INFO:teuthology.orchestra.run.smithi192.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-06-27T21:01:08.772 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-27T21:01:08.775 INFO:teuthology.orchestra.run.smithi190.stdout:(3/36): ceph-mds-19.0.0-3315.g98c31c1a.el9.x86_ 3.8 MB/s | 2.4 MB 00:00 2024-06-27T21:01:08.788 INFO:teuthology.orchestra.run.smithi192.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-06-27T21:01:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-06-27T21:01:08.824 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-27T21:01:08.838 INFO:teuthology.orchestra.run.smithi192.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-27T21:01:08.842 INFO:teuthology.orchestra.run.smithi190.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3315.g98c3 3.7 MB/s | 249 kB 00:00 2024-06-27T21:01:08.863 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-27T21:01:08.864 INFO:teuthology.orchestra.run.smithi192.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-06-27T21:01:08.888 INFO:teuthology.orchestra.run.smithi192.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-06-27T21:01:08.914 INFO:teuthology.orchestra.run.smithi192.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-06-27T21:01:08.920 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-27T21:01:08.939 INFO:teuthology.orchestra.run.smithi192.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-27T21:01:08.975 INFO:teuthology.orchestra.run.smithi192.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-06-27T21:01:08.976 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-27T21:01:08.998 INFO:teuthology.orchestra.run.smithi192.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-06-27T21:01:09.024 INFO:teuthology.orchestra.run.smithi192.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-06-27T21:01:09.029 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:09.029 INFO:teuthology.orchestra.run.smithi192.stdout:Total 9.0 MB/s | 30 MB 00:03 2024-06-27T21:01:09.092 INFO:teuthology.orchestra.run.smithi190.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-06-27T21:01:09.169 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:09.172 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-27T21:01:09.209 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:09.209 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:09.217 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-27T21:01:09.296 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-27T21:01:09.384 INFO:teuthology.orchestra.run.smithi190.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 4.2 MB/s | 1.2 MB 00:00 2024-06-27T21:01:09.385 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-27T21:01:09.424 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-27T21:01:09.434 INFO:teuthology.orchestra.run.smithi190.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-27T21:01:09.485 INFO:teuthology.orchestra.run.smithi190.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-06-27T21:01:09.543 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-27T21:01:09.649 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:09.650 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:09.706 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9. 31/36 2024-06-27T21:01:09.727 INFO:teuthology.orchestra.run.smithi190.stdout:(9/36): ceph-osd-19.0.0-3315.g98c31c1a.el9.x86_ 15 MB/s | 17 MB 00:01 2024-06-27T21:01:09.734 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/36 2024-06-27T21:01:09.752 INFO:teuthology.orchestra.run.smithi190.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 468 kB/s | 125 kB 00:00 2024-06-27T21:01:09.760 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/36 2024-06-27T21:01:09.761 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:09.761 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-27T21:01:09.761 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-27T21:01:09.761 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-27T21:01:09.761 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:09.819 INFO:teuthology.orchestra.run.smithi190.stdout:(11/36): ceph-mon-19.0.0-3315.g98c31c1a.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-06-27T21:01:09.844 INFO:teuthology.orchestra.run.smithi190.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-06-27T21:01:09.894 INFO:teuthology.orchestra.run.smithi190.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 694 kB/s | 35 kB 00:00 2024-06-27T21:01:09.953 INFO:teuthology.orchestra.run.smithi190.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 875 kB/s | 51 kB 00:00 2024-06-27T21:01:09.987 INFO:teuthology.orchestra.run.smithi190.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 828 kB/s | 215 kB 00:00 2024-06-27T21:01:10.012 INFO:teuthology.orchestra.run.smithi190.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 709 kB/s | 42 kB 00:00 2024-06-27T21:01:10.037 INFO:teuthology.orchestra.run.smithi190.stdout:(17/36): python3-mako-1.1.4-6.el9.noarch.rpm 790 kB/s | 172 kB 00:00 2024-06-27T21:01:10.121 INFO:teuthology.orchestra.run.smithi190.stdout:(18/36): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-06-27T21:01:10.154 INFO:teuthology.orchestra.run.smithi190.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.5 MB/s | 172 kB 00:00 2024-06-27T21:01:10.180 INFO:teuthology.orchestra.run.smithi190.stdout:(20/36): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-06-27T21:01:10.205 INFO:teuthology.orchestra.run.smithi190.stdout:(21/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 259 kB/s | 43 kB 00:00 2024-06-27T21:01:10.230 INFO:teuthology.orchestra.run.smithi190.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-06-27T21:01:10.280 INFO:teuthology.orchestra.run.smithi190.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 911 kB/s | 23 kB 00:00 2024-06-27T21:01:10.306 INFO:teuthology.orchestra.run.smithi190.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-06-27T21:01:10.331 INFO:teuthology.orchestra.run.smithi190.stdout:(25/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-06-27T21:01:10.356 INFO:teuthology.orchestra.run.smithi190.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 265 kB/s | 19 kB 00:00 2024-06-27T21:01:10.381 INFO:teuthology.orchestra.run.smithi190.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-06-27T21:01:10.407 INFO:teuthology.orchestra.run.smithi190.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-06-27T21:01:10.432 INFO:teuthology.orchestra.run.smithi190.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-27T21:01:10.457 INFO:teuthology.orchestra.run.smithi190.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-27T21:01:10.483 INFO:teuthology.orchestra.run.smithi190.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-27T21:01:10.508 INFO:teuthology.orchestra.run.smithi190.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-06-27T21:01:10.533 INFO:teuthology.orchestra.run.smithi190.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-27T21:01:10.558 INFO:teuthology.orchestra.run.smithi190.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-06-27T21:01:10.584 INFO:teuthology.orchestra.run.smithi190.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-27T21:01:10.609 INFO:teuthology.orchestra.run.smithi190.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-06-27T21:01:10.612 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:10.613 INFO:teuthology.orchestra.run.smithi190.stdout:Total 11 MB/s | 30 MB 00:02 2024-06-27T21:01:10.745 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:10.785 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:10.785 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:10.909 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-27T21:01:11.018 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-27T21:01:11.054 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-27T21:01:11.126 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-27T21:01:11.161 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-27T21:01:11.194 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/36 2024-06-27T21:01:11.194 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-27T21:01:11.221 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/36 2024-06-27T21:01:11.221 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:11.221 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-27T21:01:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-27T21:01:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-27T21:01:11.222 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:11.237 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:11.237 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:11.286 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-27T21:01:11.373 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-27T21:01:11.414 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-27T21:01:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-27T21:01:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-27T21:01:11.513 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-27T21:01:11.546 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-27T21:01:11.587 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-27T21:01:11.653 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-27T21:01:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 34/36 2024-06-27T21:01:11.690 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-27T21:01:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 34/36 2024-06-27T21:01:11.693 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:11.693 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-27T21:01:11.693 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-27T21:01:11.693 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-27T21:01:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:11.737 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-27T21:01:11.810 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-27T21:01:11.920 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 35/36 2024-06-27T21:01:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 35/36 2024-06-27T21:01:11.944 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:11.944 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-27T21:01:11.944 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-27T21:01:11.944 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-27T21:01:11.944 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:11.948 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-27T21:01:12.009 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 36/36 2024-06-27T21:01:12.028 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-27T21:01:12.082 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-27T21:01:12.128 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-27T21:01:12.186 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-27T21:01:12.241 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-27T21:01:12.430 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-27T21:01:12.482 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-27T21:01:12.515 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:12.560 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-27T21:01:12.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-06-27T21:01:12.607 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-06-27T21:01:12.647 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-06-27T21:01:12.658 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-27T21:01:12.698 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-27T21:01:12.719 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-06-27T21:01:12.771 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-06-27T21:01:12.804 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-06-27T21:01:12.816 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-27T21:01:12.898 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-06-27T21:01:12.981 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9. 31/36 2024-06-27T21:01:13.009 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/36 2024-06-27T21:01:13.010 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-06-27T21:01:13.034 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/36 2024-06-27T21:01:13.034 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:13.035 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-27T21:01:13.035 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-27T21:01:13.035 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-27T21:01:13.035 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:13.051 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-06-27T21:01:13.085 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-06-27T21:01:13.131 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-06-27T21:01:13.158 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-06-27T21:01:13.191 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-06-27T21:01:13.232 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-06-27T21:01:13.275 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-06-27T21:01:13.309 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-06-27T21:01:13.347 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-06-27T21:01:13.420 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-06-27T21:01:13.559 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-06-27T21:01:13.647 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-06-27T21:01:13.700 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-06-27T21:01:13.747 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-06-27T21:01:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 36/36 2024-06-27T21:01:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/36 2024-06-27T21:01:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/36 2024-06-27T21:01:13.796 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 5/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9. 6/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-27T21:01:13.797 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-27T21:01:13.798 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-27T21:01:13.799 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-27T21:01:13.800 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-27T21:01:13.801 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-27T21:01:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-06-27T21:01:14.038 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-06-27T21:01:14.092 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-06-27T21:01:14.178 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-06-27T21:01:14.267 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-06-27T21:01:14.308 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-06-27T21:01:14.424 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-06-27T21:01:14.460 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/36 2024-06-27T21:01:14.487 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/36 2024-06-27T21:01:14.487 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:14.487 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-27T21:01:14.487 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-27T21:01:14.487 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-27T21:01:14.487 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:14.583 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9. 31/36 2024-06-27T21:01:14.610 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/36 2024-06-27T21:01:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 32/36 2024-06-27T21:01:14.635 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:14.635 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-27T21:01:14.635 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-27T21:01:14.636 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-27T21:01:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:14.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-27T21:01:14.855 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:14.855 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9.noarch 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-27T21:01:14.856 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-06-27T21:01:14.857 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:14.858 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:14.928 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 34/36 2024-06-27T21:01:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 34/36 2024-06-27T21:01:14.951 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:14.952 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-27T21:01:14.952 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-27T21:01:14.952 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-27T21:01:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 35/36 2024-06-27T21:01:15.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-06-27T21:01:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 35/36 2024-06-27T21:01:15.202 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:15.202 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-27T21:01:15.202 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-27T21:01:15.202 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-27T21:01:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:15.266 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 36/36 2024-06-27T21:01:15.753 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:53 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:15.838 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:15.881 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:15.882 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:15.882 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:15.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-06-27T21:01:16.086 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/36 2024-06-27T21:01:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 33/36 2024-06-27T21:01:16.112 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:16.112 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-27T21:01:16.112 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-27T21:01:16.112 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-27T21:01:16.113 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:16.476 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:54 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:16.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 34/36 2024-06-27T21:01:16.585 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 34/36 2024-06-27T21:01:16.585 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:16.585 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-27T21:01:16.585 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-27T21:01:16.585 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-27T21:01:16.586 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:16.604 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:16.605 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:16.605 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-06-27T21:01:16.605 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:16.605 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.0.0-3315.g98c31c1a.el9 ceph-noarch 763 k 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 763 k 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 770 k 2024-06-27T21:01:16.606 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:16.829 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 35/36 2024-06-27T21:01:16.852 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 35/36 2024-06-27T21:01:16.852 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:16.852 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-27T21:01:16.852 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-27T21:01:16.852 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-27T21:01:16.852 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:16.917 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 36/36 2024-06-27T21:01:16.927 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-19.0.0-3315.g98c31c1a.el9.noarch.rpm 2.3 MB/s | 763 kB 00:00 2024-06-27T21:01:16.927 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:16.928 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.3 MB/s | 763 kB 00:00 2024-06-27T21:01:16.928 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:16.933 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:16.933 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 36/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 5/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9. 6/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-27T21:01:16.938 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-27T21:01:16.939 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-27T21:01:16.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-27T21:01:16.941 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:16.941 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:17.515 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:17.605 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9.noarch 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:17.948 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-27T21:01:17.949 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-27T21:01:17.950 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-27T21:01:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-27T21:01:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-27T21:01:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-27T21:01:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-27T21:01:17.951 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:17.951 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:17.978 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:18.241 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:18.241 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:18.241 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:18.241 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 2024-06-27T21:01:18.241 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:18.241 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:18.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-06-27T21:01:18.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-06-27T21:01:18.643 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 36/36 2024-06-27T21:01:18.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/36 2024-06-27T21:01:18.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/36 2024-06-27T21:01:18.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 3/36 2024-06-27T21:01:18.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 4/36 2024-06-27T21:01:18.643 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 5/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9. 6/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-06-27T21:01:18.644 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-06-27T21:01:18.645 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-06-27T21:01:18.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-06-27T21:01:18.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-06-27T21:01:18.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-06-27T21:01:18.646 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-06-27T21:01:18.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-06-27T21:01:18.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-06-27T21:01:18.647 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-06-27T21:01:18.842 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:56 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:18.928 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:18.970 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:18.971 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:18.971 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:19.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-06-27T21:01:19.018 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:57 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:19.150 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 145 k 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:19.151 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:19.152 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 145 k 2024-06-27T21:01:19.152 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 435 k 2024-06-27T21:01:19.152 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:19.374 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-19.0.0-3315.g98c31c 652 kB/s | 145 kB 00:00 2024-06-27T21:01:19.375 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:19.375 INFO:teuthology.orchestra.run.smithi151.stdout:Total 647 kB/s | 145 kB 00:00 2024-06-27T21:01:19.375 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:19.383 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:19.383 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:19.439 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:19.439 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:19.559 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:57 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:19.675 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:19.686 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:19.687 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:19.687 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-27T21:01:19.687 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-3315.g98c31c1a.el9 ceph-noarch 763 k 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 763 k 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 770 k 2024-06-27T21:01:19.688 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:19.701 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:19.701 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:19.701 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-27T21:01:19.701 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-3315.g98c31c1a.el9.noarch 2024-06-27T21:01:19.705 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-27T21:01:19.706 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-27T21:01:19.707 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:19.708 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:20.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-06-27T21:01:20.087 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-19.0.0-3315.g98c31c1a.el9.noarch.rpm 1.9 MB/s | 763 kB 00:00 2024-06-27T21:01:20.087 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:20.087 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.9 MB/s | 763 kB 00:00 2024-06-27T21:01:20.088 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:20.093 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:20.093 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:20.099 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:20.099 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:20.560 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:53 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:20.633 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:20.633 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:20.633 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:20.633 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:20.633 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:20.633 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:20.646 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:20.656 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:20.687 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:20.688 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:20.688 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:20.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-06-27T21:01:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:20.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-06-27T21:01:21.112 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:21.295 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:54 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:21.385 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:21.385 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:21.385 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:21.385 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 2024-06-27T21:01:21.385 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:21.385 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:21.422 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-3315.g98c31c1a.el9 ceph-noarch 763 k 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:21.423 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:21.424 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:21.424 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 763 k 2024-06-27T21:01:21.424 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 770 k 2024-06-27T21:01:21.424 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:21.449 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:59 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:21.539 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:21.581 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:21.582 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:21.582 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:21.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-06-27T21:01:21.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-06-27T21:01:21.719 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-19.0.0-3315.g98c31c1a.el9.noarch.rpm 2.5 MB/s | 763 kB 00:00 2024-06-27T21:01:21.720 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:21.720 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.5 MB/s | 763 kB 00:00 2024-06-27T21:01:21.720 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:21.725 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:21.725 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:21.731 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:21.731 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:22.193 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:00 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:22.194 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:00 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:22.278 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:22.321 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 892 k 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-06-27T21:01:22.322 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-27T21:01:22.323 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:22.323 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:22.323 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:22.323 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-06-27T21:01:22.323 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:22.323 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 973 k 2024-06-27T21:01:22.324 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.8 M 2024-06-27T21:01:22.324 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:22.325 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 145 k 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:22.326 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:22.327 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:22.327 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:22.327 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 145 k 2024-06-27T21:01:22.327 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 435 k 2024-06-27T21:01:22.327 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:22.553 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-19.0.0-3315.g98c31c 643 kB/s | 145 kB 00:00 2024-06-27T21:01:22.554 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:22.554 INFO:teuthology.orchestra.run.smithi192.stdout:Total 638 kB/s | 145 kB 00:00 2024-06-27T21:01:22.554 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:22.561 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:22.561 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:22.569 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 467 kB/s | 80 kB 00:00 2024-06-27T21:01:22.617 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:22.617 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:22.727 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-19.0.0-3315.g98c31c1a.el9.x86_ 2.6 MB/s | 892 kB 00:00 2024-06-27T21:01:22.728 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:22.729 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.3 MB/s | 973 kB 00:00 2024-06-27T21:01:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:22.740 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:22.761 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:22.762 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:22.763 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:22.829 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:22.833 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:22.834 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:22.854 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:22.854 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:22.855 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-27T21:01:22.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:22.958 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 1/1 2024-06-27T21:01:22.958 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:22.958 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:22.959 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-3315.g98c31c1a.el9.noarch 2024-06-27T21:01:22.959 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:22.959 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:23.015 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:23.146 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-27T21:01:23.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-06-27T21:01:23.208 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/2 2024-06-27T21:01:23.735 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:56 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:23.794 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:23.795 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:23.865 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:23.866 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/2 2024-06-27T21:01:23.866 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/2 2024-06-27T21:01:23.866 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:23.866 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 145 k 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 145 k 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 435 k 2024-06-27T21:01:23.867 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:24.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-06-27T21:01:24.091 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-19.0.0-3315.g98c31c 647 kB/s | 145 kB 00:00 2024-06-27T21:01:24.092 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:24.092 INFO:teuthology.orchestra.run.smithi190.stdout:Total 642 kB/s | 145 kB 00:00 2024-06-27T21:01:24.092 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:24.099 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:24.099 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:24.123 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-27T21:01:24.124 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:24.124 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:24.124 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-27T21:01:24.124 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:24.124 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:24.156 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:24.156 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:24.299 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:24.384 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:24.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-06-27T21:01:24.409 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:24.409 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:24.409 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-27T21:01:24.409 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:24.596 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:02 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:24.682 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:24.724 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:24.725 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:24.725 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:24.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-06-27T21:01:24.980 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:02 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:25.113 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-06-27T21:01:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:25.113 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 127 k 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-06-27T21:01:25.114 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:25.341 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:03 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:25.343 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a. 1/1 2024-06-27T21:01:25.343 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:25.343 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:25.343 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:25.343 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:25.344 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:25.357 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-19.0.0-3315.g98c31c1a.el9.x86_64 523 kB/s | 127 kB 00:00 2024-06-27T21:01:25.357 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:25.357 INFO:teuthology.orchestra.run.smithi151.stdout:Total 519 kB/s | 127 kB 00:00 2024-06-27T21:01:25.358 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:25.363 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:25.363 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:25.420 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:25.420 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:25.468 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:25.468 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:25.468 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-27T21:01:25.468 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 892 k 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:25.469 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-06-27T21:01:25.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:25.470 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 973 k 2024-06-27T21:01:25.470 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 M 2024-06-27T21:01:25.470 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:25.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-06-27T21:01:25.573 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:25.710 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:26.117 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:59 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:26.118 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:26.192 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 155 kB/s | 80 kB 00:00 2024-06-27T21:01:26.206 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:26.248 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:26.248 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:26.249 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:26.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-06-27T21:01:26.391 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:26.392 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:26.392 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:26.392 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:26.392 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:26.392 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:26.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-06-27T21:01:26.742 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-19.0.0-3315.g98c31c1a.el9.x86_ 836 kB/s | 892 kB 00:01 2024-06-27T21:01:26.743 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:26.743 INFO:teuthology.orchestra.run.smithi192.stdout:Total 764 kB/s | 973 kB 00:01 2024-06-27T21:01:26.757 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:26.782 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:26.783 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:26.856 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:26.857 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:26.860 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:59 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:26.988 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-06-27T21:01:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:26.988 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 892 k 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-06-27T21:01:26.989 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 973 k 2024-06-27T21:01:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.8 M 2024-06-27T21:01:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:27.037 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:27.147 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:05 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:27.168 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-27T21:01:27.230 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/2 2024-06-27T21:01:27.236 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:27.277 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:27.278 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:27.279 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:27.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-06-27T21:01:27.897 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:05 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/2 2024-06-27T21:01:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/2 2024-06-27T21:01:28.023 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:28.024 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-06-27T21:01:28.024 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 32 k 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 32 k 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 142 k 2024-06-27T21:01:28.025 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:28.044 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-fuse-19.0.0-3315.g98c31c1a.el9.x86_ 1.0 MB/s | 892 kB 00:00 2024-06-27T21:01:28.077 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 87 kB/s | 80 kB 00:00 2024-06-27T21:01:28.078 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:28.078 INFO:teuthology.orchestra.run.smithi190.stdout:Total 893 kB/s | 973 kB 00:01 2024-06-27T21:01:28.090 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:28.112 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:28.113 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-27T21:01:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:28.183 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-27T21:01:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:28.183 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:28.184 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:28.184 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:28.277 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-19.0.0-3315.g98c31c1a.el9.x86_6 126 kB/s | 32 kB 00:00 2024-06-27T21:01:28.277 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:28.278 INFO:teuthology.orchestra.run.smithi151.stdout:Total 125 kB/s | 32 kB 00:00 2024-06-27T21:01:28.278 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:28.281 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:28.281 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:28.293 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:28.293 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:28.368 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:28.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-06-27T21:01:28.464 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:28.503 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-27T21:01:28.565 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/2 2024-06-27T21:01:28.842 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:29.001 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:07 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:29.073 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:29.073 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:29.073 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:29.133 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 127 k 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:29.134 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 127 k 2024-06-27T21:01:29.135 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 456 k 2024-06-27T21:01:29.135 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:29.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-06-27T21:01:29.300 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2/2 2024-06-27T21:01:29.300 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/2 2024-06-27T21:01:29.442 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-19.0.0-3315.g98c31c1a.el9.x86_64 416 kB/s | 127 kB 00:00 2024-06-27T21:01:29.442 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:29.442 INFO:teuthology.orchestra.run.smithi192.stdout:Total 414 kB/s | 127 kB 00:00 2024-06-27T21:01:29.442 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:29.446 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:29.446 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:29.502 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:29.502 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:29.580 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-27T21:01:29.581 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:29.581 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:29.581 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-27T21:01:29.581 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:29.581 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:29.655 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:29.752 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:29.774 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:07 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:29.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-06-27T21:01:29.861 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:29.902 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:29.903 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:29.903 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:29.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-06-27T21:01:30.132 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:30.374 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:03 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:30.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:30.382 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:30.383 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:30.383 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:30.383 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:30.383 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:30.501 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:30.501 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:30.501 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 127 k 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:30.502 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:30.503 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 127 k 2024-06-27T21:01:30.503 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-06-27T21:01:30.503 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:30.522 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:08 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:30.608 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:30.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-06-27T21:01:30.649 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:30.651 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:30.651 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:30.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-06-27T21:01:30.732 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-19.0.0-3315.g98c31c1a.el9.x86_64 554 kB/s | 127 kB 00:00 2024-06-27T21:01:30.732 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:30.732 INFO:teuthology.orchestra.run.smithi190.stdout:Total 550 kB/s | 127 kB 00:00 2024-06-27T21:01:30.732 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:30.738 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:30.738 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:30.796 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:30.796 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:30.961 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:31.053 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:31.201 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:09 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:31.280 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:09 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:31.290 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:31.332 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:31.333 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:31.333 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:31.368 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:31.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-06-27T21:01:31.410 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:31.411 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:31.411 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:31.465 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:31.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-06-27T21:01:31.741 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:31.741 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:31.741 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:31.741 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:31.741 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:31.741 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:31.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-06-27T21:01:31.961 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:09 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:32.065 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:10 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:32.099 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:32.100 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:32.100 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-27T21:01:32.100 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:32.100 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:32.100 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 32 k 2024-06-27T21:01:32.100 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 32 k 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 142 k 2024-06-27T21:01:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:32.154 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:32.197 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:32.198 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:32.198 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:32.261 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-19.0.0-3315.g98c31c1a.el9.x86_6 199 kB/s | 32 kB 00:00 2024-06-27T21:01:32.261 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:32.261 INFO:teuthology.orchestra.run.smithi192.stdout:Total 197 kB/s | 32 kB 00:00 2024-06-27T21:01:32.262 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:32.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-06-27T21:01:32.265 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:32.265 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:32.278 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:32.278 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:32.360 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:32.482 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:32.535 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:05 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:32.625 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:32.669 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:32.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-06-27T21:01:32.824 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:10 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:32.852 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:32.910 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:32.951 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:32.952 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:32.952 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:33.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-06-27T21:01:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:33.072 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:33.072 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:33.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-06-27T21:01:33.284 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:06 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:33.410 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 32 k 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:33.411 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:33.412 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:33.412 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:33.412 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:33.412 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 32 k 2024-06-27T21:01:33.412 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 142 k 2024-06-27T21:01:33.412 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:33.572 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:11 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:33.652 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-19.0.0-3315.g98c31c1a.el9.x86_6 132 kB/s | 32 kB 00:00 2024-06-27T21:01:33.652 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:33.653 INFO:teuthology.orchestra.run.smithi190.stdout:Total 132 kB/s | 32 kB 00:00 2024-06-27T21:01:33.653 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:33.656 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:33.656 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:33.658 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:33.671 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:33.671 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:33.700 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:33.701 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-06-27T21:01:33.701 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:33.751 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:33.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-06-27T21:01:33.795 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:11 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:33.855 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:33.882 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:33.923 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:33.924 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:33.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-06-27T21:01:34.211 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:34.326 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:12 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:34.446 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:34.446 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:34.446 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:34.446 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:34.446 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:34.446 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:34.454 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:34.454 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 85 k 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:34.455 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:34.456 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 85 k 2024-06-27T21:01:34.456 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2024-06-27T21:01:34.456 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:34.542 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:12 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:34.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-06-27T21:01:34.627 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:34.669 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:34.670 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:34.670 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:34.700 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 350 kB/s | 85 kB 00:00 2024-06-27T21:01:34.700 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:34.701 INFO:teuthology.orchestra.run.smithi151.stdout:Total 347 kB/s | 85 kB 00:00 2024-06-27T21:01:34.701 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:34.708 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:34.708 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:34.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-06-27T21:01:34.764 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:34.765 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:34.909 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:35.017 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:35.157 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:08 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:35.243 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:35.284 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:35.285 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:35.285 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:35.301 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:13 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:35.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-06-27T21:01:35.387 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:35.394 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:35.429 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:35.430 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:35.430 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:35.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-06-27T21:01:35.638 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:35.639 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:35.639 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:35.639 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:35.639 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:35.639 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:35.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-06-27T21:01:35.901 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:08 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:35.986 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:36.028 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:36.029 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:36.029 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:36.068 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:14 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:36.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-06-27T21:01:36.153 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:36.194 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:36.195 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:36.195 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:36.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-06-27T21:01:36.433 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:14 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.1 M 2024-06-27T21:01:36.562 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 13 M 2024-06-27T21:01:36.563 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:36.649 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:09 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:36.734 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:36.776 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:36.777 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:36.777 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:36.822 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:14 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:36.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-06-27T21:01:36.906 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:36.947 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:36.948 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:36.948 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:37.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-06-27T21:01:37.386 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:10 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:37.408 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 3.7 MB/s | 3.1 MB 00:00 2024-06-27T21:01:37.409 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:37.409 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.7 MB/s | 3.1 MB 00:00 2024-06-27T21:01:37.410 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:37.419 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:37.419 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:37.470 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:37.494 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:37.494 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:37.511 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:37.512 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:37.512 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:37.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-06-27T21:01:37.572 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:15 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:37.658 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:37.699 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:37.700 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-27T21:01:37.700 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:37.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-06-27T21:01:37.916 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:37.984 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:38.009 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:38.010 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:38.010 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-27T21:01:38.010 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-27T21:01:38.010 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-27T21:01:38.010 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:38.121 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:11 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:38.206 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:38.246 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:38.247 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:38.248 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:38.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-06-27T21:01:38.327 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:16 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:38.457 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:38.457 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 85 k 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 231 k 2024-06-27T21:01:38.458 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:38.781 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 264 kB/s | 85 kB 00:00 2024-06-27T21:01:38.781 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:38.782 INFO:teuthology.orchestra.run.smithi192.stdout:Total 263 kB/s | 85 kB 00:00 2024-06-27T21:01:38.782 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:38.789 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:38.789 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:38.845 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:38.846 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:38.861 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:11 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:38.945 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 is already installed. 2024-06-27T21:01:38.980 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:38.986 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:38.987 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-06-27T21:01:38.988 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:39.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-06-27T21:01:39.060 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:39.072 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:39.073 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:39.073 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:39.073 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:39.073 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:39.073 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:39.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-06-27T21:01:39.400 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:39.605 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:12 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:39.646 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:39.646 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:39.646 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:39.647 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:39.647 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:39.647 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:39.732 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 85 k 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:39.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:39.734 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 85 k 2024-06-27T21:01:39.734 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 231 k 2024-06-27T21:01:39.734 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:39.869 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:17 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:39.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-06-27T21:01:39.997 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 171 k 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:39.998 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 171 k 2024-06-27T21:01:39.999 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 494 k 2024-06-27T21:01:39.999 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-06-27T21:01:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 265 kB/s | 85 kB 00:00 2024-06-27T21:01:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:Total 264 kB/s | 85 kB 00:00 2024-06-27T21:01:40.056 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:40.063 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:40.063 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:40.119 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:40.119 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:40.239 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 709 kB/s | 171 kB 00:00 2024-06-27T21:01:40.240 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:40.240 INFO:teuthology.orchestra.run.smithi151.stdout:Total 705 kB/s | 171 kB 00:00 2024-06-27T21:01:40.241 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-06-27T21:01:40.248 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-06-27T21:01:40.248 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-06-27T21:01:40.249 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:40.306 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-06-27T21:01:40.306 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-06-27T21:01:40.334 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:40.450 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-06-27T21:01:40.453 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:18 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:40.552 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:40.584 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:40.584 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.1 M 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:40.585 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:40.586 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.1 M 2024-06-27T21:01:40.586 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 13 M 2024-06-27T21:01:40.586 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:40.683 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:40.963 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:41.185 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 5.2 MB/s | 3.1 MB 00:00 2024-06-27T21:01:41.186 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:41.186 INFO:teuthology.orchestra.run.smithi192.stdout:Total 5.2 MB/s | 3.1 MB 00:00 2024-06-27T21:01:41.186 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:41.195 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:41.195 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:41.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-06-27T21:01:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:41.238 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-06-27T21:01:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:01:41.238 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-06-27T21:01:41.268 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:41.269 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:41.473 DEBUG:teuthology.parallel:result is None 2024-06-27T21:01:41.720 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:41.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:41.790 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:14 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:41.810 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:41.811 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:41.811 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-27T21:01:41.811 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-27T21:01:41.811 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-27T21:01:41.811 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:41.920 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 3.1 M 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:41.921 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.1 M 2024-06-27T21:01:41.922 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 13 M 2024-06-27T21:01:41.922 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:42.846 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:42.846 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:42.846 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:42.847 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:42.847 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:42.847 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:42.858 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 3.3 MB/s | 3.1 MB 00:00 2024-06-27T21:01:42.859 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:42.859 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.3 MB/s | 3.1 MB 00:00 2024-06-27T21:01:42.859 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:42.868 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:42.868 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:42.944 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:42.945 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:43.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-06-27T21:01:43.368 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:43.434 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:43.460 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:43.460 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-27T21:01:43.460 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-27T21:01:43.460 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-27T21:01:43.460 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-27T21:01:43.460 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:43.608 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:21 ago on Thu 27 Jun 2024 09:00:22 PM UTC. 2024-06-27T21:01:43.740 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-27T21:01:43.740 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:43.740 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 171 k 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 171 k 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 494 k 2024-06-27T21:01:43.741 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-27T21:01:44.080 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 505 kB/s | 171 kB 00:00 2024-06-27T21:01:44.081 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:44.081 INFO:teuthology.orchestra.run.smithi192.stdout:Total 502 kB/s | 171 kB 00:00 2024-06-27T21:01:44.081 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-27T21:01:44.089 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-27T21:01:44.089 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-27T21:01:44.145 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-27T21:01:44.145 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-27T21:01:44.297 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-27T21:01:44.408 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:44.563 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:44.563 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:44.563 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:44.563 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:44.563 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:44.563 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:44.807 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:44.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-06-27T21:01:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:45.038 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-27T21:01:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:01:45.038 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-27T21:01:45.241 DEBUG:teuthology.parallel:result is None 2024-06-27T21:01:45.386 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:18 ago on Thu 27 Jun 2024 09:00:27 PM UTC. 2024-06-27T21:01:45.516 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-06-27T21:01:45.517 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:45.517 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-06-27T21:01:45.517 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-3315.g98c31c1a.el9 ceph 171 k 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 171 k 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 494 k 2024-06-27T21:01:45.518 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-06-27T21:01:45.816 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-19.0.0-3315.g98c31c1a.el9.x86_64.rpm 573 kB/s | 171 kB 00:00 2024-06-27T21:01:45.816 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:01:45.817 INFO:teuthology.orchestra.run.smithi190.stdout:Total 571 kB/s | 171 kB 00:00 2024-06-27T21:01:45.817 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-06-27T21:01:45.824 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-06-27T21:01:45.825 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-06-27T21:01:45.884 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-06-27T21:01:45.884 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-06-27T21:01:46.026 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-06-27T21:01:46.128 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:46.528 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 1/1 2024-06-27T21:01:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:46.801 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-06-27T21:01:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-3315.g98c31c1a.el9.x86_64 2024-06-27T21:01:46.801 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-06-27T21:01:46.801 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-06-27T21:01:47.037 DEBUG:teuthology.parallel:result is None 2024-06-27T21:01:47.037 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:01:47.037 INFO:teuthology.packaging:ref: None 2024-06-27T21:01:47.037 INFO:teuthology.packaging:tag: None 2024-06-27T21:01:47.037 INFO:teuthology.packaging:branch: squid 2024-06-27T21:01:47.037 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:01:47.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-27T21:01:47.246 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-27T21:01:47.272 INFO:teuthology.orchestra.run.smithi151.stdout:19.0.0-3315.g98c31c1a.el9 2024-06-27T21:01:47.273 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3315.g98c31c1a.el9 2024-06-27T21:01:47.273 INFO:teuthology.task.install:The correct ceph version 19.0.0-3315.g98c31c1a is installed. 2024-06-27T21:01:47.275 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:01:47.275 INFO:teuthology.packaging:ref: None 2024-06-27T21:01:47.275 INFO:teuthology.packaging:tag: None 2024-06-27T21:01:47.276 INFO:teuthology.packaging:branch: squid 2024-06-27T21:01:47.276 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:01:47.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-27T21:01:47.484 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-27T21:01:47.506 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-3315.g98c31c1a.el9 2024-06-27T21:01:47.506 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3315.g98c31c1a.el9 2024-06-27T21:01:47.506 INFO:teuthology.task.install:The correct ceph version 19.0.0-3315.g98c31c1a is installed. 2024-06-27T21:01:47.508 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-27T21:01:47.509 INFO:teuthology.packaging:ref: None 2024-06-27T21:01:47.509 INFO:teuthology.packaging:tag: None 2024-06-27T21:01:47.509 INFO:teuthology.packaging:branch: squid 2024-06-27T21:01:47.509 INFO:teuthology.packaging:sha1: f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:01:47.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-27T21:01:47.712 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-27T21:01:47.739 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-3315.g98c31c1a.el9 2024-06-27T21:01:47.739 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3315.g98c31c1a.el9 2024-06-27T21:01:47.739 INFO:teuthology.task.install:The correct ceph version 19.0.0-3315.g98c31c1a is installed. 2024-06-27T21:01:47.741 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-27T21:01:47.741 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:01:47.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-27T21:01:47.771 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:01:47.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-27T21:01:47.799 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:01:47.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-27T21:01:47.830 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-27T21:01:47.830 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:01:47.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-06-27T21:01:47.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-27T21:01:47.930 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:01:47.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-06-27T21:01:47.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-27T21:01:48.021 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:01:48.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-06-27T21:01:48.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-27T21:01:48.120 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-27T21:01:48.121 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:01:48.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-27T21:01:48.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-27T21:01:48.217 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:01:48.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-27T21:01:48.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-27T21:01:48.308 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:01:48.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-27T21:01:48.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-27T21:01:48.409 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-27T21:01:48.409 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:01:48.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-06-27T21:01:48.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-27T21:01:48.506 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:01:48.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-06-27T21:01:48.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-27T21:01:48.595 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:01:48.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-06-27T21:01:48.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-27T21:01:48.695 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-27T21:01:48.800 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'} 2024-06-27T21:01:48.801 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:01:48.801 INFO:tasks.cephadm:Cluster fsid is 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:01:48.801 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-27T21:01:48.801 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.151', 'mon.b': '172.21.15.190', 'mon.c': '172.21.15.192'} 2024-06-27T21:01:48.801 INFO:tasks.cephadm:First mon is mon.a on smithi151 2024-06-27T21:01:48.802 INFO:tasks.cephadm:First mgr is y 2024-06-27T21:01:48.802 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-27T21:01:48.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-06-27T21:01:48.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-06-27T21:01:48.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-06-27T21:01:48.886 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-06-27T21:01:48.886 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:01:49.025 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/80683/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-4578-gf51acc18', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-4578.gf51acc18'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-06-26 19:03:15.778024', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri12-testing-2024-06-26-0904', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-06-27T21:01:49.166 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-yuri12-testing-2024-06-26-0904, sha1 f51acc18cc950586c82c58d666661d4391fc1122 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:01:49.168 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/x86_64/flavors/default/cephadm 2024-06-27T21:01:49.168 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/x86_64/flavors/default/cephadm 2024-06-27T21:01:49.168 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:50.002 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 783074 Jun 27 21:01 /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:50.003 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:50.804 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 783074 Jun 27 21:01 /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:50.804 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri12-testing-2024-06-26-0904/f51acc18cc950586c82c58d666661d4391fc1122/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:51.590 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 783074 Jun 27 21:01 /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:51.591 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:51.610 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:51.630 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-27T21:01:51.662 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 on all hosts... 2024-06-27T21:01:51.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 pull 2024-06-27T21:01:51.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 pull 2024-06-27T21:01:51.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 pull 2024-06-27T21:01:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122... 2024-06-27T21:01:52.537 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122... 2024-06-27T21:01:52.639 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122... 2024-06-27T21:02:26.470 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:02:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T21:02:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd", 2024-06-27T21:02:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-06-27T21:02:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849" 2024-06-27T21:02:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:02:26.471 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:02:40.344 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:02:40.344 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T21:02:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd", 2024-06-27T21:02:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-06-27T21:02:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849" 2024-06-27T21:02:40.345 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:02:40.345 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:02:45.249 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:02:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev)", 2024-06-27T21:02:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd", 2024-06-27T21:02:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-06-27T21:02:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849" 2024-06-27T21:02:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:02:45.250 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:02:45.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-06-27T21:02:45.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-06-27T21:02:45.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-06-27T21:02:45.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-06-27T21:02:45.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-06-27T21:02:45.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-06-27T21:02:45.468 INFO:tasks.cephadm:Writing seed config... 2024-06-27T21:02:45.469 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-06-27T21:02:45.469 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-06-27T21:02:45.470 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-06-27T21:02:45.471 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-06-27T21:02:45.471 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-06-27T21:02:45.471 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-06-27T21:02:45.471 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-06-27T21:02:45.471 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-06-27T21:02:45.472 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-06-27T21:02:45.473 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-06-27T21:02:45.474 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:02:45.474 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-27T21:02:45.490 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 784a4e4c-34c8-11ef-bca8-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-06-27T21:02:45.491 DEBUG:teuthology.orchestra.run.smithi151:mon.a> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a.service 2024-06-27T21:02:45.533 DEBUG:teuthology.orchestra.run.smithi151:mgr.y> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y.service 2024-06-27T21:02:45.575 INFO:tasks.cephadm:Bootstrapping... 2024-06-27T21:02:45.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 -v bootstrap --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.151 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:02:45.782 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-06-27T21:02:45.783 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122', '-v', 'bootstrap', '--fsid', '784a4e4c-34c8-11ef-bca8-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.151', '--skip-admin-label'] 2024-06-27T21:02:45.783 INFO:teuthology.orchestra.run.smithi151.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-27T21:02:45.783 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying podman|docker is present... 2024-06-27T21:02:45.808 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.1.0 2024-06-27T21:02:45.809 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying lvm2 is present... 2024-06-27T21:02:45.809 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying time synchronization is in place... 2024-06-27T21:02:45.816 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-27T21:02:45.816 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-27T21:02:45.822 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-27T21:02:45.822 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-06-27T21:02:45.829 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-06-27T21:02:45.835 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-06-27T21:02:45.835 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-06-27T21:02:45.835 INFO:teuthology.orchestra.run.smithi151.stdout:Repeating the final host check... 2024-06-27T21:02:45.859 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.1.0 2024-06-27T21:02:45.859 INFO:teuthology.orchestra.run.smithi151.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-27T21:02:45.859 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl is present 2024-06-27T21:02:45.859 INFO:teuthology.orchestra.run.smithi151.stdout:lvcreate is present 2024-06-27T21:02:45.865 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-27T21:02:45.865 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-27T21:02:45.872 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-27T21:02:45.872 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-06-27T21:02:45.878 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Host looks OK 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Cluster fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Acquiring lock 139925118536576 on /run/cephadm/784a4e4c-34c8-11ef-bca8-c7b262605968.lock 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Lock 139925118536576 acquired on /run/cephadm/784a4e4c-34c8-11ef-bca8-c7b262605968.lock 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 3300 ... 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 6789 ... 2024-06-27T21:02:45.887 INFO:teuthology.orchestra.run.smithi151.stdout:Base mon IP(s) is [172.21.15.151:3300, 172.21.15.151:6789], mon addrv is [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-06-27T21:02:45.889 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.151 metric 100 2024-06-27T21:02:45.889 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.151 metric 100 2024-06-27T21:02:45.892 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-27T21:02:45.892 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-27T21:02:45.892 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6f63/64 scope link 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-27T21:02:45.895 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-06-27T21:02:45.896 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-06-27T21:02:45.896 INFO:teuthology.orchestra.run.smithi151.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-27T21:02:45.896 INFO:teuthology.orchestra.run.smithi151.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-27T21:02:45.896 INFO:teuthology.orchestra.run.smithi151.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122... 2024-06-27T21:02:46.387 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd 2024-06-27T21:02:46.387 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122... 2024-06-27T21:02:46.387 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Getting image source signatures 2024-06-27T21:02:46.387 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:84eb551f19db5b66f5df74b73a4fce798717fa9c8018a22eb0c823f9eb7ff7cd 2024-06-27T21:02:46.387 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:0d9665cbb1e45ff458d07612450003bc3c523a9d941d88d73ef519caf59c67ab 2024-06-27T21:02:46.388 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying config sha256:8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd 2024-06-27T21:02:46.388 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-27T21:02:47.225 INFO:teuthology.orchestra.run.smithi151.stdout:ceph: stdout ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev) 2024-06-27T21:02:47.225 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph version: ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev) 2024-06-27T21:02:47.226 INFO:teuthology.orchestra.run.smithi151.stdout:Extracting ceph user uid/gid from container image... 2024-06-27T21:02:48.174 INFO:teuthology.orchestra.run.smithi151.stdout:stat: stdout 167 167 2024-06-27T21:02:48.174 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial keys... 2024-06-27T21:02:49.056 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQB4031mN5ZHHhAAr8dTyW7Uo7xwDhxyESKCXg== 2024-06-27T21:02:49.956 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQB5031mn+o+FRAAjfiqmk312zAzHMdwFuQwDg== 2024-06-27T21:02:50.880 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQB6031mGAhIERAA7xtgzuNpt+yJekx5NJlSTg== 2024-06-27T21:02:50.880 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial monmap... 2024-06-27T21:02:51.922 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-27T21:02:51.922 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-06-27T21:02:51.922 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:51.922 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-27T21:02:51.922 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool for a [v2:172.21.15.151:3300,v1:172.21.15.151:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-27T21:02:51.923 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = quincy 2024-06-27T21:02:51.923 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: set fsid to 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:51.923 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-27T21:02:51.923 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:02:51.923 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mon... 2024-06-27T21:02:52.862 INFO:teuthology.orchestra.run.smithi151.stdout:create mon.a on 2024-06-27T21:02:53.061 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-27T21:02:53.216 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-27T21:02:53.401 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968.target → /etc/systemd/system/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968.target. 2024-06-27T21:02:53.401 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968.target → /etc/systemd/system/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968.target. 2024-06-27T21:02:53.641 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a 2024-06-27T21:02:53.641 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a.service: Unit ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a.service not loaded. 2024-06-27T21:02:53.804 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968.target.wants/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a.service → /etc/systemd/system/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@.service. 2024-06-27T21:02:54.330 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-27T21:02:54.330 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-06-27T21:02:54.338 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-27T21:02:54.338 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-06-27T21:02:54.338 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-06-27T21:02:54.338 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-06-27T21:02:54.339 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon to start... 2024-06-27T21:02:54.339 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon... 2024-06-27T21:02:55.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:55 smithi151 ceph-mon[28597]: from='client.? 172.21.15.151:0/913934583' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-27T21:02:55.368 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout cluster: 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout id: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout services: 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.534724s) 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:02:55.370 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout data: 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pgs: 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:mon is available 2024-06-27T21:02:55.371 INFO:teuthology.orchestra.run.smithi151.stdout:Assimilating anything we can from ceph.conf... 2024-06-27T21:02:56.411 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-27T21:02:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-27T21:02:56.413 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:02:56.413 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-06-27T21:02:56.413 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-27T21:02:56.413 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-27T21:02:56.413 INFO:teuthology.orchestra.run.smithi151.stdout:Generating new minimal ceph.conf... 2024-06-27T21:02:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:56 smithi151 ceph-mon[28597]: from='client.? 172.21.15.151:0/1083499083' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-27T21:02:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:56 smithi151 ceph-mon[28597]: from='client.? 172.21.15.151:0/1083499083' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-27T21:02:57.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:57 smithi151 ceph-mon[28597]: from='client.? 172.21.15.151:0/4100068967' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:02:57.451 INFO:teuthology.orchestra.run.smithi151.stdout:Restarting the monitor... 2024-06-27T21:02:57.727 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:57 smithi151 systemd[1]: Stopping Ceph mon.a for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:02:57.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:57 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28593]: 2024-06-27T21:02:57.735+0000 7f68a316b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:02:57.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:57 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28593]: 2024-06-27T21:02:57.735+0000 7f68a316b640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-27T21:02:57.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:57 smithi151 podman[28842]: 2024-06-27 21:02:57.991231563 +0000 UTC m=+0.363199761 container died b80324c0388641445b59799bfadb05d9672d543c591f9eb78ad8bf8f9c5931b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:02:58.503 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 podman[28842]: 2024-06-27 21:02:58.335604114 +0000 UTC m=+0.707572313 container cleanup b80324c0388641445b59799bfadb05d9672d543c591f9eb78ad8bf8f9c5931b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:02:58.503 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 bash[28842]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a 2024-06-27T21:02:58.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 podman[28855]: 2024-06-27 21:02:58.511792794 +0000 UTC m=+0.514111875 container remove b80324c0388641445b59799bfadb05d9672d543c591f9eb78ad8bf8f9c5931b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-27T21:02:59.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 systemd[1]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-27T21:02:59.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 systemd[1]: Stopped Ceph mon.a for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:02:59.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 systemd[1]: Starting Ceph mon.a for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:02:59.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 podman[28947]: 2024-06-27 21:02:58.971108512 +0000 UTC m=+0.108064312 container create 2adefe13d0e02bd30496252c93edad231fdfb85a9c71157dfa6cbbc8224d2769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.schema-version=1.0) 2024-06-27T21:02:59.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:58 smithi151 podman[28947]: 2024-06-27 21:02:58.879883843 +0000 UTC m=+0.016839651 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 2024-06-27T21:02:59.244 INFO:teuthology.orchestra.run.smithi151.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-27T21:02:59.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 podman[28947]: 2024-06-27 21:02:59.174174368 +0000 UTC m=+0.311130183 container init 2adefe13d0e02bd30496252c93edad231fdfb85a9c71157dfa6cbbc8224d2769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 podman[28947]: 2024-06-27 21:02:59.177612235 +0000 UTC m=+0.314568035 container start 2adefe13d0e02bd30496252c93edad231fdfb85a9c71157dfa6cbbc8224d2769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240625, io.buildah.version=1.36.0) 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: set uid:gid to 167:167 (ceph:ceph) 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: ceph version 19.0.0-4578-gf51acc18 (f51acc18cc950586c82c58d666661d4391fc1122) squid (dev), process ceph-mon, pid 2 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: pidfile_write: ignore empty --pid-file 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: load: jerasure load: lrc 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: RocksDB version: 7.9.2 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Git sha 0 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Compile date 2024-06-26 17:38:35 2024-06-27T21:02:59.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: DB SUMMARY 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: DB Session ID: MAF4WBETUDRAIP386LBD 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: CURRENT file: CURRENT 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: IDENTITY file: IDENTITY 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 102326 ; 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.error_if_exists: 0 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.create_if_missing: 0 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.paranoid_checks: 1 2024-06-27T21:02:59.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-27T21:02:59.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-27T21:02:59.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-27T21:02:59.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.env: 0x5593e4dcb8c0 2024-06-27T21:02:59.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.fs: PosixFileSystem 2024-06-27T21:02:59.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.info_log: 0x5593e73e54c0 2024-06-27T21:02:59.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_file_opening_threads: 16 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.statistics: (nil) 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.use_fsync: 0 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_log_file_size: 0 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.keep_log_file_num: 1000 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.recycle_log_file_num: 0 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.allow_fallocate: 1 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.allow_mmap_reads: 0 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.allow_mmap_writes: 0 2024-06-27T21:02:59.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.use_direct_reads: 0 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.create_missing_column_families: 0 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.db_log_dir: 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.wal_dir: 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-27T21:02:59.424 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.advise_random_on_open: 1 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.db_write_buffer_size: 0 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.write_buffer_manager: 0x5593e73e8e60 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.rate_limiter: (nil) 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.wal_recovery_mode: 2 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.enable_thread_tracking: 0 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.enable_pipelined_write: 0 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.unordered_write: 0 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-27T21:02:59.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.row_cache: None 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.wal_filter: None 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.allow_ingest_behind: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.two_write_queues: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.manual_wal_flush: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.wal_compression: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.atomic_flush: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-27T21:02:59.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.log_readahead_size: 0 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.best_efforts_recovery: 0 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.allow_data_in_errors: 0 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.db_host_id: __hostname__ 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_background_jobs: 2 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_background_compactions: -1 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_subcompactions: 1 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-27T21:02:59.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_total_wal_size: 0 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_open_files: -1 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bytes_per_sync: 0 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_readahead_size: 0 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_background_flushes: -1 2024-06-27T21:02:59.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Compression algorithms supported: 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kZSTD supported: 0 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kXpressCompression supported: 0 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kBZip2Compression supported: 0 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kLZ4Compression supported: 1 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kZlibCompression supported: 1 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kLZ4HCCompression supported: 1 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: kSnappyCompression supported: 1 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-06-27T21:02:59.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.merge_operator: 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_filter: None 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_filter_factory: None 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.sst_partitioner_factory: None 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5593e73e50e0) 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-06-27T21:02:59.430 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_type: 0 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_index_type: 0 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_shortening: 1 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: checksum: 4 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: no_block_cache: 0 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache: 0x5593e73d9610 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-06-27T21:02:59.431 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_options: 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: capacity : 536870912 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_shard_bits : 4 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: strict_capacity_limit : 0 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_compressed: (nil) 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: persistent_cache: (nil) 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size: 4096 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size_deviation: 10 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_restart_interval: 16 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_block_restart_interval: 1 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: metadata_block_size: 4096 2024-06-27T21:02:59.432 INFO:journalctl@ceph.mon.a.smithi151.stdout: partition_filters: 0 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: use_delta_encoding: 1 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: filter_policy: bloomfilter 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: whole_key_filtering: 1 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: verify_compression: 0 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: format_version: 5 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_index_compression: 1 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_align: 0 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_auto_readahead_size: 262144 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: prepopulate_block_cache: 0 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: initial_auto_readahead_size: 8192 2024-06-27T21:02:59.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.write_buffer_size: 33554432 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_write_buffer_number: 2 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression: NoCompression 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression: Disabled 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.prefix_extractor: nullptr 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.num_levels: 7 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-27T21:02:59.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.level: 32767 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.strategy: 0 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-27T21:02:59.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.enabled: false 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.target_file_size_base: 67108864 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-27T21:02:59.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.arena_block_size: 1048576 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.disable_auto_compactions: 0 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-27T21:02:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.inplace_update_support: 0 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-27T21:02:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.bloom_locality: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.max_successive_merges: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.paranoid_file_checks: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.force_consistency_checks: 1 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.report_bg_io_stats: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.ttl: 2592000 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.enable_blob_files: false 2024-06-27T21:02:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.min_blob_size: 0 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.blob_file_size: 268435456 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.blob_file_starting_level: 0 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c0c28c4-93e8-4896-9713-78d2517d1f0a 2024-06-27T21:02:59.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: EVENT_LOG_v1 {"time_micros": 1719522179203574, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: EVENT_LOG_v1 {"time_micros": 1719522179204572, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 97573, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 95556, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13537, "raw_average_key_size": 50, "raw_value_size": 88214, "raw_average_value_size": 327, "num_data_blocks": 12, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1719522179, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4c0c28c4-93e8-4896-9713-78d2517d1f0a", "db_session_id": "MAF4WBETUDRAIP386LBD", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: EVENT_LOG_v1 {"time_micros": 1719522179204668, "job": 1, "event": "recovery_finished"} 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5593e7408e00 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: DB pointer 0x5593e7506000 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** DB Stats ** 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-27T21:02:59.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: L0 2/0 97.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Sum 2/0 97.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-27T21:02:59.442 INFO:journalctl@ceph.mon.a.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-27T21:02:59.443 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative compaction: 0.00 GB write, 30.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval compaction: 0.00 GB write, 30.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache BinnedLRUCache@0x5593e73d9610#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: starting mon.a rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???) e1 preinit fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:59.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).mds e1 new map 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).mds e1 print_map 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: e1 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: btime 2024-06-27T21:02:54:280563+0000 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: legacy client fscid: -1 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout: No filesystems configured 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-27T21:02:59.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 bash[28947]: 2adefe13d0e02bd30496252c93edad231fdfb85a9c71157dfa6cbbc8224d2769 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 systemd[1]: Started Ceph mon.a for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: monmap epoch 1 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: last_changed 2024-06-27T21:02:51.211266+0000 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: created 2024-06-27T21:02:51.211266+0000 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: min_mon_release 19 (squid) 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: election_strategy: 1 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-06-27T21:02:59.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: fsmap 2024-06-27T21:02:59.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: osdmap e1: 0 total, 0 up, 0 in 2024-06-27T21:02:59.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:02:59 smithi151 ceph-mon[28965]: mgrmap e1: no daemons active 2024-06-27T21:03:00.285 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-27T21:03:00.306 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:03:00.307 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mgr... 2024-06-27T21:03:00.307 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-27T21:03:00.307 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-27T21:03:00.568 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y 2024-06-27T21:03:00.568 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y.service: Unit ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y.service not loaded. 2024-06-27T21:03:00.717 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968.target.wants/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@.service. 2024-06-27T21:03:01.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3914278067' entity='client.admin' 2024-06-27T21:03:01.251 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-27T21:03:01.251 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-06-27T21:03:01.259 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-27T21:03:01.259 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-06-27T21:03:01.259 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-06-27T21:03:01.260 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-06-27T21:03:01.268 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-27T21:03:01.268 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-06-27T21:03:01.277 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-27T21:03:01.277 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-06-27T21:03:01.277 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-06-27T21:03:01.277 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-06-27T21:03:01.277 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr to start... 2024-06-27T21:03:01.278 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr... 2024-06-27T21:03:02.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1081227246' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-27T21:03:02.308 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:03:02.308 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-27T21:03:02.309 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-06-27T21:03:02.310 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:02.310 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-27T21:03:02.310 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-27T21:03:02.310 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-27T21:03:02.311 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:02.312 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "btime": "2024-06-27T21:02:54:280563+0000", 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.313 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-06-27T21:02:54.281217+0000", 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:02.314 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (1/15)... 2024-06-27T21:03:06.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2305284782' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-27T21:03:07.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: Activating manager daemon y 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: mgrmap e2: y(active, starting, since 0.00296237s) 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: Manager daemon y is now available 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-27T21:03:07.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' 2024-06-27T21:03:07.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' 2024-06-27T21:03:07.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:06 smithi151 ceph-mon[28965]: from='mgr.14100 172.21.15.151:0/1501465606' entity='mgr.y' 2024-06-27T21:03:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:08 smithi151 ceph-mon[28965]: mgrmap e3: y(active, since 1.00755s) 2024-06-27T21:03:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:09 smithi151 ceph-mon[28965]: mgrmap e4: y(active, since 2s) 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-27T21:03:12.116 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:12.117 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-27T21:03:12.118 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "btime": "2024-06-27T21:02:54:280563+0000", 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-27T21:03:12.119 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-27T21:03:12.120 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-06-27T21:02:54.281217+0000", 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:12.121 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (2/15)... 2024-06-27T21:03:15.328 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1184370932' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-27T21:03:15.599 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-27T21:03:15.600 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-27T21:03:15.601 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-27T21:03:15.605 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-27T21:03:15.605 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-27T21:03:15.605 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-27T21:03:15.605 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:15.606 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "btime": "2024-06-27T21:02:54:280563+0000", 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-06-27T21:02:54.281217+0000", 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:mgr is available 2024-06-27T21:03:16.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2790701269' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-27T21:03:16.938 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr 2024-06-27T21:03:16.198+0000 7f7ab659c640 0 --2- 172.21.15.151:0/2790701269 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7ab8107a50 0x7f7ab819b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-27T21:03:16.939 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling cephadm module... 2024-06-27T21:03:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:17 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3530065587' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-27T21:03:17.940 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:17 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: ignoring --setgroup ceph since I am not root 2024-06-27T21:03:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3530065587' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-27T21:03:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:18 smithi151 ceph-mon[28965]: mgrmap e5: y(active, since 11s) 2024-06-27T21:03:22.567 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1340443336' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-27T21:03:22.862 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-06-27T21:03:22.863 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 5... 2024-06-27T21:03:23.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: Active manager daemon y restarted 2024-06-27T21:03:23.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: Activating manager daemon y 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: osdmap e2: 0 total, 0 up, 0 in 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: mgrmap e6: y(active, starting, since 0.00298465s) 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: Manager daemon y is now available 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-27T21:03:23.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:23 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-27T21:03:24.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-mon[28965]: Found migration_current of "None". Setting to last migration. 2024-06-27T21:03:24.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:24.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-mon[28965]: mgrmap e7: y(active, since 1.00565s) 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:03:24.487+0000 7fb909c9e640 -1 log_channel(cephadm) log [ERR] : [27/Jun/2024:21:03:24] ENGINE Error in HTTPServer.serve 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: Traceback (most recent call last): 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self._connections.run(self.expiration_interval) 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self._run(expiration_interval) 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-27T21:03:24.690 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: new_conn = self._from_server_socket(self.server.socket) 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: s = self.context.wrap_socket( 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: return self.sslsocket_class._create( 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self.do_handshake() 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self._sslobj.do_handshake() 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-27T21:03:24.691 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:24 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:03:24.852 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:24.853 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-27T21:03:24.853 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-27T21:03:24.853 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:24.853 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 5 is available 2024-06-27T21:03:24.853 INFO:teuthology.orchestra.run.smithi151.stdout:Setting orchestrator backend to cephadm... 2024-06-27T21:03:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-27T21:03:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-27T21:03:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:24] ENGINE Bus STARTING 2024-06-27T21:03:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:24] ENGINE Serving on https://172.21.15.151:7150 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:24] ENGINE Error in HTTPServer.serve 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-27T21:03:25.440 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:24] ENGINE Serving on http://172.21.15.151:8765 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:24] ENGINE Bus STARTED 2024-06-27T21:03:25.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:25 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:26.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:26 smithi151 ceph-mon[28965]: mgrmap e8: y(active, since 2s) 2024-06-27T21:03:26.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:26 smithi151 ceph-mon[28965]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:26.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:26 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:26.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:26 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:27.139 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-27T21:03:27.140 INFO:teuthology.orchestra.run.smithi151.stdout:Generating ssh key... 2024-06-27T21:03:27.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-mon[28965]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:28.180 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: Generating public/private rsa key pair. 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: Your identification has been saved in /tmp/tmpmmqa7hlr/key 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: Your public key has been saved in /tmp/tmpmmqa7hlr/key.pub 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: The key fingerprint is: 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: SHA256:2VSGaiclmpfTE12P7Vr24LDP1UZqzI8FUz++EsjOyhU ceph-784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: The key's randomart image is: 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: +---[RSA 3072]----+ 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | oo.. | 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | . +o. + | 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | o *.. . +| 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | o B++ o.| 2024-06-27T21:03:28.181 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | oS+oEo +.*| 2024-06-27T21:03:28.182 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | o.B.X+| 2024-06-27T21:03:28.182 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | o.. O.*| 2024-06-27T21:03:28.182 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | . .o = *.| 2024-06-27T21:03:28.182 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: | o. =..| 2024-06-27T21:03:28.182 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: +----[SHA256]-----+ 2024-06-27T21:03:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:28 smithi151 ceph-mon[28965]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:29.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:28 smithi151 ceph-mon[28965]: Generating ssh key... 2024-06-27T21:03:29.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:28 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:29.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:28 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:29.392 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:03:29.393 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-27T21:03:29.393 INFO:teuthology.orchestra.run.smithi151.stdout:Adding key to root@localhost authorized_keys... 2024-06-27T21:03:29.393 INFO:teuthology.orchestra.run.smithi151.stdout:Adding host smithi151... 2024-06-27T21:03:30.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:29 smithi151 ceph-mon[28965]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:31.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:30 smithi151 ceph-mon[28965]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "addr": "172.21.15.151", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:32.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:31 smithi151 ceph-mon[28965]: Deploying cephadm binary to smithi151 2024-06-27T21:03:33.775 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Added host 'smithi151' with addr '172.21.15.151' 2024-06-27T21:03:33.776 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mon service... 2024-06-27T21:03:34.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:34 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:34.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:34 smithi151 ceph-mon[28965]: Added host smithi151 2024-06-27T21:03:34.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:34 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:35.142 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-27T21:03:35.142 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mgr service... 2024-06-27T21:03:35.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:35 smithi151 ceph-mon[28965]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:35.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:35 smithi151 ceph-mon[28965]: Saving service mon spec with placement count:5 2024-06-27T21:03:35.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:35 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:36.358 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-27T21:03:36.985 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:36 smithi151 ceph-mon[28965]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:36.985 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:36 smithi151 ceph-mon[28965]: Saving service mgr spec with placement count:2 2024-06-27T21:03:36.985 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:36 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:38.274 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2018849095' entity='client.admin' 2024-06-27T21:03:38.274 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:38 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:39 smithi151 ceph-mon[28965]: from='mgr.14118 172.21.15.151:0/2694789550' entity='mgr.y' 2024-06-27T21:03:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1150999970' entity='client.admin' 2024-06-27T21:03:39.725 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling the dashboard module... 2024-06-27T21:03:40.890 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2702346027' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-27T21:03:40.890 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:40 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: ignoring --setuser ceph since I am not root 2024-06-27T21:03:40.890 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:40 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: ignoring --setgroup ceph since I am not root 2024-06-27T21:03:41.879 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2702346027' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-27T21:03:41.879 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:41 smithi151 ceph-mon[28965]: mgrmap e9: y(active, since 17s) 2024-06-27T21:03:42.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:42.475 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-27T21:03:42.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-27T21:03:42.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-06-27T21:03:42.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-27T21:03:42.476 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:42.476 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-06-27T21:03:42.476 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 9... 2024-06-27T21:03:42.816 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3351591976' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-27T21:03:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: Active manager daemon y restarted 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: Activating manager daemon y 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: osdmap e3: 0 total, 0 up, 0 in 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: mgrmap e10: y(active, starting, since 0.0039502s) 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: Manager daemon y is now available 2024-06-27T21:03:46.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:46.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-27T21:03:46.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-27T21:03:46.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:46 smithi151 ceph-mon[28965]: mgrmap e11: y(active, since 1.0061s) 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:03:47.218+0000 7fc2df189640 -1 log_channel(cephadm) log [ERR] : [27/Jun/2024:21:03:47] ENGINE Error in HTTPServer.serve 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: Traceback (most recent call last): 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self._connections.run(self.expiration_interval) 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self._run(expiration_interval) 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: new_conn = self._from_server_socket(self.server.socket) 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-27T21:03:47.440 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: s = self.context.wrap_socket( 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: return self.sslsocket_class._create( 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self.do_handshake() 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: self._sslobj.do_handshake() 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-27T21:03:47.441 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:03:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:03:47.568 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-06-27T21:03:47.568 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-27T21:03:47.568 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-27T21:03:47.568 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-06-27T21:03:47.568 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 9 is available 2024-06-27T21:03:47.569 INFO:teuthology.orchestra.run.smithi151.stdout:Generating a dashboard self-signed certificate... 2024-06-27T21:03:48.832 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-27T21:03:48.832 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial admin user... 2024-06-27T21:03:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:47] ENGINE Bus STARTING 2024-06-27T21:03:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:47] ENGINE Serving on https://172.21.15.151:7150 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:47] ENGINE Error in HTTPServer.serve 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-27T21:03:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:47] ENGINE Serving on http://172.21.15.151:8765 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: [27/Jun/2024:21:03:47] ENGINE Bus STARTED 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:48.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:48.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:49 smithi151 ceph-mon[28965]: from='client.14163 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:49 smithi151 ceph-mon[28965]: mgrmap e12: y(active, since 2s) 2024-06-27T21:03:50.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:50 smithi151 ceph-mon[28965]: from='client.14165 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:03:50.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:50.927 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ghe8ragHsicrjFPBhYZTluzmgnJfDq2W/.KZaYxPf6Z.v/W/h.K86", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1719522229, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-27T21:03:50.927 INFO:teuthology.orchestra.run.smithi151.stdout:Fetching dashboard port number... 2024-06-27T21:03:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/46763518' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-27T21:03:52.125 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 8443 2024-06-27T21:03:52.136 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-27T21:03:52.136 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-06-27T21:03:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-27T21:03:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-06-27T21:03:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-06-27T21:03:52.143 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-27T21:03:52.145 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph Dashboard is now available at: 2024-06-27T21:03:52.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:52.145 INFO:teuthology.orchestra.run.smithi151.stdout: URL: https://smithi151.front.sepia.ceph.com:8443/ 2024-06-27T21:03:52.145 INFO:teuthology.orchestra.run.smithi151.stdout: User: admin 2024-06-27T21:03:52.145 INFO:teuthology.orchestra.run.smithi151.stdout: Password: 14rgpgfcvj 2024-06-27T21:03:52.145 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:52.146 INFO:teuthology.orchestra.run.smithi151.stdout:Saving cluster configuration to /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config directory 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout:Or, if you are only running a single cluster on this host: 2024-06-27T21:03:53.416 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: ceph telemetry on 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout:For more information see: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:03:53.417 INFO:teuthology.orchestra.run.smithi151.stdout:Bootstrap complete. 2024-06-27T21:03:53.456 INFO:tasks.cephadm:Fetching config... 2024-06-27T21:03:53.457 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:03:53.457 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-27T21:03:53.473 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-27T21:03:53.473 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:03:53.473 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-27T21:03:53.529 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-27T21:03:53.530 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:03:53.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-27T21:03:53.595 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-27T21:03:53.595 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:03:53.595 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-27T21:03:53.651 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-27T21:03:53.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-27T21:03:53.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:53 smithi151 ceph-mon[28965]: mgrmap e13: y(active, since 6s) 2024-06-27T21:03:53.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/517756482' entity='client.admin' 2024-06-27T21:03:53.722 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:03:53.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-27T21:03:53.775 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:03:53.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-27T21:03:53.829 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFNFQU7FMMNkebblFgjdph8tecka4+g5FAnh47k3GL8dZxQvCPYGL5I0pFSiUFZ07PQz8H8FFH6lT/0kSpGfJ76B7ArY2f++4nqCZu+IyNIuj/cWXlipkEjji3aWYhJQq1SE1bTpwCy3vlerqxXCn5nsOrpVjGN3ydDWqvh/LgngRzsjyTUWi/NS/Q9K2iCLH9GVP+tJL8hNF/bm21s5ZiiejmVJ56lMIIKISJ7l8TWmW62Ime+AFLz45Zr13U+zGgPXp9jp1DhKt7A5TuowwI5iOVOBbhdp16++sVa6efhY2AGilU6wHAlJtiI1efIHdffBSHaMRvASxkSWY4hl+yAnpMVffThrtOvaT4wvqAZ8T7uNfY5R1Mg5r8YlBNeQfEO4uhoKdw8njDzNsHS8vbJComnDINRf5YUn2vkvPK2U/2eLVW/mOxmietIIQSLA3WkusFOdpATj3NJ39db4i5X9ucacRYFUxyccQrnznDjWx6cPiOPxnhnj4V0FAJ6Ys= ceph-784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:03:53.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-27T21:03:54.096 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:03:56.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:56.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:56.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:03:56.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:56.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:56.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:56.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2671274572' entity='client.admin' 2024-06-27T21:03:56.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:03:56.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:03:56.942 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-27T21:03:56.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-27T21:03:57.190 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:03:59.603 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-06-27T21:03:59.603 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:03:59.603 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-06-27T21:03:59.622 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:03:59.622 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:03:59.680 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-06-27T21:03:59.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch host add smithi190 2024-06-27T21:03:59.934 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='client.14173 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: Updating smithi151:/etc/ceph/ceph.conf 2024-06-27T21:04:00.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: Updating smithi151:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:00.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:00.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: Updating smithi151:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.client.admin.keyring 2024-06-27T21:04:00.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:00.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:00.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:03:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:01.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:01 smithi151 ceph-mon[28965]: from='client.14175 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:02.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:02 smithi151 ceph-mon[28965]: Deploying cephadm binary to smithi190 2024-06-27T21:04:04.776 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-06-27T21:04:05.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch host ls --format=json 2024-06-27T21:04:05.638 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:06.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:06.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:05 smithi151 ceph-mon[28965]: Added host smithi190 2024-06-27T21:04:06.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:06.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:06.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:07.334 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:04:07.334 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-06-27T21:04:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:07 smithi151 ceph-mon[28965]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:07 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:07.935 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-06-27T21:04:07.936 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:04:07.936 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-06-27T21:04:07.955 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:04:07.955 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:08.013 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-06-27T21:04:08.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch host add smithi192 2024-06-27T21:04:08.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:08 smithi151 ceph-mon[28965]: from='client.14177 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:04:08.263 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:09.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:09.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:09.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:09.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:09.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:09.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:04:09.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:09.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:10.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: Updating smithi190:/etc/ceph/ceph.conf 2024-06-27T21:04:10.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: Updating smithi190:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:10.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:10.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: Updating smithi190:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.client.admin.keyring 2024-06-27T21:04:10.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:10.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:10.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:11.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:11 smithi151 ceph-mon[28965]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:11.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:11 smithi151 ceph-mon[28965]: from='client.14179 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:12.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:12 smithi151 ceph-mon[28965]: Deploying cephadm binary to smithi192 2024-06-27T21:04:13.320 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-06-27T21:04:13.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:13 smithi151 ceph-mon[28965]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:13.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch host ls --format=json 2024-06-27T21:04:14.174 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:14.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:14.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:14 smithi151 ceph-mon[28965]: Added host smithi192 2024-06-27T21:04:14.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:14.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:14.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:15.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:15 smithi151 ceph-mon[28965]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:15.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:15.961 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:04:15.961 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-06-27T21:04:16.514 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-27T21:04:16.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd crush tunables default 2024-06-27T21:04:16.767 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:17.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:17.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='client.14181 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:04:17.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:17.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:17.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:17.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:17.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:04:17.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:17.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:18 smithi151 ceph-mon[28965]: Updating smithi192:/etc/ceph/ceph.conf 2024-06-27T21:04:18.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:18 smithi151 ceph-mon[28965]: Updating smithi192:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:18.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:18.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:18.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:19.609 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2024-06-27T21:04:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:19 smithi151 ceph-mon[28965]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:19 smithi151 ceph-mon[28965]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:19 smithi151 ceph-mon[28965]: Updating smithi192:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.client.admin.keyring 2024-06-27T21:04:19.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2395129497' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-27T21:04:20.208 INFO:tasks.cephadm:Adding mon.a on smithi151 2024-06-27T21:04:20.208 INFO:tasks.cephadm:Adding mon.b on smithi190 2024-06-27T21:04:20.208 INFO:tasks.cephadm:Adding mon.c on smithi192 2024-06-27T21:04:20.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch apply mon '3;smithi151:172.21.15.151=a;smithi190:172.21.15.190=b;smithi192:172.21.15.192=c' 2024-06-27T21:04:20.459 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2395129497' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-27T21:04:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:20 smithi151 ceph-mon[28965]: osdmap e4: 0 total, 0 up, 0 in 2024-06-27T21:04:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:21 smithi151 ceph-mon[28965]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:22.157 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-06-27T21:04:22.752 DEBUG:teuthology.orchestra.run.smithi190:mon.b> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.b.service 2024-06-27T21:04:22.755 DEBUG:teuthology.orchestra.run.smithi192:mon.c> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.c.service 2024-06-27T21:04:22.759 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-27T21:04:22.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph mon dump -f json 2024-06-27T21:04:23.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:23.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='client.14185 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi151:172.21.15.151=a;smithi190:172.21.15.190=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:23.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: Saving service mon spec with placement smithi151:172.21.15.151=a;smithi190:172.21.15.190=b;smithi192:172.21.15.192=c;count:3 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:23.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:23 smithi151 ceph-mon[28965]: Deploying daemon mon.c on smithi192 2024-06-27T21:04:23.468 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:25 smithi151 ceph-mon[28965]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:25.558 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:04:25.559 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","modified":"2024-06-27T21:02:51.211266Z","created":"2024-06-27T21:02:51.211266Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-27T21:04:25.559 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-06-27T21:04:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/4037353820' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:27.218 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-27T21:04:27.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph mon dump -f json 2024-06-27T21:04:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:27 smithi151 ceph-mon[28965]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:27.566 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:04:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:04:29.143 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","modified":"2024-06-27T21:02:51.211266Z","created":"2024-06-27T21:02:51.211266Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-27T21:04:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-06-27T21:04:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:29 smithi151 ceph-mon[28965]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/2707891873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:30.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:29 smithi192 systemd[1]: Started Ceph mon.c for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:04:30.703 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-27T21:04:30.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph mon dump -f json 2024-06-27T21:04:30.954 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: mon.a calling monitor election 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: mon.c calling monitor election 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: monmap epoch 2 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: last_changed 2024-06-27T21:04:29.862871+0000 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: created 2024-06-27T21:02:51.211266+0000 2024-06-27T21:04:35.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: min_mon_release 19 (squid) 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: election_strategy: 1 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: fsmap 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: osdmap e4: 0 total, 0 up, 0 in 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: mgrmap e13: y(active, since 49s) 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: overall HEALTH_OK 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:35.182 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:35.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: mon.a calling monitor election 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: mon.c calling monitor election 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: monmap epoch 2 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: last_changed 2024-06-27T21:04:29.862871+0000 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: created 2024-06-27T21:02:51.211266+0000 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: min_mon_release 19 (squid) 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: election_strategy: 1 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: fsmap 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: osdmap e4: 0 total, 0 up, 0 in 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: mgrmap e13: y(active, since 49s) 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: overall HEALTH_OK 2024-06-27T21:04:35.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:35.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:35.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:35.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:35.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:35.647 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:04:35.647 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","modified":"2024-06-27T21:04:29.862871Z","created":"2024-06-27T21:02:51.211266Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-27T21:04:35.647 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-06-27T21:04:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:35 smithi151 ceph-mon[28965]: Deploying daemon mon.b on smithi190 2024-06-27T21:04:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/1319175148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:36.195 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:35 smithi192 ceph-mon[32956]: Deploying daemon mon.b on smithi190 2024-06-27T21:04:36.195 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/1319175148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:36.195 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:36 smithi151 ceph-mon[28965]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:37.224 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-27T21:04:37.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph mon dump -f json 2024-06-27T21:04:37.253 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:36 smithi192 ceph-mon[32956]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:37.483 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:04:38.815 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:38 smithi190 ceph-mon[32247]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-27T21:04:39.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:38 smithi190 bash[32228]: bbb336e9e7a78e5ded01419c8bf4ca05a741f62c4bcd5382c16860b53fbf0fa8 2024-06-27T21:04:39.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:38 smithi190 systemd[1]: Started Ceph mon.b for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: mon.a calling monitor election 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: mon.c calling monitor election 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: mon.b calling monitor election 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: monmap epoch 3 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: last_changed 2024-06-27T21:04:38.800497+0000 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: created 2024-06-27T21:02:51.211266+0000 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: min_mon_release 19 (squid) 2024-06-27T21:04:44.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: election_strategy: 1 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: fsmap 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: osdmap e4: 0 total, 0 up, 0 in 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: mgrmap e13: y(active, since 58s) 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: overall HEALTH_OK 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: mon.a calling monitor election 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: mon.c calling monitor election 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: mon.b calling monitor election 2024-06-27T21:04:44.256 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: monmap epoch 3 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: last_changed 2024-06-27T21:04:38.800497+0000 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: created 2024-06-27T21:02:51.211266+0000 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: min_mon_release 19 (squid) 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: election_strategy: 1 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-06-27T21:04:44.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: fsmap 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: osdmap e4: 0 total, 0 up, 0 in 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: mgrmap e13: y(active, since 58s) 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: overall HEALTH_OK 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: mon.a calling monitor election 2024-06-27T21:04:44.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: mon.c calling monitor election 2024-06-27T21:04:44.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: mon.b calling monitor election 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: monmap epoch 3 2024-06-27T21:04:44.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: last_changed 2024-06-27T21:04:38.800497+0000 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: created 2024-06-27T21:02:51.211266+0000 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: min_mon_release 19 (squid) 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: election_strategy: 1 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: 2: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: fsmap 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: osdmap e4: 0 total, 0 up, 0 in 2024-06-27T21:04:44.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: mgrmap e13: y(active, since 58s) 2024-06-27T21:04:44.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: overall HEALTH_OK 2024-06-27T21:04:44.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:44.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-27T21:04:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","modified":"2024-06-27T21:04:38.800497Z","created":"2024-06-27T21:02:51.211266Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-06-27T21:04:45.092 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-06-27T21:04:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:44 smithi151 ceph-mon[28965]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:45.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:44 smithi190 ceph-mon[32247]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:45.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:45.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:45.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:44 smithi192 ceph-mon[32956]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:45.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:45.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:04:45.745 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-27T21:04:45.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph config generate-minimal-conf 2024-06-27T21:04:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Updating smithi151:/etc/ceph/ceph.conf 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Updating smithi190:/etc/ceph/ceph.conf 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Updating smithi192:/etc/ceph/ceph.conf 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/2489507888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Updating smithi190:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Updating smithi192:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Updating smithi151:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Reconfiguring mon.a (unknown last config time)... 2024-06-27T21:04:46.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:46.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:46.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:46.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: Reconfiguring daemon mon.a on smithi151 2024-06-27T21:04:46.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:46.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Updating smithi151:/etc/ceph/ceph.conf 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Updating smithi190:/etc/ceph/ceph.conf 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Updating smithi192:/etc/ceph/ceph.conf 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/2489507888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Updating smithi190:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Updating smithi192:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Updating smithi151:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Reconfiguring mon.a (unknown last config time)... 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: Reconfiguring daemon mon.a on smithi151 2024-06-27T21:04:46.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Updating smithi151:/etc/ceph/ceph.conf 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Updating smithi190:/etc/ceph/ceph.conf 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Updating smithi192:/etc/ceph/ceph.conf 2024-06-27T21:04:46.138 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/2489507888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Updating smithi190:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Updating smithi192:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Updating smithi151:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/config/ceph.conf 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.139 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Reconfiguring mon.a (unknown last config time)... 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: Reconfiguring daemon mon.a on smithi151 2024-06-27T21:04:46.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:46.568 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:47.564 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:47 smithi151 ceph-mon[28965]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:47.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:47 smithi190 ceph-mon[32247]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:47 smithi192 ceph-mon[32956]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:48.626 INFO:teuthology.orchestra.run.smithi151.stdout:# minimal ceph.conf for 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:48.626 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-06-27T21:04:48.626 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:04:48.626 INFO:teuthology.orchestra.run.smithi151.stdout: mon_host = [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: Reconfiguring mon.b (monmap changed)... 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:48 smithi151 ceph-mon[28965]: Reconfiguring daemon mon.b on smithi190 2024-06-27T21:04:48.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:48.989 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:48.989 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: Reconfiguring mon.b (monmap changed)... 2024-06-27T21:04:48.989 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:48.989 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:48.989 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:48.989 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:48 smithi190 ceph-mon[32247]: Reconfiguring daemon mon.b on smithi190 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: Reconfiguring mon.b (monmap changed)... 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:48 smithi192 ceph-mon[32956]: Reconfiguring daemon mon.b on smithi190 2024-06-27T21:04:49.302 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-27T21:04:49.302 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:04:49.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-06-27T21:04:49.330 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:04:49.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:49.396 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:04:49.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-06-27T21:04:49.430 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:04:49.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:49.499 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:04:49.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-06-27T21:04:49.526 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:04:49.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-27T21:04:49.591 INFO:tasks.cephadm:Adding mgr.y on smithi151 2024-06-27T21:04:49.592 INFO:tasks.cephadm:Adding mgr.x on smithi190 2024-06-27T21:04:49.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch apply mgr '2;smithi151=y;smithi190=x' 2024-06-27T21:04:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/858104974' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/858104974' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/858104974' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-27T21:04:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-27T21:04:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:50.204 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: Reconfiguring mon.c (monmap changed)... 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: Reconfiguring daemon mon.c on smithi192 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: Reconfiguring mon.c (monmap changed)... 2024-06-27T21:04:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: Reconfiguring daemon mon.c on smithi192 2024-06-27T21:04:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:50.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:50.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:50.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:51.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: Reconfiguring mon.c (monmap changed)... 2024-06-27T21:04:51.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: Reconfiguring daemon mon.c on smithi192 2024-06-27T21:04:51.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:51.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:51.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:51.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:51.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:51.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:51 smithi192 ceph-mon[32956]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:51.853 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-06-27T21:04:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:51 smithi151 ceph-mon[28965]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:52.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:51 smithi190 ceph-mon[32247]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:52.424 DEBUG:teuthology.orchestra.run.smithi190:mgr.x> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.x.service 2024-06-27T21:04:52.426 DEBUG:tasks.cephadm:set 0 configs 2024-06-27T21:04:52.427 INFO:tasks.cephadm:Deploying OSDs... 2024-06-27T21:04:52.427 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:04:52.427 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-06-27T21:04:52.446 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-27T21:04:52.447 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-06-27T21:04:52.504 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-27T21:04:52.504 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-06-27 21:03:54.309830536 +0000 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-06-27 20:59:08.783021205 +0000 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-06-27 20:59:08.783021205 +0000 2024-06-27T21:04:52.505 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-06-27 20:59:08.783021205 +0000 2024-06-27T21:04:52.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-27T21:04:52.571 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-06-27T21:04:52.571 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-06-27T21:04:52.571 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000285635 s, 1.8 MB/s 2024-06-27T21:04:52.572 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-27T21:04:52.630 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-06-27 21:03:54.345830124 +0000 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-06-27 20:59:09.122017309 +0000 2024-06-27T21:04:52.687 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-06-27 20:59:09.122017309 +0000 2024-06-27T21:04:52.688 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-06-27 20:59:09.122017309 +0000 2024-06-27T21:04:52.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-27T21:04:52.754 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-06-27T21:04:52.754 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-06-27T21:04:52.754 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000242018 s, 2.1 MB/s 2024-06-27T21:04:52.755 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-27T21:04:52.813 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-06-27 21:03:54.382829701 +0000 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-06-27 20:59:09.458013446 +0000 2024-06-27T21:04:52.871 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-06-27 20:59:09.458013446 +0000 2024-06-27T21:04:52.872 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-06-27 20:59:09.458013446 +0000 2024-06-27T21:04:52.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-27T21:04:52.937 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-06-27T21:04:52.937 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-06-27T21:04:52.938 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000198768 s, 2.6 MB/s 2024-06-27T21:04:52.938 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-27T21:04:52.996 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-06-27T21:04:53.052 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-27T21:04:53.052 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:53.052 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-27T21:04:53.052 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:53.053 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:53.053 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-06-27 21:03:54.416829312 +0000 2024-06-27T21:04:53.053 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-06-27 20:59:09.804009469 +0000 2024-06-27T21:04:53.053 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-06-27 20:59:09.804009469 +0000 2024-06-27T21:04:53.053 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-06-27 20:59:09.804009469 +0000 2024-06-27T21:04:53.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-27T21:04:53.112 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: Saving service mgr spec with placement smithi151=y;smithi190=x;count:2 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-27T21:04:53.113 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:52 smithi190 ceph-mon[32247]: Deploying daemon mgr.x on smithi190 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: Saving service mgr spec with placement smithi151=y;smithi190=x;count:2 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:53.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:53.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:53.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-27T21:04:53.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-27T21:04:53.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-27T21:04:53.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:53.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:52 smithi151 ceph-mon[28965]: Deploying daemon mgr.x on smithi190 2024-06-27T21:04:53.118 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-06-27T21:04:53.118 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-06-27T21:04:53.119 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000222924 s, 2.3 MB/s 2024-06-27T21:04:53.119 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-27T21:04:53.175 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:04:53.175 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-06-27T21:04:53.190 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-27T21:04:53.190 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-06-27T21:04:53.245 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-06-27 21:04:53.228785444 +0000 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-06-27 20:59:10.614109474 +0000 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-06-27 20:59:10.614109474 +0000 2024-06-27T21:04:53.246 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-06-27 20:59:10.614109474 +0000 2024-06-27T21:04:53.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-27T21:04:53.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:53.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:04:53.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: Saving service mgr spec with placement smithi151=y;smithi190=x;count:2 2024-06-27T21:04:53.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:53.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:53.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:53.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:52 smithi192 ceph-mon[32956]: Deploying daemon mgr.x on smithi190 2024-06-27T21:04:53.443 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-06-27T21:04:53.443 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-06-27T21:04:53.443 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000205529 s, 2.5 MB/s 2024-06-27T21:04:53.444 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-27T21:04:53.460 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-06-27T21:04:53.517 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-27T21:04:53.517 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:53.517 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-06-27T21:04:53.518 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:53.518 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:53.518 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-06-27 21:04:53.410780537 +0000 2024-06-27T21:04:53.518 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-06-27 20:59:10.954100103 +0000 2024-06-27T21:04:53.518 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-06-27 20:59:10.954100103 +0000 2024-06-27T21:04:53.518 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-06-27 20:59:10.954100103 +0000 2024-06-27T21:04:53.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-27T21:04:53.585 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-06-27T21:04:53.585 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-06-27T21:04:53.585 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000181149 s, 2.8 MB/s 2024-06-27T21:04:53.587 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-27T21:04:53.647 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-06-27T21:04:53.704 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-27T21:04:53.704 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:53.704 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 713 Links: 1 2024-06-27T21:04:53.704 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:53.704 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:53.704 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-06-27 21:04:53.410780537 +0000 2024-06-27T21:04:53.705 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-06-27 20:59:11.324089905 +0000 2024-06-27T21:04:53.705 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-06-27 20:59:11.324089905 +0000 2024-06-27T21:04:53.705 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-06-27 20:59:11.324089905 +0000 2024-06-27T21:04:53.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-27T21:04:53.755 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:04:53 smithi190 podman[33489]: 2024-06-27 21:04:53.589206862 +0000 UTC m=+0.017489240 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:04:53.768 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-06-27T21:04:53.768 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-06-27T21:04:53.768 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000196024 s, 2.6 MB/s 2024-06-27T21:04:53.770 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-27T21:04:53.825 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-06-27T21:04:53.879 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-27T21:04:53.879 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:53.879 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 732 Links: 1 2024-06-27T21:04:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-06-27 21:04:53.411780511 +0000 2024-06-27T21:04:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-06-27 20:59:11.646081030 +0000 2024-06-27T21:04:53.880 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-06-27 20:59:11.646081030 +0000 2024-06-27T21:04:53.880 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-06-27 20:59:11.646081030 +0000 2024-06-27T21:04:53.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-27T21:04:53.943 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-06-27T21:04:53.943 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-06-27T21:04:53.943 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000225418 s, 2.3 MB/s 2024-06-27T21:04:53.944 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-27T21:04:54.000 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:04:54.000 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-06-27T21:04:54.020 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-27T21:04:54.020 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-06-27T21:04:54.079 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-27T21:04:54.079 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-27 21:04:27.477164045 +0000 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-27 20:59:08.190633393 +0000 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-27 20:59:08.190633393 +0000 2024-06-27T21:04:54.080 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-27 20:59:08.190633393 +0000 2024-06-27T21:04:54.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-27T21:04:54.148 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-27T21:04:54.149 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-27T21:04:54.149 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000208615 s, 2.5 MB/s 2024-06-27T21:04:54.150 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-27T21:04:54.208 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-06-27T21:04:54.255 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:04:53 smithi190 podman[33489]: 2024-06-27 21:04:53.928157608 +0000 UTC m=+0.356439976 container init e30e311f29287962ba88f6563a17828263e25c2277aa6cc0fab43a410df4096c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-27T21:04:54.255 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:04:53 smithi190 podman[33489]: 2024-06-27 21:04:53.93094102 +0000 UTC m=+0.359223393 container start e30e311f29287962ba88f6563a17828263e25c2277aa6cc0fab43a410df4096c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-x, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-06-27T21:04:54.255 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:04:53 smithi190 ceph-mgr[33615]: -- 172.21.15.190:0/3573478069 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e211fe50e0 con 0x55e211fc3400 2024-06-27T21:04:54.255 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:04:54 smithi190 bash[33489]: e30e311f29287962ba88f6563a17828263e25c2277aa6cc0fab43a410df4096c 2024-06-27T21:04:54.255 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:04:54 smithi190 systemd[1]: Started Ceph mgr.x for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:04:54.265 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-27 21:04:27.478164027 +0000 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-27 20:59:08.515627204 +0000 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-27 20:59:08.515627204 +0000 2024-06-27T21:04:54.266 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-27 20:59:08.515627204 +0000 2024-06-27T21:04:54.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-27T21:04:54.330 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-27T21:04:54.330 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-27T21:04:54.330 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000186707 s, 2.7 MB/s 2024-06-27T21:04:54.332 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-27T21:04:54.388 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-06-27T21:04:54.444 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-27T21:04:54.444 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:54.444 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-27T21:04:54.444 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:54.444 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:54.445 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-27 21:04:27.478164027 +0000 2024-06-27T21:04:54.445 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-27 20:59:08.844620939 +0000 2024-06-27T21:04:54.445 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-27 20:59:08.844620939 +0000 2024-06-27T21:04:54.445 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-27 20:59:08.844620939 +0000 2024-06-27T21:04:54.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-27T21:04:54.509 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-27T21:04:54.509 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-27T21:04:54.509 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000217923 s, 2.3 MB/s 2024-06-27T21:04:54.510 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-27T21:04:54.567 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-06-27T21:04:54.623 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-27T21:04:54.623 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-27T21:04:54.623 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-27T21:04:54.623 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-27T21:04:54.623 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-27T21:04:54.624 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-27 21:04:27.479164008 +0000 2024-06-27T21:04:54.624 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-27 20:59:09.188614388 +0000 2024-06-27T21:04:54.624 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-27 20:59:09.188614388 +0000 2024-06-27T21:04:54.624 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-27 20:59:09.188614388 +0000 2024-06-27T21:04:54.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-27T21:04:54.687 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-27T21:04:54.687 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-27T21:04:54.688 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00022076 s, 2.3 MB/s 2024-06-27T21:04:54.689 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-27T21:04:54.745 INFO:tasks.cephadm:Deploying osd.0 on smithi151 with /dev/vg_nvme/lv_4... 2024-06-27T21:04:54.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-27T21:04:55.000 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:55.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:55 smithi151 ceph-mon[28965]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:55.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:55.378 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:55 smithi192 ceph-mon[32956]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:55.378 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.378 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.378 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.378 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.378 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:55 smithi190 ceph-mon[32247]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:56.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:56.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: Reconfiguring mgr.y (unknown last config time)... 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: Reconfiguring daemon mgr.y on smithi151 2024-06-27T21:04:56.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: Reconfiguring mgr.y (unknown last config time)... 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-27T21:04:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:56.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: Reconfiguring daemon mgr.y on smithi151 2024-06-27T21:04:56.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: Reconfiguring mgr.y (unknown last config time)... 2024-06-27T21:04:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-27T21:04:56.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-27T21:04:56.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:56.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: Reconfiguring daemon mgr.y on smithi151 2024-06-27T21:04:56.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:57.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:57 smithi151 ceph-mon[28965]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:57.248 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-27T21:04:57.332 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-27T21:04:57.345 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-06-27T21:04:57.346 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-06-27T21:04:57.346 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114355 s, 917 MB/s 2024-06-27T21:04:57.346 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-06-27T21:04:57.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:57 smithi190 ceph-mon[32247]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:57.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:57 smithi192 ceph-mon[32956]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:58.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2024-06-27T21:04:58.375 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:04:58.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:58.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:58.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:58.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:58.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:58.723 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:59.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:59.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:59.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:04:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:04:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:04:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:04:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:59 smithi151 ceph-mon[28965]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:04:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:59 smithi151 ceph-mon[28965]: Standby manager daemon x started 2024-06-27T21:04:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:59 smithi151 ceph-mon[28965]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-27T21:04:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:59 smithi151 ceph-mon[28965]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-27T21:04:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:59 smithi151 ceph-mon[28965]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-27T21:04:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:04:59 smithi151 ceph-mon[28965]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-27T21:05:00.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:59 smithi190 ceph-mon[32247]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:00.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:59 smithi190 ceph-mon[32247]: Standby manager daemon x started 2024-06-27T21:05:00.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:59 smithi190 ceph-mon[32247]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-27T21:05:00.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:59 smithi190 ceph-mon[32247]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-27T21:05:00.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:59 smithi190 ceph-mon[32247]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-27T21:05:00.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:04:59 smithi190 ceph-mon[32247]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-27T21:05:00.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:59 smithi192 ceph-mon[32956]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:00.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:59 smithi192 ceph-mon[32956]: Standby manager daemon x started 2024-06-27T21:05:00.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:59 smithi192 ceph-mon[32956]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-27T21:05:00.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:59 smithi192 ceph-mon[32956]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-27T21:05:00.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:59 smithi192 ceph-mon[32956]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-27T21:05:00.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:04:59 smithi192 ceph-mon[32956]: from='mgr.? 172.21.15.190:0/2243545676' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: mgrmap e14: y(active, since 73s), standbys: x 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:00.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:01.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: mgrmap e14: y(active, since 73s), standbys: x 2024-06-27T21:05:01.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-27T21:05:01.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:01.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:01.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:01.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:01.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:01.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: mgrmap e14: y(active, since 73s), standbys: x 2024-06-27T21:05:01.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-27T21:05:01.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:01.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:01.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:01.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:01.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:03.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:02 smithi151 ceph-mon[28965]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:03.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/756082039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80c5e023-5c22-463d-86e9-2307159c858f"}]: dispatch 2024-06-27T21:05:03.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/756082039' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80c5e023-5c22-463d-86e9-2307159c858f"}]': finished 2024-06-27T21:05:03.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:02 smithi151 ceph-mon[28965]: osdmap e5: 1 total, 0 up, 1 in 2024-06-27T21:05:03.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:02 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:03.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:02 smithi190 ceph-mon[32247]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:03.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/756082039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80c5e023-5c22-463d-86e9-2307159c858f"}]: dispatch 2024-06-27T21:05:03.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/756082039' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80c5e023-5c22-463d-86e9-2307159c858f"}]': finished 2024-06-27T21:05:03.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:02 smithi190 ceph-mon[32247]: osdmap e5: 1 total, 0 up, 1 in 2024-06-27T21:05:03.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:02 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:03.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:02 smithi192 ceph-mon[32956]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:03.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/756082039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80c5e023-5c22-463d-86e9-2307159c858f"}]: dispatch 2024-06-27T21:05:03.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/756082039' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80c5e023-5c22-463d-86e9-2307159c858f"}]': finished 2024-06-27T21:05:03.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:02 smithi192 ceph-mon[32956]: osdmap e5: 1 total, 0 up, 1 in 2024-06-27T21:05:03.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:02 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:04.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2547013963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:04.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2547013963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:04.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2547013963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:05.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:05 smithi151 ceph-mon[28965]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:05 smithi190 ceph-mon[32247]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:05 smithi192 ceph-mon[32956]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:07.392 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:07 smithi151 ceph-mon[28965]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:07.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:07 smithi190 ceph-mon[32247]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:07 smithi192 ceph-mon[32956]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:09.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:09 smithi151 ceph-mon[28965]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:09.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-27T21:05:09.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:09.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:09 smithi151 ceph-mon[28965]: Deploying daemon osd.0 on smithi151 2024-06-27T21:05:09.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:09 smithi190 ceph-mon[32247]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:09.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:09 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-27T21:05:09.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:09 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:09.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:09 smithi190 ceph-mon[32247]: Deploying daemon osd.0 on smithi151 2024-06-27T21:05:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:09 smithi192 ceph-mon[32956]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:09 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-27T21:05:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:09 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:09 smithi192 ceph-mon[32956]: Deploying daemon osd.0 on smithi151 2024-06-27T21:05:11.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:11 smithi151 ceph-mon[28965]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:11.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:11 smithi190 ceph-mon[32247]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:11.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:11 smithi192 ceph-mon[32956]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:13.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:13 smithi151 ceph-mon[28965]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:13.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:13 smithi190 ceph-mon[32247]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:13 smithi192 ceph-mon[32956]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:14.232 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:14.233 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:14.233 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:14.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:14.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:14.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:15.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:15 smithi151 ceph-mon[28965]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:15 smithi190 ceph-mon[32247]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:15 smithi192 ceph-mon[32956]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:16.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:16.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.741 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 0 on host 'smithi151' 2024-06-27T21:05:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:17.330 DEBUG:teuthology.orchestra.run.smithi151:osd.0> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.0.service 2024-06-27T21:05:17.334 INFO:tasks.cephadm:Deploying osd.1 on smithi151 with /dev/vg_nvme/lv_3... 2024-06-27T21:05:17.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-27T21:05:17.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:17 smithi190 ceph-mon[32247]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:17.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:17 smithi190 ceph-mon[32247]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-27T21:05:17.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:17.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:17.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:17.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:17 smithi151 ceph-mon[28965]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:17.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:17 smithi151 ceph-mon[28965]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-27T21:05:17.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:17.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:17.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:17.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:17 smithi192 ceph-mon[32956]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:17.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:17 smithi192 ceph-mon[32956]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-27T21:05:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:18.019 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:05:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:18 smithi151 ceph-mon[28965]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-27T21:05:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:18 smithi151 ceph-mon[28965]: osdmap e6: 1 total, 0 up, 1 in 2024-06-27T21:05:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:18 smithi151 ceph-mon[28965]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:05:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:18.689 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:05:18 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0[40567]: 2024-06-27T21:05:18.427+0000 7f59ad5cd640 -1 osd.0 0 waiting for initial osdmap 2024-06-27T21:05:18.689 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:05:18 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0[40567]: 2024-06-27T21:05:18.431+0000 7f59a93f5640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:05:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:18 smithi190 ceph-mon[32247]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-27T21:05:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:18 smithi190 ceph-mon[32247]: osdmap e6: 1 total, 0 up, 1 in 2024-06-27T21:05:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:18 smithi190 ceph-mon[32247]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:05:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:18 smithi192 ceph-mon[32956]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-27T21:05:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:18 smithi192 ceph-mon[32956]: osdmap e6: 1 total, 0 up, 1 in 2024-06-27T21:05:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:18 smithi192 ceph-mon[32956]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:05:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:05:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:05:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:05:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: osdmap e7: 1 total, 0 up, 1 in 2024-06-27T21:05:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:19 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: osdmap e7: 1 total, 0 up, 1 in 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:19 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:05:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:05:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-27T21:05:19.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: from='osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:05:19.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: osdmap e7: 1 total, 0 up, 1 in 2024-06-27T21:05:19.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:19.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:19 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:20.486 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-27T21:05:20.589 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-27T21:05:20.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:20 smithi151 ceph-mon[28965]: osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429] boot 2024-06-27T21:05:20.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:20 smithi151 ceph-mon[28965]: osdmap e8: 1 total, 1 up, 1 in 2024-06-27T21:05:20.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:20.603 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-06-27T21:05:20.603 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-06-27T21:05:20.603 INFO:teuthology.orchestra.run.smithi151.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122604 s, 855 MB/s 2024-06-27T21:05:20.603 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-06-27T21:05:20.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:20 smithi190 ceph-mon[32247]: osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429] boot 2024-06-27T21:05:20.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:20 smithi190 ceph-mon[32247]: osdmap e8: 1 total, 1 up, 1 in 2024-06-27T21:05:20.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:20 smithi192 ceph-mon[32956]: osd.0 [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429] boot 2024-06-27T21:05:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:20 smithi192 ceph-mon[32956]: osdmap e8: 1 total, 1 up, 1 in 2024-06-27T21:05:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-27T21:05:21.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2024-06-27T21:05:21.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:21 smithi151 ceph-mon[28965]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:21.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:21 smithi151 ceph-mon[28965]: osdmap e9: 1 total, 1 up, 1 in 2024-06-27T21:05:21.581 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:05:21.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:21 smithi190 ceph-mon[32247]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:21.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:21 smithi190 ceph-mon[32247]: osdmap e9: 1 total, 1 up, 1 in 2024-06-27T21:05:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:21 smithi192 ceph-mon[32956]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:21 smithi192 ceph-mon[32956]: osdmap e9: 1 total, 1 up, 1 in 2024-06-27T21:05:22.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi151 2024-06-27T21:05:22.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:22.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:22.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:22.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi151 to 17088M 2024-06-27T21:05:22.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:22.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:22.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:22.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi151 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi151 to 17088M 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:23.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi151 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi151 to 17088M 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:23.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:23.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:23.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:23 smithi151 ceph-mon[28965]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:23.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:23.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:23.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:24.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:23 smithi190 ceph-mon[32247]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:24.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:24.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:24.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:24.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:23 smithi192 ceph-mon[32956]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:24.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:24.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:24.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:24.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:24 smithi151 ceph-mon[28965]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:24.937 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:24 smithi192 ceph-mon[32956]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:25.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:24 smithi190 ceph-mon[32247]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:25 smithi151 ceph-mon[28965]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3968211350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083"}]: dispatch 2024-06-27T21:05:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3968211350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083"}]': finished 2024-06-27T21:05:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:25 smithi151 ceph-mon[28965]: osdmap e10: 2 total, 1 up, 2 in 2024-06-27T21:05:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:25 smithi190 ceph-mon[32247]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3968211350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083"}]: dispatch 2024-06-27T21:05:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3968211350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083"}]': finished 2024-06-27T21:05:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:25 smithi190 ceph-mon[32247]: osdmap e10: 2 total, 1 up, 2 in 2024-06-27T21:05:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:25 smithi192 ceph-mon[32956]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3968211350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083"}]: dispatch 2024-06-27T21:05:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3968211350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083"}]': finished 2024-06-27T21:05:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:25 smithi192 ceph-mon[32956]: osdmap e10: 2 total, 1 up, 2 in 2024-06-27T21:05:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:26.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1545648394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:27.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1545648394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:27.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1545648394' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:27.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:27 smithi151 ceph-mon[28965]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:28.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:27 smithi190 ceph-mon[32247]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:28.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:27 smithi192 ceph-mon[32956]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:29.720 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:29 smithi151 ceph-mon[28965]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:30.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:29 smithi190 ceph-mon[32247]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:30.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:29 smithi192 ceph-mon[32956]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:31.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:31 smithi151 ceph-mon[28965]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:32.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:31 smithi190 ceph-mon[32247]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:32.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:31 smithi192 ceph-mon[32956]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:32.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-27T21:05:32.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:32.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:32 smithi151 ceph-mon[28965]: Deploying daemon osd.1 on smithi151 2024-06-27T21:05:33.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-27T21:05:33.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:33.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:32 smithi190 ceph-mon[32247]: Deploying daemon osd.1 on smithi151 2024-06-27T21:05:33.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-27T21:05:33.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:33.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:32 smithi192 ceph-mon[32956]: Deploying daemon osd.1 on smithi151 2024-06-27T21:05:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:33 smithi151 ceph-mon[28965]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:34.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:33 smithi190 ceph-mon[32247]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:33 smithi192 ceph-mon[32956]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:35 smithi151 ceph-mon[28965]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:36.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:35 smithi190 ceph-mon[32247]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:36.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:35 smithi192 ceph-mon[32956]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:37.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:37 smithi151 ceph-mon[28965]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:37.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:37.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:37.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:37 smithi190 ceph-mon[32247]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:38.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:37 smithi192 ceph-mon[32956]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:38.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:38.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:38.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:39.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:39 smithi151 ceph-mon[28965]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:39.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:39.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:39.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:39.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:39.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:39 smithi190 ceph-mon[32247]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:39 smithi192 ceph-mon[32956]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 1 on host 'smithi151' 2024-06-27T21:05:40.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:40 smithi151 ceph-mon[28965]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-27T21:05:40.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:40.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:40.833 DEBUG:teuthology.orchestra.run.smithi151:osd.1> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.1.service 2024-06-27T21:05:40.835 INFO:tasks.cephadm:Deploying osd.2 on smithi151 with /dev/vg_nvme/lv_2... 2024-06-27T21:05:40.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-27T21:05:41.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:40 smithi190 ceph-mon[32247]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-27T21:05:41.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:41.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:41.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:41.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:40 smithi192 ceph-mon[32956]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-27T21:05:41.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:41.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:41.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:41.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:41 smithi151 ceph-mon[28965]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:41.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:41 smithi151 ceph-mon[28965]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-27T21:05:41.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:41 smithi151 ceph-mon[28965]: osdmap e11: 2 total, 1 up, 2 in 2024-06-27T21:05:41.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:41 smithi151 ceph-mon[28965]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:05:41.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:41 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:42.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:41 smithi190 ceph-mon[32247]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:42.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:41 smithi190 ceph-mon[32247]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-27T21:05:42.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:41 smithi190 ceph-mon[32247]: osdmap e11: 2 total, 1 up, 2 in 2024-06-27T21:05:42.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:41 smithi190 ceph-mon[32247]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:05:42.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:41 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:42.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:41 smithi192 ceph-mon[32956]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-06-27T21:05:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:41 smithi192 ceph-mon[32956]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-27T21:05:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:41 smithi192 ceph-mon[32956]: osdmap e11: 2 total, 1 up, 2 in 2024-06-27T21:05:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:41 smithi192 ceph-mon[32956]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:05:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:41 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:42.354 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1[46181]: 2024-06-27T21:05:42.072+0000 7efefde2a640 -1 osd.1 0 waiting for initial osdmap 2024-06-27T21:05:42.354 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1[46181]: 2024-06-27T21:05:42.075+0000 7efef5c4a640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:05:42.626 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: osdmap e12: 2 total, 1 up, 2 in 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:05:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:05:43.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:05:43.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: osdmap e12: 2 total, 1 up, 2 in 2024-06-27T21:05:43.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:43.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:43.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' 2024-06-27T21:05:43.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:05:43.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:05:43.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: osdmap e12: 2 total, 1 up, 2 in 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028]' entity='osd.1' 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:05:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:05:43.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:05:43.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:05:43.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:05:43.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:43.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:43 smithi151 ceph-mon[28965]: osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028] boot 2024-06-27T21:05:43.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:43 smithi151 ceph-mon[28965]: osdmap e13: 2 total, 2 up, 2 in 2024-06-27T21:05:43.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:05:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:05:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:43 smithi190 ceph-mon[32247]: osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028] boot 2024-06-27T21:05:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:43 smithi190 ceph-mon[32247]: osdmap e13: 2 total, 2 up, 2 in 2024-06-27T21:05:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:43 smithi192 ceph-mon[32956]: osd.1 [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028] boot 2024-06-27T21:05:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:43 smithi192 ceph-mon[32956]: osdmap e13: 2 total, 2 up, 2 in 2024-06-27T21:05:44.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-27T21:05:44.586 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-27T21:05:44.684 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-27T21:05:44.699 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-06-27T21:05:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-06-27T21:05:44.699 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013452 s, 779 MB/s 2024-06-27T21:05:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-06-27T21:05:45.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2024-06-27T21:05:45.738 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:05:45.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:45.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: osdmap e14: 2 total, 2 up, 2 in 2024-06-27T21:05:45.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:45.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:45.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:45.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:46.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: osdmap e14: 2 total, 2 up, 2 in 2024-06-27T21:05:46.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:46.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: osdmap e14: 2 total, 2 up, 2 in 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:05:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:05:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:05:46.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:46 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi151 2024-06-27T21:05:46.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:46 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi151 to 8544M 2024-06-27T21:05:47.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:46 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi151 2024-06-27T21:05:47.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:46 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi151 to 8544M 2024-06-27T21:05:47.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:46 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi151 2024-06-27T21:05:47.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:46 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi151 to 8544M 2024-06-27T21:05:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:47 smithi151 ceph-mon[28965]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:47 smithi151 ceph-mon[28965]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:48.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:47 smithi190 ceph-mon[32247]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:48.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:47 smithi190 ceph-mon[32247]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:48.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:48.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:48.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:47 smithi192 ceph-mon[32956]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:47 smithi192 ceph-mon[32956]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:05:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:05:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:05:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:48.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:48 smithi151 ceph-mon[28965]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:49.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:48 smithi190 ceph-mon[32247]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:49.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:48 smithi192 ceph-mon[32956]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4110808291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62"}]: dispatch 2024-06-27T21:05:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4110808291' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62"}]': finished 2024-06-27T21:05:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:50 smithi151 ceph-mon[28965]: osdmap e15: 3 total, 2 up, 3 in 2024-06-27T21:05:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:05:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4110808291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62"}]: dispatch 2024-06-27T21:05:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4110808291' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62"}]': finished 2024-06-27T21:05:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:50 smithi190 ceph-mon[32247]: osdmap e15: 3 total, 2 up, 3 in 2024-06-27T21:05:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:05:50.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4110808291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62"}]: dispatch 2024-06-27T21:05:50.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4110808291' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62"}]': finished 2024-06-27T21:05:50.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:50 smithi192 ceph-mon[32956]: osdmap e15: 3 total, 2 up, 3 in 2024-06-27T21:05:50.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:05:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:51 smithi190 ceph-mon[32247]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/15083033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:51.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:51 smithi192 ceph-mon[32956]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:51.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/15083033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:51 smithi151 ceph-mon[28965]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/15083033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:05:53.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:53 smithi151 ceph-mon[28965]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:53.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:53 smithi190 ceph-mon[32247]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:53.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:53 smithi192 ceph-mon[32956]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:55.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:55 smithi151 ceph-mon[28965]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:55.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:55 smithi190 ceph-mon[32247]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:55.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:55 smithi192 ceph-mon[32956]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-27T21:05:56.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:56.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-27T21:05:56.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:56.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-27T21:05:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:05:57.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:57 smithi151 ceph-mon[28965]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:57.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:57 smithi151 ceph-mon[28965]: Deploying daemon osd.2 on smithi151 2024-06-27T21:05:57.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:57 smithi190 ceph-mon[32247]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:57.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:57 smithi190 ceph-mon[32247]: Deploying daemon osd.2 on smithi151 2024-06-27T21:05:57.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:57 smithi192 ceph-mon[32956]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:57.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:57 smithi192 ceph-mon[32956]: Deploying daemon osd.2 on smithi151 2024-06-27T21:05:59.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:05:59 smithi190 ceph-mon[32247]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:59.519 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:05:59 smithi151 ceph-mon[28965]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:05:59.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:05:59 smithi192 ceph-mon[32956]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:01.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:01 smithi190 ceph-mon[32247]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:01.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:01.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:01.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:01.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:01 smithi192 ceph-mon[32956]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:01.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:01.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:01.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:01.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:01 smithi151 ceph-mon[28965]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:01.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:01.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:01.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:03.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:03 smithi151 ceph-mon[28965]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:03.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:03 smithi190 ceph-mon[32247]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:03.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:03 smithi192 ceph-mon[32956]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:04.089 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 2 on host 'smithi151' 2024-06-27T21:06:05.112 DEBUG:teuthology.orchestra.run.smithi151:osd.2> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.2.service 2024-06-27T21:06:05.114 INFO:tasks.cephadm:Deploying osd.3 on smithi151 with /dev/vg_nvme/lv_1... 2024-06-27T21:06:05.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-27T21:06:05.423 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:06:05.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:05 smithi151 ceph-mon[28965]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:05.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:05.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:05.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:05 smithi151 ceph-mon[28965]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-27T21:06:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:05 smithi190 ceph-mon[32247]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:05 smithi190 ceph-mon[32247]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-27T21:06:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:05 smithi192 ceph-mon[32956]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:05 smithi192 ceph-mon[32956]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-27T21:06:06.500 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2[52157]: 2024-06-27T21:06:06.271+0000 7fdaf89b6640 -1 osd.2 0 waiting for initial osdmap 2024-06-27T21:06:06.500 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2[52157]: 2024-06-27T21:06:06.274+0000 7fdaf47de640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:06:06.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:06.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:06.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-mon[28965]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-27T21:06:06.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-mon[28965]: osdmap e16: 3 total, 2 up, 3 in 2024-06-27T21:06:06.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-mon[28965]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:06:06.501 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:06.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:06.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:06.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:06 smithi190 ceph-mon[32247]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-27T21:06:06.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:06 smithi190 ceph-mon[32247]: osdmap e16: 3 total, 2 up, 3 in 2024-06-27T21:06:06.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:06 smithi190 ceph-mon[32247]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:06:06.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:06 smithi192 ceph-mon[32956]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-27T21:06:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:06 smithi192 ceph-mon[32956]: osdmap e16: 3 total, 2 up, 3 in 2024-06-27T21:06:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:06 smithi192 ceph-mon[32956]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:06:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:07 smithi151 ceph-mon[28965]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:07 smithi151 ceph-mon[28965]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:06:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:07 smithi151 ceph-mon[28965]: osdmap e17: 3 total, 2 up, 3 in 2024-06-27T21:06:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:07 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:07 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:07 smithi192 ceph-mon[32956]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:07 smithi192 ceph-mon[32956]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:06:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:07 smithi192 ceph-mon[32956]: osdmap e17: 3 total, 2 up, 3 in 2024-06-27T21:06:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:07 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:07 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:07 smithi190 ceph-mon[32247]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:07 smithi190 ceph-mon[32247]: from='osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:06:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:07 smithi190 ceph-mon[32247]: osdmap e17: 3 total, 2 up, 3 in 2024-06-27T21:06:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:07 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:07 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:07.805 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-27T21:06:07.901 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-27T21:06:07.915 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-06-27T21:06:07.915 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-06-27T21:06:07.915 INFO:teuthology.orchestra.run.smithi151.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012192 s, 860 MB/s 2024-06-27T21:06:07.915 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170] boot 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: osdmap e18: 3 total, 3 up, 3 in 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:08.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170] boot 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: osdmap e18: 3 total, 3 up, 3 in 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:08.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:06:08.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_1 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: osd.2 [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170] boot 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: osdmap e18: 3 total, 3 up, 3 in 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-27T21:06:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:06:09.238 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:09 smithi192 ceph-mon[32956]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:09 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-27T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:09 smithi192 ceph-mon[32956]: osdmap e19: 3 total, 3 up, 3 in 2024-06-27T21:06:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:09 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:06:09.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:09 smithi151 ceph-mon[28965]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:09.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-27T21:06:09.638 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:09 smithi151 ceph-mon[28965]: osdmap e19: 3 total, 3 up, 3 in 2024-06-27T21:06:09.638 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:06:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:09 smithi190 ceph-mon[32247]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-06-27T21:06:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:09 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-27T21:06:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:09 smithi190 ceph-mon[32247]: osdmap e19: 3 total, 3 up, 3 in 2024-06-27T21:06:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:09 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:06:10.528 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:06:10.528 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 ceph-mon[28965]: osdmap e20: 3 total, 3 up, 3 in 2024-06-27T21:06:10.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:06:10.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:10 smithi190 ceph-mon[32247]: osdmap e20: 3 total, 3 up, 3 in 2024-06-27T21:06:10.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:06:10.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:10 smithi192 ceph-mon[32956]: osdmap e20: 3 total, 3 up, 3 in 2024-06-27T21:06:11.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-27T21:06:11.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56349]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56349]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-27T21:06:11.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56352]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56352]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56331]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56331]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56331]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56331]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56334]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56334]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56334]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.192 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56334]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56337]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56337]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56337]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56337]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56340]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56340]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56340]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56343]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.193 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56343]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.194 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56343]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.194 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56346]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-27T21:06:11.194 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.194 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56346]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.194 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:06:10 smithi151 sudo[56346]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34977]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-27T21:06:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34977]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34977]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34977]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34980]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-27T21:06:11.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:11.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34980]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:11.287 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 sudo[34980]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: osdmap e21: 3 total, 3 up, 3 in 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi151 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi151 to 5696M 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:11.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:11.472 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:11 smithi151 ceph-mon[28965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: osdmap e21: 3 total, 3 up, 3 in 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi151 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi151 to 5696M 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:11.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 ceph-mon[32247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:11.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34320]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-27T21:06:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: osdmap e21: 3 total, 3 up, 3 in 2024-06-27T21:06:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi151 2024-06-27T21:06:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi151 to 5696M 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:11 smithi192 ceph-mon[32956]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34320]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34320]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34323]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34323]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-27T21:06:12.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:11 smithi190 sudo[34323]: pam_unix(sudo:session): session closed for user root 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:12.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:12.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:12.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:12.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-27T21:06:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:13.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:13.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:13.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:13 smithi151 ceph-mon[28965]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:13 smithi151 ceph-mon[28965]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:13 smithi151 ceph-mon[28965]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-27T21:06:14.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:13 smithi190 ceph-mon[32247]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:14.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:13 smithi190 ceph-mon[32247]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:14.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:13 smithi190 ceph-mon[32247]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-27T21:06:14.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:13 smithi192 ceph-mon[32956]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:14.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:13 smithi192 ceph-mon[32956]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:14.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:13 smithi192 ceph-mon[32956]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-27T21:06:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3938013444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80"}]: dispatch 2024-06-27T21:06:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3938013444' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80"}]': finished 2024-06-27T21:06:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:14 smithi151 ceph-mon[28965]: osdmap e22: 4 total, 3 up, 4 in 2024-06-27T21:06:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:15.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3938013444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80"}]: dispatch 2024-06-27T21:06:15.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3938013444' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80"}]': finished 2024-06-27T21:06:15.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:14 smithi190 ceph-mon[32247]: osdmap e22: 4 total, 3 up, 4 in 2024-06-27T21:06:15.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:15.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3938013444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80"}]: dispatch 2024-06-27T21:06:15.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3938013444' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80"}]': finished 2024-06-27T21:06:15.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:14 smithi192 ceph-mon[32956]: osdmap e22: 4 total, 3 up, 4 in 2024-06-27T21:06:15.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:15.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:15 smithi151 ceph-mon[28965]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:15.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/882246969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:06:16.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:15 smithi190 ceph-mon[32247]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:16.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/882246969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:06:16.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:15 smithi192 ceph-mon[32956]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:16.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/882246969' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:06:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:17 smithi151 ceph-mon[28965]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:18.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:17 smithi190 ceph-mon[32247]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:18.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:17 smithi192 ceph-mon[32956]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:19 smithi151 ceph-mon[28965]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:20.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:19 smithi190 ceph-mon[32247]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:20.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:19 smithi192 ceph-mon[32956]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:20.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-27T21:06:20.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-27T21:06:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:21.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-27T21:06:21.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:21 smithi151 ceph-mon[28965]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:21 smithi151 ceph-mon[28965]: Deploying daemon osd.3 on smithi151 2024-06-27T21:06:22.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:21 smithi190 ceph-mon[32247]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:22.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:21 smithi190 ceph-mon[32247]: Deploying daemon osd.3 on smithi151 2024-06-27T21:06:22.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:21 smithi192 ceph-mon[32956]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:22.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:21 smithi192 ceph-mon[32956]: Deploying daemon osd.3 on smithi151 2024-06-27T21:06:23.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:23 smithi151 ceph-mon[28965]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:24.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:23 smithi190 ceph-mon[32247]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:24.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:23 smithi192 ceph-mon[32956]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:25.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:25 smithi151 ceph-mon[28965]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:25.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:25.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:25.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:25 smithi190 ceph-mon[32247]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:25 smithi192 ceph-mon[32956]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:26.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:27.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:27 smithi151 ceph-mon[28965]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:28.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:27 smithi190 ceph-mon[32247]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:28.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:27 smithi192 ceph-mon[32956]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:28.925 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 3 on host 'smithi151' 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:29.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:29.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:29.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-27T21:06:29.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:29.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.582 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:29.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:29.616 DEBUG:teuthology.orchestra.run.smithi151:osd.3> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.3.service 2024-06-27T21:06:29.619 INFO:tasks.cephadm:Deploying osd.4 on smithi190 with /dev/vg_nvme/lv_4... 2024-06-27T21:06:29.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-27T21:06:29.873 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:06:30.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:30 smithi192 ceph-mon[32956]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-27T21:06:30.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:30 smithi192 ceph-mon[32956]: osdmap e23: 4 total, 3 up, 4 in 2024-06-27T21:06:30.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:30 smithi192 ceph-mon[32956]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:06:30.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:30 smithi151 ceph-mon[28965]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-27T21:06:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:30 smithi151 ceph-mon[28965]: osdmap e23: 4 total, 3 up, 4 in 2024-06-27T21:06:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:30 smithi151 ceph-mon[28965]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:06:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:30.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:30 smithi190 ceph-mon[32247]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-27T21:06:30.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:30 smithi190 ceph-mon[32247]: osdmap e23: 4 total, 3 up, 4 in 2024-06-27T21:06:30.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:30 smithi190 ceph-mon[32247]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-06-27T21:06:30.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:30.955 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:06:30 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:06:30.698+0000 7f023c77f640 -1 osd.3 0 waiting for initial osdmap 2024-06-27T21:06:30.955 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:06:30 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:06:30.702+0000 7f023459f640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: osdmap e24: 4 total, 3 up, 4 in 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' 2024-06-27T21:06:31.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:31.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:06:31.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: osdmap e24: 4 total, 3 up, 4 in 2024-06-27T21:06:31.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' 2024-06-27T21:06:31.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.548 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-27T21:06:31.644 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-27T21:06:31.657 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-06-27T21:06:31.658 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-06-27T21:06:31.658 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113394 s, 925 MB/s 2024-06-27T21:06:31.658 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-06-27T21:06:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-06-27T21:06:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: osdmap e24: 4 total, 3 up, 4 in 2024-06-27T21:06:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:31.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: from='osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662]' entity='osd.3' 2024-06-27T21:06:31.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:32.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-06-27T21:06:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:06:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:06:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:32 smithi151 ceph-mon[28965]: osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662] boot 2024-06-27T21:06:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:32 smithi151 ceph-mon[28965]: osdmap e25: 4 total, 4 up, 4 in 2024-06-27T21:06:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:32.503 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:06:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:06:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:06:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:32 smithi192 ceph-mon[32956]: osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662] boot 2024-06-27T21:06:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:32 smithi192 ceph-mon[32956]: osdmap e25: 4 total, 4 up, 4 in 2024-06-27T21:06:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:32.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:06:32.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:06:32.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:32 smithi190 ceph-mon[32247]: osd.3 [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662] boot 2024-06-27T21:06:32.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:32 smithi190 ceph-mon[32247]: osdmap e25: 4 total, 4 up, 4 in 2024-06-27T21:06:32.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi151 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi151 to 4272M 2024-06-27T21:06:33.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:33.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:33.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:33 smithi190 ceph-mon[32247]: osdmap e26: 4 total, 4 up, 4 in 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi151 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi151 to 4272M 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:33.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:33 smithi151 ceph-mon[28965]: osdmap e26: 4 total, 4 up, 4 in 2024-06-27T21:06:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-06-27T21:06:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi151 2024-06-27T21:06:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi151 to 4272M 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:34.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:33 smithi192 ceph-mon[32956]: osdmap e26: 4 total, 4 up, 4 in 2024-06-27T21:06:34.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:34.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:34.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:35.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:35.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:35.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:35.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:35 smithi190 ceph-mon[32247]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:35.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:35 smithi190 ceph-mon[32247]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:35 smithi151 ceph-mon[28965]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:35 smithi151 ceph-mon[28965]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:36.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:35 smithi192 ceph-mon[32956]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:36.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:35 smithi192 ceph-mon[32956]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:36.813 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/1476271536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]: dispatch 2024-06-27T21:06:36.813 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:36 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]: dispatch 2024-06-27T21:06:36.813 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:36 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]': finished 2024-06-27T21:06:36.813 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:36 smithi190 ceph-mon[32247]: osdmap e27: 5 total, 4 up, 5 in 2024-06-27T21:06:36.813 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:36 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:36.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/1476271536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]: dispatch 2024-06-27T21:06:36.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:36 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]: dispatch 2024-06-27T21:06:36.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:36 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]': finished 2024-06-27T21:06:36.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:36 smithi151 ceph-mon[28965]: osdmap e27: 5 total, 4 up, 5 in 2024-06-27T21:06:36.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:36 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:37.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/1476271536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]: dispatch 2024-06-27T21:06:37.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:36 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]: dispatch 2024-06-27T21:06:37.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:36 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc"}]': finished 2024-06-27T21:06:37.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:36 smithi192 ceph-mon[32956]: osdmap e27: 5 total, 4 up, 5 in 2024-06-27T21:06:37.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:36 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:37.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:37 smithi151 ceph-mon[28965]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:37.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/1716382388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:06:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:37 smithi190 ceph-mon[32247]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/1716382388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:06:38.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:37 smithi192 ceph-mon[32956]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:38.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/1716382388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:06:39.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:39 smithi190 ceph-mon[32247]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:39 smithi151 ceph-mon[28965]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:39 smithi192 ceph-mon[32956]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:41.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:41 smithi190 ceph-mon[32247]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:41.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:06:41.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:06:41.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:41 smithi151 ceph-mon[28965]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:41.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:06:41.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:06:42.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:41 smithi192 ceph-mon[32956]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:06:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:06:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:42 smithi190 ceph-mon[32247]: osdmap e28: 5 total, 4 up, 5 in 2024-06-27T21:06:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-27T21:06:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:42 smithi151 ceph-mon[28965]: osdmap e28: 5 total, 4 up, 5 in 2024-06-27T21:06:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-27T21:06:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:42 smithi192 ceph-mon[32956]: osdmap e28: 5 total, 4 up, 5 in 2024-06-27T21:06:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-27T21:06:43.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:43.830 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:43 smithi190 ceph-mon[32247]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:43.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:43 smithi190 ceph-mon[32247]: Deploying daemon osd.4 on smithi190 2024-06-27T21:06:43.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:43 smithi151 ceph-mon[28965]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:43.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:43 smithi151 ceph-mon[28965]: Deploying daemon osd.4 on smithi190 2024-06-27T21:06:44.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:43 smithi192 ceph-mon[32956]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:43 smithi192 ceph-mon[32956]: Deploying daemon osd.4 on smithi190 2024-06-27T21:06:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:45 smithi190 ceph-mon[32247]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:45 smithi151 ceph-mon[28965]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:46.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:45 smithi192 ceph-mon[32956]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:47 smithi190 ceph-mon[32247]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:47 smithi151 ceph-mon[28965]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:48.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:47 smithi192 ceph-mon[32956]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:48.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:49.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:49 smithi190 ceph-mon[32247]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:49.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:49.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:49.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:49.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:49.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:49 smithi151 ceph-mon[28965]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:49.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:49 smithi192 ceph-mon[32956]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.432 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 4 on host 'smithi190' 2024-06-27T21:06:50.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:50 smithi190 ceph-mon[32247]: from='osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:06:50.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:50 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:06:50.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:50.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:50 smithi151 ceph-mon[28965]: from='osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:06:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:50 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:06:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:51.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:50 smithi192 ceph-mon[32956]: from='osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:06:51.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:50 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:06:51.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:06:51.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:51.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:51.108 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4.service 2024-06-27T21:06:51.110 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_3... 2024-06-27T21:06:51.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-27T21:06:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:51 smithi190 ceph-mon[32247]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:51 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-27T21:06:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:51 smithi190 ceph-mon[32247]: osdmap e29: 5 total, 4 up, 5 in 2024-06-27T21:06:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:51 smithi190 ceph-mon[32247]: from='osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:06:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:51 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:06:51.774 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:06:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:51 smithi151 ceph-mon[28965]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:51 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-27T21:06:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:51 smithi151 ceph-mon[28965]: osdmap e29: 5 total, 4 up, 5 in 2024-06-27T21:06:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:51 smithi151 ceph-mon[28965]: from='osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:06:51.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:51 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:06:52.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:51 smithi192 ceph-mon[32956]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:52.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:51 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-27T21:06:52.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:51 smithi192 ceph-mon[32956]: osdmap e29: 5 total, 4 up, 5 in 2024-06-27T21:06:52.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:51 smithi192 ceph-mon[32956]: from='osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:06:52.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:51 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:06:52.213 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[36726]: 2024-06-27T21:06:52.031+0000 7fe62f4b5640 -1 osd.4 0 waiting for initial osdmap 2024-06-27T21:06:52.213 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[36726]: 2024-06-27T21:06:52.036+0000 7fe626ac1640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:06:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:06:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: osdmap e30: 5 total, 4 up, 5 in 2024-06-27T21:06:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' 2024-06-27T21:06:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547] boot 2024-06-27T21:06:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: osdmap e31: 5 total, 5 up, 5 in 2024-06-27T21:06:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: osdmap e30: 5 total, 4 up, 5 in 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547] boot 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: osdmap e31: 5 total, 5 up, 5 in 2024-06-27T21:06:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: osdmap e30: 5 total, 4 up, 5 in 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: osd.4 [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547] boot 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: osdmap e31: 5 total, 5 up, 5 in 2024-06-27T21:06:52.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:06:53.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:53 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:06:53.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:53 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:06:53.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:53 smithi190 ceph-mon[32247]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:53.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:53 smithi190 ceph-mon[32247]: osdmap e32: 5 total, 5 up, 5 in 2024-06-27T21:06:53.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:53 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:06:53.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:53 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:06:53.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:53 smithi151 ceph-mon[28965]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:53.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:53 smithi151 ceph-mon[28965]: osdmap e32: 5 total, 5 up, 5 in 2024-06-27T21:06:54.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:53 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:06:54.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:53 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:06:54.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:53 smithi192 ceph-mon[32956]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-06-27T21:06:54.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:53 smithi192 ceph-mon[32956]: osdmap e32: 5 total, 5 up, 5 in 2024-06-27T21:06:54.374 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-27T21:06:54.479 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-27T21:06:54.494 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-06-27T21:06:54.494 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-06-27T21:06:54.494 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130827 s, 801 MB/s 2024-06-27T21:06:54.494 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-06-27T21:06:55.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-06-27T21:06:55.666 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:55 smithi190 ceph-mon[32247]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:55.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:55 smithi190 ceph-mon[32247]: osdmap e33: 5 total, 5 up, 5 in 2024-06-27T21:06:55.748 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:06:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:55 smithi151 ceph-mon[28965]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:55 smithi151 ceph-mon[28965]: osdmap e33: 5 total, 5 up, 5 in 2024-06-27T21:06:56.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:55 smithi192 ceph-mon[32956]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:56.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:55 smithi192 ceph-mon[32956]: osdmap e33: 5 total, 5 up, 5 in 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi190 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi190 to 17088M 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:56.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.832 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:56 smithi190 ceph-mon[32247]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:56.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi190 2024-06-27T21:06:56.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi190 to 17088M 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:56.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:56 smithi151 ceph-mon[28965]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi190 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi190 to 17088M 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:57.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:06:57.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:06:57.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:56 smithi192 ceph-mon[32956]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:58.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:57 smithi190 ceph-mon[32247]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:58.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:57 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:58.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:57 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:58.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:57 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:58.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:57 smithi192 ceph-mon[32956]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:58.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:57 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:58.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:57 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:58.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:57 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:58.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:57 smithi151 ceph-mon[28965]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:06:58.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:57 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:06:58.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:57 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:06:58.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:57 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:06:58.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:06:58 smithi190 ceph-mon[32247]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:59.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:06:58 smithi192 ceph-mon[32956]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:06:59.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:06:58 smithi151 ceph-mon[28965]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:00.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/1594381457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]: dispatch 2024-06-27T21:07:00.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:00 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]: dispatch 2024-06-27T21:07:00.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:00 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]': finished 2024-06-27T21:07:00.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:00 smithi190 ceph-mon[32247]: osdmap e34: 6 total, 5 up, 6 in 2024-06-27T21:07:00.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:00.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/1594381457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]: dispatch 2024-06-27T21:07:00.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:00 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]: dispatch 2024-06-27T21:07:00.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:00 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]': finished 2024-06-27T21:07:00.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:00 smithi192 ceph-mon[32956]: osdmap e34: 6 total, 5 up, 6 in 2024-06-27T21:07:00.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/1594381457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]: dispatch 2024-06-27T21:07:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:00 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]: dispatch 2024-06-27T21:07:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:00 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30"}]': finished 2024-06-27T21:07:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:00 smithi151 ceph-mon[28965]: osdmap e34: 6 total, 5 up, 6 in 2024-06-27T21:07:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:01.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:01 smithi190 ceph-mon[32247]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-27T21:07:01.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/461453179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:01.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:01 smithi192 ceph-mon[32956]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-27T21:07:01.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/461453179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:01 smithi151 ceph-mon[28965]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-27T21:07:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/461453179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:03.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:03 smithi190 ceph-mon[32247]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:03.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:03 smithi192 ceph-mon[32956]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:03 smithi151 ceph-mon[28965]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:05.336 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:05 smithi190 ceph-mon[32247]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-27T21:07:05.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:07:05.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:07:05.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:05 smithi190 ceph-mon[32247]: osdmap e35: 6 total, 5 up, 6 in 2024-06-27T21:07:05.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:05 smithi192 ceph-mon[32956]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-27T21:07:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:07:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:07:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:05 smithi192 ceph-mon[32956]: osdmap e35: 6 total, 5 up, 6 in 2024-06-27T21:07:05.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:05 smithi151 ceph-mon[28965]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-06-27T21:07:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:07:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:07:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:05 smithi151 ceph-mon[28965]: osdmap e35: 6 total, 5 up, 6 in 2024-06-27T21:07:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:06.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-27T21:07:06.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:06.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-27T21:07:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-27T21:07:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:07.334 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:07 smithi190 ceph-mon[32247]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:07.334 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:07 smithi190 ceph-mon[32247]: Deploying daemon osd.5 on smithi190 2024-06-27T21:07:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:07 smithi192 ceph-mon[32956]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:07.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:07 smithi192 ceph-mon[32956]: Deploying daemon osd.5 on smithi190 2024-06-27T21:07:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:07 smithi151 ceph-mon[28965]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:07 smithi151 ceph-mon[28965]: Deploying daemon osd.5 on smithi190 2024-06-27T21:07:09.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:09 smithi190 ceph-mon[32247]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:09.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:09 smithi192 ceph-mon[32956]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:09 smithi151 ceph-mon[28965]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:11.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:11 smithi190 ceph-mon[32247]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:11.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:11.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:11.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:11.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:11 smithi192 ceph-mon[32956]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:11.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:11.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:11.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:11 smithi151 ceph-mon[28965]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:13 smithi190 ceph-mon[32247]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:13.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:13.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:13.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:13 smithi192 ceph-mon[32956]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:13.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:13 smithi151 ceph-mon[28965]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:13.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:13.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:13.758 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-06-27T21:07:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:14 smithi190 ceph-mon[32247]: from='osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-27T21:07:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:14 smithi190 ceph-mon[32247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-27T21:07:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:14.352 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.5.service 2024-06-27T21:07:14.354 INFO:tasks.cephadm:Deploying osd.6 on smithi190 with /dev/vg_nvme/lv_2... 2024-06-27T21:07:14.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-27T21:07:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:14 smithi192 ceph-mon[32956]: from='osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-27T21:07:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:14 smithi192 ceph-mon[32956]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-27T21:07:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:14.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:14 smithi151 ceph-mon[28965]: from='osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-27T21:07:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:14 smithi151 ceph-mon[28965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-27T21:07:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:15.526 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:07:15.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:15 smithi151 ceph-mon[28965]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:15.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:15 smithi151 ceph-mon[28965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-27T21:07:15.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:15 smithi151 ceph-mon[28965]: from='osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:15.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:15 smithi151 ceph-mon[28965]: osdmap e36: 6 total, 5 up, 6 in 2024-06-27T21:07:15.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:15.591 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:15 smithi151 ceph-mon[28965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:15.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-mon[32247]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:15.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-mon[32247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-27T21:07:15.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-mon[32247]: from='osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:15.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-mon[32247]: osdmap e36: 6 total, 5 up, 6 in 2024-06-27T21:07:15.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:15.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-mon[32247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:15.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:15 smithi192 ceph-mon[32956]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:15 smithi192 ceph-mon[32956]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-27T21:07:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:15 smithi192 ceph-mon[32956]: from='osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:15 smithi192 ceph-mon[32956]: osdmap e36: 6 total, 5 up, 6 in 2024-06-27T21:07:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:15 smithi192 ceph-mon[32956]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:15.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:07:15.719+0000 7fc1a5699640 -1 osd.5 0 waiting for initial osdmap 2024-06-27T21:07:15.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:07:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:07:15.723+0000 7fc19d4b9640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:07:16.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:16 smithi190 ceph-mon[32247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:07:16.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:16 smithi190 ceph-mon[32247]: osdmap e37: 6 total, 5 up, 6 in 2024-06-27T21:07:16.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:16.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:16.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:16 smithi190 ceph-mon[32247]: from='osd.5 ' entity='osd.5' 2024-06-27T21:07:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:16 smithi151 ceph-mon[28965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:07:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:16 smithi151 ceph-mon[28965]: osdmap e37: 6 total, 5 up, 6 in 2024-06-27T21:07:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:16 smithi151 ceph-mon[28965]: from='osd.5 ' entity='osd.5' 2024-06-27T21:07:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:16 smithi192 ceph-mon[32956]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:07:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:16 smithi192 ceph-mon[32956]: osdmap e37: 6 total, 5 up, 6 in 2024-06-27T21:07:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:16 smithi192 ceph-mon[32956]: from='osd.5 ' entity='osd.5' 2024-06-27T21:07:17.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:07:17.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:07:17.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:17.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:17.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120] boot 2024-06-27T21:07:17.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: osdmap e38: 6 total, 6 up, 6 in 2024-06-27T21:07:17.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:17.648 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120] boot 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: osdmap e38: 6 total, 6 up, 6 in 2024-06-27T21:07:17.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:17.747 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-27T21:07:17.763 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-06-27T21:07:17.763 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-06-27T21:07:17.764 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0142064 s, 738 MB/s 2024-06-27T21:07:17.764 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-06-27T21:07:17.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:07:17.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:07:17.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-06-27T21:07:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: osd.5 [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120] boot 2024-06-27T21:07:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: osdmap e38: 6 total, 6 up, 6 in 2024-06-27T21:07:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-27T21:07:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: osdmap e39: 6 total, 6 up, 6 in 2024-06-27T21:07:18.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:18.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:18.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:18.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:18.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: osdmap e39: 6 total, 6 up, 6 in 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-06-27T21:07:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: osdmap e39: 6 total, 6 up, 6 in 2024-06-27T21:07:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:18.987 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:07:19.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:19 smithi190 ceph-mon[32247]: pgmap v135: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:19.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:19 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi190 2024-06-27T21:07:19.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:19 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi190 to 8544M 2024-06-27T21:07:19.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:19 smithi190 ceph-mon[32247]: osdmap e40: 6 total, 6 up, 6 in 2024-06-27T21:07:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:19 smithi151 ceph-mon[28965]: pgmap v135: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:19 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi190 2024-06-27T21:07:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:19 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi190 to 8544M 2024-06-27T21:07:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:19 smithi151 ceph-mon[28965]: osdmap e40: 6 total, 6 up, 6 in 2024-06-27T21:07:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:19 smithi192 ceph-mon[32956]: pgmap v135: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:19 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi190 2024-06-27T21:07:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:19 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi190 to 8544M 2024-06-27T21:07:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:19 smithi192 ceph-mon[32956]: osdmap e40: 6 total, 6 up, 6 in 2024-06-27T21:07:21.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:21 smithi190 ceph-mon[32247]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:21.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:07:21.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:07:21.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:21 smithi151 ceph-mon[28965]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:07:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:07:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:21 smithi192 ceph-mon[32956]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:07:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:07:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:22.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:22 smithi190 ceph-mon[32247]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:07:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:22 smithi151 ceph-mon[28965]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:07:22.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:22 smithi192 ceph-mon[32956]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:07:23.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:23 smithi190 ceph-mon[32247]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:23.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/950647569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]: dispatch 2024-06-27T21:07:23.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:23 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]: dispatch 2024-06-27T21:07:23.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:23 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]': finished 2024-06-27T21:07:23.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:23 smithi190 ceph-mon[32247]: osdmap e41: 7 total, 6 up, 7 in 2024-06-27T21:07:23.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:23 smithi151 ceph-mon[28965]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/950647569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]: dispatch 2024-06-27T21:07:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:23 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]: dispatch 2024-06-27T21:07:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:23 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]': finished 2024-06-27T21:07:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:23 smithi151 ceph-mon[28965]: osdmap e41: 7 total, 6 up, 7 in 2024-06-27T21:07:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:23.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:23 smithi192 ceph-mon[32956]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:23.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/950647569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]: dispatch 2024-06-27T21:07:23.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:23 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]: dispatch 2024-06-27T21:07:23.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:23 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a"}]': finished 2024-06-27T21:07:23.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:23 smithi192 ceph-mon[32956]: osdmap e41: 7 total, 6 up, 7 in 2024-06-27T21:07:23.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:24.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/3715266180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:24.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/3715266180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:24.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/3715266180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:25.687 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:25 smithi192 ceph-mon[32956]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-06-27T21:07:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:25 smithi151 ceph-mon[28965]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-06-27T21:07:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:25 smithi190 ceph-mon[32247]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-06-27T21:07:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:27 smithi190 ceph-mon[32247]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:07:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:07:27.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:27 smithi151 ceph-mon[28965]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:27.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:07:27.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:07:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:27 smithi192 ceph-mon[32956]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:27.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:07:27.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:07:28.420 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:28 smithi190 ceph-mon[32247]: osdmap e42: 7 total, 6 up, 7 in 2024-06-27T21:07:28.421 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:28 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:28.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:28 smithi151 ceph-mon[28965]: osdmap e42: 7 total, 6 up, 7 in 2024-06-27T21:07:28.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:28 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:28 smithi192 ceph-mon[32956]: osdmap e42: 7 total, 6 up, 7 in 2024-06-27T21:07:28.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:28 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:29.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:29 smithi190 ceph-mon[32247]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:29.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-27T21:07:29.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:29 smithi151 ceph-mon[28965]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-27T21:07:29.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:29 smithi192 ceph-mon[32956]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-27T21:07:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:30.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:30 smithi190 ceph-mon[32247]: Deploying daemon osd.6 on smithi190 2024-06-27T21:07:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:30 smithi151 ceph-mon[28965]: Deploying daemon osd.6 on smithi190 2024-06-27T21:07:30.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:30 smithi192 ceph-mon[32956]: Deploying daemon osd.6 on smithi190 2024-06-27T21:07:31.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:31 smithi190 ceph-mon[32247]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:31 smithi151 ceph-mon[28965]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:31 smithi192 ceph-mon[32956]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:33.424 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:33 smithi190 ceph-mon[32247]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:33.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:33 smithi151 ceph-mon[28965]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:33.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:33 smithi192 ceph-mon[32956]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:34.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:34.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:34.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:34.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:34.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:34.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:34.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:34.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:34.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:35.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:35 smithi190 ceph-mon[32247]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:35.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:35 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:07:35.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:35 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:07:35.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:35 smithi151 ceph-mon[28965]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:35.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:35 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:07:35.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:35 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:07:35.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:35 smithi192 ceph-mon[32956]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:35.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:35 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:07:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:35 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:07:37.070 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 6 on host 'smithi190' 2024-06-27T21:07:37.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:37 smithi190 ceph-mon[32247]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:37 smithi190 ceph-mon[32247]: from='osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:37 smithi190 ceph-mon[32247]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:37 smithi151 ceph-mon[28965]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:37.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:37 smithi151 ceph-mon[28965]: from='osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-27T21:07:37.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:37 smithi151 ceph-mon[28965]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-27T21:07:37.715 DEBUG:teuthology.orchestra.run.smithi190:osd.6> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.6.service 2024-06-27T21:07:37.718 INFO:tasks.cephadm:Deploying osd.7 on smithi190 with /dev/vg_nvme/lv_1... 2024-06-27T21:07:37.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-27T21:07:37.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:37 smithi192 ceph-mon[32956]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:37 smithi192 ceph-mon[32956]: from='osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-27T21:07:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:37 smithi192 ceph-mon[32956]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-27T21:07:38.029 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: osdmap e43: 7 total, 6 up, 7 in 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:39.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: from='osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:38 smithi190 ceph-mon[32247]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:39.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:39.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:39.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:39.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-27T21:07:39.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: osdmap e43: 7 total, 6 up, 7 in 2024-06-27T21:07:39.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:39.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: from='osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:39.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:38 smithi151 ceph-mon[28965]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: osdmap e43: 7 total, 6 up, 7 in 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: from='osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:39.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:38 smithi192 ceph-mon[32956]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:07:39.383 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:07:39 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:07:39.084+0000 7f77244eb640 -1 osd.6 0 waiting for initial osdmap 2024-06-27T21:07:39.383 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:07:39 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:07:39.091+0000 7f7720313640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:07:39.770 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-27T21:07:39.865 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-27T21:07:39.879 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-06-27T21:07:39.880 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-06-27T21:07:39.880 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128951 s, 813 MB/s 2024-06-27T21:07:39.880 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-06-27T21:07:40.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:07:40.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: osdmap e44: 7 total, 6 up, 7 in 2024-06-27T21:07:40.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280] boot 2024-06-27T21:07:40.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: osdmap e45: 7 total, 7 up, 7 in 2024-06-27T21:07:40.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: osdmap e44: 7 total, 6 up, 7 in 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280] boot 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: osdmap e45: 7 total, 7 up, 7 in 2024-06-27T21:07:40.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:07:40.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: osdmap e44: 7 total, 6 up, 7 in 2024-06-27T21:07:40.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:40.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: osd.6 [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280] boot 2024-06-27T21:07:40.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: osdmap e45: 7 total, 7 up, 7 in 2024-06-27T21:07:40.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-27T21:07:41.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2024-06-27T21:07:41.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:07:41.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:07:41.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:41 smithi190 ceph-mon[32247]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:41.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:41 smithi190 ceph-mon[32247]: osdmap e46: 7 total, 7 up, 7 in 2024-06-27T21:07:41.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:07:41.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:07:41.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:41 smithi151 ceph-mon[28965]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:41.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:41 smithi151 ceph-mon[28965]: osdmap e46: 7 total, 7 up, 7 in 2024-06-27T21:07:41.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:07:41.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:07:41.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:41 smithi192 ceph-mon[32956]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-06-27T21:07:41.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:41 smithi192 ceph-mon[32956]: osdmap e46: 7 total, 7 up, 7 in 2024-06-27T21:07:41.998 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:07:42.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:42 smithi190 ceph-mon[32247]: osdmap e47: 7 total, 7 up, 7 in 2024-06-27T21:07:42.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:42 smithi192 ceph-mon[32956]: osdmap e47: 7 total, 7 up, 7 in 2024-06-27T21:07:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:42 smithi151 ceph-mon[28965]: osdmap e47: 7 total, 7 up, 7 in 2024-06-27T21:07:43.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:43.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:43.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:43.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:43.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:43.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:43.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:43.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:43.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:43.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:43.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:43.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:43.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:07:43.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:44.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:44 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi190 2024-06-27T21:07:44.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:44 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi190 to 5696M 2024-06-27T21:07:44.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:44 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi190 2024-06-27T21:07:44.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:44 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi190 to 5696M 2024-06-27T21:07:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:44 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi190 2024-06-27T21:07:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:44 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi190 to 5696M 2024-06-27T21:07:45.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:45 smithi190 ceph-mon[32247]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:45.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:45 smithi190 ceph-mon[32247]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:07:45.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:07:45.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:07:45.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:45.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:45 smithi192 ceph-mon[32956]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:45 smithi192 ceph-mon[32956]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:07:45.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:45.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:45 smithi151 ceph-mon[28965]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:45.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:45 smithi151 ceph-mon[28965]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:07:45.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:07:45.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:07:45.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:47.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:47 smithi190 ceph-mon[32247]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-27T21:07:47.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/3680942659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]: dispatch 2024-06-27T21:07:47.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:47 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]: dispatch 2024-06-27T21:07:47.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:47 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]': finished 2024-06-27T21:07:47.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:47 smithi190 ceph-mon[32247]: osdmap e48: 8 total, 7 up, 8 in 2024-06-27T21:07:47.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:07:47.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:47 smithi192 ceph-mon[32956]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-27T21:07:47.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/3680942659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]: dispatch 2024-06-27T21:07:47.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:47 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]: dispatch 2024-06-27T21:07:47.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:47 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]': finished 2024-06-27T21:07:47.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:47 smithi192 ceph-mon[32956]: osdmap e48: 8 total, 7 up, 8 in 2024-06-27T21:07:47.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:07:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:47 smithi151 ceph-mon[28965]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-06-27T21:07:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/3680942659' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]: dispatch 2024-06-27T21:07:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:47 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]: dispatch 2024-06-27T21:07:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:47 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484"}]': finished 2024-06-27T21:07:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:47 smithi151 ceph-mon[28965]: osdmap e48: 8 total, 7 up, 8 in 2024-06-27T21:07:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:07:48.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/3485165472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/3485165472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:48.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/3485165472' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:07:49.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:49 smithi192 ceph-mon[32956]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-06-27T21:07:49.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:07:49.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:07:49.648 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:49 smithi190 ceph-mon[32247]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-06-27T21:07:49.649 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:07:49.649 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:07:49.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:49 smithi151 ceph-mon[28965]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-06-27T21:07:49.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:07:49.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:07:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:50 smithi190 ceph-mon[32247]: osdmap e49: 8 total, 7 up, 8 in 2024-06-27T21:07:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:07:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:50 smithi151 ceph-mon[28965]: osdmap e49: 8 total, 7 up, 8 in 2024-06-27T21:07:50.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:07:50.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:50 smithi192 ceph-mon[32956]: osdmap e49: 8 total, 7 up, 8 in 2024-06-27T21:07:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:07:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:51 smithi190 ceph-mon[32247]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:51 smithi151 ceph-mon[28965]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:51.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:51 smithi192 ceph-mon[32956]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:53.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:53 smithi190 ceph-mon[32247]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:53.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-27T21:07:53.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:53.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:53 smithi151 ceph-mon[28965]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:53.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-27T21:07:53.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:53.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:53 smithi192 ceph-mon[32956]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:07:53.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-27T21:07:53.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:07:54.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:54 smithi190 ceph-mon[32247]: Deploying daemon osd.7 on smithi190 2024-06-27T21:07:54.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:54 smithi151 ceph-mon[28965]: Deploying daemon osd.7 on smithi190 2024-06-27T21:07:54.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:54 smithi192 ceph-mon[32956]: Deploying daemon osd.7 on smithi190 2024-06-27T21:07:55.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:55 smithi190 ceph-mon[32247]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:55.686 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:55 smithi192 ceph-mon[32956]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:55 smithi151 ceph-mon[28965]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:57.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:57 smithi151 ceph-mon[28965]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:57.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:57 smithi190 ceph-mon[32247]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:57.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:57 smithi192 ceph-mon[32956]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:58.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:58.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:58.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:58.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:58.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:58.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:07:58.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:58.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:07:59.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:07:59 smithi151 ceph-mon[28965]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:59.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:07:59 smithi190 ceph-mon[32247]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:07:59.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:07:59 smithi192 ceph-mon[32956]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:00 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:00 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:00 smithi151 ceph-mon[28965]: osdmap e50: 8 total, 7 up, 8 in 2024-06-27T21:08:00.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:00.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:00 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:00.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:00 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:00.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:00 smithi190 ceph-mon[32247]: osdmap e50: 8 total, 7 up, 8 in 2024-06-27T21:08:00.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:00.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:00 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:00.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:00 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:00.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:00 smithi192 ceph-mon[32956]: osdmap e50: 8 total, 7 up, 8 in 2024-06-27T21:08:00.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:01 smithi151 ceph-mon[28965]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:01 smithi190 ceph-mon[32247]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:01.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:01.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:01.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.740 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 7 on host 'smithi190' 2024-06-27T21:08:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:01 smithi192 ceph-mon[32956]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:02.374 DEBUG:teuthology.orchestra.run.smithi190:osd.7> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.7.service 2024-06-27T21:08:02.377 INFO:tasks.cephadm:Deploying osd.8 on smithi192 with /dev/vg_nvme/lv_4... 2024-06-27T21:08:02.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-27T21:08:02.636 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:08:02.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:02 smithi151 ceph-mon[28965]: from='osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-27T21:08:02.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:02 smithi151 ceph-mon[28965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-27T21:08:02.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:02 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:02.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:02 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:02.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:02 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:02.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:02 smithi190 ceph-mon[32247]: from='osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-27T21:08:02.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:02 smithi190 ceph-mon[32247]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-27T21:08:02.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:02 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:02.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:02 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:02.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:02 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:02.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:02 smithi192 ceph-mon[32956]: from='osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-27T21:08:02.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:02 smithi192 ceph-mon[32956]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-27T21:08:02.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:02 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:02.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:02 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:02.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:02 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: from='osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: osdmap e51: 8 total, 7 up, 8 in 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: osdmap e52: 8 total, 7 up, 8 in 2024-06-27T21:08:03.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: from='osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: osdmap e51: 8 total, 7 up, 8 in 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:08:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: osdmap e52: 8 total, 7 up, 8 in 2024-06-27T21:08:03.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:03.765 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: from='osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: osdmap e51: 8 total, 7 up, 8 in 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: osdmap e52: 8 total, 7 up, 8 in 2024-06-27T21:08:03.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:04.005 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:08:03.767+0000 7f5cc0f15640 -1 osd.7 0 waiting for initial osdmap 2024-06-27T21:08:04.005 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:08:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:08:03.774+0000 7f5cb8d35640 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:08:04.375 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-27T21:08:04.469 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-27T21:08:04.483 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-06-27T21:08:04.483 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-06-27T21:08:04.483 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128108 s, 819 MB/s 2024-06-27T21:08:04.483 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-06-27T21:08:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:04 smithi151 ceph-mon[28965]: from='osd.7 ' entity='osd.7' 2024-06-27T21:08:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:04 smithi151 ceph-mon[28965]: osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685] boot 2024-06-27T21:08:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:04 smithi151 ceph-mon[28965]: osdmap e53: 8 total, 8 up, 8 in 2024-06-27T21:08:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:04.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:04.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:04 smithi190 ceph-mon[32247]: from='osd.7 ' entity='osd.7' 2024-06-27T21:08:04.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:04 smithi190 ceph-mon[32247]: osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685] boot 2024-06-27T21:08:04.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:04 smithi190 ceph-mon[32247]: osdmap e53: 8 total, 8 up, 8 in 2024-06-27T21:08:04.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:04.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:04 smithi192 ceph-mon[32956]: from='osd.7 ' entity='osd.7' 2024-06-27T21:08:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:04 smithi192 ceph-mon[32956]: osd.7 [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685] boot 2024-06-27T21:08:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:04 smithi192 ceph-mon[32956]: osdmap e53: 8 total, 8 up, 8 in 2024-06-27T21:08:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-27T21:08:05.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-06-27T21:08:05.309 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:05 smithi190 ceph-mon[32247]: osdmap e54: 8 total, 8 up, 8 in 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.652 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:05 smithi192 ceph-mon[32956]: osdmap e54: 8 total, 8 up, 8 in 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:05 smithi151 ceph-mon[28965]: osdmap e54: 8 total, 8 up, 8 in 2024-06-27T21:08:06.421 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:06 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi190 2024-06-27T21:08:06.421 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:06 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi190 to 4272M 2024-06-27T21:08:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:06 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi190 2024-06-27T21:08:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:06 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi190 to 4272M 2024-06-27T21:08:06.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:06 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi190 2024-06-27T21:08:06.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:06 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi190 to 4272M 2024-06-27T21:08:07.677 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:07 smithi192 ceph-mon[32956]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:07.677 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:07 smithi192 ceph-mon[32956]: osdmap e55: 8 total, 8 up, 8 in 2024-06-27T21:08:07.677 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:07 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:07.677 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:07 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:07.677 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:07 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:07 smithi151 ceph-mon[28965]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:07 smithi151 ceph-mon[28965]: osdmap e55: 8 total, 8 up, 8 in 2024-06-27T21:08:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:07 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:07 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:07.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:07 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:07 smithi190 ceph-mon[32247]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:07 smithi190 ceph-mon[32247]: osdmap e55: 8 total, 8 up, 8 in 2024-06-27T21:08:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:07 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:07 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:07 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:08.637 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:08 smithi192 ceph-mon[32956]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:08 smithi151 ceph-mon[28965]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:08 smithi190 ceph-mon[32247]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:09 smithi151 ceph-mon[28965]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/1368036878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]: dispatch 2024-06-27T21:08:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:09 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]: dispatch 2024-06-27T21:08:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:09 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]': finished 2024-06-27T21:08:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:09 smithi151 ceph-mon[28965]: osdmap e56: 9 total, 8 up, 9 in 2024-06-27T21:08:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:09 smithi190 ceph-mon[32247]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/1368036878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]: dispatch 2024-06-27T21:08:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:09 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]: dispatch 2024-06-27T21:08:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:09 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]': finished 2024-06-27T21:08:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:09 smithi190 ceph-mon[32247]: osdmap e56: 9 total, 8 up, 9 in 2024-06-27T21:08:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:09 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:09 smithi192 ceph-mon[32956]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/1368036878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]: dispatch 2024-06-27T21:08:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:09 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]: dispatch 2024-06-27T21:08:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:09 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243"}]': finished 2024-06-27T21:08:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:09 smithi192 ceph-mon[32956]: osdmap e56: 9 total, 8 up, 9 in 2024-06-27T21:08:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:09 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/3008872957' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:10.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/3008872957' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:10.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/3008872957' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:11 smithi151 ceph-mon[28965]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-06-27T21:08:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:11 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:11 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:11 smithi151 ceph-mon[28965]: osdmap e57: 9 total, 8 up, 9 in 2024-06-27T21:08:11.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:11.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:11 smithi190 ceph-mon[32247]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-06-27T21:08:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:11 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:11 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:11 smithi190 ceph-mon[32247]: osdmap e57: 9 total, 8 up, 9 in 2024-06-27T21:08:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:11 smithi192 ceph-mon[32956]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-06-27T21:08:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:11 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:11 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:11 smithi192 ceph-mon[32956]: osdmap e57: 9 total, 8 up, 9 in 2024-06-27T21:08:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:13.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:13 smithi192 ceph-mon[32956]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-27T21:08:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:13 smithi151 ceph-mon[28965]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-27T21:08:13.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:13 smithi190 ceph-mon[32247]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-27T21:08:15.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:15 smithi192 ceph-mon[32956]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-27T21:08:15.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-27T21:08:15.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:15.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:15 smithi151 ceph-mon[28965]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-27T21:08:15.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-27T21:08:15.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:15.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:15 smithi190 ceph-mon[32247]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-06-27T21:08:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-27T21:08:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:16.648 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:16 smithi192 ceph-mon[32956]: Deploying daemon osd.8 on smithi192 2024-06-27T21:08:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:16 smithi151 ceph-mon[28965]: Deploying daemon osd.8 on smithi192 2024-06-27T21:08:16.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:16 smithi190 ceph-mon[32247]: Deploying daemon osd.8 on smithi192 2024-06-27T21:08:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:17 smithi151 ceph-mon[28965]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:08:17.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:17 smithi190 ceph-mon[32247]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:08:17.783 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:17 smithi192 ceph-mon[32956]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:08:19.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:19 smithi192 ceph-mon[32956]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-27T21:08:19.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:19.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:19.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:19 smithi192 ceph-mon[32956]: osdmap e58: 9 total, 8 up, 9 in 2024-06-27T21:08:19.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:19 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:19 smithi151 ceph-mon[28965]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-27T21:08:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:19 smithi151 ceph-mon[28965]: osdmap e58: 9 total, 8 up, 9 in 2024-06-27T21:08:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:19 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:19.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:19 smithi190 ceph-mon[32247]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-27T21:08:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:19 smithi190 ceph-mon[32247]: osdmap e58: 9 total, 8 up, 9 in 2024-06-27T21:08:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:19 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:20.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:20.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:20.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:20.698 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:20.698 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:20.698 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:20.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:20.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:20.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:21.470 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:21 smithi192 ceph-mon[32956]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:21.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:21 smithi151 ceph-mon[28965]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:21.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:21 smithi190 ceph-mon[32247]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:23.039 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 8 on host 'smithi192' 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: osdmap e59: 9 total, 8 up, 9 in 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.634 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-27T21:08:23.635 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:23 smithi192 ceph-mon[32956]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-27T21:08:23.662 DEBUG:teuthology.orchestra.run.smithi192:osd.8> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.8.service 2024-06-27T21:08:23.665 INFO:tasks.cephadm:Deploying osd.9 on smithi192 with /dev/vg_nvme/lv_3... 2024-06-27T21:08:23.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: osdmap e59: 9 total, 8 up, 9 in 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-27T21:08:23.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:23 smithi151 ceph-mon[28965]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: osdmap e59: 9 total, 8 up, 9 in 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:23.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-27T21:08:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:23 smithi190 ceph-mon[32247]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-27T21:08:24.426 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:08:25.260 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-mon[32956]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:25.260 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-mon[32956]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-27T21:08:25.260 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-mon[32956]: from='osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:25.260 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-mon[32956]: osdmap e60: 9 total, 8 up, 9 in 2024-06-27T21:08:25.260 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:25.260 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-mon[32956]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:25.260 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:08:25.054+0000 7fa937d9b640 -1 osd.8 0 waiting for initial osdmap 2024-06-27T21:08:25.260 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:08:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:08:25.062+0000 7fa933bc3640 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:08:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:25 smithi151 ceph-mon[28965]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:25 smithi151 ceph-mon[28965]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-27T21:08:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:25 smithi151 ceph-mon[28965]: from='osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:25 smithi151 ceph-mon[28965]: osdmap e60: 9 total, 8 up, 9 in 2024-06-27T21:08:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:25 smithi151 ceph-mon[28965]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:25 smithi190 ceph-mon[32247]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:25 smithi190 ceph-mon[32247]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-27T21:08:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:25 smithi190 ceph-mon[32247]: from='osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:25 smithi190 ceph-mon[32247]: osdmap e60: 9 total, 8 up, 9 in 2024-06-27T21:08:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:25 smithi190 ceph-mon[32247]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:26.210 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:26 smithi192 ceph-mon[32956]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:08:26.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:26 smithi192 ceph-mon[32956]: osdmap e61: 9 total, 8 up, 9 in 2024-06-27T21:08:26.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:26.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:26 smithi151 ceph-mon[28965]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:26 smithi151 ceph-mon[28965]: osdmap e61: 9 total, 8 up, 9 in 2024-06-27T21:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:26.469 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-27T21:08:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:26 smithi190 ceph-mon[32247]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:08:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:26 smithi190 ceph-mon[32247]: osdmap e61: 9 total, 8 up, 9 in 2024-06-27T21:08:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:26.575 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-27T21:08:26.611 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-06-27T21:08:26.611 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-06-27T21:08:26.611 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0338426 s, 310 MB/s 2024-06-27T21:08:26.611 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942] boot 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: osdmap e62: 9 total, 9 up, 9 in 2024-06-27T21:08:27.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:27 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942] boot 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: osdmap e62: 9 total, 9 up, 9 in 2024-06-27T21:08:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:27 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: osd.8 [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942] boot 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: osdmap e62: 9 total, 9 up, 9 in 2024-06-27T21:08:27.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:27 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-27T21:08:27.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-06-27T21:08:28.191 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:08:28.396 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:28 smithi192 ceph-mon[32956]: osdmap e63: 9 total, 9 up, 9 in 2024-06-27T21:08:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:28 smithi151 ceph-mon[28965]: osdmap e63: 9 total, 9 up, 9 in 2024-06-27T21:08:28.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:28 smithi190 ceph-mon[32247]: osdmap e63: 9 total, 9 up, 9 in 2024-06-27T21:08:29.374 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: osdmap e64: 9 total, 9 up, 9 in 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi192 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi192 to 21184M 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:29.375 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: osdmap e64: 9 total, 9 up, 9 in 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi192 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi192 to 21184M 2024-06-27T21:08:29.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:29.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:29.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: osdmap e64: 9 total, 9 up, 9 in 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi192 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi192 to 21184M 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:29.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:30.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:30.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:30.518 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:30.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:30.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:30.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:31.500 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:31 smithi192 ceph-mon[32956]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-06-27T21:08:31.500 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:31 smithi192 ceph-mon[32956]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:31 smithi151 ceph-mon[28965]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-06-27T21:08:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:31 smithi151 ceph-mon[28965]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:31.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:31 smithi190 ceph-mon[32247]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-06-27T21:08:31.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:31 smithi190 ceph-mon[32247]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/1008283624' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]: dispatch 2024-06-27T21:08:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:32 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]: dispatch 2024-06-27T21:08:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:32 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]': finished 2024-06-27T21:08:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:32 smithi192 ceph-mon[32956]: osdmap e65: 10 total, 9 up, 10 in 2024-06-27T21:08:32.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:32.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/1008283624' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]: dispatch 2024-06-27T21:08:32.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:32 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]: dispatch 2024-06-27T21:08:32.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:32 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]': finished 2024-06-27T21:08:32.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:32 smithi151 ceph-mon[28965]: osdmap e65: 10 total, 9 up, 10 in 2024-06-27T21:08:32.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:32.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/1008283624' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]: dispatch 2024-06-27T21:08:32.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:32 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]: dispatch 2024-06-27T21:08:32.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:32 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd"}]': finished 2024-06-27T21:08:32.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:32 smithi190 ceph-mon[32247]: osdmap e65: 10 total, 9 up, 10 in 2024-06-27T21:08:32.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:33.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:33 smithi192 ceph-mon[32956]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-27T21:08:33.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/2989643445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:33 smithi151 ceph-mon[28965]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-27T21:08:33.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/2989643445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:33.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:33 smithi190 ceph-mon[32247]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-27T21:08:33.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/2989643445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:35.497 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:35 smithi192 ceph-mon[32956]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-06-27T21:08:35.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:35 smithi190 ceph-mon[32247]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-06-27T21:08:35.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:35 smithi151 ceph-mon[28965]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-06-27T21:08:37.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:37 smithi192 ceph-mon[32956]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:08:37.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:37 smithi151 ceph-mon[28965]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:08:37.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:37 smithi190 ceph-mon[32247]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:08:38.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-27T21:08:38.404 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-27T21:08:38.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:38.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-27T21:08:38.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:39.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:39 smithi192 ceph-mon[32956]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-06-27T21:08:39.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:39 smithi192 ceph-mon[32956]: Deploying daemon osd.9 on smithi192 2024-06-27T21:08:39.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:39.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:39.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:39 smithi192 ceph-mon[32956]: osdmap e66: 10 total, 9 up, 10 in 2024-06-27T21:08:39.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:39 smithi151 ceph-mon[28965]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-06-27T21:08:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:39 smithi151 ceph-mon[28965]: Deploying daemon osd.9 on smithi192 2024-06-27T21:08:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:39 smithi151 ceph-mon[28965]: osdmap e66: 10 total, 9 up, 10 in 2024-06-27T21:08:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:39.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:39 smithi190 ceph-mon[32247]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-06-27T21:08:39.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:39 smithi190 ceph-mon[32247]: Deploying daemon osd.9 on smithi192 2024-06-27T21:08:39.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:39.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:39.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:39 smithi190 ceph-mon[32247]: osdmap e66: 10 total, 9 up, 10 in 2024-06-27T21:08:39.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:41.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:41 smithi192 ceph-mon[32956]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:41 smithi151 ceph-mon[28965]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:41 smithi190 ceph-mon[32247]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:43.570 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:43 smithi192 ceph-mon[32956]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:43.570 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:43.570 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:43.570 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:43 smithi151 ceph-mon[28965]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:43.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:43 smithi190 ceph-mon[32247]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:44 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:08:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:44 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:08:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:08:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:08:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:08:44.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:44 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:08:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:44 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:08:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:08:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:08:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:08:44.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:44 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:08:44.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:44 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:08:44.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:08:44.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:08:44.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:46.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:46 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:46.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:46.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:46.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:46.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:46.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:46 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:46.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:46 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:46.519 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 9 on host 'smithi192' 2024-06-27T21:08:47.103 DEBUG:teuthology.orchestra.run.smithi192:osd.9> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.9.service 2024-06-27T21:08:47.105 INFO:tasks.cephadm:Deploying osd.10 on smithi192 with /dev/vg_nvme/lv_2... 2024-06-27T21:08:47.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-27T21:08:47.385 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:47 smithi192 ceph-mon[32956]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:47.386 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:47 smithi192 ceph-mon[32956]: from='osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-27T21:08:47.386 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:47 smithi192 ceph-mon[32956]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-27T21:08:47.386 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:47.386 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:47.386 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:47 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:47.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:47 smithi151 ceph-mon[28965]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:47.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:47 smithi151 ceph-mon[28965]: from='osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-27T21:08:47.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:47 smithi151 ceph-mon[28965]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-27T21:08:47.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:47.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:47.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:47 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:47.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:47 smithi190 ceph-mon[32247]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:47.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:47 smithi190 ceph-mon[32247]: from='osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-27T21:08:47.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:47 smithi190 ceph-mon[32247]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-27T21:08:47.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:08:47.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:47.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:47 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:48.166 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:08:48.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-mon[32956]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-27T21:08:48.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-mon[32956]: osdmap e67: 10 total, 9 up, 10 in 2024-06-27T21:08:48.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:48.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-mon[32956]: from='osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:48.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-mon[32956]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:48.419 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:08:48.182+0000 7f68948da640 -1 osd.9 0 waiting for initial osdmap 2024-06-27T21:08:48.419 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:08:48.182+0000 7f688c31c640 -1 osd.9 0 failed to load OSD map for epoch 13, got 0 bytes 2024-06-27T21:08:48.420 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:08:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:08:48.189+0000 7f6890702640 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:08:48.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:48 smithi151 ceph-mon[28965]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-27T21:08:48.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:48 smithi151 ceph-mon[28965]: osdmap e67: 10 total, 9 up, 10 in 2024-06-27T21:08:48.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:48.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:48 smithi151 ceph-mon[28965]: from='osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:48.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:48 smithi151 ceph-mon[28965]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:48.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:48 smithi190 ceph-mon[32247]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-27T21:08:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:48 smithi190 ceph-mon[32247]: osdmap e67: 10 total, 9 up, 10 in 2024-06-27T21:08:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:48 smithi190 ceph-mon[32247]: from='osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:48 smithi190 ceph-mon[32247]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:08:49.424 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:49 smithi192 ceph-mon[32956]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:49.424 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:49 smithi192 ceph-mon[32956]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:08:49.424 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:49 smithi192 ceph-mon[32956]: osdmap e68: 10 total, 9 up, 10 in 2024-06-27T21:08:49.424 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:49.424 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:49.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:49 smithi151 ceph-mon[28965]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:49.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:49 smithi151 ceph-mon[28965]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:08:49.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:49 smithi151 ceph-mon[28965]: osdmap e68: 10 total, 9 up, 10 in 2024-06-27T21:08:49.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:49.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:49.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:49 smithi190 ceph-mon[32247]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:49.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:49 smithi190 ceph-mon[32247]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:08:49.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:49 smithi190 ceph-mon[32247]: osdmap e68: 10 total, 9 up, 10 in 2024-06-27T21:08:49.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:49.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:50.382 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:08:50.382 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:08:50.382 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:50.382 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:50 smithi192 ceph-mon[32956]: osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799] boot 2024-06-27T21:08:50.382 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:50 smithi192 ceph-mon[32956]: osdmap e69: 10 total, 10 up, 10 in 2024-06-27T21:08:50.382 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:08:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:08:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:50 smithi151 ceph-mon[28965]: osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799] boot 2024-06-27T21:08:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:50 smithi151 ceph-mon[28965]: osdmap e69: 10 total, 10 up, 10 in 2024-06-27T21:08:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:50.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:08:50.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:08:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:50 smithi190 ceph-mon[32247]: osd.9 [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799] boot 2024-06-27T21:08:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:50 smithi190 ceph-mon[32247]: osdmap e69: 10 total, 10 up, 10 in 2024-06-27T21:08:50.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-27T21:08:51.002 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-27T21:08:51.089 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-27T21:08:51.104 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-06-27T21:08:51.104 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-06-27T21:08:51.104 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139114 s, 754 MB/s 2024-06-27T21:08:51.104 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-06-27T21:08:51.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: osdmap e70: 10 total, 10 up, 10 in 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi192 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi192 to 10592M 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:51.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: osdmap e70: 10 total, 10 up, 10 in 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi192 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi192 to 10592M 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:51.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:51.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: osdmap e70: 10 total, 10 up, 10 in 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi192 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi192 to 10592M 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:08:51.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:08:51.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-06-27T21:08:51.940 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:08:53.276 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:53 smithi192 ceph-mon[32956]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:53.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:53 smithi151 ceph-mon[28965]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:53.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:53 smithi190 ceph-mon[32247]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:54.466 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:54.466 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:54.466 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:54.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:54.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:54.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:08:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:08:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:08:55.377 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:55 smithi192 ceph-mon[32956]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:55.377 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:55 smithi192 ceph-mon[32956]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:55 smithi151 ceph-mon[28965]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:55.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:55 smithi151 ceph-mon[28965]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:55.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:55 smithi190 ceph-mon[32247]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:08:55.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:55 smithi190 ceph-mon[32247]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:56.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/603632502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]: dispatch 2024-06-27T21:08:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:56 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]: dispatch 2024-06-27T21:08:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:56 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]': finished 2024-06-27T21:08:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:56 smithi192 ceph-mon[32956]: osdmap e71: 11 total, 10 up, 11 in 2024-06-27T21:08:56.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:08:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/603632502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]: dispatch 2024-06-27T21:08:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:56 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]: dispatch 2024-06-27T21:08:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:56 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]': finished 2024-06-27T21:08:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:56 smithi151 ceph-mon[28965]: osdmap e71: 11 total, 10 up, 11 in 2024-06-27T21:08:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:08:56.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/603632502' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]: dispatch 2024-06-27T21:08:56.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:56 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]: dispatch 2024-06-27T21:08:56.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:56 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3"}]': finished 2024-06-27T21:08:56.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:56 smithi190 ceph-mon[32247]: osdmap e71: 11 total, 10 up, 11 in 2024-06-27T21:08:56.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:08:57.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:57 smithi192 ceph-mon[32956]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:57.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/817636535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:57 smithi151 ceph-mon[28965]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:57.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/817636535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:57.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:57 smithi190 ceph-mon[32247]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:57.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/817636535' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:08:59.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:08:59 smithi192 ceph-mon[32956]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:08:59 smithi151 ceph-mon[28965]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:08:59.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:08:59 smithi190 ceph-mon[32247]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:01.357 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:01 smithi192 ceph-mon[32956]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:01 smithi151 ceph-mon[28965]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:01.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:01 smithi190 ceph-mon[32247]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:03.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:03 smithi192 ceph-mon[32956]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:03.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-27T21:09:03.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:03.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:03 smithi192 ceph-mon[32956]: Deploying daemon osd.10 on smithi192 2024-06-27T21:09:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:03 smithi151 ceph-mon[28965]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-27T21:09:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:03 smithi151 ceph-mon[28965]: Deploying daemon osd.10 on smithi192 2024-06-27T21:09:03.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:03 smithi190 ceph-mon[32247]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:03.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-27T21:09:03.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:03.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:03 smithi190 ceph-mon[32247]: Deploying daemon osd.10 on smithi192 2024-06-27T21:09:05.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:05 smithi190 ceph-mon[32247]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:05.632 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:05 smithi192 ceph-mon[32956]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:05 smithi151 ceph-mon[28965]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:06 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:06 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:06 smithi192 ceph-mon[32956]: osdmap e72: 11 total, 10 up, 11 in 2024-06-27T21:09:06.536 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:06 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:06 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:06 smithi151 ceph-mon[28965]: osdmap e72: 11 total, 10 up, 11 in 2024-06-27T21:09:06.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:06.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:06 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:06.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:06 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:06.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:06 smithi190 ceph-mon[32247]: osdmap e72: 11 total, 10 up, 11 in 2024-06-27T21:09:06.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:07.641 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:07 smithi192 ceph-mon[32956]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:07 smithi151 ceph-mon[28965]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:07.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:07 smithi190 ceph-mon[32247]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:08.453 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:08.453 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:08.453 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:09 smithi151 ceph-mon[28965]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:09.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:09.744 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:09 smithi192 ceph-mon[32956]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:09.744 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:09.744 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:09 smithi190 ceph-mon[32247]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:10 smithi151 ceph-mon[28965]: osdmap e73: 11 total, 10 up, 11 in 2024-06-27T21:09:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:10.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:10 smithi192 ceph-mon[32956]: osdmap e73: 11 total, 10 up, 11 in 2024-06-27T21:09:10.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:10.692 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 10 on host 'smithi192' 2024-06-27T21:09:10.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:10 smithi190 ceph-mon[32247]: osdmap e73: 11 total, 10 up, 11 in 2024-06-27T21:09:10.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:11.367 DEBUG:teuthology.orchestra.run.smithi192:osd.10> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.10.service 2024-06-27T21:09:11.372 INFO:tasks.cephadm:Deploying osd.11 on smithi192 with /dev/vg_nvme/lv_1... 2024-06-27T21:09:11.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-27T21:09:11.669 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:09:11.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:11.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:11.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 275 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:11.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-27T21:09:11.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:11.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: osdmap e74: 11 total, 10 up, 11 in 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:12.753 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:12.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-27T21:09:12.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-27T21:09:12.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: osdmap e74: 11 total, 10 up, 11 in 2024-06-27T21:09:12.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:12.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:12.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:12.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:12.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: osdmap e74: 11 total, 10 up, 11 in 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:13.032 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:09:12.896+0000 7fe0d2734640 -1 osd.10 0 waiting for initial osdmap 2024-06-27T21:09:13.032 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:09:12.897+0000 7fe0c595a640 -1 osd.10 0 failed to load OSD map for epoch 13, got 0 bytes 2024-06-27T21:09:13.032 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:09:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:09:12.905+0000 7fe0c9d40640 -1 osd.10 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:09:13.393 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-27T21:09:13.481 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-27T21:09:13.494 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-06-27T21:09:13.494 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-06-27T21:09:13.495 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121194 s, 865 MB/s 2024-06-27T21:09:13.495 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-06-27T21:09:13.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:13 smithi190 ceph-mon[32247]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:13.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:13 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:09:13.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:13 smithi190 ceph-mon[32247]: osdmap e75: 11 total, 10 up, 11 in 2024-06-27T21:09:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:13 smithi190 ceph-mon[32247]: from='osd.10 ' entity='osd.10' 2024-06-27T21:09:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:13 smithi192 ceph-mon[32956]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:13 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:09:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:13 smithi192 ceph-mon[32956]: osdmap e75: 11 total, 10 up, 11 in 2024-06-27T21:09:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:13 smithi192 ceph-mon[32956]: from='osd.10 ' entity='osd.10' 2024-06-27T21:09:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:13 smithi151 ceph-mon[28965]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:13.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:13 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:09:13.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:13 smithi151 ceph-mon[28965]: osdmap e75: 11 total, 10 up, 11 in 2024-06-27T21:09:13.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:13.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:13.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:13 smithi151 ceph-mon[28965]: from='osd.10 ' entity='osd.10' 2024-06-27T21:09:14.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:14 smithi190 ceph-mon[32247]: osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762] boot 2024-06-27T21:09:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:14 smithi190 ceph-mon[32247]: osdmap e76: 11 total, 11 up, 11 in 2024-06-27T21:09:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:14.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:14.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:14.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:14.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:14 smithi192 ceph-mon[32956]: osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762] boot 2024-06-27T21:09:14.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:14 smithi192 ceph-mon[32956]: osdmap e76: 11 total, 11 up, 11 in 2024-06-27T21:09:14.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:14.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-06-27T21:09:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:14 smithi151 ceph-mon[28965]: osd.10 [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762] boot 2024-06-27T21:09:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:14 smithi151 ceph-mon[28965]: osdmap e76: 11 total, 11 up, 11 in 2024-06-27T21:09:14.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-27T21:09:15.419 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:09:15.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:15 smithi190 ceph-mon[32247]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:15.781 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:15 smithi192 ceph-mon[32956]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:15 smithi151 ceph-mon[28965]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 894 GiB / 894 GiB avail 2024-06-27T21:09:16.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:16 smithi190 ceph-mon[32247]: osdmap e77: 11 total, 11 up, 11 in 2024-06-27T21:09:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:16 smithi192 ceph-mon[32956]: osdmap e77: 11 total, 11 up, 11 in 2024-06-27T21:09:16.840 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:16 smithi151 ceph-mon[28965]: osdmap e77: 11 total, 11 up, 11 in 2024-06-27T21:09:17.611 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:17.612 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:17.613 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.613 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:17.613 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:17.613 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:17.613 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:17.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:17.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:18.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:18 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi192 2024-06-27T21:09:18.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:18 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi192 to 7061M 2024-06-27T21:09:18.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:09:18.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:09:18.581 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:18 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:18.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:18 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi192 2024-06-27T21:09:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:18 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi192 to 7061M 2024-06-27T21:09:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:09:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:09:18.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:18 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:18 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi192 2024-06-27T21:09:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:18 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi192 to 7061M 2024-06-27T21:09:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-27T21:09:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-27T21:09:18.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:18 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:19.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:19 smithi190 ceph-mon[32247]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:19 smithi190 ceph-mon[32247]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:09:19.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:19 smithi192 ceph-mon[32956]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:19 smithi192 ceph-mon[32956]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:09:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:19 smithi151 ceph-mon[28965]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:19 smithi151 ceph-mon[28965]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:09:20.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/1494028158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]: dispatch 2024-06-27T21:09:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:20 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]: dispatch 2024-06-27T21:09:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:20 smithi192 ceph-mon[32956]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]': finished 2024-06-27T21:09:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:20 smithi192 ceph-mon[32956]: osdmap e78: 12 total, 11 up, 12 in 2024-06-27T21:09:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/1494028158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]: dispatch 2024-06-27T21:09:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:20 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]: dispatch 2024-06-27T21:09:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:20 smithi151 ceph-mon[28965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]': finished 2024-06-27T21:09:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:20 smithi151 ceph-mon[28965]: osdmap e78: 12 total, 11 up, 12 in 2024-06-27T21:09:20.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:21.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/1494028158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]: dispatch 2024-06-27T21:09:21.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:20 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]: dispatch 2024-06-27T21:09:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:20 smithi190 ceph-mon[32247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d"}]': finished 2024-06-27T21:09:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:20 smithi190 ceph-mon[32247]: osdmap e78: 12 total, 11 up, 12 in 2024-06-27T21:09:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:21 smithi192 ceph-mon[32956]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:21.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/2951377199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:09:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:21 smithi151 ceph-mon[28965]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/2951377199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:09:22.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:21 smithi190 ceph-mon[32247]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:22.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/2951377199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-27T21:09:23.710 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:23 smithi192 ceph-mon[32956]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:23.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:23 smithi151 ceph-mon[28965]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:24.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:23 smithi190 ceph-mon[32247]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:25.712 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:25 smithi192 ceph-mon[32956]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:25 smithi151 ceph-mon[28965]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:26.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:25 smithi190 ceph-mon[32247]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:26.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-27T21:09:26.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-27T21:09:26.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:27.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-27T21:09:27.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:27 smithi192 ceph-mon[32956]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:27 smithi192 ceph-mon[32956]: Deploying daemon osd.11 on smithi192 2024-06-27T21:09:27.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:27 smithi151 ceph-mon[28965]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:27.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:27 smithi151 ceph-mon[28965]: Deploying daemon osd.11 on smithi192 2024-06-27T21:09:28.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:27 smithi190 ceph-mon[32247]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:28.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:27 smithi190 ceph-mon[32247]: Deploying daemon osd.11 on smithi192 2024-06-27T21:09:28.770 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:28.770 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:29.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:29.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:29.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:29 smithi192 ceph-mon[32956]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:29 smithi192 ceph-mon[32956]: osdmap e79: 12 total, 11 up, 12 in 2024-06-27T21:09:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:29.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:29.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:29 smithi151 ceph-mon[28965]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:29.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:29 smithi151 ceph-mon[28965]: osdmap e79: 12 total, 11 up, 12 in 2024-06-27T21:09:29.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:29.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:29.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:30.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:29 smithi190 ceph-mon[32247]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:30.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:29 smithi190 ceph-mon[32247]: osdmap e79: 12 total, 11 up, 12 in 2024-06-27T21:09:30.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:30.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:30.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:30.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:30 smithi192 ceph-mon[32956]: osdmap e80: 12 total, 11 up, 12 in 2024-06-27T21:09:30.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:30 smithi151 ceph-mon[28965]: osdmap e80: 12 total, 11 up, 12 in 2024-06-27T21:09:30.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:31.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:30 smithi190 ceph-mon[32247]: osdmap e80: 12 total, 11 up, 12 in 2024-06-27T21:09:31.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:31 smithi192 ceph-mon[32956]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:31 smithi151 ceph-mon[28965]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:32.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:31 smithi190 ceph-mon[32247]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:32.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:32.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:33.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:33.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:33.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:33.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:33.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:33.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:33.650 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:33 smithi192 ceph-mon[32956]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:33 smithi151 ceph-mon[28965]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:34.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:33 smithi190 ceph-mon[32247]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:34.710 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:34.710 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:34.710 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:34 smithi192 ceph-mon[32956]: osdmap e81: 12 total, 11 up, 12 in 2024-06-27T21:09:34.710 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:34 smithi151 ceph-mon[28965]: osdmap e81: 12 total, 11 up, 12 in 2024-06-27T21:09:34.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:35.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:35.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:35.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:34 smithi190 ceph-mon[32247]: osdmap e81: 12 total, 11 up, 12 in 2024-06-27T21:09:35.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:35.545 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 11 on host 'smithi192' 2024-06-27T21:09:35.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:35 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:35.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:35.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:35.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-27T21:09:35.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-27T21:09:35.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:35.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:35 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 302 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:35.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:35 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:36.267 DEBUG:teuthology.orchestra.run.smithi192:osd.11> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.11.service 2024-06-27T21:09:36.269 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-06-27T21:09:36.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd stat -f json 2024-06-27T21:09:36.532 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:36.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:36 smithi192 ceph-mon[32956]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-27T21:09:36.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:36 smithi192 ceph-mon[32956]: osdmap e82: 12 total, 11 up, 12 in 2024-06-27T21:09:36.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:36 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:36.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:36 smithi192 ceph-mon[32956]: from='osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:36.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:36 smithi192 ceph-mon[32956]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:36.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:36 smithi151 ceph-mon[28965]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-27T21:09:36.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:36 smithi151 ceph-mon[28965]: osdmap e82: 12 total, 11 up, 12 in 2024-06-27T21:09:36.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:36 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:36.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:36 smithi151 ceph-mon[28965]: from='osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:36.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:36 smithi151 ceph-mon[28965]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:37.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:36 smithi190 ceph-mon[32247]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-27T21:09:37.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:36 smithi190 ceph-mon[32247]: osdmap e82: 12 total, 11 up, 12 in 2024-06-27T21:09:37.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:36 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:37.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:36 smithi190 ceph-mon[32247]: from='osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:37.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:36 smithi190 ceph-mon[32247]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-06-27T21:09:37.087 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:09:37.057+0000 7fac8526e640 -1 osd.11 0 waiting for initial osdmap 2024-06-27T21:09:37.087 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:09:37.058+0000 7fac78ca8640 -1 osd.11 0 failed to load OSD map for epoch 25, got 0 bytes 2024-06-27T21:09:37.087 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:09:37.065+0000 7fac7d08e640 -1 osd.11 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:09:37.838 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-mon[32956]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:37.839 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-mon[32956]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:09:37.839 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-mon[32956]: osdmap e83: 12 total, 11 up, 12 in 2024-06-27T21:09:37.839 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:37.839 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:37.839 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:37 smithi192 ceph-mon[32956]: from='osd.11 ' entity='osd.11' 2024-06-27T21:09:37.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:37 smithi151 ceph-mon[28965]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:37.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:37 smithi151 ceph-mon[28965]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:09:37.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:37 smithi151 ceph-mon[28965]: osdmap e83: 12 total, 11 up, 12 in 2024-06-27T21:09:37.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:37.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:37 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:37.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:37 smithi151 ceph-mon[28965]: from='osd.11 ' entity='osd.11' 2024-06-27T21:09:38.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:37 smithi190 ceph-mon[32247]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:37 smithi190 ceph-mon[32247]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-06-27T21:09:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:37 smithi190 ceph-mon[32247]: osdmap e83: 12 total, 11 up, 12 in 2024-06-27T21:09:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:37 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:38.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:37 smithi190 ceph-mon[32247]: from='osd.11 ' entity='osd.11' 2024-06-27T21:09:38.472 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267] boot 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: osdmap e84: 12 total, 12 up, 12 in 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:38.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/324248537' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-27T21:09:38.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:38.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:38.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:38.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267] boot 2024-06-27T21:09:38.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: osdmap e84: 12 total, 12 up, 12 in 2024-06-27T21:09:38.835 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:38.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:38.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:38.836 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/324248537' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-27T21:09:39.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: osd.11 [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267] boot 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: osdmap e84: 12 total, 12 up, 12 in 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:39.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/324248537' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-27T21:09:39.072 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":84,"num_osds":12,"num_up_osds":12,"osd_up_since":1719522577,"num_in_osds":12,"osd_in_since":1719522560,"num_remapped_pgs":1} 2024-06-27T21:09:39.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump --format=json 2024-06-27T21:09:39.332 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:39.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:39.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: osdmap e85: 12 total, 12 up, 12 in 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:39.760 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:39 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: osdmap e85: 12 total, 12 up, 12 in 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:40.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:40.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:39 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: osdmap e85: 12 total, 12 up, 12 in 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:09:40.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:09:40.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:39 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:40 smithi151 ceph-mon[28965]: Detected new or changed devices on smithi192 2024-06-27T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:40 smithi151 ceph-mon[28965]: Adjusting osd_memory_target on smithi192 to 5296M 2024-06-27T21:09:40.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:40 smithi151 ceph-mon[28965]: osdmap e86: 12 total, 12 up, 12 in 2024-06-27T21:09:41.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:40 smithi190 ceph-mon[32247]: Detected new or changed devices on smithi192 2024-06-27T21:09:41.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:40 smithi190 ceph-mon[32247]: Adjusting osd_memory_target on smithi192 to 5296M 2024-06-27T21:09:41.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:40 smithi190 ceph-mon[32247]: osdmap e86: 12 total, 12 up, 12 in 2024-06-27T21:09:41.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:40 smithi192 ceph-mon[32956]: Detected new or changed devices on smithi192 2024-06-27T21:09:41.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:40 smithi192 ceph-mon[32956]: Adjusting osd_memory_target on smithi192 to 5296M 2024-06-27T21:09:41.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:40 smithi192 ceph-mon[32956]: osdmap e86: 12 total, 12 up, 12 in 2024-06-27T21:09:41.054 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:09:41.054 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":86,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","created":"2024-06-27T21:02:54.280751+0000","modified":"2024-06-27T21:09:39.653407+0000","last_up_change":"2024-06-27T21:09:37.640435+0000","last_in_change":"2024-06-27T21:09:20.227260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-27T21:06:07.812474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80c5e023-5c22-463d-86e9-2307159c858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6803","nonce":2519434429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6805","nonce":2519434429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6809","nonce":2519434429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6807","nonce":2519434429}]},"public_addr":"172.21.15.151:6803/2519434429","cluster_addr":"172.21.15.151:6805/2519434429","heartbeat_back_addr":"172.21.15.151:6809/2519434429","heartbeat_front_addr":"172.21.15.151:6807/2519434429","state":["exists","up"]},{"osd":1,"uuid":"d6a0fa83-bdab-4c01-b0d0-8996e8d0c083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6811","nonce":59941028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6813","nonce":59941028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6817","nonce":59941028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6815","nonce":59941028}]},"public_addr":"172.21.15.151:6811/59941028","cluster_addr":"172.21.15.151:6813/59941028","heartbeat_back_addr":"172.21.15.151:6817/59941028","heartbeat_front_addr":"172.21.15.151:6815/59941028","state":["exists","up"]},{"osd":2,"uuid":"22115999-54ff-4f34-81cc-9d90bfa4bd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6819","nonce":302970170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6821","nonce":302970170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6825","nonce":302970170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6823","nonce":302970170}]},"public_addr":"172.21.15.151:6819/302970170","cluster_addr":"172.21.15.151:6821/302970170","heartbeat_back_addr":"172.21.15.151:6825/302970170","heartbeat_front_addr":"172.21.15.151:6823/302970170","state":["exists","up"]},{"osd":3,"uuid":"88c12fa5-d2af-4144-958e-b887b4474b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6827","nonce":3555754662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6829","nonce":3555754662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6833","nonce":3555754662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6831","nonce":3555754662}]},"public_addr":"172.21.15.151:6827/3555754662","cluster_addr":"172.21.15.151:6829/3555754662","heartbeat_back_addr":"172.21.15.151:6833/3555754662","heartbeat_front_addr":"172.21.15.151:6831/3555754662","state":["exists","up"]},{"osd":4,"uuid":"97aaddb9-d5bb-4827-8c35-b3c948eae3dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6801","nonce":2107645547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6803","nonce":2107645547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6807","nonce":2107645547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6805","nonce":2107645547}]},"public_addr":"172.21.15.190:6801/2107645547","cluster_addr":"172.21.15.190:6803/2107645547","heartbeat_back_addr":"172.21.15.190:6807/2107645547","heartbeat_front_addr":"172.21.15.190:6805/2107645547","state":["exists","up"]},{"osd":5,"uuid":"7ac9bbd6-0e55-4cab-b406-6f9967eaaf30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6809","nonce":888361120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6811","nonce":888361120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6815","nonce":888361120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6813","nonce":888361120}]},"public_addr":"172.21.15.190:6809/888361120","cluster_addr":"172.21.15.190:6811/888361120","heartbeat_back_addr":"172.21.15.190:6815/888361120","heartbeat_front_addr":"172.21.15.190:6813/888361120","state":["exists","up"]},{"osd":6,"uuid":"89fae0b2-26a4-456a-9d54-cfbaec80ba6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6817","nonce":30028280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6819","nonce":30028280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6823","nonce":30028280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6821","nonce":30028280}]},"public_addr":"172.21.15.190:6817/30028280","cluster_addr":"172.21.15.190:6819/30028280","heartbeat_back_addr":"172.21.15.190:6823/30028280","heartbeat_front_addr":"172.21.15.190:6821/30028280","state":["exists","up"]},{"osd":7,"uuid":"e324ffdc-9842-46ee-a87e-5054ec265484","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6825","nonce":849289685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6827","nonce":849289685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6831","nonce":849289685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6829","nonce":849289685}]},"public_addr":"172.21.15.190:6825/849289685","cluster_addr":"172.21.15.190:6827/849289685","heartbeat_back_addr":"172.21.15.190:6831/849289685","heartbeat_front_addr":"172.21.15.190:6829/849289685","state":["exists","up"]},{"osd":8,"uuid":"43d138bf-2d82-44ea-a98f-b9099e824243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6801","nonce":2977299942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6803","nonce":2977299942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6807","nonce":2977299942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6805","nonce":2977299942}]},"public_addr":"172.21.15.192:6801/2977299942","cluster_addr":"172.21.15.192:6803/2977299942","heartbeat_back_addr":"172.21.15.192:6807/2977299942","heartbeat_front_addr":"172.21.15.192:6805/2977299942","state":["exists","up"]},{"osd":9,"uuid":"0c86c106-04c9-4eb7-bb3e-d39f563b71fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6809","nonce":3401759799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6811","nonce":3401759799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6815","nonce":3401759799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6813","nonce":3401759799}]},"public_addr":"172.21.15.192:6809/3401759799","cluster_addr":"172.21.15.192:6811/3401759799","heartbeat_back_addr":"172.21.15.192:6815/3401759799","heartbeat_front_addr":"172.21.15.192:6813/3401759799","state":["exists","up"]},{"osd":10,"uuid":"ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6817","nonce":1426195762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6819","nonce":1426195762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6823","nonce":1426195762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6821","nonce":1426195762}]},"public_addr":"172.21.15.192:6817/1426195762","cluster_addr":"172.21.15.192:6819/1426195762","heartbeat_back_addr":"172.21.15.192:6823/1426195762","heartbeat_front_addr":"172.21.15.192:6821/1426195762","state":["exists","up"]},{"osd":11,"uuid":"a8a163c4-a14c-4710-bc2e-94e2cb69df0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6825","nonce":517567267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6827","nonce":517567267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6831","nonce":517567267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6829","nonce":517567267}]},"public_addr":"172.21.15.192:6825/517567267","cluster_addr":"172.21.15.192:6827/517567267","heartbeat_back_addr":"172.21.15.192:6831/517567267","heartbeat_front_addr":"172.21.15.192:6829/517567267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:29.094249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:08:22.234358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:08:38.175071+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:04.810742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:38.259153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:08:17.977076+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:08.916632+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:27.581040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:33.534034+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:08:47.245848+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:11.518724+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:35.931629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/4089480652":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3356605898":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3908626376":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3261020553":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/2698418425":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/335837712":"2024-06-28T21:03:22.780955+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-27T21:09:41.674 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:41 smithi151 ceph-mon[28965]: pgmap v253: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:41.674 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4006981610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:09:41.703 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-27T21:09:41.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-27T21:09:41.996 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:42.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:41 smithi190 ceph-mon[32247]: pgmap v253: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:42.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4006981610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:09:42.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:41 smithi192 ceph-mon[32956]: pgmap v253: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:42.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4006981610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:09:43.789 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-06-27T21:09:43.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:43 smithi151 ceph-mon[28965]: pgmap v254: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:43.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:43.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:43.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:43 smithi151 ceph-mon[28965]: osdmap e87: 12 total, 12 up, 12 in 2024-06-27T21:09:44.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:43 smithi190 ceph-mon[32247]: pgmap v254: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:44.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:44.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:43 smithi190 ceph-mon[32247]: osdmap e87: 12 total, 12 up, 12 in 2024-06-27T21:09:44.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:43 smithi192 ceph-mon[32956]: pgmap v254: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:44.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:43 smithi192 ceph-mon[32956]: osdmap e87: 12 total, 12 up, 12 in 2024-06-27T21:09:44.495 INFO:tasks.cephadm:Setting up client nodes... 2024-06-27T21:09:44.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-27T21:09:44.747 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:44.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/768863637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-27T21:09:44.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=53,logm=11} crc {auth=2183816140,config=3313370793,health=861809967,kv=3801761488,logm=2748417159}) 2024-06-27T21:09:44.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587100466}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170030257}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671569925}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=338438872}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1183484472}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1490599505}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568742425}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1587313937}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086004151}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2290338559}) 2024-06-27T21:09:44.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436254756}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812061289}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150448854}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=226617972}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500031367}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=77,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=1} crc {logm=3317706324,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1253490410}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mgr_metadata=1,mgrstat=10,mon_config_key=89} crc {mgr_metadata=2959742413,mgrstat=575413430,mon_config_key=2648787822}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3232185981}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=99,monmap=1} crc {mon_config_key=4134672977,monmap=1643965351}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {monmap=4,osd_metadata=12,osd_pg_creating=1,osdmap=83} crc {monmap=2022633679,osd_metadata=992691294,osd_pg_creating=103948301,osdmap=1809542927}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=44} crc {osdmap=1431652577}) 2024-06-27T21:09:44.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:44.872 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/768863637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=53,logm=11} crc {auth=2183816140,config=3313370793,health=861809967,kv=3801761488,logm=2748417159}) 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587100466}) 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170030257}) 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671569925}) 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=338438872}) 2024-06-27T21:09:45.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1183484472}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1490599505}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568742425}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1587313937}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086004151}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2290338559}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436254756}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812061289}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150448854}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=226617972}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500031367}) 2024-06-27T21:09:45.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=77,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=1} crc {logm=3317706324,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1253490410}) 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mgr_metadata=1,mgrstat=10,mon_config_key=89} crc {mgr_metadata=2959742413,mgrstat=575413430,mon_config_key=2648787822}) 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3232185981}) 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=99,monmap=1} crc {mon_config_key=4134672977,monmap=1643965351}) 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {monmap=4,osd_metadata=12,osd_pg_creating=1,osdmap=83} crc {monmap=2022633679,osd_metadata=992691294,osd_pg_creating=103948301,osdmap=1809542927}) 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=44} crc {osdmap=1431652577}) 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/768863637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=53,logm=11} crc {auth=2183816140,config=3313370793,health=861809967,kv=3801761488,logm=2748417159}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2587100466}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170030257}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671569925}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=338438872}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1183484472}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1490599505}) 2024-06-27T21:09:45.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568742425}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1587313937}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2086004151}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2290338559}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2436254756}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812061289}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2150448854}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=226617972}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500031367}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=77,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=1} crc {logm=3317706324,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1253490410}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mgr_metadata=1,mgrstat=10,mon_config_key=89} crc {mgr_metadata=2959742413,mgrstat=575413430,mon_config_key=2648787822}) 2024-06-27T21:09:45.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3232185981}) 2024-06-27T21:09:45.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=99,monmap=1} crc {mon_config_key=4134672977,monmap=1643965351}) 2024-06-27T21:09:45.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {monmap=4,osd_metadata=12,osd_pg_creating=1,osdmap=83} crc {monmap=2022633679,osd_metadata=992691294,osd_pg_creating=103948301,osdmap=1809542927}) 2024-06-27T21:09:45.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=44} crc {osdmap=1431652577}) 2024-06-27T21:09:45.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:45.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:45.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:45 smithi151 ceph-mon[28965]: pgmap v256: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:45.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:45 smithi151 ceph-mon[28965]: osdmap e88: 12 total, 12 up, 12 in 2024-06-27T21:09:46.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:45 smithi190 ceph-mon[32247]: pgmap v256: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:46.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:45 smithi190 ceph-mon[32247]: osdmap e88: 12 total, 12 up, 12 in 2024-06-27T21:09:46.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:45 smithi192 ceph-mon[32956]: pgmap v256: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:46.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:45 smithi192 ceph-mon[32956]: osdmap e88: 12 total, 12 up, 12 in 2024-06-27T21:09:46.678 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-06-27T21:09:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQAa1X1mY4rtKBAAtfSCCCw1NqlZ+JtbTFz1ZQ== 2024-06-27T21:09:47.424 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-06-27T21:09:47.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-27T21:09:47.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-27T21:09:47.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-27T21:09:47.729 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:09:47.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:47 smithi190 ceph-mon[32247]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-27T21:09:47.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2357205808' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:47.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2357205808' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:47 smithi151 ceph-mon[28965]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-27T21:09:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2357205808' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2357205808' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:48.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:47 smithi192 ceph-mon[32956]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s, 0 objects/s recovering 2024-06-27T21:09:48.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2357205808' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:48.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2357205808' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:49.591 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-06-27T21:09:49.591 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAd1X1m8qOPIxAAu6143ZKN3zRQz8z2l+yM1w== 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: osdmap e89: 12 total, 12 up, 12 in 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.190:0/481942136' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:49.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:09:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: osdmap e89: 12 total, 12 up, 12 in 2024-06-27T21:09:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.190:0/481942136' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:49.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:50.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-27T21:09:50.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:50.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: osdmap e89: 12 total, 12 up, 12 in 2024-06-27T21:09:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.190:0/481942136' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:50.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:50.262 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-06-27T21:09:50.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-27T21:09:50.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-27T21:09:50.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-27T21:09:50.566 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:09:51.985 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:51 smithi192 ceph-mon[32956]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-27T21:09:51.986 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:09:51.986 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:09:51.986 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:51 smithi192 ceph-mon[32956]: osdmap e90: 12 total, 12 up, 12 in 2024-06-27T21:09:52.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:51 smithi190 ceph-mon[32247]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-27T21:09:52.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:09:52.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:09:52.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:51 smithi190 ceph-mon[32247]: osdmap e90: 12 total, 12 up, 12 in 2024-06-27T21:09:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:51 smithi151 ceph-mon[28965]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s, 0 objects/s recovering 2024-06-27T21:09:52.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:09:52.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:09:52.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:51 smithi151 ceph-mon[28965]: osdmap e90: 12 total, 12 up, 12 in 2024-06-27T21:09:52.487 INFO:teuthology.orchestra.run.smithi192.stdout:[client.2] 2024-06-27T21:09:52.487 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAg1X1m/cdhHRAAk92ep+i2bNmlhSLCu97MJg== 2024-06-27T21:09:52.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.192:0/1242044815' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:52.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:52 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:52.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:52 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:53.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.192:0/1242044815' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:53.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:52 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:53.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:52 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:53.103 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-27T21:09:53.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-06-27T21:09:53.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-06-27T21:09:53.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph config log 1 --format=json 2024-06-27T21:09:53.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.192:0/1242044815' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:53.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:52 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-27T21:09:53.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:52 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-27T21:09:53.407 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:53.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:53 smithi151 ceph-mon[28965]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:54.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:53 smithi190 ceph-mon[32247]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:54.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:53 smithi192 ceph-mon[32956]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:54.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:54 smithi151 ceph-mon[28965]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:55.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:54 smithi190 ceph-mon[32247]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:55.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:54 smithi192 ceph-mon[32956]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:55.298 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:09:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1532170025' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-06-27T21:09:55.959 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1532170025' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-06-27T21:09:55.993 INFO:teuthology.orchestra.run.smithi151.stdout:[{"version":28,"timestamp":"2024-06-27T21:09:39.270861+0000","name":"","changes":[{"name":"osd/host:smithi192/osd_memory_target","previous_value":"7404542634","new_value":"5553406976"}]}] 2024-06-27T21:09:55.993 INFO:tasks.ceph_manager:config epoch is 28 2024-06-27T21:09:55.994 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-27T21:09:55.994 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-27T21:09:55.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph mgr dump --format=json 2024-06-27T21:09:56.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1532170025' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-06-27T21:09:56.286 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:56 smithi151 ceph-mon[28965]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:57.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:56 smithi190 ceph-mon[32247]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:57.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:56 smithi192 ceph-mon[32956]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:58.175 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:09:58.927 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":15,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":155555358},{"type":"v1","addr":"172.21.15.151:6801","nonce":155555358}]},"active_addr":"172.21.15.151:6801/155555358","active_change":"2024-06-27T21:03:45.726808+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.151:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":732292861}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2061105940}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":321784879}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":2338461827}]}]} 2024-06-27T21:09:58.932 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-27T21:09:58.932 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-27T21:09:58.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump --format=json 2024-06-27T21:09:59.182 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:09:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:58 smithi151 ceph-mon[28965]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:09:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1586016321' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-27T21:09:59.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:58 smithi190 ceph-mon[32247]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:59.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:09:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1586016321' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-27T21:09:59.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:58 smithi192 ceph-mon[32956]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:09:59.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:09:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1586016321' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-27T21:10:00.550 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:00 smithi151 ceph-mon[28965]: overall HEALTH_OK 2024-06-27T21:10:00.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:00 smithi190 ceph-mon[32247]: overall HEALTH_OK 2024-06-27T21:10:00.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:00 smithi192 ceph-mon[32956]: overall HEALTH_OK 2024-06-27T21:10:01.047 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:10:01.047 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":90,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","created":"2024-06-27T21:02:54.280751+0000","modified":"2024-06-27T21:09:50.693007+0000","last_up_change":"2024-06-27T21:09:37.640435+0000","last_in_change":"2024-06-27T21:09:20.227260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-27T21:06:07.812474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80c5e023-5c22-463d-86e9-2307159c858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6803","nonce":2519434429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6805","nonce":2519434429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6809","nonce":2519434429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6807","nonce":2519434429}]},"public_addr":"172.21.15.151:6803/2519434429","cluster_addr":"172.21.15.151:6805/2519434429","heartbeat_back_addr":"172.21.15.151:6809/2519434429","heartbeat_front_addr":"172.21.15.151:6807/2519434429","state":["exists","up"]},{"osd":1,"uuid":"d6a0fa83-bdab-4c01-b0d0-8996e8d0c083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6811","nonce":59941028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6813","nonce":59941028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6817","nonce":59941028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6815","nonce":59941028}]},"public_addr":"172.21.15.151:6811/59941028","cluster_addr":"172.21.15.151:6813/59941028","heartbeat_back_addr":"172.21.15.151:6817/59941028","heartbeat_front_addr":"172.21.15.151:6815/59941028","state":["exists","up"]},{"osd":2,"uuid":"22115999-54ff-4f34-81cc-9d90bfa4bd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6819","nonce":302970170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6821","nonce":302970170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6825","nonce":302970170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6823","nonce":302970170}]},"public_addr":"172.21.15.151:6819/302970170","cluster_addr":"172.21.15.151:6821/302970170","heartbeat_back_addr":"172.21.15.151:6825/302970170","heartbeat_front_addr":"172.21.15.151:6823/302970170","state":["exists","up"]},{"osd":3,"uuid":"88c12fa5-d2af-4144-958e-b887b4474b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6827","nonce":3555754662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6829","nonce":3555754662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6833","nonce":3555754662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6831","nonce":3555754662}]},"public_addr":"172.21.15.151:6827/3555754662","cluster_addr":"172.21.15.151:6829/3555754662","heartbeat_back_addr":"172.21.15.151:6833/3555754662","heartbeat_front_addr":"172.21.15.151:6831/3555754662","state":["exists","up"]},{"osd":4,"uuid":"97aaddb9-d5bb-4827-8c35-b3c948eae3dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6801","nonce":2107645547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6803","nonce":2107645547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6807","nonce":2107645547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6805","nonce":2107645547}]},"public_addr":"172.21.15.190:6801/2107645547","cluster_addr":"172.21.15.190:6803/2107645547","heartbeat_back_addr":"172.21.15.190:6807/2107645547","heartbeat_front_addr":"172.21.15.190:6805/2107645547","state":["exists","up"]},{"osd":5,"uuid":"7ac9bbd6-0e55-4cab-b406-6f9967eaaf30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6809","nonce":888361120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6811","nonce":888361120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6815","nonce":888361120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6813","nonce":888361120}]},"public_addr":"172.21.15.190:6809/888361120","cluster_addr":"172.21.15.190:6811/888361120","heartbeat_back_addr":"172.21.15.190:6815/888361120","heartbeat_front_addr":"172.21.15.190:6813/888361120","state":["exists","up"]},{"osd":6,"uuid":"89fae0b2-26a4-456a-9d54-cfbaec80ba6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6817","nonce":30028280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6819","nonce":30028280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6823","nonce":30028280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6821","nonce":30028280}]},"public_addr":"172.21.15.190:6817/30028280","cluster_addr":"172.21.15.190:6819/30028280","heartbeat_back_addr":"172.21.15.190:6823/30028280","heartbeat_front_addr":"172.21.15.190:6821/30028280","state":["exists","up"]},{"osd":7,"uuid":"e324ffdc-9842-46ee-a87e-5054ec265484","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6825","nonce":849289685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6827","nonce":849289685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6831","nonce":849289685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6829","nonce":849289685}]},"public_addr":"172.21.15.190:6825/849289685","cluster_addr":"172.21.15.190:6827/849289685","heartbeat_back_addr":"172.21.15.190:6831/849289685","heartbeat_front_addr":"172.21.15.190:6829/849289685","state":["exists","up"]},{"osd":8,"uuid":"43d138bf-2d82-44ea-a98f-b9099e824243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6801","nonce":2977299942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6803","nonce":2977299942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6807","nonce":2977299942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6805","nonce":2977299942}]},"public_addr":"172.21.15.192:6801/2977299942","cluster_addr":"172.21.15.192:6803/2977299942","heartbeat_back_addr":"172.21.15.192:6807/2977299942","heartbeat_front_addr":"172.21.15.192:6805/2977299942","state":["exists","up"]},{"osd":9,"uuid":"0c86c106-04c9-4eb7-bb3e-d39f563b71fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6809","nonce":3401759799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6811","nonce":3401759799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6815","nonce":3401759799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6813","nonce":3401759799}]},"public_addr":"172.21.15.192:6809/3401759799","cluster_addr":"172.21.15.192:6811/3401759799","heartbeat_back_addr":"172.21.15.192:6815/3401759799","heartbeat_front_addr":"172.21.15.192:6813/3401759799","state":["exists","up"]},{"osd":10,"uuid":"ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6817","nonce":1426195762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6819","nonce":1426195762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6823","nonce":1426195762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6821","nonce":1426195762}]},"public_addr":"172.21.15.192:6817/1426195762","cluster_addr":"172.21.15.192:6819/1426195762","heartbeat_back_addr":"172.21.15.192:6823/1426195762","heartbeat_front_addr":"172.21.15.192:6821/1426195762","state":["exists","up"]},{"osd":11,"uuid":"a8a163c4-a14c-4710-bc2e-94e2cb69df0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6825","nonce":517567267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6827","nonce":517567267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6831","nonce":517567267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6829","nonce":517567267}]},"public_addr":"172.21.15.192:6825/517567267","cluster_addr":"172.21.15.192:6827/517567267","heartbeat_back_addr":"172.21.15.192:6831/517567267","heartbeat_front_addr":"172.21.15.192:6829/517567267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:29.094249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:48.275220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:44.075057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:04.810742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:38.259153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:41.863814+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:08.916632+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:27.581040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:33.534034+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:50.524628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:11.518724+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:35.931629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/4089480652":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3356605898":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3908626376":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3261020553":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/2698418425":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/335837712":"2024-06-28T21:03:22.780955+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-27T21:10:01.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:01 smithi151 ceph-mon[28965]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:01.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2283169976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:10:01.699 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-27T21:10:01.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump --format=json 2024-06-27T21:10:01.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:01 smithi190 ceph-mon[32247]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:01.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2283169976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:10:01.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:01 smithi192 ceph-mon[32956]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:01.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2283169976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:10:01.951 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:03.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:03 smithi151 ceph-mon[28965]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:03.701 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:10:03.702 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":90,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","created":"2024-06-27T21:02:54.280751+0000","modified":"2024-06-27T21:09:50.693007+0000","last_up_change":"2024-06-27T21:09:37.640435+0000","last_in_change":"2024-06-27T21:09:20.227260+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-27T21:06:07.812474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80c5e023-5c22-463d-86e9-2307159c858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6803","nonce":2519434429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6805","nonce":2519434429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6809","nonce":2519434429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6807","nonce":2519434429}]},"public_addr":"172.21.15.151:6803/2519434429","cluster_addr":"172.21.15.151:6805/2519434429","heartbeat_back_addr":"172.21.15.151:6809/2519434429","heartbeat_front_addr":"172.21.15.151:6807/2519434429","state":["exists","up"]},{"osd":1,"uuid":"d6a0fa83-bdab-4c01-b0d0-8996e8d0c083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6811","nonce":59941028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6813","nonce":59941028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6817","nonce":59941028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6815","nonce":59941028}]},"public_addr":"172.21.15.151:6811/59941028","cluster_addr":"172.21.15.151:6813/59941028","heartbeat_back_addr":"172.21.15.151:6817/59941028","heartbeat_front_addr":"172.21.15.151:6815/59941028","state":["exists","up"]},{"osd":2,"uuid":"22115999-54ff-4f34-81cc-9d90bfa4bd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6819","nonce":302970170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6821","nonce":302970170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6825","nonce":302970170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6823","nonce":302970170}]},"public_addr":"172.21.15.151:6819/302970170","cluster_addr":"172.21.15.151:6821/302970170","heartbeat_back_addr":"172.21.15.151:6825/302970170","heartbeat_front_addr":"172.21.15.151:6823/302970170","state":["exists","up"]},{"osd":3,"uuid":"88c12fa5-d2af-4144-958e-b887b4474b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6827","nonce":3555754662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6829","nonce":3555754662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6833","nonce":3555754662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6831","nonce":3555754662}]},"public_addr":"172.21.15.151:6827/3555754662","cluster_addr":"172.21.15.151:6829/3555754662","heartbeat_back_addr":"172.21.15.151:6833/3555754662","heartbeat_front_addr":"172.21.15.151:6831/3555754662","state":["exists","up"]},{"osd":4,"uuid":"97aaddb9-d5bb-4827-8c35-b3c948eae3dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6801","nonce":2107645547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6803","nonce":2107645547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6807","nonce":2107645547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2107645547},{"type":"v1","addr":"172.21.15.190:6805","nonce":2107645547}]},"public_addr":"172.21.15.190:6801/2107645547","cluster_addr":"172.21.15.190:6803/2107645547","heartbeat_back_addr":"172.21.15.190:6807/2107645547","heartbeat_front_addr":"172.21.15.190:6805/2107645547","state":["exists","up"]},{"osd":5,"uuid":"7ac9bbd6-0e55-4cab-b406-6f9967eaaf30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6809","nonce":888361120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6811","nonce":888361120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6815","nonce":888361120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6813","nonce":888361120}]},"public_addr":"172.21.15.190:6809/888361120","cluster_addr":"172.21.15.190:6811/888361120","heartbeat_back_addr":"172.21.15.190:6815/888361120","heartbeat_front_addr":"172.21.15.190:6813/888361120","state":["exists","up"]},{"osd":6,"uuid":"89fae0b2-26a4-456a-9d54-cfbaec80ba6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6817","nonce":30028280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6819","nonce":30028280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6823","nonce":30028280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6821","nonce":30028280}]},"public_addr":"172.21.15.190:6817/30028280","cluster_addr":"172.21.15.190:6819/30028280","heartbeat_back_addr":"172.21.15.190:6823/30028280","heartbeat_front_addr":"172.21.15.190:6821/30028280","state":["exists","up"]},{"osd":7,"uuid":"e324ffdc-9842-46ee-a87e-5054ec265484","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6825","nonce":849289685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6827","nonce":849289685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6831","nonce":849289685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6829","nonce":849289685}]},"public_addr":"172.21.15.190:6825/849289685","cluster_addr":"172.21.15.190:6827/849289685","heartbeat_back_addr":"172.21.15.190:6831/849289685","heartbeat_front_addr":"172.21.15.190:6829/849289685","state":["exists","up"]},{"osd":8,"uuid":"43d138bf-2d82-44ea-a98f-b9099e824243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6801","nonce":2977299942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6803","nonce":2977299942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6807","nonce":2977299942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6805","nonce":2977299942}]},"public_addr":"172.21.15.192:6801/2977299942","cluster_addr":"172.21.15.192:6803/2977299942","heartbeat_back_addr":"172.21.15.192:6807/2977299942","heartbeat_front_addr":"172.21.15.192:6805/2977299942","state":["exists","up"]},{"osd":9,"uuid":"0c86c106-04c9-4eb7-bb3e-d39f563b71fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6809","nonce":3401759799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6811","nonce":3401759799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6815","nonce":3401759799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6813","nonce":3401759799}]},"public_addr":"172.21.15.192:6809/3401759799","cluster_addr":"172.21.15.192:6811/3401759799","heartbeat_back_addr":"172.21.15.192:6815/3401759799","heartbeat_front_addr":"172.21.15.192:6813/3401759799","state":["exists","up"]},{"osd":10,"uuid":"ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6817","nonce":1426195762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6819","nonce":1426195762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6823","nonce":1426195762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6821","nonce":1426195762}]},"public_addr":"172.21.15.192:6817/1426195762","cluster_addr":"172.21.15.192:6819/1426195762","heartbeat_back_addr":"172.21.15.192:6823/1426195762","heartbeat_front_addr":"172.21.15.192:6821/1426195762","state":["exists","up"]},{"osd":11,"uuid":"a8a163c4-a14c-4710-bc2e-94e2cb69df0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6825","nonce":517567267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6827","nonce":517567267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6831","nonce":517567267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6829","nonce":517567267}]},"public_addr":"172.21.15.192:6825/517567267","cluster_addr":"172.21.15.192:6827/517567267","heartbeat_back_addr":"172.21.15.192:6831/517567267","heartbeat_front_addr":"172.21.15.192:6829/517567267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:29.094249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:48.275220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:44.075057+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:04.810742+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:38.259153+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:41.863814+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:08.916632+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:27.581040+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:33.534034+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:50.524628+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:11.518724+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:09:35.931629+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/4089480652":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3356605898":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3908626376":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3261020553":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/2698418425":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/335837712":"2024-06-28T21:03:22.780955+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-27T21:10:03.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:03 smithi190 ceph-mon[32247]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:03.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:03 smithi192 ceph-mon[32956]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:04.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-27T21:10:04.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-27T21:10:04.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-27T21:10:04.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-27T21:10:04.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-27T21:10:04.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-27T21:10:04.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-27T21:10:04.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-27T21:10:04.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-27T21:10:04.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-27T21:10:04.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-27T21:10:04.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-27T21:10:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3438408367' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:10:04.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3438408367' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:10:04.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3438408367' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:10:05.041 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:05.045 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:05.476 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:05 smithi151 ceph-mon[28965]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:05.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:05 smithi190 ceph-mon[32247]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:05.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:05 smithi192 ceph-mon[32956]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:06.425 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:07.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:07 smithi151 ceph-mon[28965]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:07.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:07 smithi192 ceph-mon[32956]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:07.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:07 smithi190 ceph-mon[32247]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:09 smithi151 ceph-mon[28965]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:09 smithi190 ceph-mon[32247]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:09 smithi192 ceph-mon[32956]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:09.945 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.945 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.954 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.956 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:09.958 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:10.464 INFO:teuthology.orchestra.run.smithi151.stdout:77309411380 2024-06-27T21:10:10.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-27T21:10:11.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:11 smithi151 ceph-mon[28965]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:11.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:11 smithi190 ceph-mon[32247]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:11 smithi192 ceph-mon[32956]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:12.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:12 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:10:12.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:12 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:10:12.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:12 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:10:12.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:12 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:10:12.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:12 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:10:12.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:12 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:10:13.063 INFO:teuthology.orchestra.run.smithi151.stdout:107374182447 2024-06-27T21:10:13.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-27T21:10:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:13 smithi151 ceph-mon[28965]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:13.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:13 smithi190 ceph-mon[32247]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:13 smithi192 ceph-mon[32956]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:15 smithi151 ceph-mon[28965]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:15.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:15 smithi190 ceph-mon[32247]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:15 smithi192 ceph-mon[32956]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:17.655 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:17 smithi151 ceph-mon[28965]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:17.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:17 smithi190 ceph-mon[32247]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:17.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:17 smithi192 ceph-mon[32956]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:19 smithi151 ceph-mon[28965]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:19.619 INFO:teuthology.orchestra.run.smithi151.stdout:326417514512 2024-06-27T21:10:19.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-27T21:10:19.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:19 smithi190 ceph-mon[32247]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:19.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:19 smithi192 ceph-mon[32956]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:21 smithi151 ceph-mon[28965]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:21.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:21 smithi190 ceph-mon[32247]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:21.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:21 smithi192 ceph-mon[32956]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:22.743 INFO:teuthology.orchestra.run.smithi151.stdout:193273528354 2024-06-27T21:10:22.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-27T21:10:23.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:23 smithi151 ceph-mon[28965]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:23.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:23 smithi190 ceph-mon[32247]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:23.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:23 smithi192 ceph-mon[32956]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:25 smithi151 ceph-mon[28965]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:25 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:25 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:25.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:25 smithi151 ceph-mon[28965]: osdmap e91: 12 total, 12 up, 12 in 2024-06-27T21:10:25.445 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:25 smithi190 ceph-mon[32247]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:25 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:25 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:25.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:25 smithi190 ceph-mon[32247]: osdmap e91: 12 total, 12 up, 12 in 2024-06-27T21:10:25.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:25 smithi192 ceph-mon[32956]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:25.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:25 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:25.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:25 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:25.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:25 smithi192 ceph-mon[32956]: osdmap e91: 12 total, 12 up, 12 in 2024-06-27T21:10:25.963 INFO:teuthology.orchestra.run.smithi151.stdout:133143986221 2024-06-27T21:10:25.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-27T21:10:26.602 INFO:teuthology.orchestra.run.smithi151.stdout:55834574908 2024-06-27T21:10:26.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-27T21:10:26.988 INFO:teuthology.orchestra.run.smithi151.stdout:227633266718 2024-06-27T21:10:26.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-27T21:10:27.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:27 smithi151 ceph-mon[28965]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:27.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:27.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:27.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:27 smithi151 ceph-mon[28965]: osdmap e92: 12 total, 12 up, 12 in 2024-06-27T21:10:27.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:27.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:27.551 INFO:teuthology.orchestra.run.smithi151.stdout:360777252876 2024-06-27T21:10:27.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-27T21:10:27.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:27 smithi190 ceph-mon[32247]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:27.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:27.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:27.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:27 smithi190 ceph-mon[32247]: osdmap e92: 12 total, 12 up, 12 in 2024-06-27T21:10:27.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:27.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:27 smithi192 ceph-mon[32956]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:27 smithi192 ceph-mon[32956]: osdmap e92: 12 total, 12 up, 12 in 2024-06-27T21:10:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:27.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:28 smithi151 ceph-mon[28965]: osdmap e93: 12 total, 12 up, 12 in 2024-06-27T21:10:28.615 INFO:teuthology.orchestra.run.smithi151.stdout:296352743446 2024-06-27T21:10:28.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-27T21:10:28.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:28 smithi190 ceph-mon[32247]: osdmap e93: 12 total, 12 up, 12 in 2024-06-27T21:10:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:28 smithi192 ceph-mon[32956]: osdmap e93: 12 total, 12 up, 12 in 2024-06-27T21:10:29.317 INFO:teuthology.orchestra.run.smithi151.stdout:266287972379 2024-06-27T21:10:29.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-27T21:10:29.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:29 smithi151 ceph-mon[28965]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:29.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:29 smithi190 ceph-mon[32247]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:29.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:29 smithi192 ceph-mon[32956]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:31.218 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:31 smithi151 ceph-mon[28965]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:31.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:31 smithi190 ceph-mon[32247]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:31 smithi192 ceph-mon[32956]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:33.438 INFO:teuthology.orchestra.run.smithi151.stdout:163208757290 2024-06-27T21:10:33.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-27T21:10:33.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:33 smithi151 ceph-mon[28965]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:33.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:33 smithi190 ceph-mon[32247]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:33.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.773 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.773 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.775 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.779 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.779 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:33 smithi192 ceph-mon[32956]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:33.813 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:33.831 INFO:teuthology.orchestra.run.smithi151.stdout:34359738434 2024-06-27T21:10:33.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-27T21:10:35.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:35 smithi151 ceph-mon[28965]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:35.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:35 smithi190 ceph-mon[32247]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:35.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:35 smithi192 ceph-mon[32956]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:36.801 INFO:teuthology.orchestra.run.smithi151.stdout:77309411385 2024-06-27T21:10:37.477 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:37 smithi151 ceph-mon[28965]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:37.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:37.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:37.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:37 smithi151 ceph-mon[28965]: osdmap e94: 12 total, 12 up, 12 in 2024-06-27T21:10:37.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/271411927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-27T21:10:37.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:37 smithi190 ceph-mon[32247]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:37.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:37.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:37.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:37 smithi190 ceph-mon[32247]: osdmap e94: 12 total, 12 up, 12 in 2024-06-27T21:10:37.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/271411927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-27T21:10:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:37 smithi192 ceph-mon[32956]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:37 smithi192 ceph-mon[32956]: osdmap e94: 12 total, 12 up, 12 in 2024-06-27T21:10:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/271411927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-27T21:10:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:39 smithi151 ceph-mon[28965]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:39.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:39 smithi190 ceph-mon[32247]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:39.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:39.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:39 smithi192 ceph-mon[32956]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:39.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:40.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:10:40.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:40 smithi151 ceph-mon[28965]: osdmap e95: 12 total, 12 up, 12 in 2024-06-27T21:10:40.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:10:40.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:10:40.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:10:40.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:10:40.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:40 smithi190 ceph-mon[32247]: osdmap e95: 12 total, 12 up, 12 in 2024-06-27T21:10:40.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:10:40.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:10:40.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:10:40.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:10:40.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:40 smithi192 ceph-mon[32956]: osdmap e95: 12 total, 12 up, 12 in 2024-06-27T21:10:40.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:10:40.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:10:40.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:10:41.199 INFO:teuthology.orchestra.run.smithi151.stdout:227633266722 2024-06-27T21:10:41.411 INFO:teuthology.orchestra.run.smithi151.stdout:55834574911 2024-06-27T21:10:41.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:41 smithi151 ceph-mon[28965]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:41.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2009020703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-27T21:10:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:41 smithi190 ceph-mon[32247]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2009020703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-27T21:10:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:41 smithi192 ceph-mon[32956]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2009020703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-27T21:10:42.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411380 got 77309411385 for osd.2 2024-06-27T21:10:42.332 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:42.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1891911603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-27T21:10:42.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1891911603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-27T21:10:42.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1891911603' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-27T21:10:42.944 INFO:teuthology.orchestra.run.smithi151.stdout:326417514517 2024-06-27T21:10:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:43 smithi151 ceph-mon[28965]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2655514637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-27T21:10:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:43 smithi190 ceph-mon[32247]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:43.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2655514637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-27T21:10:43.922 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:43 smithi192 ceph-mon[32956]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:43.922 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2655514637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-27T21:10:43.927 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266718 got 227633266722 for osd.7 2024-06-27T21:10:43.927 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:45.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:45 smithi151 ceph-mon[28965]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:45.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:45 smithi190 ceph-mon[32247]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:45.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:45 smithi192 ceph-mon[32956]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:46.230 INFO:teuthology.orchestra.run.smithi151.stdout:107374182454 2024-06-27T21:10:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:46.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:46 smithi151 ceph-mon[28965]: osdmap e96: 12 total, 12 up, 12 in 2024-06-27T21:10:46.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2530304666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-27T21:10:46.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:46.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:46.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:46 smithi190 ceph-mon[32247]: osdmap e96: 12 total, 12 up, 12 in 2024-06-27T21:10:46.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2530304666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-27T21:10:46.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:46.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:46.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:46 smithi192 ceph-mon[32956]: osdmap e96: 12 total, 12 up, 12 in 2024-06-27T21:10:46.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2530304666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-27T21:10:46.821 INFO:teuthology.orchestra.run.smithi151.stdout:296352743450 2024-06-27T21:10:47.333 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182447 got 107374182454 for osd.3 2024-06-27T21:10:47.333 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:47 smithi151 ceph-mon[28965]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1511479710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-27T21:10:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:47 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:47.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:47 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:47 smithi190 ceph-mon[32247]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1511479710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-27T21:10:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:47 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:47.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:47 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:47.764 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514512 got 326417514517 for osd.10 2024-06-27T21:10:47.764 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:47.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:47 smithi192 ceph-mon[32956]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:47.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1511479710' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-27T21:10:47.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:47 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:47.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:47 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:48.136 INFO:teuthology.orchestra.run.smithi151.stdout:133143986225 2024-06-27T21:10:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:48 smithi151 ceph-mon[28965]: osdmap e97: 12 total, 12 up, 12 in 2024-06-27T21:10:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/520004397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-27T21:10:48.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:48 smithi190 ceph-mon[32247]: osdmap e97: 12 total, 12 up, 12 in 2024-06-27T21:10:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/520004397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-27T21:10:48.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:48 smithi192 ceph-mon[32956]: osdmap e97: 12 total, 12 up, 12 in 2024-06-27T21:10:48.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/520004397' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-27T21:10:48.840 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986221 got 133143986225 for osd.4 2024-06-27T21:10:48.840 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:49.263 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574911 for osd.1 2024-06-27T21:10:49.263 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:49.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:49.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:49.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:49 smithi151 ceph-mon[28965]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:49.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:49.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:49.676 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:49 smithi151 ceph-mon[28965]: osdmap e98: 12 total, 12 up, 12 in 2024-06-27T21:10:49.702 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743446 got 296352743450 for osd.9 2024-06-27T21:10:49.702 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:49.711 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:49.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:49.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:49 smithi190 ceph-mon[32247]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:49 smithi190 ceph-mon[32247]: osdmap e98: 12 total, 12 up, 12 in 2024-06-27T21:10:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:49 smithi192 ceph-mon[32956]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:49 smithi192 ceph-mon[32956]: osdmap e98: 12 total, 12 up, 12 in 2024-06-27T21:10:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:50 smithi151 ceph-mon[28965]: osdmap e99: 12 total, 12 up, 12 in 2024-06-27T21:10:50.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:50 smithi190 ceph-mon[32247]: osdmap e99: 12 total, 12 up, 12 in 2024-06-27T21:10:50.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:50 smithi192 ceph-mon[32956]: osdmap e99: 12 total, 12 up, 12 in 2024-06-27T21:10:50.913 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:50.913 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:51.095 INFO:teuthology.orchestra.run.smithi151.stdout:193273528359 2024-06-27T21:10:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:51 smithi151 ceph-mon[28965]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3969925732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-27T21:10:51.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:51 smithi190 ceph-mon[32247]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:51.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3969925732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-27T21:10:51.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:51 smithi192 ceph-mon[32956]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:51.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3969925732' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-27T21:10:52.120 INFO:teuthology.orchestra.run.smithi151.stdout:266287972383 2024-06-27T21:10:52.722 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3074448199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-27T21:10:52.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3074448199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-27T21:10:52.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3074448199' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-27T21:10:53.139 INFO:teuthology.orchestra.run.smithi151.stdout:360777252882 2024-06-27T21:10:53.477 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528354 got 193273528359 for osd.6 2024-06-27T21:10:53.477 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:53.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:53 smithi190 ceph-mon[32247]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:53.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3601435214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-27T21:10:53.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:53 smithi192 ceph-mon[32956]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:53.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3601435214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-27T21:10:53.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:53 smithi151 ceph-mon[28965]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:53.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3601435214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-27T21:10:54.923 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972379 got 266287972383 for osd.8 2024-06-27T21:10:54.923 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:55.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:55 smithi151 ceph-mon[28965]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:55.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:55.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:55.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:55 smithi190 ceph-mon[32247]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:55.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:55.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:55.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:55 smithi192 ceph-mon[32956]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:55.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:55.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:56.275 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777252876 got 360777252882 for osd.11 2024-06-27T21:10:56.275 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:56 smithi151 ceph-mon[28965]: osdmap e100: 12 total, 12 up, 12 in 2024-06-27T21:10:56.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:56 smithi190 ceph-mon[32247]: osdmap e100: 12 total, 12 up, 12 in 2024-06-27T21:10:56.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:56 smithi192 ceph-mon[32956]: osdmap e100: 12 total, 12 up, 12 in 2024-06-27T21:10:57.529 INFO:teuthology.orchestra.run.smithi151.stdout:34359738439 2024-06-27T21:10:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:57 smithi151 ceph-mon[28965]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:57.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:57 smithi190 ceph-mon[32247]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:57.761 INFO:teuthology.orchestra.run.smithi151.stdout:163208757295 2024-06-27T21:10:57.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:57 smithi192 ceph-mon[32956]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:58.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738434 got 34359738439 for osd.0 2024-06-27T21:10:58.140 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:58.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:10:58.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:10:58.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:58 smithi190 ceph-mon[32247]: osdmap e101: 12 total, 12 up, 12 in 2024-06-27T21:10:58.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3493740600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-27T21:10:58.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1171702724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-27T21:10:58.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:10:58.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:10:58.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:58 smithi192 ceph-mon[32956]: osdmap e101: 12 total, 12 up, 12 in 2024-06-27T21:10:58.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3493740600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-27T21:10:58.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1171702724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-27T21:10:58.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:10:58.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:10:58.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:58 smithi151 ceph-mon[28965]: osdmap e101: 12 total, 12 up, 12 in 2024-06-27T21:10:58.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3493740600' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-27T21:10:58.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1171702724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-27T21:10:58.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757290 got 163208757295 for osd.5 2024-06-27T21:10:58.940 DEBUG:teuthology.parallel:result is None 2024-06-27T21:10:58.940 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-27T21:10:58.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:10:59.198 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:10:59.626 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:10:59 smithi151 ceph-mon[28965]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:59.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:10:59 smithi190 ceph-mon[32247]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:10:59.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:10:59 smithi192 ceph-mon[32956]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:01.124 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:11:01.124 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:11:01.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:01 smithi151 ceph-mon[28965]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:01.752 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-06-27T21:10:59.812780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338140,"kb_used_data":9324,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603684,"statfs":{"total":1151940427776,"available":1151594172416,"internally_reserved":0,"allocated":9547776,"data_stored":5937643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001672"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":145,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-27T21:10:57.641268+0000","last_change":"2024-06-27T21:10:10.498300+0000","last_active":"2024-06-27T21:10:57.641268+0000","last_peered":"2024-06-27T21:10:57.641268+0000","last_clean":"2024-06-27T21:10:57.641268+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:10:57.641268+0000","last_undegraded":"2024-06-27T21:10:57.641268+0000","last_fullsized":"2024-06-27T21:10:57.641268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:10:10.498201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:10:10.498201+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:11:29.197938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514521,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743452,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27784,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":647168,"data_stored":353352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528361,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1179648,"data_stored":882280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986228,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":360729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713312,"statfs":{"total":95995035648,"available":95962431488,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27784,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":647168,"data_stored":353352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:11:01.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:11:01.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:01 smithi190 ceph-mon[32247]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:01.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:01 smithi192 ceph-mon[32956]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:02.008 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:02 smithi151 ceph-mon[28965]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:02.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:02 smithi190 ceph-mon[32247]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:02.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:02 smithi192 ceph-mon[32956]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:03.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:03 smithi151 ceph-mon[28965]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:03.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:03 smithi190 ceph-mon[32247]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:03.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:03 smithi192 ceph-mon[32956]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:03.998 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:11:03.998 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:11:04.696 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-06-27T21:11:03.813615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337872,"kb_used_data":9432,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603952,"statfs":{"total":1151940427776,"available":1151594446848,"internally_reserved":0,"allocated":9658368,"data_stored":6004036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.352917"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":145,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-27T21:10:57.641268+0000","last_change":"2024-06-27T21:10:10.498300+0000","last_active":"2024-06-27T21:10:57.641268+0000","last_peered":"2024-06-27T21:10:57.641268+0000","last_clean":"2024-06-27T21:10:57.641268+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:10:57.641268+0000","last_undegraded":"2024-06-27T21:10:57.641268+0000","last_fullsized":"2024-06-27T21:10:57.641268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:10:10.498201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:10:10.498201+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:11:29.197938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514521,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743453,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986229,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31840,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713312,"statfs":{"total":95995035648,"available":95962431488,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:11:04.697 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-27T21:11:04.697 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-27T21:11:04.697 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-27T21:11:04.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph health --format=json 2024-06-27T21:11:04.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:05 smithi151 ceph-mon[28965]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:05 smithi151 ceph-mon[28965]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:05.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:05 smithi190 ceph-mon[32247]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:05.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:05 smithi190 ceph-mon[32247]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:05.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:05 smithi192 ceph-mon[32956]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:05.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:05 smithi192 ceph-mon[32956]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:11:06.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:06 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:11:06.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:11:06.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:11:06.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:11:06.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:11:06.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:11:06.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:11:06.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:06 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:11:06.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:11:06.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:11:06.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:11:06.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:11:06.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:11:06.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:11:06.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:06 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:11:06.992 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:11:06.992 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-27T21:11:07.667 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-27T21:11:07.668 INFO:tasks.cephadm:Setup complete, yielding 2024-06-27T21:11:07.668 INFO:teuthology.run_tasks:Running task thrashosds... 2024-06-27T21:11:07.681 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-06-27T21:11:07.682 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri12-testing-2024-06-26-0904'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-yuri12-testing-2024-06-26-0904', 'sha1': 'f51acc18cc950586c82c58d666661d4391fc1122'}} 2024-06-27T21:11:07.682 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-06-27T21:11:07.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-06-27T21:11:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:11:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:11:07.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:07 smithi151 ceph-mon[28965]: osdmap e102: 12 total, 12 up, 12 in 2024-06-27T21:11:07.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2126210274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-27T21:11:07.939 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:08.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:11:08.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:11:08.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:07 smithi190 ceph-mon[32247]: osdmap e102: 12 total, 12 up, 12 in 2024-06-27T21:11:08.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2126210274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-27T21:11:08.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:11:08.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:11:08.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:07 smithi192 ceph-mon[32956]: osdmap e102: 12 total, 12 up, 12 in 2024-06-27T21:11:08.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2126210274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-27T21:11:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:08 smithi151 ceph-mon[28965]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:09.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:08 smithi190 ceph-mon[32247]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:09.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:08 smithi192 ceph-mon[32956]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:10.772 INFO:tasks.thrashosds:Beginning thrashosds... 2024-06-27T21:11:10.772 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-27T21:11:10.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:11:11.027 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:11.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:10 smithi151 ceph-mon[28965]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:11.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2567889990' entity='client.admin' 2024-06-27T21:11:11.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:11:11.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:11:11.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:11:11.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:11:11.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:10 smithi190 ceph-mon[32247]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:11.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2567889990' entity='client.admin' 2024-06-27T21:11:11.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:11:11.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:11:11.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:11:11.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:11:11.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:10 smithi192 ceph-mon[32956]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:11.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2567889990' entity='client.admin' 2024-06-27T21:11:11.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:11:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:11:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:11:11.286 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:11:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:11:12.807 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:11:13.154 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:12 smithi151 ceph-mon[28965]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:13.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:12 smithi190 ceph-mon[32247]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:13.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:12 smithi192 ceph-mon[32956]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:13.565 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-06-27T21:11:11.815292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337940,"kb_used_data":9528,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603884,"statfs":{"total":1151940427776,"available":1151594377216,"internally_reserved":0,"allocated":9756672,"data_stored":6063052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002008"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":147,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-06-27T21:11:07.748550+0000","last_change":"2024-06-27T21:10:10.498300+0000","last_active":"2024-06-27T21:11:07.748550+0000","last_peered":"2024-06-27T21:11:07.748550+0000","last_clean":"2024-06-27T21:11:07.748550+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:11:07.748550+0000","last_undegraded":"2024-06-27T21:11:07.748550+0000","last_fullsized":"2024-06-27T21:11:07.748550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:10:10.498201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:10:10.498201+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:11:29.197938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972387,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1216512,"data_stored":904411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1216512,"data_stored":904411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986230,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":368106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:11:13.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:11:13.832 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:14.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:13 smithi151 ceph-mon[28965]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:14.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:13 smithi190 ceph-mon[32247]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:14.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:13 smithi192 ceph-mon[32956]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:15.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:14 smithi151 ceph-mon[28965]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:15.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:14 smithi190 ceph-mon[32247]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:15.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:14 smithi192 ceph-mon[32956]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:11:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:11:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:15 smithi151 ceph-mon[28965]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:15 smithi190 ceph-mon[32247]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:16.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:15 smithi192 ceph-mon[32956]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:11:16.407 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-06-27T21:11:13.815734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337968,"kb_used_data":9564,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603856,"statfs":{"total":1151940427776,"available":1151594348544,"internally_reserved":0,"allocated":9793536,"data_stored":6085183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":336115002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002118"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":147,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-06-27T21:11:07.748550+0000","last_change":"2024-06-27T21:10:10.498300+0000","last_active":"2024-06-27T21:11:07.748550+0000","last_peered":"2024-06-27T21:11:07.748550+0000","last_clean":"2024-06-27T21:11:07.748550+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:11:07.748550+0000","last_undegraded":"2024-06-27T21:11:07.748550+0000","last_fullsized":"2024-06-27T21:11:07.748550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:10:10.498201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:10:10.498201+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:11:29.197938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":897014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972388,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1216512,"data_stored":904411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1216512,"data_stored":904411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986231,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31848,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713304,"statfs":{"total":95995035648,"available":95962423296,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:11:16.407 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-27T21:11:16.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump 2024-06-27T21:11:16.658 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:17.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:16 smithi151 ceph-mon[28965]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:17.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:16 smithi190 ceph-mon[32247]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:17.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:16 smithi192 ceph-mon[32956]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:18.365 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 102 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-06-27T21:02:54.280751+0000 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-06-27T21:11:06.832120+0000 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:crush_version 26 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:full_ratio 0.95 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:backfillfull_ratio 0.9 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:nearfull_ratio 0.85 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:require_min_compat_client luminous 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:min_compat_client jewel 2024-06-27T21:11:18.366 INFO:teuthology.orchestra.run.smithi151.stdout:require_osd_release squid 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:stretch_mode_enabled false 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:max_osd 12 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 up in weight 1 up_from 8 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429] [v2:172.21.15.151:6804/2519434429,v1:172.21.15.151:6805/2519434429] exists,up 80c5e023-5c22-463d-86e9-2307159c858f 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 up in weight 1 up_from 13 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028] [v2:172.21.15.151:6812/59941028,v1:172.21.15.151:6813/59941028] exists,up d6a0fa83-bdab-4c01-b0d0-8996e8d0c083 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170] [v2:172.21.15.151:6820/302970170,v1:172.21.15.151:6821/302970170] exists,up 22115999-54ff-4f34-81cc-9d90bfa4bd62 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662] [v2:172.21.15.151:6828/3555754662,v1:172.21.15.151:6829/3555754662] exists,up 88c12fa5-d2af-4144-958e-b887b4474b80 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547] [v2:172.21.15.190:6802/2107645547,v1:172.21.15.190:6803/2107645547] exists,up 97aaddb9-d5bb-4827-8c35-b3c948eae3dc 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 up in weight 1 up_from 38 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120] [v2:172.21.15.190:6810/888361120,v1:172.21.15.190:6811/888361120] exists,up 7ac9bbd6-0e55-4cab-b406-6f9967eaaf30 2024-06-27T21:11:18.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280] [v2:172.21.15.190:6818/30028280,v1:172.21.15.190:6819/30028280] exists,up 89fae0b2-26a4-456a-9d54-cfbaec80ba6a 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 up in weight 1 up_from 53 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685] [v2:172.21.15.190:6826/849289685,v1:172.21.15.190:6827/849289685] exists,up e324ffdc-9842-46ee-a87e-5054ec265484 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942] [v2:172.21.15.192:6802/2977299942,v1:172.21.15.192:6803/2977299942] exists,up 43d138bf-2d82-44ea-a98f-b9099e824243 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799] [v2:172.21.15.192:6810/3401759799,v1:172.21.15.192:6811/3401759799] exists,up 0c86c106-04c9-4eb7-bb3e-d39f563b71fd 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762] [v2:172.21.15.192:6818/1426195762,v1:172.21.15.192:6819/1426195762] exists,up ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 up in weight 1 up_from 84 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267] [v2:172.21.15.192:6826/517567267,v1:172.21.15.192:6827/517567267] exists,up a8a163c4-a14c-4710-bc2e-94e2cb69df0d 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/4089480652 expires 2024-06-28T21:03:45.726698+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/3356605898 expires 2024-06-28T21:03:45.726698+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/3908626376 expires 2024-06-28T21:03:45.726698+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/3654164819 expires 2024-06-28T21:03:45.726698+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/3261020553 expires 2024-06-28T21:03:22.780955+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/3654164819 expires 2024-06-28T21:03:45.726698+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/3767407460 expires 2024-06-28T21:03:22.780955+0000 2024-06-27T21:11:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/3767407460 expires 2024-06-28T21:03:22.780955+0000 2024-06-27T21:11:18.369 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2698418425 expires 2024-06-28T21:03:22.780955+0000 2024-06-27T21:11:18.369 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/335837712 expires 2024-06-28T21:03:22.780955+0000 2024-06-27T21:11:18.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:18 smithi151 ceph-mon[28965]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:18.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3229426127' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-27T21:11:19.090 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6802/2519434429,v1:172.21.15.151:6803/2519434429] [v2:172.21.15.151:6804/2519434429,v1:172.21.15.151:6805/2519434429] exists,up 80c5e023-5c22-463d-86e9-2307159c858f', 'osd.1 up in weight 1 up_from 13 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6810/59941028,v1:172.21.15.151:6811/59941028] [v2:172.21.15.151:6812/59941028,v1:172.21.15.151:6813/59941028] exists,up d6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6818/302970170,v1:172.21.15.151:6819/302970170] [v2:172.21.15.151:6820/302970170,v1:172.21.15.151:6821/302970170] exists,up 22115999-54ff-4f34-81cc-9d90bfa4bd62', 'osd.3 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6826/3555754662,v1:172.21.15.151:6827/3555754662] [v2:172.21.15.151:6828/3555754662,v1:172.21.15.151:6829/3555754662] exists,up 88c12fa5-d2af-4144-958e-b887b4474b80', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/2107645547,v1:172.21.15.190:6801/2107645547] [v2:172.21.15.190:6802/2107645547,v1:172.21.15.190:6803/2107645547] exists,up 97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'osd.5 up in weight 1 up_from 38 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/888361120,v1:172.21.15.190:6809/888361120] [v2:172.21.15.190:6810/888361120,v1:172.21.15.190:6811/888361120] exists,up 7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/30028280,v1:172.21.15.190:6817/30028280] [v2:172.21.15.190:6818/30028280,v1:172.21.15.190:6819/30028280] exists,up 89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'osd.7 up in weight 1 up_from 53 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/849289685,v1:172.21.15.190:6825/849289685] [v2:172.21.15.190:6826/849289685,v1:172.21.15.190:6827/849289685] exists,up e324ffdc-9842-46ee-a87e-5054ec265484', 'osd.8 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/2977299942,v1:172.21.15.192:6801/2977299942] [v2:172.21.15.192:6802/2977299942,v1:172.21.15.192:6803/2977299942] exists,up 43d138bf-2d82-44ea-a98f-b9099e824243', 'osd.9 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/3401759799,v1:172.21.15.192:6809/3401759799] [v2:172.21.15.192:6810/3401759799,v1:172.21.15.192:6811/3401759799] exists,up 0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'osd.10 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/1426195762,v1:172.21.15.192:6817/1426195762] [v2:172.21.15.192:6818/1426195762,v1:172.21.15.192:6819/1426195762] exists,up ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'osd.11 up in weight 1 up_from 84 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6824/517567267,v1:172.21.15.192:6825/517567267] [v2:172.21.15.192:6826/517567267,v1:172.21.15.192:6827/517567267] exists,up a8a163c4-a14c-4710-bc2e-94e2cb69df0d'] 2024-06-27T21:11:19.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon mon.a config show 2024-06-27T21:11:19.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:18 smithi190 ceph-mon[32247]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:19.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3229426127' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-27T21:11:19.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:18 smithi192 ceph-mon[32956]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:19.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3229426127' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-27T21:11:19.345 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:21.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:20 smithi151 ceph-mon[28965]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "name": "mon.a", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster": "ceph", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "admin_socket_mode": "", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_client_required": "cephx, none", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_cluster_required": "cephx", 2024-06-27T21:11:21.159 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_debug": "true", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_service_required": "cephx", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_service_ticket_ttl": "240.000000", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_supported": "", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio": "true", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_max_queue_depth": "1024", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_poll_ms": "250", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_reap_max": "16", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_async_discard_threads": "0", 2024-06-27T21:11:21.160 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_block_size": "4096", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio": "false", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_inflight_ios": "false", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_enable_discard": "false", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_flock_retry": "3", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_flock_retry_interval": "0.100000", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_inject_crash": "0", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_inject_crash_flush_delay": "2", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring": "false", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring_hipri": "false", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring_sqthread_poll": "false", 2024-06-27T21:11:21.161 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_read_buffer_alignment": "4096", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_type": "", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_alloc_size": "1048576", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_allocator": "hybrid", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_buffered_io": "true", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_for_zeros": "false", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_volume_selector_often": "false", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_compact_log_sync": "false", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-06-27T21:11:21.162 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_compact_min_size": "16777216", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_replay_check_allocations": "true", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_max_log_runway": "4194304", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_max_prefetch": "1048576", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_min_flush_size": "524288", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_min_log_runway": "1048576", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_replay_recovery": "false", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_shared_alloc_size": "65536", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_sync_write": "false", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-06-27T21:11:21.163 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_allocation_from_file": "true", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_allocator": "hybrid", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_blobid_prealloc": "10240", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_create": "true", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_create": "false", 2024-06-27T21:11:21.164 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_path": "", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_size": "0", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_path": "", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_preallocate_file": "false", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_size": "107374182400", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_create": "false", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_path": "", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_size": "100663296", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs": "true", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_env_mirror": "false", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-06-27T21:11:21.165 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_autotune": "true", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size": "0", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-06-27T21:11:21.166 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_type": "2q", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_clone_cow": "true", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_algorithm": "snappy", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size": "0", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size": "0", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-06-27T21:11:21.167 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_mode": "none", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_csum_type": "crc32c", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_enforce_settings": "default", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_freelist": "false", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_fsck_abort": "false", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_read_err": "false", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_legacy_omap": "false", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-06-27T21:11:21.168 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_prefill": "0.000000", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_small_allocations": "0", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_default_buffered_read": "true", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_default_buffered_write": "false", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops": "0", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-06-27T21:11:21.169 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_elastic_shared_blobs": "true", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fail_eio": "false", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mkfs": "true", 2024-06-27T21:11:21.170 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mount": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_umount": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_ignore_data_csum": "false", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-06-27T21:11:21.171 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_kvbackend": "rocksdb", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_op_age": "5.000000", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_alloc_size": "0", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size": "0", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_defer_interval": "3.000000", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_deferred_txc": "32", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size": "0", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_nid_prealloc": "1024", 2024-06-27T21:11:21.172 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size": "0", 2024-06-27T21:11:21.173 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-06-27T21:11:21.173 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-06-27T21:11:21.173 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_qfsck_on_mount": "true", 2024-06-27T21:11:21.173 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_retry_disk_reads": "3", 2024-06-27T21:11:21.173 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_cf": "true", 2024-06-27T21:11:21.173 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-06-27T21:11:21.174 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-06-27T21:11:21.174 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_options_annex": "", 2024-06-27T21:11:21.174 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_coremask": "0x1", 2024-06-27T21:11:21.174 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_io_sleep": "5", 2024-06-27T21:11:21.174 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_max_io_completion": "0", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_mem": "512", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_sync_submit_transaction": "false", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_bytes": "67108864", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io": "0", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_tracing": "false", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_reserved": "0", 2024-06-27T21:11:21.175 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_zero_block_detection": "false", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_action_update_interval": "2", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-06-27T21:11:21.176 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_mount_timeout": "10", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_cluster_require_signatures": "false", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_cluster_require_version": "2", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_require_signatures": "false", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_require_version": "2", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_service_require_signatures": "false", 2024-06-27T21:11:21.177 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_service_require_version": "2", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_sign_messages": "true", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "chdir": "", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_acl_type": "", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_asio_thread_count": "2", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_cache_mid": "0.750000", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_cache_size": "16384", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_caps_release_delay": "5", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_check_pool_perm": "true", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_collect_and_send_global_metrics": "false", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_force_sync_read": "false", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_getattr_caps": "false", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_inject_features": "", 2024-06-27T21:11:21.178 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_inject_tick_delay": "0", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_die_on_failed_remount": "false", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_dirsize_rbytes": "true", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_force_lazyio": "false", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_fs": "", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_inject_fixed_oldest_tid": "false", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_inject_release_failure": "false", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_max_inline_size": "4096", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_max_retries_on_remount_failure": "5", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mds_namespace": "", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_metadata": "", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_gid": "-1", 2024-06-27T21:11:21.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_timeout": "300", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_uid": "-1", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mountpoint": "/", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_notify_timeout": "10", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc": "true", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_dirty": "104857600", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_dirty_age": "5.000000", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_objects": "1000", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_size": "209715200", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_target_dirty": "8388608", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_permissions": "true", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_quota": "true", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_quota_df": "true", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_max_bytes": "0", 2024-06-27T21:11:21.180 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_max_periods": "4", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_min": "131072", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_reconnect_stale": "false", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_shutdown_timeout": "30", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_snapdir": ".snap", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_tick_interval": "1", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_trace": "", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_try_dentry_invalidate": "false", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_use_faked_inos": "false", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "client_use_random_mds": "false", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog": "false", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog_port": "12201", 2024-06-27T21:11:21.181 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_monitors": "default=true", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog": "false", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog_level": "info", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "-", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_network": "", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_network_interface": "", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_isal": "false", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_level": "5", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_winsize": "-15", 2024-06-27T21:11:21.182 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zstd_level": "1", 2024-06-27T21:11:21.185 INFO:teuthology.orchestra.run.smithi151.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_alien_op_num_threads": "6", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_alien_thread_cpu_cores": "", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_obc_lru_size": "512", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_stat_interval": "0", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_seastar_cpu_cores": "", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_seastar_num_threads": "0", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location": "", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location_hook": "", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location_hook_timeout": "10", 2024-06-27T21:11:21.186 INFO:teuthology.orchestra.run.smithi151.stdout: "daemonize": "false", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "daos_pool": "tank", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_db_name_prefix": "dbstore", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_alienstore": "0/5", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_allow_any_pool_priority": "false", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asok": "1/5", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asok_assert_abort": "false", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asserts_on_shutdown": "true", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_auth": "1/5", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bdev": "1/3", 2024-06-27T21:11:21.187 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bluefs": "1/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bluestore": "1/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_buffer": "0/1", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ceph_dedup": "0/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ceph_exporter": "1/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cephfs_mirror": "0/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cephsqlite": "0/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_client": "0/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_compressor": "1/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_context": "0/1", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crimson_interrupt": "0/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crush": "1/1", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crypto": "1/5", 2024-06-27T21:11:21.188 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cyanstore": "0/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_deliberately_leak_memory": "false", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_disable_randomized_ping": "false", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_dpdk": "1/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_eventtrace": "1/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_filer": "0/1", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_filestore": "1/3", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_finisher": "1/1", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_fuse": "1/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_heartbeat_testing_span": "0", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_heartbeatmap": "1/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_immutable_obj_cache": "0/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_javaclient": "1/5", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_journal": "1/3", 2024-06-27T21:11:21.189 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_journaler": "0/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_kstore": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_lockdep": "0/1", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mclock": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_balancer": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_locker": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_log": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_log_expire": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_migrator": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_quiesce": "3/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_memstore": "1/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mgr": "2/5", 2024-06-27T21:11:21.190 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mgrc": "1/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mon": "20/20", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_monc": "0/10", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ms": "1/1", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_none": "0/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objclass": "0/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objectcacher": "0/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objecter": "0/1", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_optracker": "0/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_osd": "1/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_paxos": "20/20", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_perfcounter": "1/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_prioritycache": "1/5", 2024-06-27T21:11:21.191 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rados": "0/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd": "0/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_mirror": "0/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_pwl": "0/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_replay": "0/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_refs": "0/0", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_reserver": "1/1", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_access": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_datacache": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_dbstore": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_flight": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_lifecycle": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_notification": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_sync": "1/5", 2024-06-27T21:11:21.192 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rocksdb": "4/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_backref": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_cache": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_cleaner": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_device": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_epm": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_journal": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_lba": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_odata": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_omap": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_onode": "0/5", 2024-06-27T21:11:21.193 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_t": "0/5", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_tm": "0/5", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_striper": "0/1", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_test": "0/5", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_throttle": "1/1", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_timer": "0/1", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_tp": "0/5", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_trace": "1/5", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "device_failure_prediction_mode": "none", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_graylog": "false", 2024-06-27T21:11:21.194 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_journald": "false", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_stderr": "true", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_syslog": "false", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event_tracing": "false", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_addr": "0.0.0.0", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_http_port": "9926", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_prio_limit": "5", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_sort_metrics": "true", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_stats_period": "5", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "fake_statfs_for_testing": "0", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "fatal_signal_handlers": "true", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "filer_max_purge_ops": "10", 2024-06-27T21:11:21.195 INFO:teuthology.orchestra.run.smithi151.stdout: "filer_max_truncate_ops": "128", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_apply_finisher_threads": "1", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_blackhole": "false", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_btrfs_clone_range": "true", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_btrfs_snap": "true", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_caller_concurrency": "10", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_collect_device_partition_information": "true", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_commit_timeout": "600.000000", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_inject_read_err": "false", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_omap_check": "false", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_verify_split": "false", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_dump_file": "", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-06-27T21:11:21.196 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fadvise": "true", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fail_eio": "true", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fd_cache_shards": "16", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fd_cache_size": "128", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fiemap": "false", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fiemap_threshold": "4096", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_index_retry_probability": "0.000000", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_inject_stall": "0", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_parallel": "false", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_trailing": "false", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_writeahead": "false", 2024-06-27T21:11:21.197 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_kill_at": "0", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size": "0", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs": "0", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_other": "2", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_sync_interval": "5.000000", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size": "0", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-06-27T21:11:21.198 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_merge_threshold": "-10", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_min_sync_interval": "0.010000", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_odsync_write": "false", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_backend": "rocksdb", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_backend_path": "", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_header_cache_size": "1024", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_ondisk_finisher_threads": "1", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_thread_timeout": "60", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_threads": "2", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_punch_hole": "false", 2024-06-27T21:11:21.199 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_bytes": "104857600", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_ops": "50", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_seek_data_hole": "false", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_sloppy_crc": "false", 2024-06-27T21:11:21.200 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_splice": "false", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_split_multiple": "2", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_split_rand_factor": "20", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_update_to": "1000", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_enable": "true", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-06-27T21:11:21.201 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_xfs_extsize": "false", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_zfs_snap": "false", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fio_dir": "/tmp/fio", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_allow_other": "true", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_atomic_o_trunc": "true", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_big_writes": "true", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_debug": "false", 2024-06-27T21:11:21.202 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_default_permissions": "false", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_disable_pagecache": "false", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_max_write": "0", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_multithreaded": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_require_active_mds": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_set_user_groups": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_move": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_read": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_write": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_syncfs_on_mksnap": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_use_invalidate_cb": "true", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "gss_target_name": "ceph", 2024-06-27T21:11:21.203 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_file": "", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_inject_failure": "0", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_interval": "5", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "host": "smithi151", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_max_size": "1073741824", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-06-27T21:11:21.204 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_watermark": "0.900000", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "inject_early_sigterm": "false", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "jaeger_agent_port": "6799", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "jaeger_tracing_enable": "false", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_aio": "true", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_align_min_size": "65536", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_block_align": "true", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_block_size": "4096", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_dio": "true", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_discard": "false", 2024-06-27T21:11:21.205 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_force_aio": "false", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_ignore_corruption": "false", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_max_write_bytes": "10485760", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_max_write_entries": "100", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_replay_from": "0", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_high_multiple": "0.000000", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_max_multiple": "0.000000", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_write_header_frequency": "0", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_zero_on_create": "false", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_prefetch_periods": "10", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_prezero_periods": "5", 2024-06-27T21:11:21.206 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_write_head_interval": "15", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "key": "", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "keyfile": "", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_backend": "rocksdb", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_default_stripe_size": "65536", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_fsck_on_mount": "false", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_fsck_on_mount_deep": "true", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_max_bytes": "67108864", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_max_ops": "512", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_nid_prealloc": "1024", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_onode_map_size": "1024", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_sync_submit_transaction": "false", 2024-06-27T21:11:21.207 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_sync_transaction": "false", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "librados_thread_count": "2", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "lockdep": "false", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "lockdep_force_backtrace": "false", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_coarse_timestamps": "true", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_flush_on_exit": "false", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_graylog_host": "127.0.0.1", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_graylog_port": "12201", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_max_new": "1000", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_max_recent": "10000", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_stderr_prefix": "", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_stop_at_utilization": "0.970000", 2024-06-27T21:11:21.208 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_file": "true", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_graylog": "false", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_journald": "false", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_stderr": "false", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_syslog": "false", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "max_rotating_auth_attempts": "10", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_action_on_write_error": "1", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_alternate_name_max": "8192", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_asio_thread_count": "2", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_export_pin": "true", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_dirs": "true", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-06-27T21:11:21.209 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_interval": "5", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_size_max": "100000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_idle_threshold": "0.000000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_interval": "10", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_max": "-1", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_max_until": "-1", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_merge_size": "50", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_midchunk": "0.300000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_min_rebalance": "0.100000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_min_start": "0.200000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_minchunk": "0.001000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_mode": "0", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_need_max": "1.200000", 2024-06-27T21:11:21.210 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_need_min": "0.800000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_overload_epochs": "2", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_sample_interval": "3.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_bits": "3", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_rd": "25000.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_size": "10000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_wr": "10000.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_target_decay": "10.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_grace": "15.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_interval": "4.000000", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_mon_down_grace": "60", 2024-06-27T21:11:21.211 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_memory_limit": "4294967296", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_mid": "0.700000", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_delay": "0", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_sleep": "200", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_splitauth": "true", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_threshold": "524288", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_release_free_interval": "10", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_reservation": "0.050000", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_interval": "1", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_threshold": "262144", 2024-06-27T21:11:21.212 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_delegate_inos_pct": "50", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_prealloc_inos": "1000", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_connect_bootstrapping": "false", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_damage_table_max_entries": "10000", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_auth_pins": "false", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_frag": "false", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_scatterstat": "false", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_subtrees": "false", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_decay_halflife": "5.000000", 2024-06-27T21:11:21.213 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_default_dir_hash": "2", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_defer_session_stale": "true", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_deny_all_reconnect": "false", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_keys_per_op": "16384", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_max_commit_size": "10", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_max_entries": "0", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_prefetch": "true", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dirstat_min_interval": "1.000000", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_after_rejoin": "false", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_on_map": "false", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_threshold_file": "0", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_early_reply": "true", 2024-06-27T21:11:21.214 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_enable_op_tracker": "true", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_enforce_unique_name": "true", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_distributed": "true", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_random": "true", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_forward_all_requests_to_auth": "false", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_health_cache_threshold": "1.500000", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_health_summarize_threshold": "10", 2024-06-27T21:11:21.215 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_heartbeat_grace": "15.000000", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_heartbeat_reset_grace": "1000", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_health_dummy": "false", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_migrator_session_race": "false", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_join_fs": "", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_journal_format": "1", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_create_at": "0", 2024-06-27T21:11:21.216 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_dirfrag_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_export_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_import_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_expire_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_replay_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_link_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_mdstable_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_openc_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_rename_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_shutdown_at": "0", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_event_large_threshold": "524288", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_events_per_segment": "1024", 2024-06-27T21:11:21.217 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_major_segment_event_ratio": "12", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_max_events": "-1", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_max_segments": "128", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_pause": "false", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_segment_size": "0", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_skip_corrupt_events": "false", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_skip_unbounded_events": "false", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_threshold": "128", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_warn_factor": "2.000000", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_caps_per_client": "1048576", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_completed_flushes": "100000", 2024-06-27T21:11:21.218 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_completed_requests": "100000", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_export_size": "20971520", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_file_recover": "32", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_files": "64", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_ops": "8192", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_snaps_per_dir": "100", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_metrics_update_interval": "2", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_min_caps_per_client": "100", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_min_caps_working_set": "10000", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_numa_node": "-1", 2024-06-27T21:11:21.219 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_complaint_time": "30.000000", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_duration": "600", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_size": "20", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_slow_op_size": "20", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_slow_op_threshold": "10", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_log_threshold": "5", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_ping_grace": "15", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_ping_interval": "5", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_caps": "30000", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-06-27T21:11:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_decay_threshold": "131072", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_warning_threshold": "262144", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_reconnect_timeout": "45.000000", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_replay_interval": "1.000000", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_root_ino_gid": "0", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_root_ino_uid": "0", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_blocklist_on_evict": "true", 2024-06-27T21:11:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_blocklist_on_timeout": "true", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_metadata_threshold": "16777216", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_shutdown_check": "0", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_skip_ino": "0", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_sleep_rank_change": "0.000000", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_max_uid": "4294967294", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_min_uid": "0", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_rstat": "false", 2024-06-27T21:11:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_standby_replay_damaged": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_symlink_recovery": "true", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_task_status_update_interval": "2.000000", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_thrash_exports": "0", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_thrash_fragments": "0", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_tick_interval": "5.000000", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_valgrind_exit": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_verify_backtrace": "1", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_verify_scatter": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_wipe_ino_prealloc": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_wipe_sessions": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "mempool_debug": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_debug_omit_block_device_write": "false", 2024-06-27T21:11:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_device_bytes": "1073741824", 2024-06-27T21:11:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_page_set": "false", 2024-06-27T21:11:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_page_size": "65536", 2024-06-27T21:11:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_bytes": "134217728", 2024-06-27T21:11:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_messages": "512", 2024-06-27T21:11:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-06-27T21:11:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_connect_retry_interval": "1.000000", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_disabled_modules": "", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_enable_op_tracker": "true", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_max_pg_creating": "1024", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_max_pg_num_change": "128", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mds_bytes": "134217728", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mds_messages": "128", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mon_bytes": "134217728", 2024-06-27T21:11:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mon_messages": "128", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_num_op_tracker_shard": "32", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_complaint_time": "30.000000", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_duration": "600", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_size": "20", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_slow_op_size": "20", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_log_threshold": "5", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_osd_bytes": "536870912", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_osd_messages": "8192", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_pool": "true", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_service_beacon_grace": "60.000000", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_standby_modules": "true", 2024-06-27T21:11:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_stats_period": "5", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_stats_threshold": "5", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_tick_period": "2", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_accept_timeout_factor": "2.000000", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_allow_pool_delete": "true", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_allow_pool_size_one": "false", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_auth_validate_all_caps": "true", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_bytes": "104857600", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_directed_command_retry": "5", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval": "3.000000", 2024-06-27T21:11:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_parallel": "3", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_log_interval": "1.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_ping_interval": "10.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_ping_timeout": "30.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_target_rank": "-1", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clock_drift_allowed": "1.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_level": "debug", 2024-06-27T21:11:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_file": "true", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog": "false", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_journald": "true", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_stderr": "false", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_bootstrap": "false", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_start": "false", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_trim": "true", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_con_tracker_persist_interval": "10", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_con_tracker_score_halflife": "43200", 2024-06-27T21:11:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_config_key_max_entry_size": "65536", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cpu_threads": "4", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_crush_min_required_version": "hammer", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_daemon_bytes": "419430400", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_avail_crit": "5", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_avail_warn": "5", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_size_warn": "16106127360", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_block_osdmap_trim": "false", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_json": "false", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_transactions": "false", 2024-06-27T21:11:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_extra_checks": "false", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_reef": "false", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_squid": "false", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_delta_reset_interval": "10.000000", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_dns_srv_name": "ceph-mon", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_down_mkfs_grace": "60", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_down_uptime_grace": "60", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_election_default_strategy": "3", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_election_timeout": "5.000000", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ping_divisor": "2", 2024-06-27T21:11:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ping_timeout": "2.000000", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_enable_op_tracker": "true", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_fake_pool_delete": "false", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_force_quorum_join": "false", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_fsmap_prune_threshold": "300", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_globalid_prealloc": "10000", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_detail_to_clog": "true", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_log_update_period": "5", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_max_detail": "50", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog": "true", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog_interval": "600", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_host": "[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]", 2024-06-27T21:11:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_host_override": "", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_initial_members": "", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_keyvaluedb": "rocksdb", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease": "5.000000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_full_interval": "50", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_max": "10000", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_max_summary": "50", 2024-06-27T21:11:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_log_entries_per_event": "4096", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_log_epochs": "500", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_mdsmap_epochs": "500", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_mgrmap_epochs": "500", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_osd": "10000", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_pg_per_osd": "10000", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_pool_pg_num": "65536", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_force_trim_to": "0", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_skip_sanity": "false", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_memory_autotune": "true", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_memory_target": "2147483648", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_beacon_grace": "30", 2024-06-27T21:11:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_digest_period": "5", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_inactive_grace": "60", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_mkfs_grace": "240", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_min_osdmap_epochs": "50", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_complaint_time": "30", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_duration": "600", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_size": "20", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_slow_op_size": "20", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_slow_op_threshold": "10", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_log_threshold": "5", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-06-27T21:11:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_in": "false", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_new_in": "true", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_cache_size": "500", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_cache_size_min": "134217728", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_crush_smoke_test": "true", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_destroyed_out_interval": "600", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_down_out_interval": "600", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_force_trim_to": "0", 2024-06-27T21:11:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_full_ratio": "0.950000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_halflife": "3600", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_max_interval": "300", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_weight": "0.300000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_max_initial_pgs": "1024", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_down_reporters": "2", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp": "true", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-06-27T21:11:21.238 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_report_timeout": "900", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_warn_num_repaired": "10", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_warn_op_age": "32.000000", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_interval": "2", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_min": "15", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_stuck_threshold": "60", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_objects": "10000", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_per_osd": "0", 2024-06-27T21:11:21.239 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pool_quota_crit_threshold": "0", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pool_quota_warn_threshold": "0", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_probe_timeout": "2.000000", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_max_change": "0.050000", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_max_osds": "4", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_interval": "300", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_max_keys": "100", 2024-06-27T21:11:21.240 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_timeout": "300", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_session_timeout": "300", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_smart_report_timeout": "5", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stat_smooth_intervals": "6", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_pool_min_size": "2", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_pool_size": "4", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_subscribe_interval": "86400.000000", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_debug": "false", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_max_payload_keys": "2000", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_max_payload_size": "1048576", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_provider_kill_at": "0", 2024-06-27T21:11:21.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_requester_kill_at": "0", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_timeout": "60.000000", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_target_pg_per_osd": "100", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_tick_interval": "5", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_timecheck_interval": "300.000000", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_use_min_delay_socket": "false", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_older_version_delay": "604800", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_filestore_osds": "true", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-06-27T21:11:21.242 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_misplaced": "false", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_older_version": "true", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_app": "true", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_too_few_osds": "false", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-06-27T21:11:21.243 INFO:teuthology.orchestra.run.smithi151.stdout: "monmap": "", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_tracing_enabled": "false", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_op_threads": "3", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_buffer_size": "131072", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_cm": "false", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_device_name": "", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_dscp": "96", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-06-27T21:11:21.244 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_gid_idx": "0", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_local_gid": "", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_polling_us": "1000", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_port_num": "1", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_roce_ver": "1", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_send_buffers": "1024", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_sl": "3", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_support_srq": "true", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_type": "ib", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_reap_threshold": "5", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_before_connect": "false", 2024-06-27T21:11:21.245 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_ipv4": "true", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_ipv6": "false", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_msgr1": "true", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_msgr2": "true", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_port_max": "7568", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_port_min": "6800", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_prefer_ipv4": "false", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_retry_count": "3", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_retry_delay": "5", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_client": "false", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mds": "false", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mgr": "false", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mon": "false", 2024-06-27T21:11:21.246 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_osd": "false", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_client_mode": "crc secure", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_cluster_mode": "crc secure", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_cluster_type": "", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_compress_secure": "false", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_connection_idle_timeout": "900", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_connection_ready_timeout": "10", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_crc_data": "true", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_crc_header": "true", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_bad_msg": "false", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_bug": "true", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_old_message": "true", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_skipped_message": "false", 2024-06-27T21:11:21.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_unhandled_msg": "false", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_coremask": "0xF", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_devs_allowlist": "", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_enable_tso": "true", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hugepages": "", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hw_flow_control": "true", 2024-06-27T21:11:21.248 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_lro": "true", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_memory_channel": "4", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_pmd": "", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_port_id": "0", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dump_corrupt_message_level": "1", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dump_on_send": "false", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_initial_backoff": "0.200000", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_max": "1.000000", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_probability": "0.005000", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_type": "osd", 2024-06-27T21:11:21.249 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_internal_delays": "0.002000", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_network_congestion": "0", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_socket_failures": "2500", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_learn_addr_from_peer": "true", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_max_accept_failures": "4", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_max_backoff": "15.000000", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_client_mode": "secure crc", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_cluster_mode": "secure crc", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_service_mode": "secure crc", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compress_min_size": "1024", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compress_mode": "none", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compression_algorithm": "snappy", 2024-06-27T21:11:21.250 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_pq_min_cost": "65536", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_public_type": "", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_service_mode": "crc secure", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_listen_backlog": "512", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_nodelay": "true", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_rcvbuf": "0", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_time_events_min_wait_interval": "1000", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_type": "async+posix", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "no_config_file": "false", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_completion_locks_per_session": "32", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_debug_inject_relock_delay": "false", 2024-06-27T21:11:21.251 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inflight_op_bytes": "104857600", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inflight_ops": "1024", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inject_no_watch_ping": "false", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_tick_interval": "5.000000", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_timeout": "10.000000", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objectstore_blackhole": "false", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "openssl_engine_opts": "", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_delay_time": "5.000000", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_hist_halflife": "1000", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_max_low_ops": "2", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_max_ops": "4", 2024-06-27T21:11:21.252 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_quantize_effort": "0.100000", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_slop": "0.020000", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_allow_recovery_below_min_size": "true", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_asio_thread_count": "2", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_async_recovery_min_cost": "100", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_auto_mark_unfound_lost": "false", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_retry_interval": "30.000000", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_scan_max": "512", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_scan_min": "64", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_degraded": "false", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_peering": "false", 2024-06-27T21:11:21.253 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_unfound": "true", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_beacon_report_interval": "300", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_duration": "30", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_max_block_size": "67108864", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_small_size_max_iops": "100", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_blkin_trace_all": "false", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_blocked_scrub_grace_period": "120", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_check_for_log_corruption": "false", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-27T21:11:21.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_update_on_start": "true", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_message_cap": "256", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_message_size_cap": "524288000", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_op_priority": "63", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_watch_timeout": "30", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_command_max_records": "256", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_compact_on_start": "false", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_chooseleaf_type": "1", 2024-06-27T21:11:21.255 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_initial_weight": "-1.000000", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_update_on_start": "true", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_update_weight_set": "true", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_drop_ping_duration": "0", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_feed_pullee": "-1", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_misdirected_ops": "false", 2024-06-27T21:11:21.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_no_acting_change": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_no_purge_strays": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_op_order": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_pg_log_writeout": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_pretend_recovery_active": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_shutdown": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_trim_objects": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_cached_snaps": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_missing_on_start": "false", 2024-06-27T21:11:21.257 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_snaps": "false", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_keys": "1024", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_stride": "524288", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_default_data_pool_replay_window": "45", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_default_notify_timeout": "30", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep": "0.000000", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-06-27T21:11:21.258 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_discard_disconnected_ops": "true", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_enable_op_tracker": "true", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_extblkdev_plugins": "vdo", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_info": "true", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown": "true", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown_timeout": "15", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-06-27T21:11:21.259 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_function_tracing": "false", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_grace": "20", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_interval": "6", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_peers": "10", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_size": "2000", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_stale": "600", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_max_size": "100000", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_min_size": "1000", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-06-27T21:11:21.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal_flush_on_shutdown": "true", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal_size": "5120", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_kill_backfill_at": "0", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_loop_before_reset_tphandle": "64", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_cache_size": "50", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_dedup": "true", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_message_max": "40", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_message_max_bytes": "10485760", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_share_max_epochs": "40", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_attr_name_len": "100", 2024-06-27T21:11:21.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_attr_size": "0", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_backfills": "1", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_markdown_count": "5", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_markdown_period": "600", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_name_len": "2048", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_namespace_len": "256", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_size": "134217728", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_omap_entries_per_request": "1024", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_blocked_by": "16", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_log_entries": "10000", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pgls": "1024", 2024-06-27T21:11:21.262 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_push_cost": "8388608", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_push_objects": "10", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_scrubs": "3", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_trimming_pgs": "2", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_write_op_reply_len": "64", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_write_size": "90", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-06-27T21:11:21.263 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_override_recovery_settings": "false", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_profile": "balanced", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-06-27T21:11:21.264 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_skip_benchmark": "false", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_base": "805306368", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_cache_min": "134217728", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target": "4294967296", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target_autotune": "false", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_min_pg_log_entries": "250", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_min_recovery_priority": "0", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_heartbeat_interval": "30", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_report_interval": "5", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_report_max_in_flight": "2", 2024-06-27T21:11:21.265 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_num_cache_shards": "32", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_num_op_tracker_shard": "32", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_auto_affinity": "true", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_node": "-1", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_prefer_iface": "true", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objecter_finishers": "1", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore": "bluestore", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_fuse": "false", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_ideal_list_max": "64", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_tracing": "false", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_complaint_time": "30.000000", 2024-06-27T21:11:21.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_duration": "600", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_size": "20", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_slow_op_size": "20", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_log_threshold": "5", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards": "0", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards_hdd": "5", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards_ssd": "8", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard": "0", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_pq_min_cost": "65536", 2024-06-27T21:11:21.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_queue": "mclock_scheduler", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_queue_cut_off": "high", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_thread_suicide_timeout": "150", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_thread_timeout": "15", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_open_classes_on_start": "true", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_os_flags": "0", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_peering_op_priority": "255", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_delete_cost": "1048576", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_delete_priority": "5", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_dups_tracked": "3000", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_trim_max": "10000", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_trim_min": "100", 2024-06-27T21:11:21.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_object_context_cache_count": "64", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_crimson": "false", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_crush_rule": "-1", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_ec_fast_read": "false", 2024-06-27T21:11:21.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_bulk": "false", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nodelete": "false", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flags": "0", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_min_size": "2", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pg_num": "32", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pgp_num": "0", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-06-27T21:11:21.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_read_ratio": "70", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_size": "3", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_type": "replicated", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_use_gmt_hitset": "true", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_push_per_object_cost": "1000", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_read_ec_check_for_errors": "false", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recover_clone_overlap": "true", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recover_clone_overlap_limit": "10", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_cost": "20971520", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_delay_start": "0.000000", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active": "0", 2024-06-27T21:11:21.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active_hdd": "3", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active_ssd": "10", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_chunk": "8388608", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_single_start": "1", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_op_priority": "3", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_op_warn_multiple": "16", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_priority": "5", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_retry_interval": "30.000000", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep": "0.000000", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-06-27T21:11:21.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_repair_during_recovery": "false", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_requested_scrub_priority": "120", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_rollback_to_cluster_snap": "", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_auto_repair": "false", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_begin_hour": "0", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_begin_week_day": "0", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_chunk_max": "25", 2024-06-27T21:11:21.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_chunk_min": "5", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_cost": "52428800", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_during_recovery": "false", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_end_hour": "0", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_end_week_day": "0", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_event_cost": "4096", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_invalid_stats": "true", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_load_threshold": "0.500000", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_max_interval": "604800.000000", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_max_preemptions": "5", 2024-06-27T21:11:21.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_min_interval": "86400.000000", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_priority": "5", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_sleep": "0.000000", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shutdown_pgref_assert": "false", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_skip_data_digest": "false", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_smart_report_timeout": "5", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_cost": "1048576", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_priority": "5", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep": "0.000000", 2024-06-27T21:11:21.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_stats_update_period_scrubbing": "15", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_target_transaction_size": "30", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-06-27T21:11:21.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_mode": "writeback", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tracing": "false", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_use_stale_snap": "false", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osdc_blkin_trace_all": "false", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_kill_at": "0", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_max_join_drift": "10", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_min": "500", 2024-06-27T21:11:21.277 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_min_wait": "0.050000", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_propose_interval": "1.000000", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_max": "500", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_max_multiplier": "20", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_min": "10", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_stash_full_interval": "25", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_trim_max": "500", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_trim_min": "250", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "perf": "true", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "pid_file": "", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "-", 2024-06-27T21:11:21.278 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrv": "", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "public_bind_addr": "-", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "public_network": "", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "public_network_interface": "", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_enabled": "false", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_session_max_number": "256", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_mon_op_timeout": "0", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_osd_op_timeout": "0", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_tracing": "false", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_atime_update_interval": "60", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_balance_parent_reads": "false", 2024-06-27T21:11:21.279 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_balance_snap_reads": "false", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blkin_trace_all": "false", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blocklist_expire_seconds": "0", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blocklist_on_break_lock": "true", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache": "true", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_block_writes_upfront": "false", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty": "25165824", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty_object": "0", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_policy": "writearound", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_size": "33554432", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_target_dirty": "16777216", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-06-27T21:11:21.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_clone_copy_on_read": "false", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_compression_hint": "none", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_concurrent_management_ops": "10", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_clone_format": "auto", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_data_pool": "", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_features": "61", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_format": "2", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_map_options": "", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_order": "22", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_pool": "rbd", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_stripe_count": "0", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_stripe_unit": "0", 2024-06-27T21:11:21.281 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_disable_zero_copy_writes": "true", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_discard_granularity_bytes": "65536", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_enable_alloc_hint": "true", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_io_scheduler": "simple", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_commit_age": "5.000000", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_interval": "0", 2024-06-27T21:11:21.282 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_order": "24", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_pool": "", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_splay_width": "4", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_localize_parent_reads": "false", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_localize_snap_reads": "false", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-06-27T21:11:21.283 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_type": "simple", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-06-27T21:11:21.284 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_autotune": "true", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_base": "805306368", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_target": "4294967296", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-06-27T21:11:21.285 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_delete_delay": "0", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_replay_delay": "0", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_to_trash_on_remove": "false", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mtime_update_interval": "60", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_non_blocking_aio": "true", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_op_thread_timeout": "60", 2024-06-27T21:11:21.286 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_op_threads": "1", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_parent_cache_enabled": "false", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_mode": "disabled", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_path": "/tmp", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_size": "1073741824", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_plugins": "", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_burst": "0", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_limit": "0", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_exclude_ops": "0", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_burst": "0", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_limit": "0", 2024-06-27T21:11:21.287 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_burst": "0", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_limit": "0", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_burst": "0", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_limit": "0", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_schedule_tick_min": "50", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_burst": "0", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_limit": "0", 2024-06-27T21:11:21.288 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_burst": "0", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_limit": "0", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_quiesce_notification_attempts": "10", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_read_from_replica_policy": "default", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_max_bytes": "524288", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_trigger_requests": "10", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_request_timed_out_seconds": "30", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_skip_partial_discard": "true", 2024-06-27T21:11:21.289 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_tracing": "false", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_validate_names": "true", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_validate_pool": "true", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "restapi_base_url": "", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "restapi_log_level": "", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_account_default_quota_max_size": "-1", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_acl_grants_max_num": "100", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_admin_entry": "admin", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_backend_store": "rados", 2024-06-27T21:11:21.290 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_barbican_url": "", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_beast_enable_async": "true", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_counters_cache": "false", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_index_max_aio": "128", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_quota_ttl": "600", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_enabled": "true", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_expiry_interval": "900", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_lru_size": "25000", 2024-06-27T21:11:21.291 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_config_store": "rados", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_content_length_compat": "false", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_copy_obj_progress": "true", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cors_rules_max_num": "100", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cross_domain_policy": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_default_encryption_key": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_addr": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_ca_path": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_client_cert": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_client_key": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-06-27T21:11:21.292 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_password": "", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_username": "", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_require_ssl": "true", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-06-27T21:11:21.293 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_suppress_logs": "true", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_addr": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_auth": "token", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_namespace": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_prefix": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-06-27T21:11:21.294 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_token_file": "", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_buffersize": "524288", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_low_speed_limit": "1024", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_low_speed_time": "300", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_tcp_keepalive": "0", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-06-27T21:11:21.295 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_fadvise": "4", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_fadvise": "4", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-06-27T21:11:21.296 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_changes_size": "1000", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_num_shards": "128", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_window": "30", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_notify_interval_msec": "0", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_sync_poll_interval": "20", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_sync_spawn_window": "20", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_data_log_backing": "fifo", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_region_info_oid": "default.region", 2024-06-27T21:11:21.297 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_defer_to_bucket_acls": "", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_res": "500.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-06-27T21:11:21.298 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dns_name": "", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dns_s3website_name": "", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding": "true", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_gc_threads": "true", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_lc_threads": "true", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_mdsearch": "true", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_ops_log": "false", 2024-06-27T21:11:21.299 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_quota_threads": "true", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_static_website": "false", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_usage_log": "false", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enforce_swift_acls": "true", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_exit_timeout_secs": "120", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_expose_bucket": "false", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_extended_http_attrs": "", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_filter": "none", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_frontends": "beast port=7480", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_concurrent_io": "10", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_deferred": "50", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-06-27T21:11:21.300 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_objs": "32", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_queue_size": "134213632", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_trim_chunk": "16", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_obj_min_wait": "7200", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_processor_max_time": "3600", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_processor_period": "3600", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_get_obj_window_size": "16777216", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_healthcheck_disabling_path": "", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ignore_get_invalid_range": "false", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_init_timeout": "300", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-06-27T21:11:21.301 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_connection_idle": "300", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_message_timeout": "5000", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_sleep_timeout": "10", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_domain": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_password": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_password_path": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_project": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_tenant": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_token": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_token_path": "", 2024-06-27T21:11:21.302 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_user": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_api_version": "2", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_domain": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_password": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_project": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_tenant": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_user": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_implicit_tenants": "false", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_service_token_enabled": "false", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_token_cache_size": "10000", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_url": "", 2024-06-27T21:11:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_verify_ssl": "true", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_debug_interval": "-1", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_lock_max_time": "90", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_objs": "32", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_rules": "1000", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_worker": "3", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_wp_worker": "3", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_thread_delay": "0", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_dnattr": "uid", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_searchfilter": "", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_uri": "", 2024-06-27T21:11:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lfuda_sync_frequency": "60", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_http_headers": "", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_nonexistent_bucket": "false", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_object_name_utc": "false", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attr_name_len": "0", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attr_size": "0", 2024-06-27T21:11:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attrs_num_in_req": "0", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_chunk_size": "4194304", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_concurrent_requests": "1024", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_dynamic_shards": "1999", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_listing_results": "1000", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_notify_retries": "10", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_objs_per_shard": "100000", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_put_param_size": "1048576", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_put_size": "5368709120", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_slo_entries": "1000", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_md_log_max_shards": "64", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_md_notify_interval_msec": "200", 2024-06-27T21:11:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_meta_sync_poll_interval": "20", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_meta_sync_spawn_window": "20", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_mp_lock_max_time": "600", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multipart_min_part_size": "5242880", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_fhcache_size": "2017", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_lru_lanes": "5", 2024-06-27T21:11:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_max_gc": "300", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_gc_threads": "false", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_lc_threads": "false", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_quota_threads": "false", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_sync_thread": "false", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_num_async_rados_threads": "32", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_num_control_oids": "8", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_numa_node": "-1", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_obj_stripe_size": "4194304", 2024-06-27T21:11:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_chunk_size": "100", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_gc_interval": "600", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_hints_num_shards": "127", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_thread_timeout": "600", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_tracing": "false", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_token": "", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_url": "", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_verify_ssl": "true", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-06-27T21:11:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_rados": "false", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_socket_path": "", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_push_interval": "2.000000", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_push_interval_max": "30.000000", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_root_pool": ".rgw.root", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_lanes": "3", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-06-27T21:11:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_max_buckets": "100", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_partitions": "3", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_print_continue": "true", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_print_prohibited_content_length": "false", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_tracing": "false", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm": "", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm_id": "", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-06-27T21:11:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_region": "", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_region_root_pool": ".rgw.root", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_relaxed_region_enforcement": "false", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_request_uri": "", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_batch_size": "64", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_debug_interval": "-1", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_max_aio": "128", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_num_logs": "16", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_thread_interval": "600", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_resolve_cname": "false", 2024-06-27T21:11:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rest_getusage_op_compat": "false", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_run_sync_thread": "true", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_keystone": "false", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_ldap": "false", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_rados": "true", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_sts": "false", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_success_create_obj_status": "0", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_scheduler_type": "throttler", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_script_uri": "", 2024-06-27T21:11:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_service_provider_name": "", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_client_id": "", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_client_secret": "", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_entry": "sts", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_key": "sts", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_max_session_duration": "43200", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_min_session_duration": "900", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_token_introspection_url": "", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_account_in_url": "false", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_auth_entry": "auth", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_auth_url": "", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_custom_header": "", 2024-06-27T21:11:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_enforce_content_length": "false", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_need_stats": "true", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_tenant_name": "", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_token_expiration": "86400", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_url": "", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_url_prefix": "swift", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_versioning_enabled": "false", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_lease_period": "120", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_interval": "1200", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-06-27T21:11:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_obj_etag_verify": "false", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_history_size": "4096", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_thread_pool_size": "512", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_max_retries": "0", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_require_publish_policy": "false", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_comment": "", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_createby": "", 2024-06-27T21:11:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_encoding": "", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_flag": "false", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_max_size": "5368709120", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_origin": "", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_sha_unit": "524288", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_tracker": "", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_trust_forwarded_https": "false", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_log_tick_interval": "30", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_max_shards": "32", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_max_user_shards": "1", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_use_opa_authz": "false", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_counters_cache": "false", 2024-06-27T21:11:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_counters_cache_size": "10000", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_default_quota_max_size": "-1", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_max_buckets": "1000", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_policies_max_num": "100", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_interval": "86400", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_unique_email": "true", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_verify_ssl": "true", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_website_routing_rules_max_num": "50", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone": "", 2024-06-27T21:11:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone_id": "", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup": "", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup_id": "", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rgwlc_auto_session_clear": "true", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rgwlc_skip_bucket_step": "false", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_block_size": "4096", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_shard_bits": "4", 2024-06-27T21:11:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_size": "536870912", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_type": "binned_lru", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_compaction_stats": "false", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_extended_stats": "false", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_memory_stats": "false", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_index_type": "binary_search", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_log_to_ceph_log": "true", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_metadata_block_size": "4096", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_partition_filters": "false", 2024-06-27T21:11:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_perf": "false", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "rotating_keys_renewal_timeout": "10", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "run_dir": "/var/run/ceph", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_block_create": "true", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_cache_lru_size": "67108864", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_cbjournal_size": "5368709120", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_data_delta_based_overwrite": "0", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_default_max_object_size": "16777216", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_device_size": "53687091200", 2024-06-27T21:11:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_full_integrity_check": "false", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_capacity": "16", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_iodepth_limit": "5", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_main_device_type": "SSD", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_max_concurrent_transactions": "8", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_max_data_allocation_size": "32768", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_segment_size": "67108864", 2024-06-27T21:11:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "set_keepcaps": "false", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "setgroup": "ceph", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "setuser": "ceph", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "setuser_match_path": "", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_misplaced_ratio": "0.050000", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "thp": "false", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "threadpool_default_timeout": "60", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "threadpool_empty_queue_max_wait": "2", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "throttler_perf_counter": "true", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "tmp_dir": "/tmp", 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX" 2024-06-27T21:11:21.323 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:11:21.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:20 smithi192 ceph-mon[32956]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:21.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:20 smithi190 ceph-mon[32247]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:22.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-06-27T21:11:22.465 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:22 smithi151 ceph-mon[28965]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:23.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:22 smithi190 ceph-mon[32247]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:23.285 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:22 smithi192 ceph-mon[32956]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:24.372 INFO:teuthology.orchestra.run.smithi151.stderr:2024-06-27T21:11:24.381+0000 7f2626ffd640 0 --2- 172.21.15.151:0/3027063967 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f26281011f0 0x7f26281982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-27T21:11:24.390 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a: {} 2024-06-27T21:11:24.391 INFO:teuthology.orchestra.run.smithi151.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-27T21:11:24.405 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c: {} 2024-06-27T21:11:24.405 INFO:teuthology.orchestra.run.smithi151.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-27T21:11:24.419 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b: {} 2024-06-27T21:11:24.420 INFO:teuthology.orchestra.run.smithi151.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-27T21:11:25.072 INFO:teuthology.run_tasks:Running task full_sequential... 2024-06-27T21:11:25.092 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:11:25.094 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:11:25.094 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-06-27T21:11:25.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-06-27T21:11:25.095 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-06-27T21:11:25.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:11:25.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:11:25.095 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-06-27T21:11:25.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-06-27T21:11:25.096 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-06-27T21:11:25.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:25.096 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-06-27T21:11:25.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:11:25.097 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-06-27T21:11:25.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:11:25.097 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-06-27T21:11:25.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:11:25.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:25.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:25.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:25.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:25.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:25.416 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:25.416 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:25.419 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:25.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:25.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:25.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:25 smithi151 ceph-mon[28965]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:25.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3027063967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:11:25.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:25.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:25.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:25 smithi190 ceph-mon[32247]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3027063967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:11:25.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:25.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:25 smithi192 ceph-mon[32956]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:25.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3027063967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:11:25.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:25.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:25.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:25.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:26.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:26.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:26.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:26.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:26.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:26.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:26.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:26.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:26.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:26.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:26.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:26.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:26.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:26.888 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:26.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:26.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:26.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:27.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:27.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:27.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:27.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:27.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:27.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:27.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:27.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:27.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:27.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:27.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:27.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:27.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:27 smithi151 ceph-mon[28965]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:27.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:27 smithi190 ceph-mon[32247]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:27.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:27 smithi192 ceph-mon[32956]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:27.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:27.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:27.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:27.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:28.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:28.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:28.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:28.172 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:28.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:28.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:28.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:28.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:28.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:28.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:28.573 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:11:28.573 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:11:28.573 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:28 smithi151 ceph-mon[28965]: osdmap e103: 12 total, 12 up, 12 in 2024-06-27T21:11:28.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:28.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:11:28.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:11:28.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:28 smithi190 ceph-mon[32247]: osdmap e103: 12 total, 12 up, 12 in 2024-06-27T21:11:28.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:11:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:11:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:28 smithi192 ceph-mon[32956]: osdmap e103: 12 total, 12 up, 12 in 2024-06-27T21:11:28.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:28.896 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:28.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:28.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:29.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:29.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:29.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:29.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:29.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:29.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:29.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:29 smithi151 ceph-mon[28965]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:29.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:29.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:29.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:29.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:29.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:29 smithi190 ceph-mon[32247]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:29.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:29 smithi192 ceph-mon[32956]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:29.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:29.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:29.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:29.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:29.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:30.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:30.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:30.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:30.317 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_0' created 2024-06-27T21:11:30.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:30.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:30.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:30.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/852513008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-27T21:11:30.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:30.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:30.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:30.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/852513008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-27T21:11:30.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/852513008' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-06-27T21:11:30.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:30.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:30.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:30.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:30.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:31.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:31.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-06-27T21:11:31.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:31.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:31.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:31.318 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:11:31.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:31 smithi151 ceph-mon[28965]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:31.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:31 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:11:31.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:31 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:11:31.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/852513008' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-27T21:11:31.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:31 smithi151 ceph-mon[28965]: osdmap e104: 12 total, 12 up, 12 in 2024-06-27T21:11:31.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2326962672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:11:31.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:31.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:31.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:31.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:31.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:31.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:31.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:31.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:31 smithi190 ceph-mon[32247]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:31.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:31 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:11:31.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:31 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:11:31.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/852513008' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-27T21:11:31.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:31 smithi190 ceph-mon[32247]: osdmap e104: 12 total, 12 up, 12 in 2024-06-27T21:11:31.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2326962672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:11:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:31 smithi192 ceph-mon[32956]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:31 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:11:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:31 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:11:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/852513008' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-06-27T21:11:31.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:31 smithi192 ceph-mon[32956]: osdmap e104: 12 total, 12 up, 12 in 2024-06-27T21:11:31.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2326962672' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:11:31.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:31.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:31.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:31.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:31.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:32.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:32.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:32.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:32.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:32.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:32.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:32.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:32.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:32 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2326962672' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:11:32.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:32 smithi151 ceph-mon[28965]: osdmap e105: 12 total, 12 up, 12 in 2024-06-27T21:11:32.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:32.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:32.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:32 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:32.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2326962672' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:11:32.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:32 smithi190 ceph-mon[32247]: osdmap e105: 12 total, 12 up, 12 in 2024-06-27T21:11:32.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:32 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:32.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2326962672' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:11:32.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:32 smithi192 ceph-mon[32956]: osdmap e105: 12 total, 12 up, 12 in 2024-06-27T21:11:32.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:32.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:32.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:32.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:32.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:33.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:33.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:33.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:33.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:33.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:33.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:33.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:11:33.469 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:11:33.469 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:33 smithi151 ceph-mon[28965]: pgmap v328: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:33.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:33 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:11:33.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:33 smithi151 ceph-mon[28965]: osdmap e106: 12 total, 12 up, 12 in 2024-06-27T21:11:33.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:33.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:11:33.491 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:11:33.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:33.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:11:33.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:11:33.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:11:33.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:11:33.555 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:11:33.555 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:11:33.577 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:11:33.577 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:11:33.598 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:11:33.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:11:33.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:33.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:11:33.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:11:33.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:33.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:11:33.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:11:33.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:11:33.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:11:33.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:11:33.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:11:33.710 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:11:33.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:11:33.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:33.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:33.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:33 smithi190 ceph-mon[32247]: pgmap v328: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:33.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:33 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:11:33.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:33 smithi190 ceph-mon[32247]: osdmap e106: 12 total, 12 up, 12 in 2024-06-27T21:11:33.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:33 smithi192 ceph-mon[32956]: pgmap v328: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:33.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:33 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:11:33.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:33 smithi192 ceph-mon[32956]: osdmap e106: 12 total, 12 up, 12 in 2024-06-27T21:11:33.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:33.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:33.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:33.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:33.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:34.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:34.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:34.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:34.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:34.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:34.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:34.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:34.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:11:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:34.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:34.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2613537518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:34.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:34.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:34.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2613537518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:34.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:34.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2613537518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:34.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:34.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:34.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:34.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:35.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:35.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:35.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:35.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:35.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:35.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:35.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:35 smithi151 ceph-mon[28965]: pgmap v330: 17 pgs: 6 unknown, 11 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:35.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:35.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-06-27T21:11:35.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:35.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:35.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:35.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:35 smithi190 ceph-mon[32247]: pgmap v330: 17 pgs: 6 unknown, 11 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:35.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:35.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:35.776 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:35.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:35 smithi192 ceph-mon[32956]: pgmap v330: 17 pgs: 6 unknown, 11 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:35.796 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:11:35.796 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:11:35.809 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:35.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:35.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:35.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:35.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:36.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:36.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:36.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:36.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:36.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:36.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:36.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:36.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:36.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:36.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:11:36.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:36.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:36.662 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:36.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:36.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:36.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:36.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:36.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:37.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:37.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:37.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:11:37.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:37.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:37.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:37.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:37.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:37 smithi151 ceph-mon[28965]: pgmap v331: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:37.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:37.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:37.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:37.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:37.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:37.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:37 smithi190 ceph-mon[32247]: pgmap v331: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:37.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:37 smithi192 ceph-mon[32956]: pgmap v331: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:37.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:37.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:37.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:37.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:37.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:38.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:38.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:38.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:38.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:38.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:38.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:38.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:38.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:38.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:38.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:38.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:38.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:38.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:38.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:38.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:38.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:39.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:39.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:39.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:39.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:39.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:39.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:39.349 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:39.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:39.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:39.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:39.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:39.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:39 smithi151 ceph-mon[28965]: pgmap v332: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:39.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:39.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:39.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:39 smithi190 ceph-mon[32247]: pgmap v332: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:39.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:39 smithi192 ceph-mon[32956]: pgmap v332: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:39.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:39.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:39.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:39.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:39.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:40.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:40.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:40.197 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:40.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:40.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:40.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:40.355 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-06-27T21:11:40.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2586176644' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:11:40.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:40.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:40.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:40.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:40.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:40.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:40.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2586176644' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:11:40.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2586176644' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:11:40.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:40.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:40.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:40.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:40.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:41.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:41.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:41.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:41.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:41.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:41.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:41.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:41.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:41.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:41.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:41.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:41 smithi151 ceph-mon[28965]: pgmap v333: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:41.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:11:41.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:11:41.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2586176644' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:11:41.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:41 smithi151 ceph-mon[28965]: osdmap e107: 12 total, 12 up, 12 in 2024-06-27T21:11:41.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:41 smithi190 ceph-mon[32247]: pgmap v333: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:11:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:11:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2586176644' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:11:41.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:41 smithi190 ceph-mon[32247]: osdmap e107: 12 total, 12 up, 12 in 2024-06-27T21:11:41.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:41 smithi192 ceph-mon[32956]: pgmap v333: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:11:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:11:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2586176644' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:11:41.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:41 smithi192 ceph-mon[32956]: osdmap e107: 12 total, 12 up, 12 in 2024-06-27T21:11:41.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:41.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:41.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:41.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-06-27T21:11:41.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:42.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:42.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:42.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:42.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:42.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:42.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:42.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:42.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:42.516 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:42.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:42.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:42 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:11:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/343675873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:11:42.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:42.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:42 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:11:42.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/343675873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:11:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:42.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:42.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:42 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:11:42.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/343675873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:11:42.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:42.895 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:42.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:42.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:43.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:43.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:43.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:43.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:43.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:43.359 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:11:43.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:43.400 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:43.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:43.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:43.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:43 smithi151 ceph-mon[28965]: pgmap v335: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:43.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:11:43.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:11:43.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:43 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:43.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/343675873' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:11:43.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:43 smithi151 ceph-mon[28965]: osdmap e108: 12 total, 12 up, 12 in 2024-06-27T21:11:43.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:43.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:43.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:43 smithi190 ceph-mon[32247]: pgmap v335: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:11:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:11:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:43 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/343675873' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:11:43.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:43 smithi190 ceph-mon[32247]: osdmap e108: 12 total, 12 up, 12 in 2024-06-27T21:11:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:43.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:43 smithi192 ceph-mon[32956]: pgmap v335: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:43.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:11:43.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:11:43.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:43 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:43.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/343675873' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:11:43.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:43 smithi192 ceph-mon[32956]: osdmap e108: 12 total, 12 up, 12 in 2024-06-27T21:11:43.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:43.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:43.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:44.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:44.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:44.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:44.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:44.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.285 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:44.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:44.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:44.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:44.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:44.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:44.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:44.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:44.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:44.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:44.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:44.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:44.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:45.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:45.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:45.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:45.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:45.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:45.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:45.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:45.420 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:45.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:45.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:45.627 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:45.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:11:45.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:11:45.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:45 smithi151 ceph-mon[28965]: pgmap v337: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:45.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:45.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:11:45.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:11:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:45.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:45.680 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:11:45.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:11:45.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:11:45.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:11:45.724 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:11:45.724 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:11:45.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:11:45.747 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:11:45.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:45 smithi190 ceph-mon[32247]: pgmap v337: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:45.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:11:45.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:11:45.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:45.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:45 smithi192 ceph-mon[32956]: pgmap v337: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:45.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:11:45.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:11:45.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:45.815 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:11:45.815 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:11:45.838 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:11:45.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:11:45.874 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:11:45.875 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:11:45.897 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:11:45.897 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:11:45.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:45.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:45.923 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:11:45.924 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:11:45.924 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:11:45.924 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:11:45.924 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:11:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:45.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:46.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:46.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:46.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:46.159 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-06-27T21:11:46.162 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:11:46.162 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:11:46.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:46.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:11:46.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:11:46.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:11:46.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_59818 2024-06-27T21:11:46.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:11:46.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:11:46.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:46.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:46.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:46.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1118435678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:46.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:46.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:46.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:46.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:46.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:46.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1118435678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:46.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1118435678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:46.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:46.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:46.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:46.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:46.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:47.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:47.078 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:47.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:47.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:47.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 2109 2093 130.805 130.812 0.0229598 0.00749131 2024-06-27T21:11:47.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:11:47.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:47.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:47.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:47.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:47.477 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:47 smithi151 ceph-mon[28965]: pgmap v338: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:47.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:47.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:47.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:47.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:47.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:47 smithi190 ceph-mon[32247]: pgmap v338: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:47.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:47 smithi192 ceph-mon[32956]: pgmap v338: 17 pgs: 17 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:11:47.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:47.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:47.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:47.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:47.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:48.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:48.076 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:48.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:48.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:48.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:48.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:48.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 3638 3622 113.18 95.5625 0.00160073 0.00881016 2024-06-27T21:11:48.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:48.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:48.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:48.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:48.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:48.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:48.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:48.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:48.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:11:48.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:48.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:48.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:48.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:48.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:49.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:49.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:49.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:49.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:49.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 16 5389 5373 111.93 109.438 0.00243322 0.00892072 2024-06-27T21:11:49.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:49.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:49.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:49.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:49.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:49.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:49.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:49 smithi151 ceph-mon[28965]: pgmap v339: 17 pgs: 17 active+clean; 39 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s wr, 76 op/s 2024-06-27T21:11:49.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:49.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:49.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:49 smithi190 ceph-mon[32247]: pgmap v339: 17 pgs: 17 active+clean; 39 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s wr, 76 op/s 2024-06-27T21:11:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:49 smithi192 ceph-mon[32956]: pgmap v339: 17 pgs: 17 active+clean; 39 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s wr, 76 op/s 2024-06-27T21:11:49.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:49.864 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:49.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:49.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:49.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:49.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:50.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:50.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:50.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:50.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 7697 7681 120.007 144.25 0.00144902 0.00826457 2024-06-27T21:11:50.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:50.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:50.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:50.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:50.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:50.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-06-27T21:11:50.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:50.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:50.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:50.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:50.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:50.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:50.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:50.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:50.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:51.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:51.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:51.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:51.232 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 8733 8717 108.954 64.75 0.00436901 0.00916528 2024-06-27T21:11:51.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:51.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:51.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:51.390 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:51.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:51.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:51.605 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:51 smithi151 ceph-mon[28965]: pgmap v340: 17 pgs: 17 active+clean; 208 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 350 op/s 2024-06-27T21:11:51.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:51.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:51.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:51.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:51 smithi190 ceph-mon[32247]: pgmap v340: 17 pgs: 17 active+clean; 208 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 350 op/s 2024-06-27T21:11:51.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:51.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:51 smithi192 ceph-mon[32956]: pgmap v340: 17 pgs: 17 active+clean; 208 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 350 op/s 2024-06-27T21:11:51.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:51.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:51.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:51.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:52.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:52.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:52.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:52.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:52.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 10580 10564 110.033 115.438 0.0114603 0.00907839 2024-06-27T21:11:52.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:52.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:52.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:52.370 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:11:52.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:52.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:52.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:52.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:52.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:52 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:11:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:52 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:11:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:52 smithi151 ceph-mon[28965]: osdmap e109: 12 total, 12 up, 12 in 2024-06-27T21:11:52.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:52.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:52 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:11:52.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:52 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:11:52.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:52 smithi190 ceph-mon[32247]: osdmap e109: 12 total, 12 up, 12 in 2024-06-27T21:11:52.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:52.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:52 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:11:52.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:52 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:11:52.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:52 smithi192 ceph-mon[32956]: osdmap e109: 12 total, 12 up, 12 in 2024-06-27T21:11:52.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:52.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:52.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:52.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:53.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:53.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:53.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:53.176 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:11:53.176 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:11:53.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:11:53.176 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:2056 2.5 e108/104 2:a6ee9a82:::benchmark_data_smithi192_59818_object2055:head v 108'110, mlcod=108'107)", 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.185061+0000", 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.9981728780000001, 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050941536000000003, 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.185061+0000", 2024-06-27T21:11:53.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.185061+0000", 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.185062+0000", 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4300000000000002e-07 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.185136+0000", 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4281000000000003e-05 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.235580+0000", 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050443880000000003 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.235585+0000", 2024-06-27T21:11:53.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8799999999999999e-06 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.235629+0000", 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3699999999999998e-05 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.235648+0000", 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9701e-05 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.235960+0000", 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000311976 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.236003+0000", 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2375e-05 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:2497 2.5 e108/104 2:a88b1b05:::benchmark_data_smithi192_59818_object2496:head v 108'137, mlcod=108'133)", 2024-06-27T21:11:53.181 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.427439+0000", 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7557951259999998, 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13009257900000001, 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.427439+0000", 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.427439+0000", 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.427440+0000", 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4199999999999996e-07 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.427555+0000", 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011547799999999999 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.556991+0000", 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.129435472 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.556995+0000", 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7910000000000004e-06 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557039+0000", 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4085000000000002e-05 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557061+0000", 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1325999999999999e-05 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557486+0000", 2024-06-27T21:11:53.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000424795 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557532+0000", 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6090000000000001e-05 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:2528 2.8 e108/104 2:18cf28ea:::benchmark_data_smithi192_59818_object2527:head v 108'163, mlcod=108'159)", 2024-06-27T21:11:53.186 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.441563+0000", 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.7416715480000002, 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.115963999, 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.441563+0000", 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.441563+0000", 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.441563+0000", 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0299999999999998e-07 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.441634+0000", 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0784000000000005e-05 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.556936+0000", 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11530138199999999 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.556942+0000", 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8800000000000002e-06 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.556996+0000", 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3538000000000001e-05 2024-06-27T21:11:53.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557021+0000", 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5219999999999999e-05 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557491+0000", 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046967100000000001 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.557527+0000", 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5821000000000001e-05 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:2542 2.5 e108/104 2:afadc6c7:::benchmark_data_smithi192_59818_object2541:head v 108'138, mlcod=108'136)", 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.570010+0000", 2024-06-27T21:11:53.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.6132242489999999, 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.078421226999999996, 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.570010+0000", 2024-06-27T21:11:53.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.570010+0000", 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.570011+0000", 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7800000000000001e-07 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.570111+0000", 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100813 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.647936+0000", 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.077824716000000002 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.647943+0000", 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6649999999999997e-06 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.647995+0000", 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2293999999999998e-05 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.648009+0000", 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4401e-05 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.648392+0000", 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000382567 2024-06-27T21:11:53.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.648431+0000", 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9193000000000002e-05 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:2546 2.8 e108/104 2:13a0a00c:::benchmark_data_smithi192_59818_object2545:head v 108'165, mlcod=108'159)", 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.587649+0000", 2024-06-27T21:11:53.196 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.5955846950000003, 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.063479637000000005, 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.587649+0000", 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.587649+0000", 2024-06-27T21:11:53.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.587651+0000", 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3230000000000001e-06 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.587684+0000", 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3152999999999999e-05 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.650632+0000", 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062948303999999997 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.650637+0000", 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5870000000000003e-06 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.650683+0000", 2024-06-27T21:11:53.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5933999999999998e-05 2024-06-27T21:11:53.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.650706+0000", 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3061000000000002e-05 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.651100+0000", 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000393815 2024-06-27T21:11:53.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.651129+0000", 2024-06-27T21:11:53.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9459999999999999e-05 2024-06-27T21:11:53.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:3304 2.5 e108/104 2:aa6b3d1f:::benchmark_data_smithi192_59818_object3303:head v 108'194, mlcod=108'191)", 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.989302+0000", 2024-06-27T21:11:53.203 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.1939320020000004, 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042526118000000002, 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.989302+0000", 2024-06-27T21:11:53.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.989302+0000", 2024-06-27T21:11:53.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.989303+0000", 2024-06-27T21:11:53.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.009e-06 2024-06-27T21:11:53.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.989409+0000", 2024-06-27T21:11:53.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105577 2024-06-27T21:11:53.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.031397+0000", 2024-06-27T21:11:53.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041988206 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.031402+0000", 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7119999999999999e-06 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.031458+0000", 2024-06-27T21:11:53.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5818e-05 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.031470+0000", 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2746e-05 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.031804+0000", 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033412900000000001 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.031828+0000", 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3921e-05 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:3375 2.5 e108/104 2:af0c2d6d:::benchmark_data_smithi192_59818_object3374:head v 108'204, mlcod=108'201)", 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:48.092475+0000", 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.0907593120000003, 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.065194474000000002, 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.211 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.092475+0000", 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.092475+0000", 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.092476+0000", 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5400000000000001e-07 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.092577+0000", 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010178 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.157215+0000", 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.064637312000000002 2024-06-27T21:11:53.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.157224+0000", 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8570000000000007e-06 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.157275+0000", 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0272999999999998e-05 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.157289+0000", 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4538999999999999e-05 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.157626+0000", 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033700400000000001 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.157669+0000", 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3055e-05 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:3771 2.5 e108/104 2:ad2fb82c:::benchmark_data_smithi192_59818_object3770:head v 108'226, mlcod=108'223)", 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:48.313023+0000", 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8702111510000003, 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043743951000000003, 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.313023+0000", 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.313023+0000", 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.313024+0000", 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8800000000000002e-07 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.313116+0000", 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2342999999999993e-05 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.356345+0000", 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.043229525999999997 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.356351+0000", 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.215e-06 2024-06-27T21:11:53.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.356397+0000", 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6301999999999998e-05 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.356414+0000", 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7195000000000001e-05 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.356738+0000", 2024-06-27T21:11:53.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032424899999999998 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.356767+0000", 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8532999999999999e-05 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:4991 2.8 e108/104 2:10c14527:::benchmark_data_smithi192_59818_object4990:head v 108'294, mlcod=108'291)", 2024-06-27T21:11:53.220 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:48.981781+0000", 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.2014536790000001, 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.045018449000000002, 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.981781+0000", 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.981781+0000", 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.981781+0000", 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9599999999999999e-07 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.981872+0000", 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1153e-05 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.026274+0000", 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044401523999999998 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.026281+0000", 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1230000000000002e-06 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.026325+0000", 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3807999999999998e-05 2024-06-27T21:11:53.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.026344+0000", 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9215000000000002e-05 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.026769+0000", 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042507200000000003 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.026799+0000", 2024-06-27T21:11:53.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9958e-05 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:5317 2.8 e108/104 2:1b245e9c:::benchmark_data_smithi192_59818_object5316:head v 108'309, mlcod=108'306)", 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:49.169064+0000", 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.0141701689999998, 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062492086000000002, 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.169064+0000", 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.169064+0000", 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.169065+0000", 2024-06-27T21:11:53.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5499999999999998e-07 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.169139+0000", 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3961000000000006e-05 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.231121+0000", 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061982634000000002 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.231127+0000", 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2360000000000004e-06 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.231163+0000", 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6084000000000002e-05 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.231174+0000", 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1007e-05 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.231519+0000", 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034522399999999999 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.231556+0000", 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7285e-05 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:6416 2.c e108/104 2:34131a38:::benchmark_data_smithi192_59818_object6415:head v 108'398, mlcod=108'395)", 2024-06-27T21:11:53.229 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:49.690090+0000", 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.4931438030000002, 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.052166333000000002, 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.690090+0000", 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.690090+0000", 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.690091+0000", 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-07 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.690209+0000", 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118309 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.741702+0000", 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051492992000000001 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.741708+0000", 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.857e-06 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.741751+0000", 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2388999999999997e-05 2024-06-27T21:11:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.741772+0000", 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0903999999999998e-05 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.742220+0000", 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000448792 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.742257+0000", 2024-06-27T21:11:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6328999999999997e-05 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:6418 2.c e108/104 2:3cda2e42:::benchmark_data_smithi192_59818_object6417:head v 108'399, mlcod=108'395)", 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:49.691379+0000", 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.4918549030000001, 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.056126351999999997, 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.691379+0000", 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.691379+0000", 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.691380+0000", 2024-06-27T21:11:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4199999999999995e-07 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.691470+0000", 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9678000000000004e-05 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.746925+0000", 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.055455210999999997 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.746932+0000", 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8399999999999997e-06 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.746981+0000", 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9570999999999997e-05 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.746999+0000", 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7472000000000001e-05 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.747468+0000", 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046965100000000002 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:49.747506+0000", 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7286999999999997e-05 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:9048 2.8 e108/104 2:1e318175:::benchmark_data_smithi192_59818_object9047:head v 108'540, mlcod=108'537)", 2024-06-27T21:11:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:51.378050+0000", 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.8051840449999998, 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051710951999999998, 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.378050+0000", 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.378050+0000", 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.378051+0000", 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.09e-07 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.378117+0000", 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6309000000000005e-05 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 12252 12236 109.242 104.5 0.00204514 0.00914273 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.429231+0000", 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051114043999999997 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.429236+0000", 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1919999999999996e-06 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.429289+0000", 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2769999999999998e-05 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.429304+0000", 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4997999999999999e-05 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.429732+0000", 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042743700000000002 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.429761+0000", 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9393000000000001e-05 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:9275 2.5 e109/104 2:a4dd34a9:::benchmark_data_smithi192_59818_object9274:head v 109'522, mlcod=109'519)", 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:51.571650+0000", 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.6115842659999999, 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044967162999999997, 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.571650+0000", 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.571650+0000", 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.571650+0000", 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2600000000000002e-07 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.571757+0000", 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106838 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.615849+0000", 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044091409999999998 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.615855+0000", 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4849999999999997e-06 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.615910+0000", 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790999999999998e-05 2024-06-27T21:11:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.615937+0000", 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7500000000000001e-05 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.616591+0000", 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00065316699999999996 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.616617+0000", 2024-06-27T21:11:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6446000000000001e-05 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:9371 2.8 e109/104 2:1598370d:::benchmark_data_smithi192_59818_object9370:head v 109'560, mlcod=109'556)", 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:51.645077+0000", 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.538157566, 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.045588429999999999, 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.645077+0000", 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.645077+0000", 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.645077+0000", 2024-06-27T21:11:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3900000000000004e-07 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.645202+0000", 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124478 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.689699+0000", 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044496833 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.689710+0000", 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1569000000000001e-05 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.689760+0000", 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9545000000000001e-05 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.689776+0000", 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6266999999999999e-05 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.690633+0000", 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000857171 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.690665+0000", 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1928000000000003e-05 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:9539 2.5 e109/104 2:ad5cc767:::benchmark_data_smithi192_59818_object9538:head v 109'540, mlcod=109'537)", 2024-06-27T21:11:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:51.744790+0000", 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.438443726, 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.074043755000000003, 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.744790+0000", 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.744790+0000", 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.744791+0000", 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3200000000000004e-07 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.744895+0000", 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104283 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818299+0000", 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.073403520999999999 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818306+0000", 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8519999999999998e-06 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818345+0000", 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9632000000000003e-05 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818358+0000", 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2856e-05 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818808+0000", 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045005700000000002 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818834+0000", 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5822000000000001e-05 2024-06-27T21:11:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:9548 2.5 e109/104 2:acb5e00a:::benchmark_data_smithi192_59818_object9547:head v 109'541, mlcod=109'537)", 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:51.748960+0000", 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.4342737569999999, 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.098593754000000006, 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.748960+0000", 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.748960+0000", 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.748961+0000", 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-06-27T21:11:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.749079+0000", 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117657 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.847013+0000", 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.097933888999999996 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.847017+0000", 2024-06-27T21:11:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6260000000000003e-06 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.847058+0000", 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0522999999999998e-05 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.847073+0000", 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4748e-05 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.847526+0000", 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045350300000000001 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.847554+0000", 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8178999999999999e-05 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:9667 2.5 e109/104 2:abca0e31:::benchmark_data_smithi192_59818_object9666:head v 109'542, mlcod=109'538)", 2024-06-27T21:11:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:51.818188+0000", 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.365046099, 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050931944999999999, 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818188+0000", 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818188+0000", 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818189+0000", 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0299999999999998e-07 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.818272+0000", 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3028999999999994e-05 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.868446+0000", 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050173888999999999 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.868454+0000", 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7819999999999999e-06 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.868520+0000", 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5913000000000004e-05 2024-06-27T21:11:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.868544+0000", 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4045000000000001e-05 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.869051+0000", 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00050662000000000005 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:51.869120+0000", 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8963999999999999e-05 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:11664 2.5 e109/104 2:a40d06c4:::benchmark_data_smithi192_59818_object11663:head v 109'671, mlcod=109'668)", 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:52.883405+0000", 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.29982906199999998, 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.046161992999999998, 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.883405+0000", 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.883405+0000", 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.883406+0000", 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5700000000000002e-07 2024-06-27T21:11:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.883503+0000", 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7221999999999999e-05 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.928984+0000", 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.045481264 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.928989+0000", 2024-06-27T21:11:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4669999999999998e-06 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.929029+0000", 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0611999999999997e-05 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.929042+0000", 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3117e-05 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.929535+0000", 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049226499999999998 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.929567+0000", 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2388999999999998e-05 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:11672 2.5 e109/104 2:a3ae44cb:::benchmark_data_smithi192_59818_object11671:head v 109'672, mlcod=109'669)", 2024-06-27T21:11:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:52.889267+0000", 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.29396749700000002, 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.050252894999999999, 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.889267+0000", 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:11:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.889267+0000", 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.889267+0000", 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1799999999999995e-07 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.889379+0000", 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111313 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.938930+0000", 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049551216000000002 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.938937+0000", 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.464e-06 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.938988+0000", 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0807999999999999e-05 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.939010+0000", 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2002999999999999e-05 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.939494+0000", 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000483556 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:52.939520+0000", 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5916999999999999e-05 2024-06-27T21:11:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:11:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:11:53.276 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:11:53.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:53.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:53.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:53.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:53.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:53.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:53.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:53.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:53.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:53 smithi151 ceph-mon[28965]: pgmap v342: 17 pgs: 17 active+clean; 267 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 503 op/s 2024-06-27T21:11:53.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:53 smithi190 ceph-mon[32247]: pgmap v342: 17 pgs: 17 active+clean; 267 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 503 op/s 2024-06-27T21:11:53.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:53 smithi192 ceph-mon[32956]: pgmap v342: 17 pgs: 17 active+clean; 267 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 503 op/s 2024-06-27T21:11:53.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:53.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:53.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:53.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:53.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:54.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:54.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:54.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:54.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:54.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 13653 13637 106.531 87.5625 0.00309513 0.00936694 2024-06-27T21:11:54.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:54.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:54.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:54.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:54.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:54.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:54.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:54.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:54.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:54.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:54.953 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:54.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:54.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:11:55.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:55.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:55.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:55.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:55.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 15286 15270 106.034 102.062 0.0134007 0.00941434 2024-06-27T21:11:55.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:55.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:55.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:55.448 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:11:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:55.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:55.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:55.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:55.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:55 smithi151 ceph-mon[28965]: pgmap v343: 17 pgs: 17 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 954 op/s 2024-06-27T21:11:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2855953610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:11:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:55 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:11:55.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:55.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:55 smithi190 ceph-mon[32247]: pgmap v343: 17 pgs: 17 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 954 op/s 2024-06-27T21:11:55.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2855953610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:11:55.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:55 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:11:55.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:55 smithi192 ceph-mon[32956]: pgmap v343: 17 pgs: 17 active+clean; 597 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 954 op/s 2024-06-27T21:11:55.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2855953610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:11:55.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:55 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:11:55.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:55.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:55.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:55.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:55.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:56.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:56.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:56.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:56.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:56.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 16170 16154 100.955 55.25 0.00188497 0.00987034 2024-06-27T21:11:56.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:56.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:56.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:56.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:56.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:11:56.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:56.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:11:56.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:56.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:56.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:56 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:56.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:56 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:11:56.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:56 smithi151 ceph-mon[28965]: osdmap e110: 12 total, 12 up, 12 in 2024-06-27T21:11:56.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:56 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:56.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:56 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:11:56.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:56 smithi190 ceph-mon[32247]: osdmap e110: 12 total, 12 up, 12 in 2024-06-27T21:11:56.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:56 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:11:56.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:56 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:11:56.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:56 smithi192 ceph-mon[32956]: osdmap e110: 12 total, 12 up, 12 in 2024-06-27T21:11:56.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:56.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:56.903 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:11:56.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:11:56.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:11:56.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:11:56.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:56.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:11:56.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:11:56.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:56.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:57.005 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:11:57.005 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:11:57.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:11:57.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:11:57.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:11:57.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:11:57.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:11:57.092 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:11:57.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:11:57.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:11:57.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:57.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 17770 17754 100.867 100 0.00638663 0.00987582 2024-06-27T21:11:57.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:57.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:11:57.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:11:57.339 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:11:57.339 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:11:57.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:57.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:57.363 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:11:57.363 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:11:57.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:11:57.384 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:11:57.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:11:57.412 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:11:57.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:57.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:57.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:57.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:57 smithi151 ceph-mon[28965]: pgmap v345: 17 pgs: 17 active+clean; 784 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-27T21:11:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1215143491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:57.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:11:57.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:11:57.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:57 smithi190 ceph-mon[32247]: pgmap v345: 17 pgs: 17 active+clean; 784 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-27T21:11:57.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1215143491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:57.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:57 smithi192 ceph-mon[32956]: pgmap v345: 17 pgs: 17 active+clean; 784 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-27T21:11:57.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1215143491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:11:57.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:11:57.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:57.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:11:57.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:58.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:58.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:58.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:58.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:58.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 19019 19003 98.9657 78.0625 0.00498879 0.010094 2024-06-27T21:11:58.257 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:58.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:58.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:58.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:58.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:58.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:58.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:11:58.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:58.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:58.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:11:58.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:11:58.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:58.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:11:58.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:11:59.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:11:59.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:11:59.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:11:59.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 20378 20362 97.8862 84.9375 0.00903842 0.0102047 2024-06-27T21:11:59.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:59.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:59.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:11:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:59.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:11:59.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:11:59.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:11:59.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:11:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:11:59.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:11:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:59 smithi151 ceph-mon[28965]: pgmap v346: 17 pgs: 17 active+clean; 922 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:11:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:11:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:11:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:11:59.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:59 smithi190 ceph-mon[32247]: pgmap v346: 17 pgs: 17 active+clean; 922 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:11:59.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:11:59.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:11:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:11:59.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:11:59.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:11:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:11:59.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:59 smithi192 ceph-mon[32956]: pgmap v346: 17 pgs: 17 active+clean; 922 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:11:59.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:11:59.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:11:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:11:59.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:11:59.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:11:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:11:59.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:00.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:00.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:00.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:00.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:00.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 21880 21864 97.5992 93.875 0.0100074 0.010235 2024-06-27T21:12:00.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:00.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:00.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:00.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:00.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:00.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:00.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:00.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:00.711 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:00 smithi151 ceph-mon[28965]: osdmap e111: 12 total, 12 up, 12 in 2024-06-27T21:12:00.719 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:00.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:00 smithi190 ceph-mon[32247]: osdmap e111: 12 total, 12 up, 12 in 2024-06-27T21:12:00.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:00.770 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:00.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:00 smithi192 ceph-mon[32956]: osdmap e111: 12 total, 12 up, 12 in 2024-06-27T21:12:00.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:00.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:00.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:00.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:01.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:01.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:01.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:01.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 23123 23107 96.2711 77.6875 0.010499 0.0103576 2024-06-27T21:12:01.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:01.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:01.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:01.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:01.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:01.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:01.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:01.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:01 smithi151 ceph-mon[28965]: pgmap v348: 17 pgs: 17 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-27T21:12:01.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:01 smithi190 ceph-mon[32247]: pgmap v348: 17 pgs: 17 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-27T21:12:01.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:01 smithi192 ceph-mon[32956]: pgmap v348: 17 pgs: 17 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 112 MiB/s wr, 1.80k op/s 2024-06-27T21:12:01.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:01.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:01.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:01.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:01.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:02.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:02.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:02.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:02.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:02.233 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 24583 24567 95.9566 91.25 0.00241109 0.010413 2024-06-27T21:12:02.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:02.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:02.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:02.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:02.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:02.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:02.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:12:02.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:02.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:02.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:02.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:02.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:02.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:02.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:03.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:03.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:03.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:03.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:03.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 26194 26178 96.2345 100.688 0.0227429 0.0103839 2024-06-27T21:12:03.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:03.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:03.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:03.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:03.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:03.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:03 smithi151 ceph-mon[28965]: pgmap v349: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-27T21:12:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:03.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:03.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:03.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:03.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:03 smithi190 ceph-mon[32247]: pgmap v349: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-27T21:12:03.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:03.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:03.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:03 smithi192 ceph-mon[32956]: pgmap v349: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-06-27T21:12:03.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:03.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:03.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:03.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:03.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:03.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:03.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:04.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:04.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:04.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:04.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:04.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 16 27443 27427 95.2245 78.0625 0.0953441 0.0104346 2024-06-27T21:12:04.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-06-27T21:12:04.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:04.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:04.402 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:04.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:04.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:04.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:04 smithi151 ceph-mon[28965]: osdmap e112: 12 total, 12 up, 12 in 2024-06-27T21:12:04.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:04.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:04.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:04.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:04.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:04.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:04.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:04.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:04 smithi190 ceph-mon[32247]: osdmap e112: 12 total, 12 up, 12 in 2024-06-27T21:12:04.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:04.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:04.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:04 smithi192 ceph-mon[32956]: osdmap e112: 12 total, 12 up, 12 in 2024-06-27T21:12:04.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:04.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:04.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:04.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:04.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:04.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:04.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:05.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:05.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:05.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:05.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 28918 28902 95.0643 92.1875 0.00519952 0.0105119 2024-06-27T21:12:05.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:05.311 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:12:05.311 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:12:05.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:05.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:05.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:05.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:05.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:05.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:05.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:05 smithi151 ceph-mon[28965]: pgmap v351: 17 pgs: 17 active+clean; 1.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:12:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:05 smithi151 ceph-mon[28965]: osdmap e113: 12 total, 12 up, 12 in 2024-06-27T21:12:05.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:05.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:05.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:05 smithi190 ceph-mon[32247]: pgmap v351: 17 pgs: 17 active+clean; 1.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:12:05.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:05 smithi190 ceph-mon[32247]: osdmap e113: 12 total, 12 up, 12 in 2024-06-27T21:12:05.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:05 smithi192 ceph-mon[32956]: pgmap v351: 17 pgs: 17 active+clean; 1.5 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:12:05.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:05 smithi192 ceph-mon[32956]: osdmap e113: 12 total, 12 up, 12 in 2024-06-27T21:12:05.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:05.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:05.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:05.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:05.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:06.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:06.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:06.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:06.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:12:06.244532+0000 min lat: 0.00102028 max lat: 0.436856 avg lat: 0.0105975 2024-06-27T21:12:06.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:12:06.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 29619 29603 92.5015 43.8125 0.0564087 0.0105975 2024-06-27T21:12:06.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:06.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:06.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:06.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:06.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:06.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:06.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:06.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:06.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:06.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:06.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:06.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:06.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:07.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:07.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:07.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:07.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:07.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 16 29774 29758 88.5577 9.6875 0.00172232 0.0106439 2024-06-27T21:12:07.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:07.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:07.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:12:07.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.475 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:07.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:07.480 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:12:07.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.602 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:07.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:07 smithi151 ceph-mon[28965]: pgmap v353: 17 pgs: 17 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:12:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2563772899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:07.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:07.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:07.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:07 smithi190 ceph-mon[32247]: pgmap v353: 17 pgs: 17 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:12:07.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2563772899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:07.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:07 smithi192 ceph-mon[32956]: pgmap v353: 17 pgs: 17 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:12:07.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2563772899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:07.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:07.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:07.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:07.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:07.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:08.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:08.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:08.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:08.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:08.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 29774 29758 84.5322 0 - 0.0106439 2024-06-27T21:12:08.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:08.333 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:08.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:08.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:12:08.427 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:12:08.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:12:08.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:12:08.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:08.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:12:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:12:08.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:08.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:12:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:12:08.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:12:08.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:12:08.538 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:12:08.538 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:12:08.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:12:08.560 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:12:08.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:08.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:12:08.585 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:12:08.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:08.606 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:12:08.606 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:12:08.630 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:12:08.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:12:08.651 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:12:08.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:12:08.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:12:08.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:12:08.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:08.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:08 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:12:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:08 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:12:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2563772899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:08.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:08 smithi190 ceph-mon[32247]: osdmap e114: 12 total, 12 up, 12 in 2024-06-27T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/125962062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:08.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:08 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:12:08.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:08 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:12:08.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2563772899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:08.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:08 smithi192 ceph-mon[32956]: osdmap e114: 12 total, 12 up, 12 in 2024-06-27T21:12:08.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/125962062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:08.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:08.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:08 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:12:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:08 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:12:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2563772899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:08 smithi151 ceph-mon[28965]: osdmap e114: 12 total, 12 up, 12 in 2024-06-27T21:12:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/125962062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:08.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:08.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:08.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:09.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:09.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:09.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:09.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:09.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 29774 29758 80.857 0 - 0.0106439 2024-06-27T21:12:09.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:09.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:09.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:09.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:09.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:09.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:09.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: pgmap v355: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:12:09.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:09.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:09.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: osdmap e115: 12 total, 12 up, 12 in 2024-06-27T21:12:09.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: osdmap e116: 12 total, 12 up, 12 in 2024-06-27T21:12:09.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:09.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:09.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:09.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: pgmap v355: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:12:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:09.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: osdmap e115: 12 total, 12 up, 12 in 2024-06-27T21:12:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: osdmap e116: 12 total, 12 up, 12 in 2024-06-27T21:12:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: pgmap v355: 17 pgs: 17 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: osdmap e115: 12 total, 12 up, 12 in 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: osdmap e116: 12 total, 12 up, 12 in 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:09.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:09.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:09.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:09.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:09.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:09.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:09.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:10.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:10.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:10.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:10.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:10.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 16 29774 29758 77.4879 0 - 0.0106439 2024-06-27T21:12:10.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:12:10.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:10.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:10.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:10.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:10.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:10.593 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:10.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:10.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:10.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:12:10.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:10 smithi190 ceph-mon[32247]: osdmap e117: 12 total, 12 up, 12 in 2024-06-27T21:12:10.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:12:10.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:10 smithi192 ceph-mon[32956]: osdmap e117: 12 total, 12 up, 12 in 2024-06-27T21:12:10.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:12:10.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:10 smithi151 ceph-mon[28965]: osdmap e117: 12 total, 12 up, 12 in 2024-06-27T21:12:10.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:10.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:10.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:10.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:10.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:11.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:11.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:11.137 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:11.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:11.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:11.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 29774 29758 74.3883 0 - 0.0106439 2024-06-27T21:12:11.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:11.343 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:11.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:11.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:11.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:11.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:11.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:11.601 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:11 smithi151 ceph-mon[28965]: pgmap v358: 17 pgs: 17 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 772 op/s 2024-06-27T21:12:11.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-06-27T21:12:11.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:11.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:11.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:11 smithi190 ceph-mon[32247]: pgmap v358: 17 pgs: 17 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 772 op/s 2024-06-27T21:12:11.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:11 smithi192 ceph-mon[32956]: pgmap v358: 17 pgs: 17 active+clean; 1.8 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 772 op/s 2024-06-27T21:12:11.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:11.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:11.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:11.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:11.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:12.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:12.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:12.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:12.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:12.234 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 29774 29758 71.5271 0 - 0.0106439 2024-06-27T21:12:12.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:12.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:12.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:12.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:12.491 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:12.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:12.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:12.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:12.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:12.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:12:12.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:12:12.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:12:12.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:12:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:12:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:12:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:12:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:12:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:12:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:12:12.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:12:12.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:12:12.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:12:12.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:12:12.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:12:12.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:12:12.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:12:12.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:12:12.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:12:12.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:12:12.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:12.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:12.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:12.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:12:12.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:13.004 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:12:13.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:12:13.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:13.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:13.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:13.235 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 16 29774 29758 68.8778 0 - 0.0106439 2024-06-27T21:12:13.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:13.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:13.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:13.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:13.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:13.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:13.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:13.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:13.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:13.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:13.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:13.888 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:13.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:13.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:14.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:14.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:14.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:14.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:14.235 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 29774 29758 66.4178 0 - 0.0106439 2024-06-27T21:12:14.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.291 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:14.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:14.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:14.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:14.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:14.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:14.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:14.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:14.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:14.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:14.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:14.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:14.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:14.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:15.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:15.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:15.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:15.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:15.235 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 29774 29758 64.1274 0 - 0.0106439 2024-06-27T21:12:15.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:15.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:15.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:15.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:15.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:15.542 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:12:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:12:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:12:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:12:15.542 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:12:15.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:15.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:15.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:15.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:15 smithi151 ceph-mon[28965]: pgmap v361: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 286 op/s 2024-06-27T21:12:15.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:15.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:15.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:15 smithi151 ceph-mon[28965]: osdmap e118: 12 total, 12 up, 12 in 2024-06-27T21:12:15.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:15 smithi190 ceph-mon[32247]: pgmap v361: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 286 op/s 2024-06-27T21:12:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:15 smithi190 ceph-mon[32247]: osdmap e118: 12 total, 12 up, 12 in 2024-06-27T21:12:15.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:15 smithi192 ceph-mon[32956]: pgmap v361: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 286 op/s 2024-06-27T21:12:15.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:15.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:15 smithi192 ceph-mon[32956]: osdmap e118: 12 total, 12 up, 12 in 2024-06-27T21:12:15.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:15.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:15.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:15.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:15.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:16.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:16.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:16.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:16.235 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 29774 29758 61.9899 0 - 0.0106439 2024-06-27T21:12:16.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:16.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:16 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:16 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:16 smithi151 ceph-mon[28965]: osdmap e119: 12 total, 12 up, 12 in 2024-06-27T21:12:16.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:16.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:16.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:16.600 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:16.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:16.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:16 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:16 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:16 smithi190 ceph-mon[32247]: osdmap e119: 12 total, 12 up, 12 in 2024-06-27T21:12:16.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:16 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:16.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:16 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:16.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:16 smithi192 ceph-mon[32956]: osdmap e119: 12 total, 12 up, 12 in 2024-06-27T21:12:16.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:16.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:16.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:16.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:16.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:17.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:17.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:17.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:17.235 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 29774 29758 59.9901 0 - 0.0106439 2024-06-27T21:12:17.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:17.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:17.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:17.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:17 smithi151 ceph-mon[28965]: pgmap v364: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 21 op/s 2024-06-27T21:12:17.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:17.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:17.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:17.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:17.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:17.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:17.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:17 smithi190 ceph-mon[32247]: pgmap v364: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 21 op/s 2024-06-27T21:12:17.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:17 smithi192 ceph-mon[32956]: pgmap v364: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 21 op/s 2024-06-27T21:12:17.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:17.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:17.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:17.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:18.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:18.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:18.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:18.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:18.235 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 16 29774 29758 58.1154 0 - 0.0106439 2024-06-27T21:12:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:18.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:18.319 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:12:18.319 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:12:18.321 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:12:18.321 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:12:18.333 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:12:18.344 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:12:18.345 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:12:18.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:18.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:12:18.368 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:12:18.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:18.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:12:18.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:12:18.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:12:18.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:12:18.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:12:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:12:18.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:12:18.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:12:18.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:18.492 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:12:18.492 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:12:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:18.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:18.513 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:12:18.514 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:12:18.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:12:18.537 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:12:18.559 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:12:18.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:12:18.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/320479715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:12:18.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2514693030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:18.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:18.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:18.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:18.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/320479715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:12:18.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2514693030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:18.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:18.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/320479715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:12:18.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2514693030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:18.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:18.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:18.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:18.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:19.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:19.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:19.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:19.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 29774 29758 56.3543 0 - 0.0106439 2024-06-27T21:12:19.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:19.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:19.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:19.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:19.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:12:19.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:19.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:19.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:19.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:19 smithi151 ceph-mon[28965]: pgmap v365: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 1 op/s 2024-06-27T21:12:19.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:19 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:19.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/320479715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:12:19.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:19 smithi151 ceph-mon[28965]: osdmap e120: 12 total, 12 up, 12 in 2024-06-27T21:12:19.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:19 smithi151 ceph-mon[28965]: osdmap e121: 12 total, 12 up, 12 in 2024-06-27T21:12:19.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:19.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:19 smithi190 ceph-mon[32247]: pgmap v365: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 1 op/s 2024-06-27T21:12:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:19 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/320479715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:12:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:19 smithi190 ceph-mon[32247]: osdmap e120: 12 total, 12 up, 12 in 2024-06-27T21:12:19.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:19 smithi190 ceph-mon[32247]: osdmap e121: 12 total, 12 up, 12 in 2024-06-27T21:12:19.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:19.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:19 smithi192 ceph-mon[32956]: pgmap v365: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s wr, 1 op/s 2024-06-27T21:12:19.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:19 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:19.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/320479715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:12:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:19 smithi192 ceph-mon[32956]: osdmap e120: 12 total, 12 up, 12 in 2024-06-27T21:12:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:19 smithi192 ceph-mon[32956]: osdmap e121: 12 total, 12 up, 12 in 2024-06-27T21:12:19.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:19.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:19.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:20.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:20.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:20.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:20.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:20.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 29774 29758 54.6968 0 - 0.0106439 2024-06-27T21:12:20.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:20.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:20.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:20.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:20.420 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:20.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:20.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:20.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:20.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:20.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:20.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:20.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-06-27T21:12:20.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:20.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:20.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:20.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:12:21.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:21.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:21.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:21.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:21.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 16 30750 30734 54.8767 4.35714 0.0182355 0.0181829 2024-06-27T21:12:21.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:21.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:21.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:21.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:21 smithi151 ceph-mon[28965]: pgmap v368: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:12:21.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:21.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:21.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:21.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:21.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:21.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:21 smithi190 ceph-mon[32247]: pgmap v368: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:12:21.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:21 smithi192 ceph-mon[32956]: pgmap v368: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:12:21.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:21.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:21.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:12:21.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:21.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:21.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:22.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:22.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:22.156 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:22.169 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:22.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:22.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:22.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 32529 32513 56.4406 111.188 0.0161693 0.0177106 2024-06-27T21:12:22.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:22.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:22.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:22.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:22.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:22.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:22.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:22.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:22.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:22.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:22.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:22.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:22.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:23.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:23.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:23.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:23.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:23.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 33807 33791 57.0738 79.875 0.0440048 0.0175008 2024-06-27T21:12:23.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:23.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:23.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:23.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:23.521 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:12:23.521 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:12:23.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:23.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:23.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:23 smithi151 ceph-mon[28965]: pgmap v369: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:12:23.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:23.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:23.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:23 smithi190 ceph-mon[32247]: pgmap v369: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:12:23.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:23 smithi192 ceph-mon[32956]: pgmap v369: 17 pgs: 17 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:12:23.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:23.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:23.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:23.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:23.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:24.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:24.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:24.207 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:24.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:24.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 35323 35307 58.0651 94.75 0.0182537 0.0172146 2024-06-27T21:12:24.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:24.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:24.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:24.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:24.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:24.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:24.605 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:24.605 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:24.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:24.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:24.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:24.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:24.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:24.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:24.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:24.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:24.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:24.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:24.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:24.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:25.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:25.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:25.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:25.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 36458 36442 58.395 70.9375 0.0135844 0.0171168 2024-06-27T21:12:25.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:25.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:25.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:25.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:25.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:25.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:12:25.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:25.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:25.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:25 smithi151 ceph-mon[28965]: pgmap v370: 17 pgs: 17 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 279 op/s 2024-06-27T21:12:25.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:25 smithi151 ceph-mon[28965]: osdmap e122: 12 total, 12 up, 12 in 2024-06-27T21:12:25.686 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:25 smithi192 ceph-mon[32956]: pgmap v370: 17 pgs: 17 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 279 op/s 2024-06-27T21:12:25.686 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:25 smithi192 ceph-mon[32956]: osdmap e122: 12 total, 12 up, 12 in 2024-06-27T21:12:25.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:25.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:25 smithi190 ceph-mon[32247]: pgmap v370: 17 pgs: 17 active+clean; 2.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 279 op/s 2024-06-27T21:12:25.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:25 smithi190 ceph-mon[32247]: osdmap e122: 12 total, 12 up, 12 in 2024-06-27T21:12:25.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:25.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:25.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:25.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:25.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:26.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:26.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:26.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:26.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:12:26.246659+0000 min lat: 0.00102028 max lat: 15.0116 avg lat: 0.0170532 2024-06-27T21:12:26.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:12:26.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 37491 37475 58.5491 64.5625 0.0723433 0.0170532 2024-06-27T21:12:26.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:26.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:26.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:26.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:26.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:26.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:26.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:26.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:26.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:26.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:26.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:26.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:26.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:27.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:27.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:27.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:27.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:27.236 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 16 38627 38611 58.8526 71 0.00207708 0.0169688 2024-06-27T21:12:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:27.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:12:27.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:27.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:27 smithi151 ceph-mon[28965]: pgmap v372: 17 pgs: 17 active+clean; 2.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 593 op/s 2024-06-27T21:12:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1033245448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:27.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:27.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:27.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:27.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:27.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:27.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:27 smithi190 ceph-mon[32247]: pgmap v372: 17 pgs: 17 active+clean; 2.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 593 op/s 2024-06-27T21:12:27.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1033245448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:27.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:27 smithi192 ceph-mon[32956]: pgmap v372: 17 pgs: 17 active+clean; 2.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 593 op/s 2024-06-27T21:12:27.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1033245448' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:27.788 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:27.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:27.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:27.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:27.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:28.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:28.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:28.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:28.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:28.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 40123 40107 59.6773 93.5 0.015639 0.0167481 2024-06-27T21:12:28.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:28.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:28.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:28.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:28.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:28.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:28 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:28.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1033245448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:28.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:28 smithi151 ceph-mon[28965]: osdmap e123: 12 total, 12 up, 12 in 2024-06-27T21:12:28.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:28.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:28.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:28.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:28 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:28.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1033245448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:28.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:28 smithi190 ceph-mon[32247]: osdmap e123: 12 total, 12 up, 12 in 2024-06-27T21:12:28.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:28.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:28 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1033245448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:28.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:28 smithi192 ceph-mon[32956]: osdmap e123: 12 total, 12 up, 12 in 2024-06-27T21:12:28.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:28.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:28.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:28.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:29.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:29.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:29.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:29.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:29.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 41291 41275 59.987 73 0.0125699 0.0166596 2024-06-27T21:12:29.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:29.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:29.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:12:29.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:12:29.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:12:29.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:12:29.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:12:29.367 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:12:29.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:12:29.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:12:29.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:29.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:29.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:12:29.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:12:29.446 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:12:29.447 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:12:29.475 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:12:29.475 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:12:29.506 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:12:29.506 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:12:29.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:29.542 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:12:29.542 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:12:29.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:29.572 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:12:29.573 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:12:29.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:12:29.605 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:12:29.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:12:29.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:12:29.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:29 smithi151 ceph-mon[28965]: pgmap v374: 17 pgs: 17 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 798 op/s 2024-06-27T21:12:29.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4099620230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:29.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:29.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:29.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:29 smithi190 ceph-mon[32247]: pgmap v374: 17 pgs: 17 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 798 op/s 2024-06-27T21:12:29.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4099620230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:29.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:29.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-06-27T21:12:29.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:29 smithi192 ceph-mon[32956]: pgmap v374: 17 pgs: 17 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 798 op/s 2024-06-27T21:12:29.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4099620230' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:29.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:29.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:29.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:29.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:30.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:30.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:30.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:30.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:30.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 42605 42589 60.49 82.125 0.00202304 0.0165239 2024-06-27T21:12:30.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:30.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:30.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:30.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:30.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:30.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:30.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:12:30.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:30.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:30.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:30.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:30.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:30.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:30.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:31.000 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:31.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:31.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:31.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:31.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:31.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 43583 43567 60.504 61.125 0.0182699 0.016522 2024-06-27T21:12:31.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:31.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:31.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:31.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:31 smithi151 ceph-mon[28965]: pgmap v375: 17 pgs: 17 active+clean; 2.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:12:31.442 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:31.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:31.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:31.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:31.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:31 smithi190 ceph-mon[32247]: pgmap v375: 17 pgs: 17 active+clean; 2.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:12:31.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:31 smithi192 ceph-mon[32956]: pgmap v375: 17 pgs: 17 active+clean; 2.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:12:31.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:31.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:31.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:31.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:31.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:32.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:32.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:32.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:32.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:32.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 45125 45109 61.2837 96.375 0.0210795 0.0163117 2024-06-27T21:12:32.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:32.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:32.437 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:32.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:32.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:32.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:12:32.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:32.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:32.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:32.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:32.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:32.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:32.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:32.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:32.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:33.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:33.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:33.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:33.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:33.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 46448 46432 61.7389 82.6875 0.00268944 0.0161911 2024-06-27T21:12:33.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:33.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:33.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:33.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:33.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:33.616 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:33 smithi151 ceph-mon[28965]: pgmap v376: 17 pgs: 17 active+clean; 2.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24508.0:47048 2.1 2:8d158191:::benchmark_data_smithi192_59818_object47047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:33.617437+0000", 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0072305019999999998, 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072657099999999999, 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24508", 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47048 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617437+0000", 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617437+0000", 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617439+0000", 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.468e-06 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617477+0000", 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8130000000000003e-05 2024-06-27T21:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617491+0000", 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4324e-05 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617499+0000", 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5650000000000004e-06 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617592+0000", 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2911000000000005e-05 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617725+0000", 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013307999999999999 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.617876+0000", 2024-06-27T21:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015102900000000001 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.622449+0000", 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045727279999999999 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.622804+0000", 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035546500000000001 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24508.0:47050 2.1 2:8367801f:::benchmark_data_smithi192_59818_object47049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:33.618624+0000", 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.006044107, 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061607570000000002, 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24508", 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47050 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618624+0000", 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618624+0000", 2024-06-27T21:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618625+0000", 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5900000000000005e-07 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618654+0000", 2024-06-27T21:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9204e-05 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618667+0000", 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3269e-05 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618675+0000", 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9780000000000005e-06 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.619298+0000", 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000622892 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.619414+0000", 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011633099999999999 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.619529+0000", 2024-06-27T21:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114469 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.622455+0000", 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029256680000000002 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.622875+0000", 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041989600000000001 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24508.0:47051 2.1 2:8af51b1a:::benchmark_data_smithi192_59818_object47050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:33.618862+0000", 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0058058110000000001, 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059764329999999997, 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24508", 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47051 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618862+0000", 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618862+0000", 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618863+0000", 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7400000000000002e-07 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618909+0000", 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6394999999999999e-05 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618933+0000", 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4139e-05 2024-06-27T21:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.618941+0000", 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9349999999999994e-06 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.620792+0000", 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018506690000000001 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.620917+0000", 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012468599999999999 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.621068+0000", 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015110899999999999 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.622674+0000", 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001606564 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24508.0:47059 2.1 2:847bd1a1:::benchmark_data_smithi192_59818_object47058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:33.623022+0000", 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0016457500000000001, 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001887529, 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24508", 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 47059 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623022+0000", 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623022+0000", 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623024+0000", 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4249999999999999e-06 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623075+0000", 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1502999999999998e-05 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623094+0000", 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8465999999999999e-05 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623103+0000", 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0860000000000008e-06 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623152+0000", 2024-06-27T21:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9095999999999999e-05 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623282+0000", 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130271 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623424+0000", 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014159700000000001 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24508.0:47048 2.1 e123/104)", 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:33.623135+0000", 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0015327100000000001, 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00183647, 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623135+0000", 2024-06-27T21:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623135+0000", 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623136+0000", 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4000000000000001e-07 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623139+0000", 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4379999999999999e-06 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623172+0000", 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2317000000000002e-05 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623178+0000", 2024-06-27T21:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7479999999999996e-06 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:47060 2.3 e123/104 2:cf09f968:::benchmark_data_smithi192_59818_object47059:head v 123'2909, mlcod=123'2907)", 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:33.623787+0000", 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00088059900000000003, 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012249030000000001, 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623787+0000", 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623787+0000", 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623788+0000", 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4799999999999997e-07 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.623833+0000", 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5182999999999998e-05 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.624042+0000", 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020860599999999999 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.624049+0000", 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3289999999999996e-06 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.624102+0000", 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2658999999999998e-05 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:33.624122+0000", 2024-06-27T21:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9803e-05 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 6 2024-06-27T21:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:12:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:33.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:33.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:33.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:33 smithi190 ceph-mon[32247]: pgmap v376: 17 pgs: 17 active+clean; 2.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:12:33.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:33.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:33 smithi192 ceph-mon[32956]: pgmap v376: 17 pgs: 17 active+clean; 2.5 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:12:33.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:33.916 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:33.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:34.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:34.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:34.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:34.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:34.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 48205 48189 62.7403 109.812 0.016263 0.0159316 2024-06-27T21:12:34.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:34.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:34.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:34.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:34.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:34.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:34.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:34.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:34.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:34.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:34.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:34.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:34.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:35.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:35.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:35.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:35.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:35.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 16 49127 49111 62.6358 57.625 0.00910037 0.0159589 2024-06-27T21:12:35.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:35.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:35.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:35.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:35.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:35.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:35.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:35.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:35.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:35 smithi151 ceph-mon[28965]: pgmap v377: 17 pgs: 17 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:12:35.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:35 smithi190 ceph-mon[32247]: pgmap v377: 17 pgs: 17 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:12:35.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:35 smithi192 ceph-mon[32956]: pgmap v377: 17 pgs: 17 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:12:35.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:35.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:35.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:35.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:35.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:36.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:36.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:36.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:36.199 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:36.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:12:36.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 50367 50351 62.9329 77.5 0.0134713 0.015884 2024-06-27T21:12:36.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:36.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:36.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:36.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:36.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:36.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:36.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:36.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:36.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:36.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:36.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:36.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:37.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:37.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:37.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:37.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:37.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 51651 51635 63.2724 80.25 0.00251515 0.0157939 2024-06-27T21:12:37.249 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:37.249 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:37.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:37.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:37.368 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:12:37.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:37.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:37.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:37.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:37.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:37 smithi151 ceph-mon[28965]: pgmap v378: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-27T21:12:37.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/747565200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:12:37.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:37.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:37 smithi190 ceph-mon[32247]: pgmap v378: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-27T21:12:37.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/747565200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:12:37.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:37 smithi192 ceph-mon[32956]: pgmap v378: 17 pgs: 17 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-06-27T21:12:37.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/747565200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:12:37.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:37.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:37.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:37.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:37.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:38.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:38.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:38.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:38.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:38.237 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 52692 52676 63.3067 65.0625 0.0075265 0.0157873 2024-06-27T21:12:38.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:38.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:38.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:38.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:38.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:38.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:38 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/747565200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:38 smithi151 ceph-mon[28965]: osdmap e124: 12 total, 12 up, 12 in 2024-06-27T21:12:38.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:38.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:38.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:38 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:38.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/747565200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:12:38.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:38 smithi190 ceph-mon[32247]: osdmap e124: 12 total, 12 up, 12 in 2024-06-27T21:12:38.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:38 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:12:38.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/747565200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:12:38.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:38 smithi192 ceph-mon[32956]: osdmap e124: 12 total, 12 up, 12 in 2024-06-27T21:12:38.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:38.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:38.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:38.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:39.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:39.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:39.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:39.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:39.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 54259 54243 63.9599 97.9375 0.0448801 0.015629 2024-06-27T21:12:39.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:39.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:39.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:39.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:39.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:39.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:39.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:39.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:39 smithi151 ceph-mon[28965]: pgmap v380: 17 pgs: 17 active+clean; 3.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:12:39.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:39.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:39 smithi190 ceph-mon[32247]: pgmap v380: 17 pgs: 17 active+clean; 3.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:12:39.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:39.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:39.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:39 smithi192 ceph-mon[32956]: pgmap v380: 17 pgs: 17 active+clean; 3.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:12:39.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:39.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:39.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:39.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:40.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:40.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:40.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:40.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 55386 55370 64.0798 70.4375 0.0112571 0.015599 2024-06-27T21:12:40.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:40.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:40.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:40.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:40.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:40.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:40.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:40.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:40.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:40.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:40.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:40.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:40.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:41.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:41.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:41.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:41.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:12:41.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:41.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 56751 56735 64.4657 85.3125 0.0828425 0.0155029 2024-06-27T21:12:41.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:41.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:41.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:41.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:41.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:41.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:41.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:41.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:41 smithi151 ceph-mon[28965]: pgmap v381: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:12:41.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-06-27T21:12:41.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:41 smithi190 ceph-mon[32247]: pgmap v381: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:12:41.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:41.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:41 smithi192 ceph-mon[32956]: pgmap v381: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:12:41.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:41.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:41.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:41.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:42.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:42.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:42.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:42.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:42.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 57747 57731 64.426 62.25 0.0165932 0.0155143 2024-06-27T21:12:42.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:42.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:42.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:42.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:42.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:42.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:42.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:42 smithi151 ceph-mon[28965]: 2.0 scrub starts 2024-06-27T21:12:42.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:42 smithi151 ceph-mon[28965]: 2.f scrub starts 2024-06-27T21:12:42.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:42.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:42.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:42 smithi190 ceph-mon[32247]: 2.0 scrub starts 2024-06-27T21:12:42.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:42 smithi190 ceph-mon[32247]: 2.f scrub starts 2024-06-27T21:12:42.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:42 smithi192 ceph-mon[32956]: 2.0 scrub starts 2024-06-27T21:12:42.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:42 smithi192 ceph-mon[32956]: 2.f scrub starts 2024-06-27T21:12:42.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:42.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:42.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:42.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:42.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:43.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:43.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:43.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:43.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:43.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 58845 58829 64.4996 68.625 0.00863967 0.015499 2024-06-27T21:12:43.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:43.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:43.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:43.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:43.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:43.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: 2.0 scrub ok 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: pgmap v382: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: 2.3 scrub starts 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: 2.9 scrub starts 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: 2.a scrub starts 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: 2.f scrub ok 2024-06-27T21:12:43.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:43 smithi151 ceph-mon[28965]: 2.a scrub ok 2024-06-27T21:12:43.658 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:43.659 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:43.695 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:12:43.695 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:12:43.695 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:12:43.695 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:12:43.695 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:12:43.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:43.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: 2.0 scrub ok 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: pgmap v382: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: 2.3 scrub starts 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: 2.9 scrub starts 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: 2.a scrub starts 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: 2.f scrub ok 2024-06-27T21:12:43.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:43 smithi190 ceph-mon[32247]: 2.a scrub ok 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: 2.0 scrub ok 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: pgmap v382: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: 2.3 scrub starts 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: 2.9 scrub starts 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: 2.a scrub starts 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: 2.f scrub ok 2024-06-27T21:12:43.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:43 smithi192 ceph-mon[32956]: 2.a scrub ok 2024-06-27T21:12:43.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:43.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:43.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:43.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:44.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:44.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:44.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:44.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:44.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 59961 59945 64.5901 69.75 0.00298148 0.0154757 2024-06-27T21:12:44.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:44.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:44.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:44.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:44.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:44.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:44.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:44.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:44.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:44 smithi151 ceph-mon[28965]: 2.3 scrub ok 2024-06-27T21:12:44.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:44 smithi151 ceph-mon[28965]: 2.4 scrub starts 2024-06-27T21:12:44.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:44 smithi151 ceph-mon[28965]: 2.d scrub starts 2024-06-27T21:12:44.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:44 smithi151 ceph-mon[28965]: 2.7 scrub starts 2024-06-27T21:12:44.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:44 smithi151 ceph-mon[28965]: 2.9 scrub ok 2024-06-27T21:12:44.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:44 smithi151 ceph-mon[28965]: 2.d scrub ok 2024-06-27T21:12:44.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:44 smithi190 ceph-mon[32247]: 2.3 scrub ok 2024-06-27T21:12:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:44 smithi190 ceph-mon[32247]: 2.4 scrub starts 2024-06-27T21:12:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:44 smithi190 ceph-mon[32247]: 2.d scrub starts 2024-06-27T21:12:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:44 smithi190 ceph-mon[32247]: 2.7 scrub starts 2024-06-27T21:12:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:44 smithi190 ceph-mon[32247]: 2.9 scrub ok 2024-06-27T21:12:44.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:44 smithi190 ceph-mon[32247]: 2.d scrub ok 2024-06-27T21:12:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:44.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:44.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:44 smithi192 ceph-mon[32956]: 2.3 scrub ok 2024-06-27T21:12:44.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:44 smithi192 ceph-mon[32956]: 2.4 scrub starts 2024-06-27T21:12:44.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:44 smithi192 ceph-mon[32956]: 2.d scrub starts 2024-06-27T21:12:44.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:44 smithi192 ceph-mon[32956]: 2.7 scrub starts 2024-06-27T21:12:44.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:44 smithi192 ceph-mon[32956]: 2.9 scrub ok 2024-06-27T21:12:44.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:44 smithi192 ceph-mon[32956]: 2.d scrub ok 2024-06-27T21:12:44.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:44.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:44.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:45.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:45.008 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:12:45.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:12:45.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:45.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:12:45.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:12:45.059 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:12:45.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:12:45.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:12:45.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:12:45.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:12:45.106 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:12:45.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:45.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:12:45.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:12:45.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:45.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:12:45.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:12:45.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:12:45.188 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:12:45.217 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:12:45.218 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:12:45.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 61145 61129 64.7495 74 0.0174427 0.0154376 2024-06-27T21:12:45.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:12:45.247 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:12:45.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:45.276 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:12:45.276 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:12:45.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:45.303 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:12:45.303 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:12:45.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:45.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:45.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.538 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:45.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:45.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:45.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:45 smithi151 ceph-mon[28965]: pgmap v383: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:12:45.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:45 smithi151 ceph-mon[28965]: 2.4 scrub ok 2024-06-27T21:12:45.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:45 smithi151 ceph-mon[28965]: 2.7 scrub ok 2024-06-27T21:12:45.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3708209468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:45.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:45.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:45 smithi190 ceph-mon[32247]: pgmap v383: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:12:45.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:45 smithi190 ceph-mon[32247]: 2.4 scrub ok 2024-06-27T21:12:45.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:45 smithi190 ceph-mon[32247]: 2.7 scrub ok 2024-06-27T21:12:45.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3708209468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:45.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:45.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:45 smithi192 ceph-mon[32956]: pgmap v383: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:12:45.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:45 smithi192 ceph-mon[32956]: 2.4 scrub ok 2024-06-27T21:12:45.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:45 smithi192 ceph-mon[32956]: 2.7 scrub ok 2024-06-27T21:12:45.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3708209468' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:45.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:45.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:45.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:45.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:46.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:46.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:46.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:46.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:46.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:12:46.248224+0000 min lat: 0.00102028 max lat: 15.0116 avg lat: 0.0153861 2024-06-27T21:12:46.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:12:46.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 62393 62377 64.9702 78 0.0276787 0.0153861 2024-06-27T21:12:46.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:46.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:46.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:46.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:46.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:46.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:46.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:46.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:46.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:46 smithi151 ceph-mon[28965]: 2.c scrub starts 2024-06-27T21:12:46.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:46 smithi151 ceph-mon[28965]: 2.1 scrub starts 2024-06-27T21:12:46.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:46 smithi151 ceph-mon[28965]: 2.c scrub ok 2024-06-27T21:12:46.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:46 smithi190 ceph-mon[32247]: 2.c scrub starts 2024-06-27T21:12:46.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:46 smithi190 ceph-mon[32247]: 2.1 scrub starts 2024-06-27T21:12:46.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:46 smithi190 ceph-mon[32247]: 2.c scrub ok 2024-06-27T21:12:46.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:46.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:46 smithi192 ceph-mon[32956]: 2.c scrub starts 2024-06-27T21:12:46.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:46 smithi192 ceph-mon[32956]: 2.1 scrub starts 2024-06-27T21:12:46.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:46 smithi192 ceph-mon[32956]: 2.c scrub ok 2024-06-27T21:12:46.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:46.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:46.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:46.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:47.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:47.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:47.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:47.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:47.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 15 63356 63341 64.8927 60.25 0.00296529 0.0154028 2024-06-27T21:12:47.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:47.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:47.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:47.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:47.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:47.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:47.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:12:47.660 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:47 smithi151 ceph-mon[28965]: pgmap v384: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s 2024-06-27T21:12:47.660 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:47 smithi151 ceph-mon[28965]: 2.1 scrub ok 2024-06-27T21:12:47.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:47.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:47.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:47 smithi190 ceph-mon[32247]: pgmap v384: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s 2024-06-27T21:12:47.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:47 smithi190 ceph-mon[32247]: 2.1 scrub ok 2024-06-27T21:12:47.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:47 smithi192 ceph-mon[32956]: pgmap v384: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.22k op/s 2024-06-27T21:12:47.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:47 smithi192 ceph-mon[32956]: 2.1 scrub ok 2024-06-27T21:12:47.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:47.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:47.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:47.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:47.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:48.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:48.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:48.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:48.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:48.238 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 64327 64311 64.8238 60.625 0.014521 0.0154204 2024-06-27T21:12:48.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:48.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:48.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:48.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:48.503 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:48.503 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:48.503 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:48 smithi151 ceph-mon[28965]: 2.2 scrub starts 2024-06-27T21:12:48.504 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:48 smithi151 ceph-mon[28965]: 2.e scrub starts 2024-06-27T21:12:48.504 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:48 smithi151 ceph-mon[28965]: osdmap e125: 12 total, 12 up, 12 in 2024-06-27T21:12:48.504 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:48 smithi151 ceph-mon[28965]: 2.e scrub ok 2024-06-27T21:12:48.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:48.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:48.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:48.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:48 smithi190 ceph-mon[32247]: 2.2 scrub starts 2024-06-27T21:12:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:48 smithi190 ceph-mon[32247]: 2.e scrub starts 2024-06-27T21:12:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:48 smithi190 ceph-mon[32247]: osdmap e125: 12 total, 12 up, 12 in 2024-06-27T21:12:48.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:48 smithi190 ceph-mon[32247]: 2.e scrub ok 2024-06-27T21:12:48.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:12:48.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:48.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:48.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:48 smithi192 ceph-mon[32956]: 2.2 scrub starts 2024-06-27T21:12:48.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:48 smithi192 ceph-mon[32956]: 2.e scrub starts 2024-06-27T21:12:48.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:48 smithi192 ceph-mon[32956]: osdmap e125: 12 total, 12 up, 12 in 2024-06-27T21:12:48.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:48 smithi192 ceph-mon[32956]: 2.e scrub ok 2024-06-27T21:12:48.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:48.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:48.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:48.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:48.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:49.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:49.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:49.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:49.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 65411 65395 64.8702 67.75 0.00317092 0.0154089 2024-06-27T21:12:49.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:49.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:49.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:49.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:49.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:49.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:49.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:49 smithi151 ceph-mon[28965]: pgmap v386: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:12:49.622 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:49 smithi151 ceph-mon[28965]: 2.2 scrub ok 2024-06-27T21:12:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:49.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:49.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:49 smithi190 ceph-mon[32247]: pgmap v386: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:12:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:49 smithi190 ceph-mon[32247]: 2.2 scrub ok 2024-06-27T21:12:49.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:49 smithi192 ceph-mon[32956]: pgmap v386: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:12:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:49 smithi192 ceph-mon[32956]: 2.2 scrub ok 2024-06-27T21:12:49.854 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:49.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:49.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:49.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:49.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:50.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:50.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:50.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:50.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 66575 66559 64.9932 72.75 0.00426853 0.0153796 2024-06-27T21:12:50.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:50.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:50.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-06-27T21:12:50.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:50.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:50.422 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:12:50.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:50.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:50.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:50.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:50.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/789132594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:50.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:50.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/789132594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:50.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:50.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:50.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/789132594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:50.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:12:50.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:50.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:50.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:50.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:51.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:51.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:51.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:51.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:51.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 67858 67842 65.2269 80.1875 0.0298335 0.0153231 2024-06-27T21:12:51.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:51.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:51.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:51.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:51.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:51.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:51.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:51.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:51.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: pgmap v387: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:51 smithi151 ceph-mon[28965]: osdmap e126: 12 total, 12 up, 12 in 2024-06-27T21:12:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: pgmap v387: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:12:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:12:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:12:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:12:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:12:51.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:51.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:51 smithi190 ceph-mon[32247]: osdmap e126: 12 total, 12 up, 12 in 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: pgmap v387: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:12:51.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:51 smithi192 ceph-mon[32956]: osdmap e126: 12 total, 12 up, 12 in 2024-06-27T21:12:51.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:51.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:51.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:51.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:51.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:52.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:52.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:52.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:52.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:52.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 69297 69281 65.6012 89.9375 0.00265141 0.015236 2024-06-27T21:12:52.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:52.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:52.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:52.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:52.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:52.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:52.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:52.636 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:52.660 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:52.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:52.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:52.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:52.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:52.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:52.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:52.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:53.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:53.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:53.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:53.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:53.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 15 70668 70653 65.9018 85.75 0.00282458 0.0151695 2024-06-27T21:12:53.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:53.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:53.420 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:12:53.421 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:12:53.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:53.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:53.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:12:53.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:53.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:53.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:53 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:53.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:53 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:53.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:53 smithi151 ceph-mon[28965]: pgmap v389: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 943 op/s 2024-06-27T21:12:53.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:53 smithi151 ceph-mon[28965]: osdmap e127: 12 total, 12 up, 12 in 2024-06-27T21:12:53.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:53.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:53 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:53.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:53 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:53.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:53 smithi190 ceph-mon[32247]: pgmap v389: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 943 op/s 2024-06-27T21:12:53.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:53 smithi190 ceph-mon[32247]: osdmap e127: 12 total, 12 up, 12 in 2024-06-27T21:12:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:53.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:53.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:53 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:53.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:53 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:53.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:53 smithi192 ceph-mon[32956]: pgmap v389: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 943 op/s 2024-06-27T21:12:53.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:53 smithi192 ceph-mon[32956]: osdmap e127: 12 total, 12 up, 12 in 2024-06-27T21:12:53.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:53.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:53.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:53.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:54.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:54.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:54.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:54.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 16 72060 72044 66.2111 86.9375 0.0501836 0.0150961 2024-06-27T21:12:54.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:54.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:54.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:54.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:54.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:54.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:54.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:54.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:54 smithi151 ceph-mon[28965]: 2.5 deep-scrub starts 2024-06-27T21:12:54.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:54 smithi190 ceph-mon[32247]: 2.5 deep-scrub starts 2024-06-27T21:12:54.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:54.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:54 smithi192 ceph-mon[32956]: 2.5 deep-scrub starts 2024-06-27T21:12:54.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:54.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:54.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:54.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:55.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:55.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:55.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:55.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 73069 73053 66.1654 63.0625 0.0965829 0.0150961 2024-06-27T21:12:55.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:55.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:55.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:55.425 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:55.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:55.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:55.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:55.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:55 smithi151 ceph-mon[28965]: pgmap v391: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:12:55.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:55 smithi151 ceph-mon[28965]: 2.8 scrub starts 2024-06-27T21:12:55.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:55.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:55 smithi190 ceph-mon[32247]: pgmap v391: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:12:55.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:55 smithi190 ceph-mon[32247]: 2.8 scrub starts 2024-06-27T21:12:55.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:55 smithi192 ceph-mon[32956]: pgmap v391: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:12:55.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:55 smithi192 ceph-mon[32956]: 2.8 scrub starts 2024-06-27T21:12:55.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:55.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:55.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:55.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:55.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:56.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:56.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:56.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:56.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:56.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 73927 73911 65.9862 53.625 0.00377303 0.0151491 2024-06-27T21:12:56.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:56.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:56.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:56.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:56.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:56.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:56.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:56 smithi151 ceph-mon[28965]: 2.6 scrub starts 2024-06-27T21:12:56.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:56 smithi151 ceph-mon[28965]: 2.8 scrub ok 2024-06-27T21:12:56.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:56.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:56 smithi190 ceph-mon[32247]: 2.6 scrub starts 2024-06-27T21:12:56.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:56 smithi190 ceph-mon[32247]: 2.8 scrub ok 2024-06-27T21:12:56.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:56 smithi192 ceph-mon[32956]: 2.6 scrub starts 2024-06-27T21:12:56.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:56 smithi192 ceph-mon[32956]: 2.8 scrub ok 2024-06-27T21:12:56.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:12:56.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:12:56.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:56.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:56.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:57.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:57.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:57.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:57.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:57.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 75073 75057 66.0655 71.625 0.00833131 0.0151304 2024-06-27T21:12:57.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:57.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:57.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:57.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:12:57.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:57.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:57.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:57.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:57.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:12:57.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:57 smithi151 ceph-mon[28965]: pgmap v392: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:12:57.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:57 smithi151 ceph-mon[28965]: 2.6 scrub ok 2024-06-27T21:12:57.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:57 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:12:57.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:57 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:12:57.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:12:57.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:57 smithi190 ceph-mon[32247]: pgmap v392: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:12:57.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:57 smithi190 ceph-mon[32247]: 2.6 scrub ok 2024-06-27T21:12:57.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:57 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:12:57.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:57 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:12:57.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:57 smithi192 ceph-mon[32956]: pgmap v392: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:12:57.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:57 smithi192 ceph-mon[32956]: 2.6 scrub ok 2024-06-27T21:12:57.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:57 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:12:57.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:57 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:12:57.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:12:57.836 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:12:57.909 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:12:57.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:57.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-06-27T21:12:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:57.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:57.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:58.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:58.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:58.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:58.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:58.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 75845 75829 65.818 48.25 0.013884 0.0151854 2024-06-27T21:12:58.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:12:58.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:12:58.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:58.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:58.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:12:58.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:12:58.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:58 smithi151 ceph-mon[28965]: osdmap e128: 12 total, 12 up, 12 in 2024-06-27T21:12:58.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:12:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:58.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:12:58.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:58 smithi190 ceph-mon[32247]: osdmap e128: 12 total, 12 up, 12 in 2024-06-27T21:12:58.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:58 smithi192 ceph-mon[32956]: osdmap e128: 12 total, 12 up, 12 in 2024-06-27T21:12:58.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:58.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:58.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:58.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:58.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:59.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:12:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:59.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:12:59.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:59.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:12:59.238 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:12:59.239 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 76729 76713 65.6732 55.25 0.0270246 0.0152217 2024-06-27T21:12:59.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:59.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:12:59.269 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:12:59.296 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:12:59.296 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:12:59.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:12:59.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:12:59.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:12:59.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:12:59.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:12:59.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:12:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:12:59.388 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:12:59.388 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:12:59.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:12:59.417 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:12:59.449 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:12:59.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:12:59.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:12:59.484 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:12:59.484 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:12:59.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:12:59.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:12:59.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:12:59.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:12:59.549 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:12:59.581 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:12:59.581 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:12:59.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:12:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:12:59.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:12:59.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:59 smithi190 ceph-mon[32247]: pgmap v394: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:12:59.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:12:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/379830917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:59.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:12:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.763 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:12:59.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:59 smithi192 ceph-mon[32956]: pgmap v394: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:12:59.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:12:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/379830917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:59.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:12:59.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:12:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:12:59.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:12:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:59 smithi151 ceph-mon[28965]: pgmap v394: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:12:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:12:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/379830917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:12:59.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:00.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:00.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:00.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:00.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 15 77811 77796 65.7003 67.6875 0.0199586 0.0152143 2024-06-27T21:13:00.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:00.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:00.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:00.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:00.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:00.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:00.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:00.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:00.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:00.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:00.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:00.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:00.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:01.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:01.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:01.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:01.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:01.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:01.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 78545 78529 65.4351 45.8125 0.0325675 0.0152739 2024-06-27T21:13:01.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:01.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:01.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:01.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:01.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:01.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:01.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:01.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:01.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:01 smithi190 ceph-mon[32247]: pgmap v395: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-27T21:13:01.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:01.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:01 smithi151 ceph-mon[28965]: pgmap v395: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-27T21:13:01.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:01 smithi192 ceph-mon[32956]: pgmap v395: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-27T21:13:01.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:01.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:01.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:01.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:02.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.044 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:02.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:02.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:02.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:02.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 79270 79254 65.1703 45.3125 0.00611322 0.0153397 2024-06-27T21:13:02.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:02.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:02.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:02.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:02.522 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:02.523 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:02.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:02.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:02.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:02.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:02.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:02.823 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:02.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:02.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:02.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:03.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:03.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:03.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:03.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:03.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 80167 80151 65.0519 56.0625 0.0034662 0.0153651 2024-06-27T21:13:03.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:03.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:03.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:03.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:03.472 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:13:03.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:03.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:03.623 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:03 smithi151 ceph-mon[28965]: pgmap v396: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:13:03.623 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:03 smithi151 ceph-mon[28965]: 2.b deep-scrub starts 2024-06-27T21:13:03.623 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2872690014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:03.623 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:03.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:03.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:03.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:03 smithi190 ceph-mon[32247]: pgmap v396: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:13:03.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:03 smithi190 ceph-mon[32247]: 2.b deep-scrub starts 2024-06-27T21:13:03.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2872690014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:03.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:03.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:03 smithi192 ceph-mon[32956]: pgmap v396: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:13:03.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:03 smithi192 ceph-mon[32956]: 2.b deep-scrub starts 2024-06-27T21:13:03.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2872690014' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:03.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:03.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:03.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:03.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:03.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:03.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:04.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:04.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:04.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:04.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:04.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 81125 81109 64.9855 59.875 0.0466384 0.0153798 2024-06-27T21:13:04.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:04.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:04.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:04.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:04.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:13:04.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:04.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:04.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:04.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:04.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:04 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:04.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:13:04.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:04 smithi190 ceph-mon[32247]: osdmap e129: 12 total, 12 up, 12 in 2024-06-27T21:13:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:04 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:13:04.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:04 smithi192 ceph-mon[32956]: osdmap e129: 12 total, 12 up, 12 in 2024-06-27T21:13:04.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:04 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:04.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:13:04.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:04 smithi151 ceph-mon[28965]: osdmap e129: 12 total, 12 up, 12 in 2024-06-27T21:13:04.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:04.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:04.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:04.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:04.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:05.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:05.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:05.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:05.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 82025 82009 64.8749 56.25 0.00397992 0.0154068 2024-06-27T21:13:05.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:05.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:05.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:05.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:05.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:05.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:05.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:05.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:05 smithi151 ceph-mon[28965]: pgmap v398: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s 2024-06-27T21:13:05.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:05.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:05 smithi190 ceph-mon[32247]: pgmap v398: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s 2024-06-27T21:13:05.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:05.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:05 smithi192 ceph-mon[32956]: pgmap v398: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s 2024-06-27T21:13:05.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:05.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:05.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:05.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:06.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:06.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:06.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:06.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:13:06.249804+0000 min lat: 0.00102028 max lat: 15.0116 avg lat: 0.0154333 2024-06-27T21:13:06.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:13:06.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 82919 82903 64.7623 55.875 0.00838132 0.0154333 2024-06-27T21:13:06.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:06.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:06.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:06.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:06.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:06.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:06.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:06.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:06.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:06.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:06.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:06.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:06.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:06.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:07.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:07.047 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:07.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:07.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:07.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:07.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 83887 83871 64.7096 60.5 0.0465589 0.0154461 2024-06-27T21:13:07.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:07.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:07.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:07.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:07.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:07.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:07.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:07.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:07.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:07 smithi190 ceph-mon[32247]: pgmap v399: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 890 op/s 2024-06-27T21:13:07.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:07 smithi192 ceph-mon[32956]: pgmap v399: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 890 op/s 2024-06-27T21:13:07.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:07.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:07.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:07.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:07 smithi151 ceph-mon[28965]: pgmap v399: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 890 op/s 2024-06-27T21:13:07.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:07.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:08.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:08.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:08.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:08.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:08.240 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 84747 84731 64.5759 53.75 0.0674874 0.0154793 2024-06-27T21:13:08.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:08.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:08.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:08.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:08.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:08.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:08.664 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:13:08.665 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:13:08.665 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:13:08.665 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:13:08.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:08.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:08.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:08.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:13:08.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:08.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:08.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:09.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:09.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:09.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:09.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:09.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 85727 85711 64.5358 61.25 0.00352116 0.0154895 2024-06-27T21:13:09.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:09.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:09.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:09.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:09.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:09.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-06-27T21:13:09.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:09.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:09.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:09 smithi151 ceph-mon[28965]: pgmap v400: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 899 op/s 2024-06-27T21:13:09.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:09 smithi151 ceph-mon[28965]: 2.5 deep-scrub ok 2024-06-27T21:13:09.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:09 smithi190 ceph-mon[32247]: pgmap v400: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 899 op/s 2024-06-27T21:13:09.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:09 smithi190 ceph-mon[32247]: 2.5 deep-scrub ok 2024-06-27T21:13:09.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:09 smithi192 ceph-mon[32956]: pgmap v400: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 899 op/s 2024-06-27T21:13:09.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:09 smithi192 ceph-mon[32956]: 2.5 deep-scrub ok 2024-06-27T21:13:09.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:09.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:09.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:09.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:10.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:10.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:10.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:10.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:10.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 86782 86766 64.5524 65.9375 0.003549 0.0154858 2024-06-27T21:13:10.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:10.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:10.308 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:13:10.308 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:13:10.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:13:10.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:13:10.333 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:13:10.333 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:13:10.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:13:10.363 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:13:10.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:13:10.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:13:10.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:10.423 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:13:10.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:13:10.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:10.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:13:10.453 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:13:10.486 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:13:10.487 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:13:10.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:13:10.514 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:13:10.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:10.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:13:10.544 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:13:10.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:10.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:13:10.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:13:10.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:13:10.604 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:13:10.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:10.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3946998567' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:10.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3946998567' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:10.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3946998567' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:10.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:10.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:10.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:10.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:11.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:11.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:11.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:11.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:11.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 87635 87619 64.4201 53.3125 0.0341795 0.0155106 2024-06-27T21:13:11.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:11.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:13:11.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:11.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:11.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:11.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:11.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:11.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:11.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:11 smithi151 ceph-mon[28965]: pgmap v401: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 889 op/s 2024-06-27T21:13:11.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:11 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:11.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:11 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:11.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:11.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:11 smithi190 ceph-mon[32247]: pgmap v401: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 889 op/s 2024-06-27T21:13:11.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:11 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:11.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:11 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:11 smithi192 ceph-mon[32956]: pgmap v401: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 889 op/s 2024-06-27T21:13:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:11 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:11.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:11 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:11.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:11.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:11.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:11.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:11.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:12.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:12.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:12.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:12.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:12.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 88425 88409 64.2451 49.375 0.0131295 0.0155596 2024-06-27T21:13:12.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:12.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:12.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:12.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:12.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:12.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:12.649 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:12.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:12.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:12.734 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:12.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:12 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:12.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:12 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:12.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:12 smithi190 ceph-mon[32247]: osdmap e130: 12 total, 12 up, 12 in 2024-06-27T21:13:12.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:13:12.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:12 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:12 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:12 smithi151 ceph-mon[28965]: osdmap e130: 12 total, 12 up, 12 in 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:12 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:12 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:12 smithi192 ceph-mon[32956]: osdmap e130: 12 total, 12 up, 12 in 2024-06-27T21:13:12.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:13:12.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:12.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:12.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:12.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:12.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:13.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:13.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:13.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:13.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 89320 89304 64.1496 55.9375 0.0115022 0.0155828 2024-06-27T21:13:13.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:13.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:13.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:13.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:13.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:13.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:13.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:13.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:13.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:13 smithi190 ceph-mon[32247]: pgmap v403: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 849 op/s 2024-06-27T21:13:13.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:13:13.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:13:13.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:13.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:13.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:13 smithi151 ceph-mon[28965]: pgmap v403: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 849 op/s 2024-06-27T21:13:13.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:13:13.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:13:13.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:13 smithi192 ceph-mon[32956]: pgmap v403: 17 pgs: 2 active+clean+scrubbing+deep, 15 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 849 op/s 2024-06-27T21:13:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:13:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:13:13.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:13.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:13.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:13.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:13.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:14.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:14.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:14.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:14.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 90235 90219 64.0704 57.1875 0.0040751 0.0156 2024-06-27T21:13:14.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:14.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:14.319 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:14.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:14.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:14.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:14.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:14.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:14.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:13:14.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:14.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:14.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:14.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:14.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:14.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:14.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:14.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:14.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:14.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:15.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:15.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:15.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:15.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:15.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 16 91055 91039 63.9263 51.25 0.0127616 0.015637 2024-06-27T21:13:15.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:15.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:15.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:15.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:15.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:15.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:15.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:15.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:15.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:15.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:15 smithi190 ceph-mon[32247]: pgmap v404: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s 2024-06-27T21:13:15.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:15 smithi190 ceph-mon[32247]: osdmap e131: 12 total, 12 up, 12 in 2024-06-27T21:13:15.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:15.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:15.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:15 smithi192 ceph-mon[32956]: pgmap v404: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s 2024-06-27T21:13:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:15 smithi192 ceph-mon[32956]: osdmap e131: 12 total, 12 up, 12 in 2024-06-27T21:13:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:15.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:15.825 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:15 smithi151 ceph-mon[28965]: pgmap v404: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s 2024-06-27T21:13:15.825 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:15 smithi151 ceph-mon[28965]: osdmap e131: 12 total, 12 up, 12 in 2024-06-27T21:13:15.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:15.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:15.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:15.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:15.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:16.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:16.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:16.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:16.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:16.241 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 11 92072 92061 63.9257 63.875 0.00610893 0.0156367 2024-06-27T21:13:16.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:16.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:16.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:16.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:16.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:16.524 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:13:16.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:16.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:16 smithi151 ceph-mon[28965]: osdmap e132: 12 total, 12 up, 12 in 2024-06-27T21:13:16.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/855834405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:16.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:16 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:16.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:16.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:16.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:16 smithi192 ceph-mon[32956]: osdmap e132: 12 total, 12 up, 12 in 2024-06-27T21:13:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/855834405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:16.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:16 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:16.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:16.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:16.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:16.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:17.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:16 smithi190 ceph-mon[32247]: osdmap e132: 12 total, 12 up, 12 in 2024-06-27T21:13:17.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/855834405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:17.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:16 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:17.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:17.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:17.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:17.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.7231 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 92072 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 63.4293 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 33.7722 2024-06-27T21:13:17.242 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 144.25 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 0 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1014 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 540.37 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 2308 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 0 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0156736 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.192298 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 15.0116 2024-06-27T21:13:17.243 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00102028 2024-06-27T21:13:17.254 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:13:17.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:17.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:17.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:17.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:17.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:17.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:17.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:17.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:17.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:17 smithi151 ceph-mon[28965]: pgmap v407: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 852 op/s 2024-06-27T21:13:17.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:17 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:17.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:17 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:17.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:17 smithi151 ceph-mon[28965]: osdmap e133: 12 total, 12 up, 12 in 2024-06-27T21:13:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:17 smithi192 ceph-mon[32956]: pgmap v407: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 852 op/s 2024-06-27T21:13:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:17 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:17 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:17 smithi192 ceph-mon[32956]: osdmap e133: 12 total, 12 up, 12 in 2024-06-27T21:13:17.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:17.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:17.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:17.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:17.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:18.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:17 smithi190 ceph-mon[32247]: pgmap v407: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 852 op/s 2024-06-27T21:13:18.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:17 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:18.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:17 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:18.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:17 smithi190 ceph-mon[32247]: osdmap e133: 12 total, 12 up, 12 in 2024-06-27T21:13:18.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:18.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:18.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:18.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:18.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:18.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:18.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:18.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:18.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:18.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:18.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:18 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:18.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:18 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:18.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:18 smithi151 ceph-mon[28965]: osdmap e134: 12 total, 12 up, 12 in 2024-06-27T21:13:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:18.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:18.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:18 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:18 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:18.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:18 smithi192 ceph-mon[32956]: osdmap e134: 12 total, 12 up, 12 in 2024-06-27T21:13:18.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:18.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.840 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:18.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:18.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:18.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:19.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:18 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:19.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:18 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:19.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:18 smithi190 ceph-mon[32247]: osdmap e134: 12 total, 12 up, 12 in 2024-06-27T21:13:19.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:19.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:19.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:19.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:19.231 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:19.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:19.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:19.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:19.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:19.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:19.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:19.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:19.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:19.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:19.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:19 smithi192 ceph-mon[32956]: pgmap v410: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 711 op/s 2024-06-27T21:13:19.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:19.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:19.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:19 smithi151 ceph-mon[28965]: pgmap v410: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 711 op/s 2024-06-27T21:13:19.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:19.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:19.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:20.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:19 smithi190 ceph-mon[32247]: pgmap v410: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 711 op/s 2024-06-27T21:13:20.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:20.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:20.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:20.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:20.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:20.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:20.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:20.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:20.533 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:13:20.533 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:13:20.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:20.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:20.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:20 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:20.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:20.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:20.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:20 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:20.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:20.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:13:20.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:20.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:20.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:20.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:21.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:20 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:21.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:21.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:21.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:21.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:21.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:21.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:21.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:21.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:21.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:21.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:21.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:21.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:21 smithi151 ceph-mon[28965]: pgmap v411: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 740 op/s 2024-06-27T21:13:21.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:21 smithi151 ceph-mon[28965]: osdmap e135: 12 total, 12 up, 12 in 2024-06-27T21:13:21.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:21.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:21.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:21.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:21.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:21.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:22.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:21 smithi190 ceph-mon[32247]: pgmap v411: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 740 op/s 2024-06-27T21:13:22.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:21 smithi190 ceph-mon[32247]: osdmap e135: 12 total, 12 up, 12 in 2024-06-27T21:13:22.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:13:22.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:22.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:21 smithi192 ceph-mon[32956]: pgmap v411: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 740 op/s 2024-06-27T21:13:22.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:21 smithi192 ceph-mon[32956]: osdmap e135: 12 total, 12 up, 12 in 2024-06-27T21:13:22.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:22.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:22.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:22.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:22.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:22.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:22.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:22.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:22.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:22.649 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:13:22.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:13:22.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:22.681 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:22.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:13:22.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:13:22.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:13:22.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:13:22.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:22.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:13:22.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:13:22.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:22.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:22.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:22.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:22.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:22.983 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:13:22.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:13:23.011 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:13:23.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:13:23.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:23.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:13:23.036 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:13:23.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:23.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:13:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:13:23.092 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:13:23.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:13:23.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:13:23.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:13:23.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:23.143 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:13:23.144 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:13:23.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:23.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:13:23.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:13:23.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:23.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:23.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:23.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:23.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:23.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:23.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:23.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:23 smithi151 ceph-mon[28965]: pgmap v413: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 26 MiB/s wr, 662 op/s 2024-06-27T21:13:23.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3027716794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:23.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:23.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:23.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:23.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:23.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:23.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:24.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:23 smithi190 ceph-mon[32247]: pgmap v413: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 26 MiB/s wr, 662 op/s 2024-06-27T21:13:24.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3027716794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:24.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:23 smithi192 ceph-mon[32956]: pgmap v413: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 26 MiB/s wr, 662 op/s 2024-06-27T21:13:24.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3027716794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:24.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:24.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:24.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:24.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:24.237 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:13:24.237 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:13:24.238 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-27T21:13:24.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:13:24.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:24.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:24.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:24.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:24.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:24.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:24.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:24.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:24.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:24.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:24 smithi151 ceph-mon[28965]: osdmap e136: 12 total, 12 up, 12 in 2024-06-27T21:13:24.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:24.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:24.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:24.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:24.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:24.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:24.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:24.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:24.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:25.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:25.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:25.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:24 smithi190 ceph-mon[32247]: osdmap e136: 12 total, 12 up, 12 in 2024-06-27T21:13:25.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:25.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:25.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:25.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:25.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:24 smithi192 ceph-mon[32956]: osdmap e136: 12 total, 12 up, 12 in 2024-06-27T21:13:25.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:25.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:25.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:25.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:25.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:25.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:25.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:25.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:25.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:25.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:25.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:25.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:25.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:25.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:25.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:25.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:25.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:25.938 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:25 smithi192 ceph-mon[32956]: pgmap v415: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 162 B/s rd, 13 MiB/s wr, 963 op/s 2024-06-27T21:13:25.938 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:25 smithi192 ceph-mon[32956]: osdmap e137: 12 total, 12 up, 12 in 2024-06-27T21:13:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:25 smithi151 ceph-mon[28965]: pgmap v415: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 162 B/s rd, 13 MiB/s wr, 963 op/s 2024-06-27T21:13:25.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:25 smithi151 ceph-mon[28965]: osdmap e137: 12 total, 12 up, 12 in 2024-06-27T21:13:25.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:25.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:26.005 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:25 smithi190 ceph-mon[32247]: pgmap v415: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 162 B/s rd, 13 MiB/s wr, 963 op/s 2024-06-27T21:13:26.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:25 smithi190 ceph-mon[32247]: osdmap e137: 12 total, 12 up, 12 in 2024-06-27T21:13:26.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:26.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:26.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:26.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:26.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:26.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:26.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:26.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:26.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:26.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:26.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:26.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:26.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:26.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:26.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:26.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:26.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:27.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:27.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:27.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:27.133 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:3287 2.d e108/104 2:bc4e09aa:::benchmark_data_smithi192_59818_object3286:head v 108'210, mlcod=108'207)", 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:47.977817+0000", 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 99.159869353000005, 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090792053999999997, 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.134 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.977817+0000", 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.977817+0000", 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.977818+0000", 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:47.977909+0000", 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1003999999999998e-05 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.068178+0000", 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090268800999999996 2024-06-27T21:13:27.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.068183+0000", 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6330000000000001e-06 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.068230+0000", 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6529999999999997e-05 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.068245+0000", 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4987999999999999e-05 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.068580+0000", 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033505600000000003 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:48.068609+0000", 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9552000000000001e-05 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:12928 2.6 e109/104 2:629b93d8:::benchmark_data_smithi192_59818_object12927:head v 109'805, mlcod=109'802)", 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:53.720632+0000", 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 93.417054164999996, 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.117089314, 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.139 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.720632+0000", 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.720632+0000", 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.720633+0000", 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6300000000000004e-07 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.720651+0000", 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7980999999999999e-05 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.836958+0000", 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11630710499999999 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.836967+0000", 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1789999999999997e-06 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.837023+0000", 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5472e-05 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.837044+0000", 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1228000000000001e-05 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.837684+0000", 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000639549 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:53.837722+0000", 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8037000000000002e-05 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.144 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:15316 2.1 e109/104 2:8d2a2e52:::benchmark_data_smithi192_59818_object15315:head v 109'1013, mlcod=109'1011)", 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:55.342809+0000", 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 91.794877940999996, 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.122061514, 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.342809+0000", 2024-06-27T21:13:27.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.342809+0000", 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.342809+0000", 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6899999999999997e-07 2024-06-27T21:13:27.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.342830+0000", 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0676999999999999e-05 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.463906+0000", 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.121075616 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.463913+0000", 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5320000000000001e-06 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.464057+0000", 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014391 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.464081+0000", 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4165999999999999e-05 2024-06-27T21:13:27.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.464822+0000", 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00074081500000000003 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:55.464870+0000", 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8128999999999999e-05 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:16003 2.d e110/104 2:b52b3994:::benchmark_data_smithi192_59818_object16002:head v 110'973, mlcod=110'968)", 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:56.123900+0000", 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 91.013786550999995, 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13125125700000001, 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.123900+0000", 2024-06-27T21:13:27.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.123900+0000", 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.123901+0000", 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999995e-07 2024-06-27T21:13:27.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.123977+0000", 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6771000000000001e-05 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.254454+0000", 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13047647500000001 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.254459+0000", 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6739999999999996e-06 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.254509+0000", 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0874000000000001e-05 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.254534+0000", 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4082000000000001e-05 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.255108+0000", 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057430199999999995 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.255151+0000", 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3467999999999998e-05 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:16237 2.9 e110/104 2:9eb599a7:::benchmark_data_smithi192_59818_object16236:head v 110'1055, mlcod=110'1052)", 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:56.319518+0000", 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.818169122, 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14686606999999999, 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.319518+0000", 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.319518+0000", 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.319518+0000", 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.319536+0000", 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8054999999999999e-05 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.465695+0000", 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14615850399999999 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.465699+0000", 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1679999999999997e-06 2024-06-27T21:13:27.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.465745+0000", 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5859000000000003e-05 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.465758+0000", 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3003000000000001e-05 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.466358+0000", 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060058699999999997 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.466384+0000", 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5232000000000002e-05 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:16238 2.d e110/104 2:ba869e7a:::benchmark_data_smithi192_59818_object16237:head v 110'980, mlcod=110'975)", 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:11:56.321268+0000", 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.816419073999995, 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13375520099999999, 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.321268+0000", 2024-06-27T21:13:27.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.321268+0000", 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.321268+0000", 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5700000000000002e-07 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.321327+0000", 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9157999999999998e-05 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.454274+0000", 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13294677799999999 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.454282+0000", 2024-06-27T21:13:27.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6009999999999997e-06 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.454333+0000", 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1147999999999999e-05 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.454352+0000", 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8970999999999998e-05 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.454989+0000", 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000637614 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:11:56.455023+0000", 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3374000000000003e-05 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:48528 2.d e123/104 2:b099c484:::benchmark_data_smithi192_59818_object48527:head v 123'3023, mlcod=123'3019)", 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:34.502712+0000", 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.634974483000001, 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.100332326, 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.502712+0000", 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.502712+0000", 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.502713+0000", 2024-06-27T21:13:27.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.502829+0000", 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011651500000000001 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601621+0000", 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.098791601000000007 2024-06-27T21:13:27.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601629+0000", 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3769999999999996e-06 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601688+0000", 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8452999999999998e-05 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601711+0000", 2024-06-27T21:13:27.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3195000000000001e-05 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.603006+0000", 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012946819999999999 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.603044+0000", 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8742999999999998e-05 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.170 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:48558 2.d e123/104 2:b968b99f:::benchmark_data_smithi192_59818_object48557:head v 123'3024, mlcod=123'3021)", 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:34.601519+0000", 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 52.536167691000003, 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13976770499999999, 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601519+0000", 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601519+0000", 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601521+0000", 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6139999999999999e-06 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.601601+0000", 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0129999999999993e-05 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.738847+0000", 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.137245853 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.738851+0000", 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8579999999999999e-06 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.739453+0000", 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060150399999999999 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.739478+0000", 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5077000000000001e-05 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.741242+0000", 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017642039999999999 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:34.741287+0000", 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4465000000000002e-05 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.175 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:50238 2.6 e123/104 2:62fa5ea0:::benchmark_data_smithi192_59818_object50237:head v 123'3186, mlcod=123'3183)", 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:35.983742+0000", 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 51.153944758000002, 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.129351679, 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:35.983742+0000", 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:35.983742+0000", 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:35.983743+0000", 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:35.983761+0000", 2024-06-27T21:13:27.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8488999999999999e-05 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:36.111570+0000", 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12780924399999999 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:36.111579+0000", 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5189999999999992e-06 2024-06-27T21:13:27.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:36.111629+0000", 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.003e-05 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:36.111647+0000", 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7816e-05 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:36.113055+0000", 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014082389999999999 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:36.113094+0000", 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8646999999999998e-05 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:51527 2.a e123/104 2:52187585:::benchmark_data_smithi192_59818_object51526:head v 123'3204, mlcod=123'3200)", 2024-06-27T21:13:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:37.029516+0000", 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.108170901000001, 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10146372300000001, 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.029516+0000", 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.029516+0000", 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.029516+0000", 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8299999999999996e-07 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.029563+0000", 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6826000000000002e-05 2024-06-27T21:13:27.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.129658+0000", 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.100095242 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.129665+0000", 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4609999999999996e-06 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.129710+0000", 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5485000000000002e-05 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.129725+0000", 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.467e-05 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.130949+0000", 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012234329999999999 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.130979+0000", 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0923000000000002e-05 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:52305 2.1 e124/104 2:8e52240a:::benchmark_data_smithi192_59818_object52304:head v 124'3431, mlcod=124'3428)", 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:37.862759+0000", 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 49.274928062000001, 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.109058989, 2024-06-27T21:13:27.185 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.862759+0000", 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.862759+0000", 2024-06-27T21:13:27.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.862759+0000", 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.78e-07 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.862782+0000", 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2993000000000002e-05 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.969902+0000", 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.107119886 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.969909+0000", 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4609999999999996e-06 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.970004+0000", 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5492e-05 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.970023+0000", 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.893e-05 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.971775+0000", 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017516459999999999 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:37.971818+0000", 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2703000000000001e-05 2024-06-27T21:13:27.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:56577 2.a e124/104 2:54377030:::benchmark_data_smithi192_59818_object56576:head v 124'3525, mlcod=124'3520)", 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:41.084869+0000", 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.052817288, 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14215898799999999, 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.190 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.084869+0000", 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.084869+0000", 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.084870+0000", 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1900000000000002e-07 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.084935+0000", 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5450000000000005e-05 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.225661+0000", 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.140725565 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.225666+0000", 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8130000000000003e-06 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.225708+0000", 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2518999999999997e-05 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.225722+0000", 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3882e-05 2024-06-27T21:13:27.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.227001+0000", 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012783740000000001 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.227028+0000", 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7766000000000001e-05 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:56633 2.a e124/104 2:5ecf20c0:::benchmark_data_smithi192_59818_object56632:head v 124'3526, mlcod=124'3523)", 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:41.123361+0000", 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.014325714999998, 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11116835899999999, 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.123361+0000", 2024-06-27T21:13:27.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.123361+0000", 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.123365+0000", 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7409999999999998e-06 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.123421+0000", 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6444999999999999e-05 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.233122+0000", 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.109700599 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.233127+0000", 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2719999999999997e-06 2024-06-27T21:13:27.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.233162+0000", 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4789999999999997e-05 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.233175+0000", 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.309e-05 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.234497+0000", 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013218559999999999 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.234529+0000", 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2566e-05 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:57137 2.a e124/104 2:5f7ac0a5:::benchmark_data_smithi192_59818_object57136:head v 124'3553, mlcod=124'3549)", 2024-06-27T21:13:27.199 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:41.565337+0000", 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.572349504000002, 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.089560573000000004, 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.565337+0000", 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.565337+0000", 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.565338+0000", 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000005e-07 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.565385+0000", 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7459000000000001e-05 2024-06-27T21:13:27.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.653493+0000", 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.088107656000000006 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.653498+0000", 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7269999999999998e-06 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.653568+0000", 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0458999999999997e-05 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.653583+0000", 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4559000000000001e-05 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.654868+0000", 2024-06-27T21:13:27.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012848169999999999 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:41.654898+0000", 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0147000000000001e-05 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:58724 2.6 e124/104 2:6f0e57f1:::benchmark_data_smithi192_59818_object58723:head v 124'3772, mlcod=124'3769)", 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:12:43.016762+0000", 2024-06-27T21:13:27.204 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 44.12092492, 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16848517700000001, 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.016762+0000", 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.016762+0000", 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.016763+0000", 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1999999999999999e-06 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.016785+0000", 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2209999999999999e-05 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.182795+0000", 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.166009984 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.182801+0000", 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.117e-06 2024-06-27T21:13:27.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.182868+0000", 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6804999999999995e-05 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.182884+0000", 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6274999999999999e-05 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.185165+0000", 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022805870000000002 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:12:43.185247+0000", 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1998999999999999e-05 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:79866 2.d e128/104 2:bf972344:::benchmark_data_smithi192_59818_object79865:head v 128'4992, mlcod=128'4989)", 2024-06-27T21:13:27.209 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:02.786763+0000", 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.350923721000001, 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.094454000999999996, 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.786763+0000", 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.786763+0000", 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.786764+0000", 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9699999999999995e-07 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.786857+0000", 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3640999999999993e-05 2024-06-27T21:13:27.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.878809+0000", 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.091952033000000002 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.878814+0000", 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3780000000000003e-06 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.878866+0000", 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2086000000000002e-05 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.878886+0000", 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0373e-05 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.881176+0000", 2024-06-27T21:13:27.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002289386 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:02.881217+0000", 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1406999999999998e-05 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:80148 2.d e128/104 2:b81e3a60:::benchmark_data_smithi192_59818_object80147:head v 128'5008, mlcod=128'5004)", 2024-06-27T21:13:27.214 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:03.225618+0000", 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.912068653999999, 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.124419127, 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.225618+0000", 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.225618+0000", 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.225619+0000", 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6200000000000006e-07 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.225672+0000", 2024-06-27T21:13:27.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3359999999999997e-05 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.348193+0000", 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.122520988 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.348199+0000", 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7010000000000001e-06 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.348252+0000", 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2893000000000003e-05 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.348275+0000", 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2889000000000001e-05 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.349995+0000", 2024-06-27T21:13:27.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017206439999999999 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:03.350037+0000", 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1690000000000002e-05 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:98639 2.a e135/104 2:5cf8774b:::benchmark_data_smithi192_59818_object6564:head v 135'6136, mlcod=135'6131)", 2024-06-27T21:13:27.219 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:23.347375+0000", 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.7903117069999999, 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.094934858999999996, 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.347375+0000", 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.347375+0000", 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.347376+0000", 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1799999999999995e-07 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.347380+0000", 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6260000000000003e-06 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.439177+0000", 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.091796618999999996 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.439182+0000", 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2469999999999999e-06 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.439242+0000", 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0182000000000001e-05 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.439262+0000", 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0058999999999999e-05 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.442263+0000", 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030007609999999998 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:23.442310+0000", 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6746999999999998e-05 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:101896 2.a e137/104 2:52f29e22:::benchmark_data_smithi192_59818_object9821:head v 137'6341, mlcod=137'6336)", 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:26.321566+0000", 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.81612084299999998, 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.092443898999999996, 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.224 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.321566+0000", 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.321566+0000", 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.321567+0000", 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3829999999999999e-06 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.321577+0000", 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3999999999999998e-06 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.411731+0000", 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090154849999999995 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.411738+0000", 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1800000000000001e-06 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.411803+0000", 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5227000000000004e-05 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.411827+0000", 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4541000000000001e-05 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.413979+0000", 2024-06-27T21:13:27.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021516479999999999 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.414010+0000", 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0670000000000003e-05 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:101990 2.a e137/104 2:59958f96:::benchmark_data_smithi192_59818_object9915:head v 137'6349, mlcod=137'6345)", 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:26.470709+0000", 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.66697723399999997, 2024-06-27T21:13:27.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.118877595, 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.470709+0000", 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.470709+0000", 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.470710+0000", 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4300000000000003e-07 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.470714+0000", 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.726e-06 2024-06-27T21:13:27.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.587328+0000", 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11661475 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.587333+0000", 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5669999999999996e-06 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.587380+0000", 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7278999999999999e-05 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.587395+0000", 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5153e-05 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.589540+0000", 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021448539999999999 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:27.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:26.589587+0000", 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6723e-05 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:27.234 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:13:27.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:27.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:27.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:27.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:27.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:27.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:27.555 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:13:27.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:27.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:27.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:13:27.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:27.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:27.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:27.894 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:27.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:27.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:27 smithi151 ceph-mon[28965]: pgmap v417: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.3 MiB/s wr, 972 op/s 2024-06-27T21:13:27.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1189107586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:13:27.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:27 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:13:27.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:27 smithi190 ceph-mon[32247]: pgmap v417: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.3 MiB/s wr, 972 op/s 2024-06-27T21:13:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1189107586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:13:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:27 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:13:28.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:28.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:27 smithi192 ceph-mon[32956]: pgmap v417: 17 pgs: 17 active+clean; 5.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.3 MiB/s wr, 972 op/s 2024-06-27T21:13:28.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1189107586' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:13:28.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:27 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:13:28.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:28.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:28.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:28.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:28.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:28.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:28.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:28.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:28.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:28.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:28.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:28.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:28.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:28.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:28 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:28 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:13:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:28 smithi151 ceph-mon[28965]: osdmap e138: 12 total, 12 up, 12 in 2024-06-27T21:13:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:28 smithi151 ceph-mon[28965]: 2.b scrub starts 2024-06-27T21:13:29.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:28 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:29.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:28 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:13:29.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:28 smithi190 ceph-mon[32247]: osdmap e138: 12 total, 12 up, 12 in 2024-06-27T21:13:29.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:28 smithi190 ceph-mon[32247]: 2.b scrub starts 2024-06-27T21:13:29.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:29.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:28 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:29.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:28 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:13:29.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:28 smithi192 ceph-mon[32956]: osdmap e138: 12 total, 12 up, 12 in 2024-06-27T21:13:29.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:28 smithi192 ceph-mon[32956]: 2.b scrub starts 2024-06-27T21:13:29.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:29.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:29.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:29.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:29.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:29.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:29.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:29.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:29.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:29.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:29.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:29.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:29.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:29 smithi151 ceph-mon[28965]: pgmap v419: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-06-27T21:13:29.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:13:29.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:29.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:29.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:29.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:29.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:30.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:29 smithi190 ceph-mon[32247]: pgmap v419: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-06-27T21:13:30.025 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:30.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:29 smithi192 ceph-mon[32956]: pgmap v419: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s 2024-06-27T21:13:30.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:30.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:30.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:30.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:30.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:30.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:30.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:30.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:30.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:13:30.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:30.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:30 smithi151 ceph-mon[28965]: 2.b scrub ok 2024-06-27T21:13:30.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:30.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:30 smithi190 ceph-mon[32247]: 2.b scrub ok 2024-06-27T21:13:30.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:30.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:30.838 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:30.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:30.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:30.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:31.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:30 smithi192 ceph-mon[32956]: 2.b scrub ok 2024-06-27T21:13:31.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:31.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:31.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:31.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:31.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:31.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:31.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:31.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:31.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:31.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:31.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:31.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:31.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:31 smithi190 ceph-mon[32247]: pgmap v420: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-06-27T21:13:31.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:31 smithi151 ceph-mon[28965]: pgmap v420: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-06-27T21:13:31.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 138, 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:13:27.559311+0000", 2024-06-27T21:13:31.861 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:09:37.640435+0000", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:09:20.227260+0000", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-06-27T21:13:31.862 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 2, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:13:31.863 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:13:31.864 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 1, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 1, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 1, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 1, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 1, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:13:31.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "21", 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:13:31.866 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:13:31.867 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:13:31.868 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 12, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 12, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.25, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 3, 2024-06-27T21:13:31.869 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 3, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 2, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_0", 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:11:29.737013+0000", 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:13:31.870 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-06-27T21:13:31.871 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "107", 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:13:31.872 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:13:31.873 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:13:31.874 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:13:31.875 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.25, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.25, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.25, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.25, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-06-27T21:13:31.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.877 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 53, 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:13:31.879 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.881 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.882 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.883 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.884 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:13:31.885 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.886 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.887 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.890 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.891 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.892 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:13:31.894 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.895 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:13:31.896 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.897 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.902 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.903 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 31, 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.904 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.906 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.907 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/2107645547", 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/2107645547", 2024-06-27T21:13:31.908 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/2107645547", 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/2107645547", 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.909 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.911 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.912 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.913 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:13:31.914 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.915 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.916 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.917 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.918 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:13:31.919 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.921 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.922 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.924 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:13:31.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.927 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.928 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.929 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:13:31.930 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.931 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.932 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.933 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.934 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.935 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.936 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.937 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:13:31.938 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.939 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.939 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.939 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.939 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:31.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:31.940 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.940 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:13:31.940 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-06-27T21:13:31.940 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.940 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.940 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:13:31.943 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.944 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.945 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:13:31.946 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:13:31.947 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:13:31.950 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:13:31.951 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.952 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:13.789023+0000", 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:12:23.781513+0000", 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:14.985857+0000", 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:13:31.954 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:12:51.772248+0000", 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.955 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:24.358917+0000", 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:23.314643+0000", 2024-06-27T21:13:31.956 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:12:56.580143+0000", 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:20.352174+0000", 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:13:31.958 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:10.920857+0000", 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.959 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:11.332489+0000", 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.960 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:12:46.919930+0000", 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:17.219093+0000", 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.961 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:13:31.962 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:13:31.963 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:13:31.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:13:31.965 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:13:32.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:31 smithi192 ceph-mon[32956]: pgmap v420: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s 2024-06-27T21:13:32.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:32.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:32.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:32.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:32.298 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:13:32.298 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:13:32.298 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:13:32.298 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:13:32.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:32.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:32.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:32.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:32.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:32.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:32.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:32.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:32.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:32.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:32.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1264757266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:13:32.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:32.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:32.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:33.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1264757266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:13:33.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1264757266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:13:33.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:33.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:33.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:13:33.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:33.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:33.288 INFO:tasks.thrashosds.thrasher:j is {'epoch': 138, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:13:27.559311+0000', 'last_up_change': '2024-06-27T21:09:37.640435+0000', 'last_in_change': '2024-06-27T21:09:20.227260+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-06-27T21:11:29.737013+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 2107645547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 2107645547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 2107645547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 2107645547}]}, 'public_addr': '172.21.15.190:6801/2107645547', 'cluster_addr': '172.21.15.190:6803/2107645547', 'heartbeat_back_addr': '172.21.15.190:6807/2107645547', 'heartbeat_front_addr': '172.21.15.190:6805/2107645547', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:13.789023+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:12:23.781513+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:14.985857+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:12:51.772248+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:24.358917+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:23.314643+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:12:56.580143+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:20.352174+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:10.920857+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:11.332489+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:12:46.919930+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:17.219093+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:13:33.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:13:33.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:33.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:33.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:33.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:33.526 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:33.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:33.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:33.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:33.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:33.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:33.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:33 smithi151 ceph-mon[28965]: pgmap v421: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-06-27T21:13:33.899 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:33 smithi190 ceph-mon[32247]: pgmap v421: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-06-27T21:13:33.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:33.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:33.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:33.993 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:34.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:34.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:34.036 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:33 smithi192 ceph-mon[32956]: pgmap v421: 17 pgs: 17 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 897 op/s 2024-06-27T21:13:34.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:34.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:34.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:34.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:34.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:34.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:34.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:34.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:34.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:34.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:34.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:34.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:34.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:34.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:34.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:34.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:34.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:34.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:35.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:35.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:35.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:35.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:35.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:35.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:35.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:35.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:35.387 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:13:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:13:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:13:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:13:35.388 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:13:35.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:35.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:35.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:35.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:35.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:35.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:35 smithi151 ceph-mon[28965]: pgmap v422: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 989 op/s 2024-06-27T21:13:35.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:35 smithi151 ceph-mon[28965]: osdmap e139: 12 total, 12 up, 12 in 2024-06-27T21:13:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:35.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:35.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:35.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:35.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:13:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:13:35.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:13:35.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:13:35.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:35.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:13:35.946 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:13:35.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:35.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:35.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:35 smithi190 ceph-mon[32247]: pgmap v422: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 989 op/s 2024-06-27T21:13:35.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:35 smithi190 ceph-mon[32247]: osdmap e139: 12 total, 12 up, 12 in 2024-06-27T21:13:35.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:13:35.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:13:36.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:13:36.010 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:13:36.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:35 smithi192 ceph-mon[32956]: pgmap v422: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 989 op/s 2024-06-27T21:13:36.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:35 smithi192 ceph-mon[32956]: osdmap e139: 12 total, 12 up, 12 in 2024-06-27T21:13:36.041 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:13:36.041 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:13:36.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:13:36.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:36.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:13:36.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:13:36.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:36.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:13:36.106 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:13:36.136 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:13:36.136 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:13:36.167 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:13:36.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:13:36.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:36.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:13:36.198 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:13:36.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:36.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:13:36.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:13:36.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:36.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:36.383 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:36.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:36.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:36.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:36.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:36.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:36.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:36.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1062658390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:36.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:36.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:36.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1062658390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:36.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:36.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:36.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:37.032 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1062658390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:37.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:37.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:37.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:37.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:37.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:37.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:37.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:37.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:37.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:37.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:37.707 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:13:37.707 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:13:37.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:37.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:37.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:37.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:37.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:37.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:37 smithi151 ceph-mon[28965]: pgmap v424: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-06-27T21:13:37.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:38.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:38.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:37 smithi190 ceph-mon[32247]: pgmap v424: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-06-27T21:13:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:38.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:37 smithi192 ceph-mon[32956]: pgmap v424: 17 pgs: 17 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-06-27T21:13:38.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:38.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:38.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:38.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:38.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:38.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:38.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:38.521 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-06-27T21:13:35.843333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4906156600,"num_objects":74856,"num_object_clones":0,"num_object_copies":224568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74856,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":109415,"num_write_kb":5894759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114376,"ondisk_log_size":114376,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15145428,"kb_used_data":14440884,"kb_used_omap":18,"kb_used_meta":704365,"kb_avail":1109796396,"statfs":{"total":1151940427776,"available":1136431509504,"internally_reserved":0,"allocated":14787465216,"data_stored":14784273976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":721270074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,7,3,6,2,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-578617344,"num_objects":-8829,"num_object_clones":0,"num_object_copies":-26487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.282246"},"pg_stats":[{"pgid":"2.f","version":"138'6812","reported_seq":6934,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:32.177167+0000","last_change":"2024-06-27T21:12:42.522399+0000","last_active":"2024-06-27T21:13:32.177167+0000","last_peered":"2024-06-27T21:13:32.177167+0000","last_clean":"2024-06-27T21:13:32.177167+0000","last_became_active":"2024-06-27T21:11:31.339208+0000","last_became_peered":"2024-06-27T21:11:31.339208+0000","last_unstale":"2024-06-27T21:13:32.177167+0000","last_undegraded":"2024-06-27T21:13:32.177167+0000","last_fullsized":"2024-06-27T21:13:32.177167+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3650","last_scrub_stamp":"2024-06-27T21:12:42.522239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.522239+0000","objects_scrubbed":3627,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:01.818239+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6812,"num_write_kb":374016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"138'6620","reported_seq":6744,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:33.316019+0000","last_change":"2024-06-27T21:12:48.076158+0000","last_active":"2024-06-27T21:13:33.316019+0000","last_peered":"2024-06-27T21:13:33.316019+0000","last_clean":"2024-06-27T21:13:33.316019+0000","last_became_active":"2024-06-27T21:11:31.337998+0000","last_became_peered":"2024-06-27T21:11:31.337998+0000","last_unstale":"2024-06-27T21:13:33.316019+0000","last_undegraded":"2024-06-27T21:13:33.316019+0000","last_fullsized":"2024-06-27T21:13:33.316019+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'3883","last_scrub_stamp":"2024-06-27T21:12:48.075948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:48.075948+0000","objects_scrubbed":3871,"log_size":6620,"log_dups_size":0,"ondisk_log_size":6620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:52.588505+0000","scrub_duration":667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296484864,"num_objects":4524,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6620,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"139'6913","reported_seq":7038,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-27T21:13:35.193047+0000","last_change":"2024-06-27T21:12:44.116062+0000","last_active":"2024-06-27T21:13:35.193047+0000","last_peered":"2024-06-27T21:13:35.193047+0000","last_clean":"2024-06-27T21:13:35.193047+0000","last_became_active":"2024-06-27T21:11:31.339229+0000","last_became_peered":"2024-06-27T21:11:31.339229+0000","last_unstale":"2024-06-27T21:13:35.193047+0000","last_undegraded":"2024-06-27T21:13:35.193047+0000","last_fullsized":"2024-06-27T21:13:35.193047+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3764","last_scrub_stamp":"2024-06-27T21:12:44.115896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.115896+0000","objects_scrubbed":3741,"log_size":6913,"log_dups_size":0,"ondisk_log_size":6913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:55.572851+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383128,"num_objects":4615,"num_object_clones":0,"num_object_copies":13845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6913,"num_write_kb":368833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"138'6903","reported_seq":7025,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:34.301142+0000","last_change":"2024-06-27T21:12:45.902924+0000","last_active":"2024-06-27T21:13:34.301142+0000","last_peered":"2024-06-27T21:13:34.301142+0000","last_clean":"2024-06-27T21:13:34.301142+0000","last_became_active":"2024-06-27T21:11:31.556192+0000","last_became_peered":"2024-06-27T21:11:31.556192+0000","last_unstale":"2024-06-27T21:13:34.301142+0000","last_undegraded":"2024-06-27T21:13:34.301142+0000","last_fullsized":"2024-06-27T21:13:34.301142+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3891","last_scrub_stamp":"2024-06-27T21:12:45.902837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:45.902837+0000","objects_scrubbed":3854,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:48.868840+0000","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304414720,"num_objects":4645,"num_object_clones":0,"num_object_copies":13935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":369536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":235,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:27.563195+0000","last_change":"2024-06-27T21:12:50.081819+0000","last_active":"2024-06-27T21:13:27.563195+0000","last_peered":"2024-06-27T21:13:27.563195+0000","last_clean":"2024-06-27T21:13:27.563195+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:13:27.563195+0000","last_undegraded":"2024-06-27T21:13:27.563195+0000","last_fullsized":"2024-06-27T21:13:27.563195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:12:50.081718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:12:50.081718+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:52.027615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"138'6767","reported_seq":6888,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:32.326799+0000","last_change":"2024-06-27T21:12:42.932200+0000","last_active":"2024-06-27T21:13:32.326799+0000","last_peered":"2024-06-27T21:13:32.326799+0000","last_clean":"2024-06-27T21:13:32.326799+0000","last_became_active":"2024-06-27T21:11:31.337613+0000","last_became_peered":"2024-06-27T21:11:31.337613+0000","last_unstale":"2024-06-27T21:13:32.326799+0000","last_undegraded":"2024-06-27T21:13:32.326799+0000","last_fullsized":"2024-06-27T21:13:32.326799+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3620","last_scrub_stamp":"2024-06-27T21:12:42.932079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.932079+0000","objects_scrubbed":3600,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:58.545974+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315031552,"num_objects":4807,"num_object_clones":0,"num_object_copies":14421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6767,"num_write_kb":370368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"138'6781","reported_seq":6901,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:34.409174+0000","last_change":"2024-06-27T21:12:41.700470+0000","last_active":"2024-06-27T21:13:34.409174+0000","last_peered":"2024-06-27T21:13:34.409174+0000","last_clean":"2024-06-27T21:13:34.409174+0000","last_became_active":"2024-06-27T21:11:31.341500+0000","last_became_peered":"2024-06-27T21:11:31.341500+0000","last_unstale":"2024-06-27T21:13:34.409174+0000","last_undegraded":"2024-06-27T21:13:34.409174+0000","last_fullsized":"2024-06-27T21:13:34.409174+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3530","last_scrub_stamp":"2024-06-27T21:12:41.700386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:41.700386+0000","objects_scrubbed":3520,"log_size":6781,"log_dups_size":0,"ondisk_log_size":6781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:48.568993+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296157184,"num_objects":4519,"num_object_clones":0,"num_object_copies":13557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6781,"num_write_kb":361600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"138'7070","reported_seq":7197,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:33.321891+0000","last_change":"2024-06-27T21:12:46.629126+0000","last_active":"2024-06-27T21:13:33.321891+0000","last_peered":"2024-06-27T21:13:33.321891+0000","last_clean":"2024-06-27T21:13:33.321891+0000","last_became_active":"2024-06-27T21:11:31.556127+0000","last_became_peered":"2024-06-27T21:11:31.556127+0000","last_unstale":"2024-06-27T21:13:33.321891+0000","last_undegraded":"2024-06-27T21:13:33.321891+0000","last_fullsized":"2024-06-27T21:13:33.321891+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'4084","last_scrub_stamp":"2024-06-27T21:12:46.629019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:46.629019+0000","objects_scrubbed":4059,"log_size":7070,"log_dups_size":0,"ondisk_log_size":7070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:16.370058+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314441728,"num_objects":4798,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"138'6840","reported_seq":6965,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:32.323504+0000","last_change":"2024-06-27T21:12:47.918243+0000","last_active":"2024-06-27T21:13:32.323504+0000","last_peered":"2024-06-27T21:13:32.323504+0000","last_clean":"2024-06-27T21:13:32.323504+0000","last_became_active":"2024-06-27T21:11:31.337786+0000","last_became_peered":"2024-06-27T21:11:31.337786+0000","last_unstale":"2024-06-27T21:13:32.323504+0000","last_undegraded":"2024-06-27T21:13:32.323504+0000","last_fullsized":"2024-06-27T21:13:32.323504+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'3978","last_scrub_stamp":"2024-06-27T21:12:47.918098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:47.918098+0000","objects_scrubbed":3946,"log_size":6840,"log_dups_size":0,"ondisk_log_size":6840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:06.555581+0000","scrub_duration":897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314441728,"num_objects":4798,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6840,"num_write_kb":372416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"138'6861","reported_seq":6982,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:33.715792+0000","last_change":"2024-06-27T21:12:44.158051+0000","last_active":"2024-06-27T21:13:33.715792+0000","last_peered":"2024-06-27T21:13:33.715792+0000","last_clean":"2024-06-27T21:13:33.715792+0000","last_became_active":"2024-06-27T21:11:31.556366+0000","last_became_peered":"2024-06-27T21:11:31.556366+0000","last_unstale":"2024-06-27T21:13:33.715792+0000","last_undegraded":"2024-06-27T21:13:33.715792+0000","last_fullsized":"2024-06-27T21:13:33.715792+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3740","last_scrub_stamp":"2024-06-27T21:12:44.157914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.157914+0000","objects_scrubbed":3704,"log_size":6861,"log_dups_size":0,"ondisk_log_size":6861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:55.134407+0000","scrub_duration":1005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301662208,"num_objects":4603,"num_object_clones":0,"num_object_copies":13809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6861,"num_write_kb":366848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'9185","reported_seq":9462,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:32.197148+0000","last_change":"2024-06-27T21:13:08.494332+0000","last_active":"2024-06-27T21:13:32.197148+0000","last_peered":"2024-06-27T21:13:32.197148+0000","last_clean":"2024-06-27T21:13:32.197148+0000","last_became_active":"2024-06-27T21:11:31.556322+0000","last_became_peered":"2024-06-27T21:11:31.556322+0000","last_unstale":"2024-06-27T21:13:32.197148+0000","last_undegraded":"2024-06-27T21:13:32.197148+0000","last_fullsized":"2024-06-27T21:13:32.197148+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7832","last_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_deep_scrub":"129'7832","last_deep_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_clean_scrub_stamp":"2024-06-27T21:13:08.494150+0000","objects_scrubbed":4784,"log_size":9185,"log_dups_size":0,"ondisk_log_size":9185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:33.871589+0000","scrub_duration":14657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311754752,"num_objects":4757,"num_object_clones":0,"num_object_copies":14271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6601,"num_write_kb":363456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"138'6960","reported_seq":7091,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:33.623198+0000","last_change":"2024-06-27T21:12:56.263993+0000","last_active":"2024-06-27T21:13:33.623198+0000","last_peered":"2024-06-27T21:13:33.623198+0000","last_clean":"2024-06-27T21:13:33.623198+0000","last_became_active":"2024-06-27T21:11:31.338960+0000","last_became_peered":"2024-06-27T21:11:31.338960+0000","last_unstale":"2024-06-27T21:13:33.623198+0000","last_undegraded":"2024-06-27T21:13:33.623198+0000","last_fullsized":"2024-06-27T21:13:33.623198+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4740","last_scrub_stamp":"2024-06-27T21:12:56.263856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:56.263856+0000","objects_scrubbed":4707,"log_size":6960,"log_dups_size":0,"ondisk_log_size":6960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:10.594628+0000","scrub_duration":1215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314572800,"num_objects":4800,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6960,"num_write_kb":376320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"139'6844","reported_seq":6969,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-06-27T21:13:35.122269+0000","last_change":"2024-06-27T21:12:44.319499+0000","last_active":"2024-06-27T21:13:35.122269+0000","last_peered":"2024-06-27T21:13:35.122269+0000","last_clean":"2024-06-27T21:13:35.122269+0000","last_became_active":"2024-06-27T21:11:31.943105+0000","last_became_peered":"2024-06-27T21:11:31.943105+0000","last_unstale":"2024-06-27T21:13:35.122269+0000","last_undegraded":"2024-06-27T21:13:35.122269+0000","last_fullsized":"2024-06-27T21:13:35.122269+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3747","last_scrub_stamp":"2024-06-27T21:12:44.319331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.319331+0000","objects_scrubbed":3704,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:59.241125+0000","scrub_duration":901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698432,"num_objects":4512,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6844,"num_write_kb":363392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"138'6806","reported_seq":6936,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:34.318667+0000","last_change":"2024-06-27T21:12:55.073348+0000","last_active":"2024-06-27T21:13:34.318667+0000","last_peered":"2024-06-27T21:13:34.318667+0000","last_clean":"2024-06-27T21:13:34.318667+0000","last_became_active":"2024-06-27T21:11:31.556186+0000","last_became_peered":"2024-06-27T21:11:31.556186+0000","last_unstale":"2024-06-27T21:13:34.318667+0000","last_undegraded":"2024-06-27T21:13:34.318667+0000","last_fullsized":"2024-06-27T21:13:34.318667+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4512","last_scrub_stamp":"2024-06-27T21:12:55.073215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:55.073215+0000","objects_scrubbed":4477,"log_size":6806,"log_dups_size":0,"ondisk_log_size":6806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:03.225629+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300417024,"num_objects":4584,"num_object_clones":0,"num_object_copies":13752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":364480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"138'6926","reported_seq":7046,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:33.690142+0000","last_change":"2024-06-27T21:12:43.447627+0000","last_active":"2024-06-27T21:13:33.690142+0000","last_peered":"2024-06-27T21:13:33.690142+0000","last_clean":"2024-06-27T21:13:33.690142+0000","last_became_active":"2024-06-27T21:11:31.337888+0000","last_became_peered":"2024-06-27T21:11:31.337888+0000","last_unstale":"2024-06-27T21:13:33.690142+0000","last_undegraded":"2024-06-27T21:13:33.690142+0000","last_fullsized":"2024-06-27T21:13:33.690142+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3700","last_scrub_stamp":"2024-06-27T21:12:43.447399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:43.447399+0000","objects_scrubbed":3670,"log_size":6926,"log_dups_size":0,"ondisk_log_size":6926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:44.702380+0000","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305004544,"num_objects":4654,"num_object_clones":0,"num_object_copies":13962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6926,"num_write_kb":370560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"138'6817","reported_seq":6936,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:34.248632+0000","last_change":"2024-06-27T21:12:42.976967+0000","last_active":"2024-06-27T21:13:34.248632+0000","last_peered":"2024-06-27T21:13:34.248632+0000","last_clean":"2024-06-27T21:13:34.248632+0000","last_became_active":"2024-06-27T21:11:31.340369+0000","last_became_peered":"2024-06-27T21:11:31.340369+0000","last_unstale":"2024-06-27T21:13:34.248632+0000","last_undegraded":"2024-06-27T21:13:34.248632+0000","last_fullsized":"2024-06-27T21:13:34.248632+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3653","last_scrub_stamp":"2024-06-27T21:12:42.976769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.976769+0000","objects_scrubbed":3623,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:53.267872+0000","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300875776,"num_objects":4591,"num_object_clones":0,"num_object_copies":13773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6817,"num_write_kb":365056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"138'9179","reported_seq":9474,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-06-27T21:13:32.197455+0000","last_change":"2024-06-27T21:13:29.274450+0000","last_active":"2024-06-27T21:13:32.197455+0000","last_peered":"2024-06-27T21:13:32.197455+0000","last_clean":"2024-06-27T21:13:32.197455+0000","last_became_active":"2024-06-27T21:11:31.337915+0000","last_became_peered":"2024-06-27T21:11:31.337915+0000","last_unstale":"2024-06-27T21:13:32.197455+0000","last_undegraded":"2024-06-27T21:13:32.197455+0000","last_fullsized":"2024-06-27T21:13:32.197455+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9000","last_scrub_stamp":"2024-06-27T21:13:29.274287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:13:29.274287+0000","objects_scrubbed":5002,"log_size":9179,"log_dups_size":0,"ondisk_log_size":9179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:47.960430+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312672256,"num_objects":4771,"num_object_clones":0,"num_object_copies":14313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6771,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4905566232,"num_objects":74854,"num_object_clones":0,"num_object_copies":224562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109292,"num_write_kb":5892609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14663888896,"data_stored":14663876680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114284,"ondisk_log_size":114284,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252915,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949100,"kb_used_data":891556,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92796052,"statfs":{"total":95995035648,"available":95023157248,"internally_reserved":0,"allocated":912953344,"data_stored":912685743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920996,"kb_used_data":875932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92824156,"statfs":{"total":95995035648,"available":95051935744,"internally_reserved":0,"allocated":896954368,"data_stored":896690339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743484,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1838952,"kb_used_data":1763360,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91906200,"statfs":{"total":95995035648,"available":94111948800,"internally_reserved":0,"allocated":1805680640,"data_stored":1805412539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972416,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1316708,"kb_used_data":1241116,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92428444,"statfs":{"total":95995035648,"available":94646726656,"internally_reserved":0,"allocated":1270902784,"data_stored":1270638755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1545260,"kb_used_data":1482148,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92199892,"statfs":{"total":95995035648,"available":94412689408,"internally_reserved":0,"allocated":1517719552,"data_stored":1517451971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528393,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625772,"kb_used_data":594596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93119380,"statfs":{"total":95995035648,"available":95354245120,"internally_reserved":0,"allocated":608866304,"data_stored":608598723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654948,"kb_used_data":619612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93090204,"statfs":{"total":95995035648,"available":95324368896,"internally_reserved":0,"allocated":634482688,"data_stored":634218659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":986216,"kb_used_data":928672,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92758936,"statfs":{"total":95995035648,"available":94985150464,"internally_reserved":0,"allocated":950960128,"data_stored":950692027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914984,"kb_used_data":1839328,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91830168,"statfs":{"total":95995035648,"available":94034092032,"internally_reserved":0,"allocated":1883471872,"data_stored":1883203771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266340,"kb_used_data":1212956,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92478812,"statfs":{"total":95995035648,"available":94698303488,"internally_reserved":0,"allocated":1242066944,"data_stored":1241802915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582820,"kb_used_data":1511388,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92162332,"statfs":{"total":95995035648,"available":94374227968,"internally_reserved":0,"allocated":1547661312,"data_stored":1547397283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1543332,"kb_used_data":1480220,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92201820,"statfs":{"total":95995035648,"available":94414663680,"internally_reserved":0,"allocated":1515745280,"data_stored":1515481251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1494089728,"data_stored":1494089728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1537212416,"data_stored":1537212416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1223884800,"data_stored":1223884800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1875447808,"data_stored":1875443736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":939724800,"data_stored":939720728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":629342208,"data_stored":629342208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":600178688,"data_stored":600178688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1512046592,"data_stored":1512046592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1264451584,"data_stored":1264451584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1782845440,"data_stored":1782841368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":894173184,"data_stored":894173184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":910491648,"data_stored":910491648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:13:38.523 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [11, 8, 6] 2024-06-27T21:13:38.523 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.0', '11', '8', '6'] 2024-06-27T21:13:38.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap 2.0 11 8 6 2024-06-27T21:13:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:38.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:38.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:38.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:38.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:38.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:38.860 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:38 smithi190 ceph-mon[32247]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:13:38.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:38.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:38.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:38.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:38 smithi151 ceph-mon[28965]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:13:38.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:13:39.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:39.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:38 smithi192 ceph-mon[32956]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:13:39.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:39.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:39.142 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:13:39.143 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:13:39.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:39.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:39.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:39.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:39.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:39.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:39.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:39.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:39.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:39.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:39.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:39.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:39 smithi151 ceph-mon[28965]: pgmap v425: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:13:39.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:39.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:39.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:40.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:39 smithi190 ceph-mon[32247]: pgmap v425: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:13:40.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:40.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:39 smithi192 ceph-mon[32956]: pgmap v425: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:13:40.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:40.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:40.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:40.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:13:40.190 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:40.190 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:40.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:40.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:40.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:40.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:40.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:40.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:40.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:40.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:40.611 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:13:40.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3579555856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:40.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:40 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:40.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:40.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:40.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:40.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:40.855 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3579555856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:40.855 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:40 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:40.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:40.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:40.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:41.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3579555856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:41.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:40 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:41.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:41.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:41.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:41.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:41.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:41.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:41.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:41.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:41.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:41.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:41.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:41 smithi151 ceph-mon[28965]: pgmap v426: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:13:41.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:41 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:13:41.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:41 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:13:41.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:41 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:41.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:41 smithi151 ceph-mon[28965]: osdmap e140: 12 total, 12 up, 12 in 2024-06-27T21:13:41.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:41.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:41.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:41.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:41.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:41 smithi190 ceph-mon[32247]: pgmap v426: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:13:41.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:41 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:13:41.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:41 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:13:41.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:41 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:41.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:41 smithi190 ceph-mon[32247]: osdmap e140: 12 total, 12 up, 12 in 2024-06-27T21:13:41.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:41.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:41.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:42.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:41 smithi192 ceph-mon[32956]: pgmap v426: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:13:42.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:41 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:13:42.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:41 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:13:42.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:41 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:42.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:41 smithi192 ceph-mon[32956]: osdmap e140: 12 total, 12 up, 12 in 2024-06-27T21:13:42.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:42.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:42.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:42.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:42.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:42.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:42.407 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:13:42.408 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:13:42.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:42.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:42.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:42.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:42.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:42.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:42.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:42.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:42.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:42.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:43.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:43.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:13:43.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:43.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:43.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:43.263 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:13:43.264 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:13:43.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:43.294 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:13:43.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:13:43.325 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:13:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:13:43.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:13:43.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:13:43.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:43.384 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:43.387 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:13:43.388 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:13:43.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:43.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:13:43.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:13:43.444 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:13:43.444 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:13:43.475 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:13:43.476 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:13:43.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:43.502 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:13:43.502 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:13:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:43.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:43.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:13:43.531 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:13:43.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:13:43.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:13:43.592 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:13:43.592 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:13:43.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:43.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:43.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:43.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:43.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:43 smithi190 ceph-mon[32247]: pgmap v428: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s 2024-06-27T21:13:43.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3705392955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:43.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:43.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:43.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:43.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:43 smithi151 ceph-mon[28965]: pgmap v428: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s 2024-06-27T21:13:43.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3705392955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:44.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:44.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:44.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:43 smithi192 ceph-mon[32956]: pgmap v428: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s 2024-06-27T21:13:44.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3705392955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:44.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:44.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:44.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:44.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:44.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:44.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:44.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:44.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:44.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:13:44.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:44.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:44.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:44.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:44.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:44.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:44.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:45.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:45.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:45.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:45.177 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:13:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:13:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:13:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:13:45.177 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:13:45.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:45.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:45.300 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:45.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:45.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:45.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:45.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:45.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:45.619 INFO:teuthology.orchestra.run.smithi151.stderr:set 2.0 pg_upmap mapping to [11,8,6] 2024-06-27T21:13:45.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:45.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:45.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:45 smithi151 ceph-mon[28965]: pgmap v429: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s 2024-06-27T21:13:45.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3854943027' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [11, 8, 6]}]: dispatch 2024-06-27T21:13:45.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:45.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:45.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:45.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:45.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:46.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:45 smithi190 ceph-mon[32247]: pgmap v429: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s 2024-06-27T21:13:46.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3854943027' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [11, 8, 6]}]: dispatch 2024-06-27T21:13:46.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:13:46.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:45 smithi192 ceph-mon[32956]: pgmap v429: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s 2024-06-27T21:13:46.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3854943027' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [11, 8, 6]}]: dispatch 2024-06-27T21:13:46.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:46.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:46.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:46.287 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:46.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:46.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:13:46.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:46.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:46.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:46.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:46.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:46.684 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3854943027' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [11, 8, 6]}]': finished 2024-06-27T21:13:46.684 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:46 smithi190 ceph-mon[32247]: osdmap e141: 12 total, 12 up, 12 in 2024-06-27T21:13:46.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:46.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:46.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:46.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:46.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3854943027' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [11, 8, 6]}]': finished 2024-06-27T21:13:46.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:46 smithi151 ceph-mon[28965]: osdmap e141: 12 total, 12 up, 12 in 2024-06-27T21:13:47.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:47.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3854943027' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.0", "id": [11, 8, 6]}]': finished 2024-06-27T21:13:47.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:46 smithi192 ceph-mon[32956]: osdmap e141: 12 total, 12 up, 12 in 2024-06-27T21:13:47.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:47.112 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:47.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:47.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:47.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:47.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:47.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:47.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:47.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:47.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:47.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:47.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:47.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:47.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:47.842 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:47 smithi190 ceph-mon[32247]: pgmap v431: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-06-27T21:13:47.842 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:47 smithi190 ceph-mon[32247]: osdmap e142: 12 total, 12 up, 12 in 2024-06-27T21:13:47.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:47 smithi151 ceph-mon[28965]: pgmap v431: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-06-27T21:13:47.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:47 smithi151 ceph-mon[28965]: osdmap e142: 12 total, 12 up, 12 in 2024-06-27T21:13:47.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:47.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:47.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:7684 2.2 2:41003e73:::benchmark_data_smithi192_59818_object7683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:13:47.957 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:50.220063+0000", 2024-06-27T21:13:47.958 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 117.743219946, 2024-06-27T21:13:47.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24628781299999999, 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 7684 2024-06-27T21:13:47.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220063+0000", 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220063+0000", 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220064+0000", 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1900000000000001e-07 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220089+0000", 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5174999999999999e-05 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220100+0000", 2024-06-27T21:13:47.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0672999999999999e-05 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220105+0000", 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3589999999999999e-06 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220146+0000", 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0272000000000003e-05 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220220+0000", 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4742999999999999e-05 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220302+0000", 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1630000000000003e-05 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220753+0000", 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045050800000000002 2024-06-27T21:13:47.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466242+0000", 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24548952800000001 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466294+0000", 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1595999999999999e-05 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:47.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466327+0000", 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3390999999999999e-05 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466351+0000", 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4119000000000001e-05 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.965 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:7684 2.2 e108/104 ondisk, result = 0)", 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:50.220902+0000", 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 117.742380973, 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.245454533, 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220902+0000", 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220902+0000", 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220903+0000", 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.82e-07 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.220906+0000", 2024-06-27T21:13:47.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6010000000000002e-06 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466241+0000", 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24533508800000001 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466245+0000", 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.403e-06 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466284+0000", 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9050000000000001e-05 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466291+0000", 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4389999999999997e-06 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:50.466357+0000", 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6370000000000003e-05 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.969 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:16223 2.2 2:46296cd0:::benchmark_data_smithi192_59818_object16222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:56.297207+0000", 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 111.666076506, 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20902547299999999, 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:47.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 16223 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297207+0000", 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297207+0000", 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297209+0000", 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.447e-06 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297258+0000", 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9098999999999999e-05 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297288+0000", 2024-06-27T21:13:47.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9929000000000001e-05 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297303+0000", 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4677e-05 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297419+0000", 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011574999999999999 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297575+0000", 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015599900000000001 2024-06-27T21:13:47.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.297659+0000", 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4571000000000001e-05 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.298291+0000", 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00063167599999999996 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.298511+0000", 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000220072 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506130+0000", 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.207619044 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506190+0000", 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9629000000000001e-05 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506232+0000", 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2580000000000002e-05 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:16223 2.2 e110/104 ondisk, result = 0)", 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:56.298585+0000", 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 111.664697847, 2024-06-27T21:13:47.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20765287900000001, 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.298585+0000", 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.298585+0000", 2024-06-27T21:13:47.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.298586+0000", 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3099999999999997e-07 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.298589+0000", 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1039999999999998e-06 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506052+0000", 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20746235499999999 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506057+0000", 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0100000000000003e-06 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506117+0000", 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0025999999999998e-05 2024-06-27T21:13:47.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506124+0000", 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.622e-06 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:56.506238+0000", 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000114131 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:43160 2.3 2:c85ba3d8:::benchmark_data_smithi192_59818_object43159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:30.813189+0000", 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 77.150093902999998, 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14836608100000001, 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:47.982 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 43160 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813189+0000", 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813189+0000", 2024-06-27T21:13:47.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813191+0000", 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.376e-06 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813219+0000", 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8583000000000001e-05 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813237+0000", 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7354999999999999e-05 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813261+0000", 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4695999999999999e-05 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813326+0000", 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4732000000000003e-05 2024-06-27T21:13:47.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813482+0000", 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000155826 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.813632+0000", 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000150197 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:47.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.814901+0000", 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012686959999999999 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.815343+0000", 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044173599999999997 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961490+0000", 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14614686199999999 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961526+0000", 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6319000000000002e-05 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961556+0000", 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9703e-05 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:47.988 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:43160 2.3 e123/104 ondisk, result = 0)", 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:30.815149+0000", 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 77.148134061999997, 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14641116900000001, 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.815149+0000", 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.815149+0000", 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.815150+0000", 2024-06-27T21:13:47.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0599999999999999e-07 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.815156+0000", 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3729999999999998e-06 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961445+0000", 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14628886299999999 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961451+0000", 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3079999999999998e-06 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961481+0000", 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0448000000000001e-05 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961486+0000", 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0309999999999998e-06 2024-06-27T21:13:47.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.961560+0000", 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4339999999999996e-05 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:48486 2.2 2:42f900d0:::benchmark_data_smithi192_59818_object48485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.454017+0000", 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 73.509266381000003, 2024-06-27T21:13:47.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15779775400000001, 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 48486 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:47.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454017+0000", 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454017+0000", 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454018+0000", 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.035e-06 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454064+0000", 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6254000000000001e-05 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454075+0000", 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0849999999999999e-05 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454081+0000", 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0820000000000004e-06 2024-06-27T21:13:47.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454124+0000", 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3074000000000001e-05 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454221+0000", 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6910000000000006e-05 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:47.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.454328+0000", 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106974 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.455585+0000", 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001257298 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.455945+0000", 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035964999999999999 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611755+0000", 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.155810159 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611790+0000", 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4459999999999999e-05 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611815+0000", 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5007999999999998e-05 2024-06-27T21:13:47.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:48486 2.2 e123/104 ondisk, result = 0)", 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.456050+0000", 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 73.507233428000006, 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.155770033, 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.456050+0000", 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.456050+0000", 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.456051+0000", 2024-06-27T21:13:48.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2399999999999998e-07 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.456054+0000", 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4060000000000001e-06 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611710+0000", 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15565648000000001 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611715+0000", 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2690000000000001e-06 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611746+0000", 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1735e-05 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611751+0000", 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4279999999999998e-06 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.611820+0000", 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9091000000000006e-05 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:48824 2.2 2:4f69430c:::benchmark_data_smithi192_59818_object48823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.911412+0000", 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 73.051870871999995, 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15164235500000001, 2024-06-27T21:13:48.004 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 48824 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911412+0000", 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911412+0000", 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911414+0000", 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4470000000000001e-06 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911460+0000", 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6369000000000003e-05 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911474+0000", 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3995999999999999e-05 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911483+0000", 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9069999999999996e-06 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911537+0000", 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3384999999999998e-05 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911637+0000", 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100692 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.911728+0000", 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0786999999999997e-05 2024-06-27T21:13:48.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913029+0000", 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001300945 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062752+0000", 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14972250600000001 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062948+0000", 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019603700000000001 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.063008+0000", 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0646000000000002e-05 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.063055+0000", 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6638000000000003e-05 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.010 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:48824 2.2 e123/104 ondisk, result = 0)", 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.913145+0000", 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 73.050138227000005, 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14961248199999999, 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913145+0000", 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913145+0000", 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913146+0000", 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0299999999999998e-07 2024-06-27T21:13:48.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913149+0000", 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6399999999999999e-06 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062707+0000", 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.149557146 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062711+0000", 2024-06-27T21:13:48.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2860000000000001e-06 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062744+0000", 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2895000000000003e-05 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062748+0000", 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3800000000000004e-06 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062758+0000", 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4320000000000005e-06 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:48824 2.2 e123/104 ondisk, result = 0)", 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.913361+0000", 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 73.049922809999998, 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14970185999999999, 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.015 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913361+0000", 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913361+0000", 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913361+0000", 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2900000000000003e-07 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.913365+0000", 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1749999999999998e-06 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062880+0000", 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14951494700000001 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062885+0000", 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4279999999999998e-06 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062930+0000", 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5581999999999998e-05 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.062940+0000", 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7550000000000006e-06 2024-06-27T21:13:48.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:35.063062+0000", 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000122244 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:79075 2.3 2:c80e5dc0:::benchmark_data_smithi192_59818_object79074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:01.895805+0000", 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 46.067478674, 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23156829400000001, 2024-06-27T21:13:48.020 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 79075 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895805+0000", 2024-06-27T21:13:48.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895805+0000", 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895805+0000", 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5300000000000003e-07 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895864+0000", 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8943000000000001e-05 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895874+0000", 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9529999999999994e-06 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895881+0000", 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4409999999999998e-06 2024-06-27T21:13:48.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.895935+0000", 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4295000000000001e-05 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.896058+0000", 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000122917 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:13:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.896185+0000", 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012703000000000001 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.898100+0000", 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001914607 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.899087+0000", 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00098742300000000008 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.127266+0000", 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22817900399999999 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.127326+0000", 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9821999999999997e-05 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.127373+0000", 2024-06-27T21:13:48.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7105999999999999e-05 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:109746 2.2 2:44f7e819:::benchmark_data_smithi192_59818_object17671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:33.917672+0000", 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.045611508, 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.164118976, 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.027 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 109746 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917672+0000", 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917672+0000", 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917674+0000", 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0640000000000001e-06 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917681+0000", 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6170000000000003e-06 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917696+0000", 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5967000000000001e-05 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917705+0000", 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0399999999999993e-06 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917766+0000", 2024-06-27T21:13:48.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1474000000000001e-05 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.917944+0000", 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017785899999999999 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.918072+0000", 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128563 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.921267+0000", 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031944600000000001 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:33.921448+0000", 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000181029 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.081725+0000", 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.160277109 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.081760+0000", 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4728999999999999e-05 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.081791+0000", 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1065000000000002e-05 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:109929 2.2 2:4a417644:::benchmark_data_smithi192_59818_object17854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:34.307966+0000", 2024-06-27T21:13:48.033 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.655316854000001, 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.149696737, 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 109929 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.307966+0000", 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.307966+0000", 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.307968+0000", 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1090000000000001e-06 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.307972+0000", 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5340000000000001e-06 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.307985+0000", 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2449000000000001e-05 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.307992+0000", 2024-06-27T21:13:48.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4270000000000002e-06 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.308051+0000", 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9237000000000002e-05 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.308179+0000", 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012736400000000001 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.308298+0000", 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011961200000000001 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.310758+0000", 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024593890000000002 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.310942+0000", 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000184592 2024-06-27T21:13:48.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.457595+0000", 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14665284100000001 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.457636+0000", 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1016999999999997e-05 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:34.457663+0000", 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7166e-05 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121207 2.3 2:c63983c9:::benchmark_data_smithi192_59818_object29132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.266363+0000", 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.696920553, 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.172002877, 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.040 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121207 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266363+0000", 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266363+0000", 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266364+0000", 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.426e-06 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266370+0000", 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8359999999999997e-06 2024-06-27T21:13:48.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266384+0000", 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3723999999999999e-05 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266391+0000", 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1740000000000003e-06 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266443+0000", 2024-06-27T21:13:48.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2015000000000003e-05 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266522+0000", 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9183000000000004e-05 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.266600+0000", 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7705000000000002e-05 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.269145+0000", 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025448440000000001 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.270220+0000", 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001075723 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.438304+0000", 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.168083594 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.438336+0000", 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1853000000000001e-05 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.438366+0000", 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9799999999999999e-05 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121390 2.2 2:41402c32:::benchmark_data_smithi192_59818_object29315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.820966+0000", 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1423171010000002, 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18226844, 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121390 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.047 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.820966+0000", 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.820966+0000", 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.820968+0000", 2024-06-27T21:13:48.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.308e-06 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.820973+0000", 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7080000000000002e-06 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.820986+0000", 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3035e-05 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.820992+0000", 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0950000000000001e-06 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.821041+0000", 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8131999999999999e-05 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.821125+0000", 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4043999999999997e-05 2024-06-27T21:13:48.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.821209+0000", 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4401999999999996e-05 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.824470+0000", 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032614020000000001 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.975790+0000", 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15131987699999999 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.003157+0000", 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.027366364000000001 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.003191+0000", 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4372000000000002e-05 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.003235+0000", 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3701000000000001e-05 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:121390 2.2 e141/104 ondisk, result = 0)", 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.824663+0000", 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1386201950000001, 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15114200999999999, 2024-06-27T21:13:48.053 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.824663+0000", 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.824663+0000", 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.824664+0000", 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.824667+0000", 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5389999999999999e-06 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.975587+0000", 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15091990699999999 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.975608+0000", 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1060000000000002e-05 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.975763+0000", 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015430899999999999 2024-06-27T21:13:48.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.975780+0000", 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7343e-05 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.975805+0000", 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5176000000000001e-05 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121391 2.3 2:c32609d9:::benchmark_data_smithi192_59818_object29316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.825052+0000", 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1382316709999998, 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.165897511, 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.058 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121391 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825052+0000", 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825052+0000", 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825052+0000", 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3900000000000004e-07 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825056+0000", 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5360000000000001e-06 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825066+0000", 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0348e-05 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825071+0000", 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1189999999999996e-06 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825115+0000", 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3761000000000003e-05 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825190+0000", 2024-06-27T21:13:48.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4772999999999997e-05 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.825257+0000", 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7152999999999999e-05 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.827693+0000", 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002435413 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.828244+0000", 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00055181700000000002 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.990886+0000", 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16264161699999999 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.990918+0000", 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2178999999999998e-05 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.990949+0000", 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1056000000000002e-05 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121442 2.2 2:478d92b6:::benchmark_data_smithi192_59818_object29367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:46.100577+0000", 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.862706489, 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.223586971, 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.064 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121442 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100577+0000", 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100577+0000", 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100578+0000", 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0869999999999999e-06 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100582+0000", 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3869999999999997e-06 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100594+0000", 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2021000000000001e-05 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100600+0000", 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3430000000000003e-06 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100657+0000", 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7036000000000001e-05 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100783+0000", 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125826 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100911+0000", 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128477 2024-06-27T21:13:48.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.104303+0000", 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033924519999999998 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.104493+0000", 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000189614 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.324088+0000", 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21959461799999999 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.324130+0000", 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2110999999999997e-05 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.324164+0000", 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3998999999999998e-05 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121449 2.3 2:cd1bd32e:::benchmark_data_smithi192_59818_object29374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:46.119375+0000", 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.8439087590000001, 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.342841958, 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:48.112 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121449 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119375+0000", 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119375+0000", 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119376+0000", 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6750000000000001e-06 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119383+0000", 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8410000000000002e-06 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119401+0000", 2024-06-27T21:13:48.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7441000000000001e-05 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119410+0000", 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5370000000000003e-06 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119470+0000", 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0226999999999998e-05 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:48.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119554+0000", 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3463999999999999e-05 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119622+0000", 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8194999999999993e-05 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.122203+0000", 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025806240000000001 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.122441+0000", 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023860299999999999 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.462142+0000", 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33970106700000002 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.462182+0000", 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9298999999999998e-05 2024-06-27T21:13:48.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.462217+0000", 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4984999999999998e-05 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:48.118 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:13:48.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:48.119 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:47 smithi192 ceph-mon[32956]: pgmap v431: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-06-27T21:13:48.119 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:47 smithi192 ceph-mon[32956]: osdmap e142: 12 total, 12 up, 12 in 2024-06-27T21:13:48.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:48.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:48.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:48.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:48.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:48.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:48.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:48.638 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:13:48.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:48.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:48.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:48.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:48.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:13:48.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:48.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:48.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:48 smithi151 ceph-mon[28965]: osdmap e143: 12 total, 12 up, 12 in 2024-06-27T21:13:48.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3343274566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:49.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:48 smithi190 ceph-mon[32247]: osdmap e143: 12 total, 12 up, 12 in 2024-06-27T21:13:49.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3343274566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:49.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:49.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:48 smithi192 ceph-mon[32956]: osdmap e143: 12 total, 12 up, 12 in 2024-06-27T21:13:49.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3343274566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:13:49.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:49.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:49.160 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:49.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:49.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:49.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:49.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:49.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:49.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:49.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:49.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:49.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:49.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:49.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:49.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:49 smithi151 ceph-mon[28965]: pgmap v434: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-27T21:13:49.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:49 smithi151 ceph-mon[28965]: 2.9 scrub starts 2024-06-27T21:13:49.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:49 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:49.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3343274566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:13:49.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:49 smithi151 ceph-mon[28965]: osdmap e144: 12 total, 12 up, 12 in 2024-06-27T21:13:49.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:49.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:49.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:50.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:49 smithi190 ceph-mon[32247]: pgmap v434: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-27T21:13:50.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:49 smithi190 ceph-mon[32247]: 2.9 scrub starts 2024-06-27T21:13:50.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:49 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3343274566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:13:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:49 smithi190 ceph-mon[32247]: osdmap e144: 12 total, 12 up, 12 in 2024-06-27T21:13:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:49 smithi192 ceph-mon[32956]: pgmap v434: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-27T21:13:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:49 smithi192 ceph-mon[32956]: 2.9 scrub starts 2024-06-27T21:13:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:49 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3343274566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:13:50.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:49 smithi192 ceph-mon[32956]: osdmap e144: 12 total, 12 up, 12 in 2024-06-27T21:13:50.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:13:50.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:13:50.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:50.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:13:50.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:13:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:50.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:50.092 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:13:50.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:13:50.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:13:50.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:13:50.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:13:50.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:13:50.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:50.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:13:50.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:13:50.199 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:50.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:13:50.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:13:50.231 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:13:50.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:13:50.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:13:50.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:13:50.285 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:13:50.285 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:13:50.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:50.309 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:13:50.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:13:50.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:50.337 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:13:50.338 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:13:50.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:50.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:50.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:50.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:50.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:50.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:50.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:50.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:50.878 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:13:50.879 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:13:50.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2681495769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:50.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:50.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2681495769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:50.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:50.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:51.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2681495769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:51.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:51.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:51.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:51.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:51.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:51.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:13:51.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:51.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:51.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:13:51.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:51.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:51.644 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:13:51.645 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:13:51.645 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-06-27T21:13:51.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:13:51.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:51 smithi151 ceph-mon[28965]: pgmap v436: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 11304/186765 objects misplaced (6.053%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:13:51.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:51.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:51.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:51.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:51.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:51.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:51.903 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:51.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:52.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:52.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:51 smithi190 ceph-mon[32247]: pgmap v436: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 11304/186765 objects misplaced (6.053%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:13:52.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:51 smithi192 ceph-mon[32956]: pgmap v436: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 11304/186765 objects misplaced (6.053%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:13:52.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:52.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:52.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:52.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:52.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:52.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:52.514 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:52.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:52.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:52.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:52.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:52.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:52.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:52.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:52.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:13:52.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:52.990 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:52.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:53.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:13:53.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:13:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:53.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:53.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:13:53.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:13:53.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:13:53.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:13:53.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:53.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:53.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:13:53.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:13:53.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:13:53.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:53.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:53.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:53.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:53.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:53.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:53.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:53.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:53.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:13:53.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:53.653 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:13:53.653 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:13:53.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:53.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:53.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:53.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:53.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:53.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:53.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:53 smithi151 ceph-mon[28965]: pgmap v437: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 423 op/s; 11304/186012 objects misplaced (6.077%); 778 KiB/s, 12 objects/s recovering 2024-06-27T21:13:53.976 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:53 smithi190 ceph-mon[32247]: pgmap v437: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 423 op/s; 11304/186012 objects misplaced (6.077%); 778 KiB/s, 12 objects/s recovering 2024-06-27T21:13:54.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:54.037 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:53 smithi192 ceph-mon[32956]: pgmap v437: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 423 op/s; 11304/186012 objects misplaced (6.077%); 778 KiB/s, 12 objects/s recovering 2024-06-27T21:13:54.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:54.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:54.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:54.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:54.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:54.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:13:54.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:54.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:54.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:54.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:54.644 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:54.662 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:13:54.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:54.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:54.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:54.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:54.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:54.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:54.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:55.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:55.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:55.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:55.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:55.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:55.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:55.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: pgmap v438: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 11304/179805 objects misplaced (6.287%); 647 KiB/s, 10 objects/s recovering 2024-06-27T21:13:55.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:13:55.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:13:55.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3216075882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:55.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:55.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:55.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:55.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:55 smithi151 ceph-mon[28965]: osdmap e145: 12 total, 12 up, 12 in 2024-06-27T21:13:55.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:55.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:55.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:55.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:55.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:55.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:55.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: pgmap v438: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 11304/179805 objects misplaced (6.287%); 647 KiB/s, 10 objects/s recovering 2024-06-27T21:13:55.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:13:55.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:13:55.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3216075882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:55.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:55.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:55.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:55.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:55 smithi190 ceph-mon[32247]: osdmap e145: 12 total, 12 up, 12 in 2024-06-27T21:13:55.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: pgmap v438: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 11304/179805 objects misplaced (6.287%); 647 KiB/s, 10 objects/s recovering 2024-06-27T21:13:55.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:13:55.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:13:55.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3216075882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:55.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:13:55.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:13:55.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:13:55.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:55 smithi192 ceph-mon[32956]: osdmap e145: 12 total, 12 up, 12 in 2024-06-27T21:13:55.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:55.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:55.947 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:55.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:55.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:56.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:56.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:56.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:56.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:56.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:56.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:56.417 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:13:56.417 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:13:56.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:13:56.514 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:13:56.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:56.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:56.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:56.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:56.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:56.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:56.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:56.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:57.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:57.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:57.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:13:57.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:57.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:57.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:13:57.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:13:57.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:57.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:57.467 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:13:57.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:57 smithi190 ceph-mon[32247]: pgmap v440: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 10629/175884 objects misplaced (6.043%); 2.3 MiB/s, 37 objects/s recovering 2024-06-27T21:13:57.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:57 smithi190 ceph-mon[32247]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:13:57.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:57.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:57.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:57 smithi151 ceph-mon[28965]: pgmap v440: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 10629/175884 objects misplaced (6.043%); 2.3 MiB/s, 37 objects/s recovering 2024-06-27T21:13:57.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:57 smithi151 ceph-mon[28965]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:13:57.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:57.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:13:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:57.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:57.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:57 smithi192 ceph-mon[32956]: pgmap v440: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 10629/175884 objects misplaced (6.043%); 2.3 MiB/s, 37 objects/s recovering 2024-06-27T21:13:57.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:57 smithi192 ceph-mon[32956]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:13:57.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:57.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:57.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:13:57.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:57.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:13:58.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:13:58.079 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-06-27T21:13:55.852134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3842638392,"num_objects":58628,"num_object_clones":0,"num_object_copies":175884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10629,"num_objects_unfound":0,"num_objects_dirty":58628,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":125643,"num_write_kb":5894759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":22768832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130204,"ondisk_log_size":130204,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12109748,"kb_used_data":11342676,"kb_used_omap":18,"kb_used_meta":766829,"kb_avail":1112832076,"statfs":{"total":1151940427776,"available":1139540045824,"internally_reserved":0,"allocated":11614900224,"data_stored":11611382236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":785233210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-322109440,"num_objects":-4915,"num_object_clones":0,"num_object_copies":-14745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10629,"num_objects_unfound":0,"num_objects_dirty":-4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002847"},"pg_stats":[{"pgid":"2.f","version":"144'7893","reported_seq":8027,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:52.132625+0000","last_change":"2024-06-27T21:12:42.522399+0000","last_active":"2024-06-27T21:13:52.132625+0000","last_peered":"2024-06-27T21:13:52.132625+0000","last_clean":"2024-06-27T21:13:52.132625+0000","last_became_active":"2024-06-27T21:11:31.339208+0000","last_became_peered":"2024-06-27T21:11:31.339208+0000","last_unstale":"2024-06-27T21:13:52.132625+0000","last_undegraded":"2024-06-27T21:13:52.132625+0000","last_fullsized":"2024-06-27T21:13:52.132625+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3650","last_scrub_stamp":"2024-06-27T21:12:42.522239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.522239+0000","objects_scrubbed":3627,"log_size":7893,"log_dups_size":0,"ondisk_log_size":7893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:01.818239+0000","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7893,"num_write_kb":374016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"144'7607","reported_seq":7744,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:53.295340+0000","last_change":"2024-06-27T21:12:48.076158+0000","last_active":"2024-06-27T21:13:53.295340+0000","last_peered":"2024-06-27T21:13:53.295340+0000","last_clean":"2024-06-27T21:13:53.295340+0000","last_became_active":"2024-06-27T21:11:31.337998+0000","last_became_peered":"2024-06-27T21:11:31.337998+0000","last_unstale":"2024-06-27T21:13:53.295340+0000","last_undegraded":"2024-06-27T21:13:53.295340+0000","last_fullsized":"2024-06-27T21:13:53.295340+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'3883","last_scrub_stamp":"2024-06-27T21:12:48.075948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:48.075948+0000","objects_scrubbed":3871,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231800832,"num_objects":3537,"num_object_clones":0,"num_object_copies":10611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7607,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"145'7912","reported_seq":8050,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-06-27T21:13:55.079394+0000","last_change":"2024-06-27T21:12:44.116062+0000","last_active":"2024-06-27T21:13:55.079394+0000","last_peered":"2024-06-27T21:13:55.079394+0000","last_clean":"2024-06-27T21:13:55.079394+0000","last_became_active":"2024-06-27T21:11:31.339229+0000","last_became_peered":"2024-06-27T21:11:31.339229+0000","last_unstale":"2024-06-27T21:13:55.079394+0000","last_undegraded":"2024-06-27T21:13:55.079394+0000","last_fullsized":"2024-06-27T21:13:55.079394+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3764","last_scrub_stamp":"2024-06-27T21:12:44.115896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.115896+0000","objects_scrubbed":3741,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:55.572851+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236912664,"num_objects":3616,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7912,"num_write_kb":368833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"144'7920","reported_seq":8054,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:54.283184+0000","last_change":"2024-06-27T21:12:45.902924+0000","last_active":"2024-06-27T21:13:54.283184+0000","last_peered":"2024-06-27T21:13:54.283184+0000","last_clean":"2024-06-27T21:13:54.283184+0000","last_became_active":"2024-06-27T21:11:31.556192+0000","last_became_peered":"2024-06-27T21:11:31.556192+0000","last_unstale":"2024-06-27T21:13:54.283184+0000","last_undegraded":"2024-06-27T21:13:54.283184+0000","last_fullsized":"2024-06-27T21:13:54.283184+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3891","last_scrub_stamp":"2024-06-27T21:12:45.902837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:45.902837+0000","objects_scrubbed":3854,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237764608,"num_objects":3628,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":369536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:48.651587+0000","last_change":"2024-06-27T21:12:50.081819+0000","last_active":"2024-06-27T21:13:48.651587+0000","last_peered":"2024-06-27T21:13:48.651587+0000","last_clean":"2024-06-27T21:13:48.651587+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:13:48.651587+0000","last_undegraded":"2024-06-27T21:13:48.651587+0000","last_fullsized":"2024-06-27T21:13:48.651587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:12:50.081718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:12:50.081718+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:52.027615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"144'7792","reported_seq":7925,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:52.303529+0000","last_change":"2024-06-27T21:12:42.932200+0000","last_active":"2024-06-27T21:13:52.303529+0000","last_peered":"2024-06-27T21:13:52.303529+0000","last_clean":"2024-06-27T21:13:52.303529+0000","last_became_active":"2024-06-27T21:11:31.337613+0000","last_became_peered":"2024-06-27T21:11:31.337613+0000","last_unstale":"2024-06-27T21:13:52.303529+0000","last_undegraded":"2024-06-27T21:13:52.303529+0000","last_fullsized":"2024-06-27T21:13:52.303529+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3620","last_scrub_stamp":"2024-06-27T21:12:42.932079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.932079+0000","objects_scrubbed":3600,"log_size":7792,"log_dups_size":0,"ondisk_log_size":7792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:58.545974+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247857152,"num_objects":3782,"num_object_clones":0,"num_object_copies":11346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7792,"num_write_kb":370368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"144'7756","reported_seq":9089,"reported_epoch":144,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:13:54.424635+0000","last_change":"2024-06-27T21:13:47.955243+0000","last_active":"2024-06-27T21:13:54.424635+0000","last_peered":"2024-06-27T21:13:54.424635+0000","last_clean":"2024-06-27T21:13:45.604588+0000","last_became_active":"2024-06-27T21:13:47.901845+0000","last_became_peered":"2024-06-27T21:13:47.901845+0000","last_unstale":"2024-06-27T21:13:54.424635+0000","last_undegraded":"2024-06-27T21:13:54.424635+0000","last_fullsized":"2024-06-27T21:13:47.640498+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3530","last_scrub_stamp":"2024-06-27T21:12:41.700386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:41.700386+0000","objects_scrubbed":3520,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232259584,"num_objects":3544,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10629,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7756,"num_write_kb":361600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[7,6],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"6,7","objects":1},{"shards":"7,9,10","objects":3543}],"blocked_by":[],"up_primary":11,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"144'8174","reported_seq":8315,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:53.335371+0000","last_change":"2024-06-27T21:12:46.629126+0000","last_active":"2024-06-27T21:13:53.335371+0000","last_peered":"2024-06-27T21:13:53.335371+0000","last_clean":"2024-06-27T21:13:53.335371+0000","last_became_active":"2024-06-27T21:11:31.556127+0000","last_became_peered":"2024-06-27T21:11:31.556127+0000","last_unstale":"2024-06-27T21:13:53.335371+0000","last_undegraded":"2024-06-27T21:13:53.335371+0000","last_fullsized":"2024-06-27T21:13:53.335371+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'4084","last_scrub_stamp":"2024-06-27T21:12:46.629019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:46.629019+0000","objects_scrubbed":4059,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:16.370058+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242089984,"num_objects":3694,"num_object_clones":0,"num_object_copies":11082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8174,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"144'7830","reported_seq":7966,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:52.323503+0000","last_change":"2024-06-27T21:12:47.918243+0000","last_active":"2024-06-27T21:13:52.323503+0000","last_peered":"2024-06-27T21:13:52.323503+0000","last_clean":"2024-06-27T21:13:52.323503+0000","last_became_active":"2024-06-27T21:11:31.337786+0000","last_became_peered":"2024-06-27T21:11:31.337786+0000","last_unstale":"2024-06-27T21:13:52.323503+0000","last_undegraded":"2024-06-27T21:13:52.323503+0000","last_fullsized":"2024-06-27T21:13:52.323503+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'3978","last_scrub_stamp":"2024-06-27T21:12:47.918098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:47.918098+0000","objects_scrubbed":3946,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:06.555581+0000","scrub_duration":897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":372416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"144'7848","reported_seq":7982,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:53.694996+0000","last_change":"2024-06-27T21:12:44.158051+0000","last_active":"2024-06-27T21:13:53.694996+0000","last_peered":"2024-06-27T21:13:53.694996+0000","last_clean":"2024-06-27T21:13:53.694996+0000","last_became_active":"2024-06-27T21:11:31.556366+0000","last_became_peered":"2024-06-27T21:11:31.556366+0000","last_unstale":"2024-06-27T21:13:53.694996+0000","last_undegraded":"2024-06-27T21:13:53.694996+0000","last_fullsized":"2024-06-27T21:13:53.694996+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3740","last_scrub_stamp":"2024-06-27T21:12:44.157914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.157914+0000","objects_scrubbed":3704,"log_size":7848,"log_dups_size":0,"ondisk_log_size":7848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:55.134407+0000","scrub_duration":1005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236978176,"num_objects":3616,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":366848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'10242","reported_seq":10529,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:52.187490+0000","last_change":"2024-06-27T21:13:08.494332+0000","last_active":"2024-06-27T21:13:52.187490+0000","last_peered":"2024-06-27T21:13:52.187490+0000","last_clean":"2024-06-27T21:13:52.187490+0000","last_became_active":"2024-06-27T21:11:31.556322+0000","last_became_peered":"2024-06-27T21:11:31.556322+0000","last_unstale":"2024-06-27T21:13:52.187490+0000","last_undegraded":"2024-06-27T21:13:52.187490+0000","last_fullsized":"2024-06-27T21:13:52.187490+0000","mapping_epoch":104,"log_start":"108'200","ondisk_log_start":"108'200","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7832","last_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_deep_scrub":"129'7832","last_deep_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_clean_scrub_stamp":"2024-06-27T21:13:08.494150+0000","objects_scrubbed":4784,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:33.871589+0000","scrub_duration":14657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242483200,"num_objects":3700,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7658,"num_write_kb":363456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"144'8014","reported_seq":8158,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:53.646779+0000","last_change":"2024-06-27T21:12:56.263993+0000","last_active":"2024-06-27T21:13:53.646779+0000","last_peered":"2024-06-27T21:13:53.646779+0000","last_clean":"2024-06-27T21:13:53.646779+0000","last_became_active":"2024-06-27T21:11:31.338960+0000","last_became_peered":"2024-06-27T21:11:31.338960+0000","last_unstale":"2024-06-27T21:13:53.646779+0000","last_undegraded":"2024-06-27T21:13:53.646779+0000","last_fullsized":"2024-06-27T21:13:53.646779+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4740","last_scrub_stamp":"2024-06-27T21:12:56.263856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:56.263856+0000","objects_scrubbed":4707,"log_size":8014,"log_dups_size":0,"ondisk_log_size":8014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:10.594628+0000","scrub_duration":1215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245497856,"num_objects":3746,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":376320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"145'7801","reported_seq":7939,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-06-27T21:13:55.116510+0000","last_change":"2024-06-27T21:12:44.319499+0000","last_active":"2024-06-27T21:13:55.116510+0000","last_peered":"2024-06-27T21:13:55.116510+0000","last_clean":"2024-06-27T21:13:55.116510+0000","last_became_active":"2024-06-27T21:11:31.943105+0000","last_became_peered":"2024-06-27T21:11:31.943105+0000","last_unstale":"2024-06-27T21:13:55.116510+0000","last_undegraded":"2024-06-27T21:13:55.116510+0000","last_fullsized":"2024-06-27T21:13:55.116510+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3747","last_scrub_stamp":"2024-06-27T21:12:44.319331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.319331+0000","objects_scrubbed":3704,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:13:59.241125+0000","scrub_duration":901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232980480,"num_objects":3555,"num_object_clones":0,"num_object_copies":10665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7801,"num_write_kb":363392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"144'7800","reported_seq":7943,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:54.366697+0000","last_change":"2024-06-27T21:12:55.073348+0000","last_active":"2024-06-27T21:13:54.366697+0000","last_peered":"2024-06-27T21:13:54.366697+0000","last_clean":"2024-06-27T21:13:54.366697+0000","last_became_active":"2024-06-27T21:11:31.556186+0000","last_became_peered":"2024-06-27T21:11:31.556186+0000","last_unstale":"2024-06-27T21:13:54.366697+0000","last_undegraded":"2024-06-27T21:13:54.366697+0000","last_fullsized":"2024-06-27T21:13:54.366697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4512","last_scrub_stamp":"2024-06-27T21:12:55.073215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:55.073215+0000","objects_scrubbed":4477,"log_size":7800,"log_dups_size":0,"ondisk_log_size":7800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:03.225629+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235274240,"num_objects":3590,"num_object_clones":0,"num_object_copies":10770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":364480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"144'7850","reported_seq":8006,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:53.700768+0000","last_change":"2024-06-27T21:13:48.695746+0000","last_active":"2024-06-27T21:13:53.700768+0000","last_peered":"2024-06-27T21:13:53.700768+0000","last_clean":"2024-06-27T21:13:53.700768+0000","last_became_active":"2024-06-27T21:11:31.337888+0000","last_became_peered":"2024-06-27T21:11:31.337888+0000","last_unstale":"2024-06-27T21:13:53.700768+0000","last_undegraded":"2024-06-27T21:13:53.700768+0000","last_fullsized":"2024-06-27T21:13:53.700768+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3700","last_scrub_stamp":"2024-06-27T21:12:43.447399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:43.447399+0000","objects_scrubbed":2099,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244449280,"num_objects":3730,"num_object_clones":0,"num_object_copies":11190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":370560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"144'7828","reported_seq":7961,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:54.307732+0000","last_change":"2024-06-27T21:12:42.976967+0000","last_active":"2024-06-27T21:13:54.307732+0000","last_peered":"2024-06-27T21:13:54.307732+0000","last_clean":"2024-06-27T21:13:54.307732+0000","last_became_active":"2024-06-27T21:11:31.340369+0000","last_became_peered":"2024-06-27T21:11:31.340369+0000","last_unstale":"2024-06-27T21:13:54.307732+0000","last_undegraded":"2024-06-27T21:13:54.307732+0000","last_fullsized":"2024-06-27T21:13:54.307732+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3653","last_scrub_stamp":"2024-06-27T21:12:42.976769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.976769+0000","objects_scrubbed":3623,"log_size":7828,"log_dups_size":0,"ondisk_log_size":7828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234618880,"num_objects":3580,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7828,"num_write_kb":365056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"144'10245","reported_seq":10552,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-06-27T21:13:52.163395+0000","last_change":"2024-06-27T21:13:29.274450+0000","last_active":"2024-06-27T21:13:52.163395+0000","last_peered":"2024-06-27T21:13:52.163395+0000","last_clean":"2024-06-27T21:13:52.163395+0000","last_became_active":"2024-06-27T21:11:31.337915+0000","last_became_peered":"2024-06-27T21:11:31.337915+0000","last_unstale":"2024-06-27T21:13:52.163395+0000","last_undegraded":"2024-06-27T21:13:52.163395+0000","last_fullsized":"2024-06-27T21:13:52.163395+0000","mapping_epoch":104,"log_start":"108'200","ondisk_log_start":"108'200","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9000","last_scrub_stamp":"2024-06-27T21:13:29.274287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:13:29.274287+0000","objects_scrubbed":5002,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:47.960430+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242810880,"num_objects":3705,"num_object_clones":0,"num_object_copies":11115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7837,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3842048024,"num_objects":58626,"num_object_clones":0,"num_object_copies":175878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10629,"num_objects_unfound":0,"num_objects_dirty":58626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125520,"num_write_kb":5892609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11519602688,"data_stored":11519590472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130112,"ondisk_log_size":130112,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252919,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775156,"kb_used_data":703724,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92969996,"statfs":{"total":95995035648,"available":95201275904,"internally_reserved":0,"allocated":720613376,"data_stored":720318546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764076,"kb_used_data":706532,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92981076,"statfs":{"total":95995035648,"available":95212621824,"internally_reserved":0,"allocated":723488768,"data_stored":723197510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743488,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1464112,"kb_used_data":1396840,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92281040,"statfs":{"total":95995035648,"available":94495784960,"internally_reserved":0,"allocated":1430364160,"data_stored":1430068830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972420,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043692,"kb_used_data":963940,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":92701460,"statfs":{"total":95995035648,"available":94926295040,"internally_reserved":0,"allocated":987074560,"data_stored":986783302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266761,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240692,"kb_used_data":1173420,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92504460,"statfs":{"total":95995035648,"available":94724567040,"internally_reserved":0,"allocated":1201582080,"data_stored":1201287270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528397,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":514612,"kb_used_data":469484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93230540,"statfs":{"total":95995035648,"available":95468072960,"internally_reserved":0,"allocated":480751616,"data_stored":480456806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533356,"kb_used_data":489700,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93211796,"statfs":{"total":95995035648,"available":95448879104,"internally_reserved":0,"allocated":501452800,"data_stored":501161542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986264,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775280,"kb_used_data":721896,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92969872,"statfs":{"total":95995035648,"available":95201148928,"internally_reserved":0,"allocated":739221504,"data_stored":738926174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1502192,"kb_used_data":1430760,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92242960,"statfs":{"total":95995035648,"available":94456791040,"internally_reserved":0,"allocated":1465098240,"data_stored":1464802910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015596,"kb_used_data":944100,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92729556,"statfs":{"total":95995035648,"available":94955065344,"internally_reserved":0,"allocated":966758400,"data_stored":966467142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1262188,"kb_used_data":1190756,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92482964,"statfs":{"total":95995035648,"available":94702555136,"internally_reserved":0,"allocated":1219334144,"data_stored":1219042886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218796,"kb_used_data":1151524,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92526356,"statfs":{"total":95995035648,"available":94746988544,"internally_reserved":0,"allocated":1179160576,"data_stored":1178869318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1162149888,"data_stored":1162149888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1212547072,"data_stored":1212547072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":958464000,"data_stored":958464000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1460604928,"data_stored":1460600856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":732958720,"data_stored":732954648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":497418240,"data_stored":497418240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":474611712,"data_stored":474611712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1187774464,"data_stored":1187774464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":983433216,"data_stored":983433216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1419120640,"data_stored":1419116568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":721616896,"data_stored":721616896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":708902912,"data_stored":708902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:13:58.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-06-27T21:13:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:58.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:13:58.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:58.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:58.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:13:58.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:13:58.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:13:58.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:13:58.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:13:58.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:13:58.641 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:13:58.651 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:13:58.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:13:58.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:13:58.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:13:58.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:58.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:13:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:13:58.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:58.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:13:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:13:58.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:13:58.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:13:58.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:13:58.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:13:58.852 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:13:58.852 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:13:58.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:13:58.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:13:58.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:58.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:58.903 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:13:58.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:13:58.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:13:58.926 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:13:58.954 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:13:58.954 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:13:58.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:13:58.981 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:13:58.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:59.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:59.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:13:59.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:13:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:13:59.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:13:59.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:13:59.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:13:59.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:13:59.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:13:59.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:59 smithi190 ceph-mon[32247]: pgmap v441: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 10629/172152 objects misplaced (6.174%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:13:59.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:13:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3674026104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:59.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:13:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:13:59.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:13:59.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:59 smithi151 ceph-mon[28965]: pgmap v441: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 10629/172152 objects misplaced (6.174%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:13:59.638 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:13:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3674026104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:59.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:13:59.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:59.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:13:59.785 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:13:59.785 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:13:59.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.785 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:130508 2.8 2:1823ead2:::benchmark_data_smithi192_59818_object38433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-27T21:13:59.785 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.780415+0000", 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.010880885, 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010908806, 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130508 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.780415+0000", 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.780415+0000", 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.780416+0000", 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1829999999999999e-06 2024-06-27T21:13:59.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.780421+0000", 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8679999999999998e-06 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.780433+0000", 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1147e-05 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.780439+0000", 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4729999999999997e-06 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786300+0000", 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058609700000000001 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786400+0000", 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100295 2024-06-27T21:13:59.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,9", 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786492+0000", 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2058000000000005e-05 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:130509 2.8 2:1bf3855e:::benchmark_data_smithi192_59818_object38434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.781337+0000", 2024-06-27T21:13:59.790 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0099591309999999995, 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010052349, 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130509 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.781337+0000", 2024-06-27T21:13:59.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.781337+0000", 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.781338+0000", 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2600000000000002e-07 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.781341+0000", 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3680000000000002e-06 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.781350+0000", 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4689999999999997e-06 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.781358+0000", 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4919999999999993e-06 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790184+0000", 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088257000000000006 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790310+0000", 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012597500000000001 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,9", 2024-06-27T21:13:59.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790410+0000", 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9726e-05 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:130510 2.8 2.e170b3f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.782947+0000", 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0083491799999999994, 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084973719999999996, 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.795 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130510 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.782947+0000", 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.782947+0000", 2024-06-27T21:13:59.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.782948+0000", 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9800000000000003e-07 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.782953+0000", 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0710000000000002e-06 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.782965+0000", 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2490999999999999e-05 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.782974+0000", 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8920000000000006e-06 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.798 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:130483 2.0 2.86879680 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.786319+0000", 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0049768449999999997, 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051669009999999998, 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130483 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786319+0000", 2024-06-27T21:13:59.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:59 smithi192 ceph-mon[32956]: pgmap v441: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 10629/172152 objects misplaced (6.174%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:13:59.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:13:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3674026104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786319+0000", 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786320+0000", 2024-06-27T21:13:59.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0950000000000001e-06 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786325+0000", 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.583e-06 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786334+0000", 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3389999999999999e-06 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786341+0000", 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9879999999999998e-06 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(2.0 145/142 [PushReplyOp(2:08ea422c:::benchmark_data_smithi192_59818_object38391:head)])", 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.754844+0000", 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.036452058000000002, 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.036684686000000001, 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.803 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.754844+0000", 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.754844+0000", 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.754845+0000", 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5099999999999998e-07 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.754852+0000", 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.267e-06 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.754893+0000", 2024-06-27T21:13:59.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0864e-05 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.754904+0000", 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0570999999999999e-05 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_backoff(2.0 ack-block id 89 [2:0758639c:::benchmark_data_smithi192_59818_object38424:head,2:0758639c:::benchmark_data_smithi192_59818_object38424:head) e145)", 2024-06-27T21:13:59.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.786452+0000", 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0048444910000000002, 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051150869999999999, 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 0 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786452+0000", 2024-06-27T21:13:59.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786452+0000", 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786452+0000", 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2099999999999996e-07 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786457+0000", 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5310000000000003e-06 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786485+0000", 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8065999999999999e-05 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786587+0000", 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000102081 2024-06-27T21:13:59.809 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:130513 2.9 e145/104 2:9e956796:::benchmark_data_smithi192_59818_object38438:head v 145'8174, mlcod=145'8172)", 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.789653+0000", 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0016433750000000001, 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019568229999999999, 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789653+0000", 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789653+0000", 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789654+0000", 2024-06-27T21:13:59.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0640000000000001e-06 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789661+0000", 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8379999999999996e-06 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789692+0000", 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0787000000000002e-05 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789699+0000", 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.153e-06 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789753+0000", 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4172999999999998e-05 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789775+0000", 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1985999999999999e-05 2024-06-27T21:13:59.813 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:130427 2.0 2:07c0d4f4:::benchmark_data_smithi192_59818_object38352:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.756950+0000", 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.034346178999999998, 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.034707373, 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:13:59.814 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130427 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.756950+0000", 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.756950+0000", 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.756951+0000", 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3600000000000002e-07 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.756956+0000", 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5380000000000002e-06 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.756969+0000", 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2223e-05 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.756977+0000", 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:59.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.767198+0000", 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010221317000000001 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.767335+0000", 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013736 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,8,11", 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.767445+0000", 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109293 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.783413+0000", 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015968830999999999 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.783474+0000", 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0463e-05 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.786196+0000", 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027217809999999999 2024-06-27T21:13:59.819 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(2.0 145/142 [PushReplyOp(2:0758639c:::benchmark_data_smithi192_59818_object38424:head)])", 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.789703+0000", 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.001593484, 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020218369999999999, 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:13:59.820 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789703+0000", 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789703+0000", 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789703+0000", 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.75e-07 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789709+0000", 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6550000000000001e-06 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789739+0000", 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0386e-05 2024-06-27T21:13:59.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789747+0000", 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2830000000000004e-06 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790175+0000", 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042862999999999999 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(2.0 145/142 [PushReplyOp(2:08ea422c:::benchmark_data_smithi192_59818_object38391:head)])", 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.757235+0000", 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.034061069999999999, 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.034528577999999997, 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.757235+0000", 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.757235+0000", 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.757236+0000", 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.47e-07 2024-06-27T21:13:59.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.757241+0000", 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9919999999999998e-06 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.757270+0000", 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9210999999999999e-05 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.757279+0000", 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2669999999999995e-06 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:130508 2.8 e145/104)", 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.789936+0000", 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0013600249999999999, 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018606670000000001, 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.827 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789936+0000", 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789936+0000", 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789937+0000", 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1099999999999995e-07 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789944+0000", 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0029999999999997e-06 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789972+0000", 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8467e-05 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.789979+0000", 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.545e-06 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(2.0 145/142 [PushReplyOp(2:0758639c:::benchmark_data_smithi192_59818_object38424:head)])", 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.790280+0000", 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0010163819999999999, 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015628599999999999, 2024-06-27T21:13:59.830 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790280+0000", 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790280+0000", 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790280+0000", 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9100000000000003e-07 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790285+0000", 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9880000000000004e-06 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790310+0000", 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4862e-05 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790316+0000", 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0820000000000004e-06 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.833 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:130508 2.8 e145/104)", 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.790428+0000", 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.00086799499999999999, 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001447804, 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790428+0000", 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790428+0000", 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790429+0000", 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9900000000000001e-07 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790432+0000", 2024-06-27T21:13:59.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.14e-06 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790460+0000", 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8206999999999999e-05 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.790465+0000", 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0200000000000002e-06 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(2.0 145/142 [PushReplyOp(2:08ea422c:::benchmark_data_smithi192_59818_object38391:head)])", 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:59.761724+0000", 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.029572068, 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.030187818000000002, 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:13:59.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.761724+0000", 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.761724+0000", 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.761725+0000", 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3399999999999997e-07 2024-06-27T21:13:59.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.761731+0000", 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6380000000000001e-06 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.761758+0000", 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6811000000000001e-05 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:13:59.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:59.761765+0000", 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6809999999999998e-06 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 14 2024-06-27T21:13:59.840 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:13:59.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:13:59.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:13:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:13:59.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:00.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:00.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:00.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:00.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:00.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:00.283 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:00.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:00.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:00.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:00.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:00.542 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:00.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:14:00.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:00.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:00.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:00.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:00.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:00.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:00.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:00.926 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:14:01.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:01.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:01.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:01.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:01.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:01.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:01.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:01.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:01.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:01.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:01 smithi192 ceph-mon[32956]: pgmap v442: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 9960/165357 objects misplaced (6.023%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:01.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:01.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:01.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:01.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:14:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:01.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:01 smithi151 ceph-mon[28965]: pgmap v442: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 9960/165357 objects misplaced (6.023%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:01.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:01 smithi190 ceph-mon[32247]: pgmap v442: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 9960/165357 objects misplaced (6.023%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:01.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:01.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:01.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:01.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:01.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:02.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:02.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:02.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:02.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:02.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:02.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:02.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:02.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:02.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:02.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:02.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.712 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:02.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:02.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:02.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:02.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:02.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:02.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:02.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:03.056 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:14:03.056 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:14:03.056 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:14:03.056 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:14:03.056 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:14:03.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:03.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:03.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:03.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:03.247 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:03.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:03.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:03.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:03.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:03.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:03 smithi151 ceph-mon[28965]: pgmap v443: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 9960/163692 objects misplaced (6.085%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:03.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:03.605 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:03 smithi190 ceph-mon[32247]: pgmap v443: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 9960/163692 objects misplaced (6.085%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:03.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:03.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:03.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:03.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:03 smithi192 ceph-mon[32956]: pgmap v443: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 9960/163692 objects misplaced (6.085%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:03.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:14:03.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:03.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:03.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:03.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:04.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:04.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:04.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:04.166 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:14:04.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:04.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:04.350 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 2 pgp_num to 16 2024-06-27T21:14:04.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:04.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:04.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:04.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:04.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:04.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:04.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4282740865' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:14:04.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:04.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:04.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:04.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:04.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4282740865' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:14:04.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:04.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:04.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:04.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:04.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4282740865' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:14:04.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:04.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:04.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:05.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:05.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:05.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:05.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:05.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:05.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:05.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:05.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:05.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:05.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:05.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:05.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:05 smithi151 ceph-mon[28965]: pgmap v444: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 9960/156885 objects misplaced (6.349%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4282740865' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:14:05.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:05 smithi151 ceph-mon[28965]: osdmap e146: 12 total, 12 up, 12 in 2024-06-27T21:14:05.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:05 smithi190 ceph-mon[32247]: pgmap v444: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 9960/156885 objects misplaced (6.349%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:05.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4282740865' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:14:05.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:05 smithi190 ceph-mon[32247]: osdmap e146: 12 total, 12 up, 12 in 2024-06-27T21:14:05.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:05.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:05 smithi192 ceph-mon[32956]: pgmap v444: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 9960/156885 objects misplaced (6.349%); 2.8 MiB/s, 44 objects/s recovering 2024-06-27T21:14:05.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4282740865' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:14:05.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:05 smithi192 ceph-mon[32956]: osdmap e146: 12 total, 12 up, 12 in 2024-06-27T21:14:05.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:05.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:05.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:06.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:06.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:06.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:06.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:06.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:06.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:14:06.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:14:06.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:06.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:14:06.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:14:06.350 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:14:06.351 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:14:06.364 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:14:06.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:14:06.377 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:14:06.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:06.409 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:14:06.410 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:14:06.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:06.433 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:14:06.433 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:14:06.455 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:14:06.455 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:14:06.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:14:06.480 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:14:06.505 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:14:06.505 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:14:06.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1328194311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:06.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:06 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:06.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3443519672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:06.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:06.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:14:06.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:14:06.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:06.562 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:14:06.562 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:14:06.563 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1328194311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:06.563 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:06 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:06.563 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3443519672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:06.593 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:14:06.593 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:14:06.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:06.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:06.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:06.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1328194311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:06.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:06 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:06.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3443519672' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:06.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:06.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:06.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:06.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:07.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:07.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:07.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:07.189 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:07.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:07.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:07.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:07.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:07.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:07.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:07.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:07.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:07 smithi151 ceph-mon[28965]: pgmap v446: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 9238/152670 objects misplaced (6.051%); 2.9 MiB/s, 46 objects/s recovering 2024-06-27T21:14:07.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:07 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:07.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:07 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:14:07.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:07 smithi151 ceph-mon[28965]: osdmap e147: 12 total, 12 up, 12 in 2024-06-27T21:14:07.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:07.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:07 smithi190 ceph-mon[32247]: pgmap v446: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 9238/152670 objects misplaced (6.051%); 2.9 MiB/s, 46 objects/s recovering 2024-06-27T21:14:07.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:07 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:07.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:07 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:14:07.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:07 smithi190 ceph-mon[32247]: osdmap e147: 12 total, 12 up, 12 in 2024-06-27T21:14:07.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:07 smithi192 ceph-mon[32956]: pgmap v446: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 9238/152670 objects misplaced (6.051%); 2.9 MiB/s, 46 objects/s recovering 2024-06-27T21:14:07.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:07 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:07.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:07 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:14:07.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:07 smithi192 ceph-mon[32956]: osdmap e147: 12 total, 12 up, 12 in 2024-06-27T21:14:07.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:07.837 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:07.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:07.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:07.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:08.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:08.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:08.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:08.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:08.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:08.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:08.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:08.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:08.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:08.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:08.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:08 smithi151 ceph-mon[28965]: 2.2 scrub starts 2024-06-27T21:14:08.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:08 smithi151 ceph-mon[28965]: 2.d scrub starts 2024-06-27T21:14:08.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:08.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:08.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:08 smithi190 ceph-mon[32247]: 2.2 scrub starts 2024-06-27T21:14:08.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:08 smithi190 ceph-mon[32247]: 2.d scrub starts 2024-06-27T21:14:08.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:08 smithi192 ceph-mon[32956]: 2.2 scrub starts 2024-06-27T21:14:08.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:08 smithi192 ceph-mon[32956]: 2.d scrub starts 2024-06-27T21:14:08.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:08.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:08.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:08.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:08.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:09.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:09.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:09.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:09.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:09.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:09.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:09.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:09.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:14:09.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:09.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:09.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:09.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:09.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:09 smithi151 ceph-mon[28965]: pgmap v448: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 9238/149529 objects misplaced (6.178%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:09.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:09 smithi151 ceph-mon[28965]: 2.2 scrub ok 2024-06-27T21:14:09.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:09 smithi151 ceph-mon[28965]: 2.d scrub ok 2024-06-27T21:14:09.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:09.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:09.712 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:09.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:09.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:09 smithi190 ceph-mon[32247]: pgmap v448: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 9238/149529 objects misplaced (6.178%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:09.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:09 smithi190 ceph-mon[32247]: 2.2 scrub ok 2024-06-27T21:14:09.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:09 smithi190 ceph-mon[32247]: 2.d scrub ok 2024-06-27T21:14:09.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:09.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:09.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:09.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:09 smithi192 ceph-mon[32956]: pgmap v448: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 9238/149529 objects misplaced (6.178%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:09.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:09 smithi192 ceph-mon[32956]: 2.2 scrub ok 2024-06-27T21:14:09.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:09 smithi192 ceph-mon[32956]: 2.d scrub ok 2024-06-27T21:14:09.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:09.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:09.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:14:09.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:09.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:09.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:09.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:10.008 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:14:10.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:14:10.008 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-27T21:14:10.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:14:10.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:10.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:10.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:10.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:10.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:10.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:10.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:10.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:10 smithi151 ceph-mon[28965]: osdmap e148: 12 total, 12 up, 12 in 2024-06-27T21:14:10.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:10.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:10.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:10.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:10.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:10 smithi190 ceph-mon[32247]: osdmap e148: 12 total, 12 up, 12 in 2024-06-27T21:14:10.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:10.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:10.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:10 smithi192 ceph-mon[32956]: osdmap e148: 12 total, 12 up, 12 in 2024-06-27T21:14:10.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:10.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:10.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:11.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:11.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:11.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:11.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:11.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:11.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:11.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:11.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:11.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:11.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:11.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:11.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:11.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:11 smithi151 ceph-mon[28965]: pgmap v450: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.9 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 8637/143856 objects misplaced (6.004%); 4.6 MiB/s, 73 objects/s recovering 2024-06-27T21:14:11.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:11 smithi190 ceph-mon[32247]: pgmap v450: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.9 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 8637/143856 objects misplaced (6.004%); 4.6 MiB/s, 73 objects/s recovering 2024-06-27T21:14:11.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:11.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:11 smithi192 ceph-mon[32956]: pgmap v450: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.9 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 8637/143856 objects misplaced (6.004%); 4.6 MiB/s, 73 objects/s recovering 2024-06-27T21:14:11.788 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:11.791 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:11.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:11.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:11.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:11.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:12.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:12.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:12.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:12.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:12.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:12.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:12.364 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:12.364 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:12.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:12.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:12.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:12 smithi190 ceph-mon[32247]: 2.1 scrub starts 2024-06-27T21:14:12.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:12 smithi190 ceph-mon[32247]: 2.9 scrub starts 2024-06-27T21:14:12.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:12.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:12.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:12.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:12 smithi151 ceph-mon[28965]: 2.1 scrub starts 2024-06-27T21:14:12.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:12 smithi151 ceph-mon[28965]: 2.9 scrub starts 2024-06-27T21:14:12.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:12.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:12 smithi192 ceph-mon[32956]: 2.1 scrub starts 2024-06-27T21:14:12.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:12 smithi192 ceph-mon[32956]: 2.9 scrub starts 2024-06-27T21:14:12.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:12.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:12.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:12.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:12.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:13.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:13.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:13.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:13.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:13.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:14:13.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:13.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:13.429 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:14:13.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:13.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:13.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:13.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:13.628 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:13 smithi151 ceph-mon[28965]: 2.1 scrub ok 2024-06-27T21:14:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:13 smithi151 ceph-mon[28965]: pgmap v451: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 8637/142611 objects misplaced (6.056%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:14:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:13 smithi151 ceph-mon[28965]: 2.4 scrub starts 2024-06-27T21:14:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:13 smithi151 ceph-mon[28965]: 2.e scrub starts 2024-06-27T21:14:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1431641474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:13 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:13.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:13.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:13 smithi190 ceph-mon[32247]: 2.1 scrub ok 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:13 smithi190 ceph-mon[32247]: pgmap v451: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 8637/142611 objects misplaced (6.056%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:13 smithi190 ceph-mon[32247]: 2.4 scrub starts 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:13 smithi190 ceph-mon[32247]: 2.e scrub starts 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1431641474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:13 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:13.757 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:13 smithi192 ceph-mon[32956]: 2.1 scrub ok 2024-06-27T21:14:13.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:13 smithi192 ceph-mon[32956]: pgmap v451: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 8637/142611 objects misplaced (6.056%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:14:13.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:13 smithi192 ceph-mon[32956]: 2.4 scrub starts 2024-06-27T21:14:13.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:13 smithi192 ceph-mon[32956]: 2.e scrub starts 2024-06-27T21:14:13.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1431641474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:13.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:13 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:13.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:13.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:13.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:13.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:13.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:14.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:14.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:14.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:14.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:14.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:14.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:14.436 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 150, 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:14:14.431066+0000", 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:09:37.640435+0000", 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:09:20.227260+0000", 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:14:14.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5799936, 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 2, 2024-06-27T21:14:14.438 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:14:14.439 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 1, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 1, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 1, 2024-06-27T21:14:14.440 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 1, 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 1, 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "21", 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:14:14.441 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:14:14.442 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:14:14.443 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:14:14.444 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 12, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 12, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.25, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 3, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 3, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-06-27T21:14:14.445 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 2, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_0", 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:11:29.737013+0000", 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:14:14.446 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:14:14.447 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "146", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:14:14.448 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:14:14.449 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:14:14.450 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.25, 2024-06-27T21:14:14.451 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.25, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.25, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.25, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:14:14.452 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 53, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.453 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.454 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.455 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.457 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.458 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.459 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.460 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.461 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:14:14.464 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.465 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:14:14.466 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.467 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.469 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:14:14.470 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.471 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:14:14.472 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:14:14.473 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.474 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:14:14.476 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:14:14.477 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 31, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.478 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.479 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:14:14.482 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/2107645547", 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/2107645547", 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/2107645547", 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/2107645547", 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.483 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.484 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.485 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.486 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.487 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:14:14.488 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.489 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.490 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:14:14.491 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:14:14.494 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.495 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:14:14.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:14.496 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.497 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 142, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:14:14.498 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.499 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:14:14.502 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.503 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:14:14.504 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:14:14.505 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.508 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.509 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:14:14.510 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:14:14.511 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:14:14.514 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.515 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:14:14.516 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.518 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:14:14.521 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 141, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:14:14.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.524 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.525 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:14:14.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:13.789023+0000", 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.528 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:33.739899+0000", 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.529 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:14.985857+0000", 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:14:03.511794+0000", 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:24.358917+0000", 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.531 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:23.314643+0000", 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.532 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:14:08.596918+0000", 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:20.352174+0000", 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:10.920857+0000", 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.534 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:14:14.329055+0000", 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.535 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:54.229058+0000", 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:13:17.219093+0000", 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:14:14.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "2.0", 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: 8, 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: 6 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:14:14.537 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "2.0", 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: 6 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:14:14.538 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:14:14.539 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:14:14.540 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:14:14.541 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:14:14.541 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:14:14.541 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:14:14.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:14:14.541 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:14:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:14.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:14.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: 2.9 scrub ok 2024-06-27T21:14:14.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: 2.4 scrub ok 2024-06-27T21:14:14.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:14:14.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:14:14.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:14.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: osdmap e149: 12 total, 12 up, 12 in 2024-06-27T21:14:14.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:14 smithi151 ceph-mon[28965]: 2.e scrub ok 2024-06-27T21:14:14.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:14.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:14.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: 2.9 scrub ok 2024-06-27T21:14:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: 2.4 scrub ok 2024-06-27T21:14:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:14:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:14:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: osdmap e149: 12 total, 12 up, 12 in 2024-06-27T21:14:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:14 smithi190 ceph-mon[32247]: 2.e scrub ok 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: 2.9 scrub ok 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: 2.4 scrub ok 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: osdmap e149: 12 total, 12 up, 12 in 2024-06-27T21:14:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:14 smithi192 ceph-mon[32956]: 2.e scrub ok 2024-06-27T21:14:14.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:14.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:14.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:14.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:14.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:15.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:15.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:15.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:15.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:15.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:15.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:15.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:15.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:15.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:15 smithi192 ceph-mon[32956]: pgmap v453: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 15 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 8637/135801 objects misplaced (6.360%); 1.7 MiB/s, 26 objects/s recovering 2024-06-27T21:14:15.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:15.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:15.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:15 smithi192 ceph-mon[32956]: osdmap e150: 12 total, 12 up, 12 in 2024-06-27T21:14:15.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1769160356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:14:15.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:15.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:15.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:15 smithi151 ceph-mon[28965]: pgmap v453: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 15 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 8637/135801 objects misplaced (6.360%); 1.7 MiB/s, 26 objects/s recovering 2024-06-27T21:14:15.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:15.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:15.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:15 smithi151 ceph-mon[28965]: osdmap e150: 12 total, 12 up, 12 in 2024-06-27T21:14:15.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1769160356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:14:15.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:15.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:15.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:15 smithi190 ceph-mon[32247]: pgmap v453: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 15 active+clean; 2.8 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 8637/135801 objects misplaced (6.360%); 1.7 MiB/s, 26 objects/s recovering 2024-06-27T21:14:15.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:15.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:15.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:15 smithi190 ceph-mon[32247]: osdmap e150: 12 total, 12 up, 12 in 2024-06-27T21:14:15.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1769160356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:14:15.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:15.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:15.904 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:15.905 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:15.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:15.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:15.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:16.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:16.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:16.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:16.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:16.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:16.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:16.385 INFO:tasks.thrashosds.thrasher:j is {'epoch': 150, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:14:14.431066+0000', 'last_up_change': '2024-06-27T21:09:37.640435+0000', 'last_in_change': '2024-06-27T21:09:20.227260+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-06-27T21:11:29.737013+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 2107645547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 2107645547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 2107645547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 2107645547}]}, 'public_addr': '172.21.15.190:6801/2107645547', 'cluster_addr': '172.21.15.190:6803/2107645547', 'heartbeat_back_addr': '172.21.15.190:6807/2107645547', 'heartbeat_front_addr': '172.21.15.190:6805/2107645547', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 141, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:13.789023+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:33.739899+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:14.985857+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:14:03.511794+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:24.358917+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:23.314643+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:14:08.596918+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:20.352174+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:10.920857+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:14:14.329055+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:54.229058+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:13:17.219093+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [11, 8, 6]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.0', 'osds': [7, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:14:16.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:14:16.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:16.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:16.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:16 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:16.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:16 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:16.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:16 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:16.537 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:16 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:16.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:16.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:16.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:14:16.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:16 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:16.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:16 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:16.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:16.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:16.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:16.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:16.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:16.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:17.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:17.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:17.033 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:17.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:17.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:17.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:17.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:17.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:17.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:17.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:17.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:17.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:17.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:17.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:14:17.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:17 smithi151 ceph-mon[28965]: pgmap v455: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 8023/131901 objects misplaced (6.083%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:14:17.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:17 smithi151 ceph-mon[28965]: osdmap e151: 12 total, 12 up, 12 in 2024-06-27T21:14:17.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:17.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:17 smithi190 ceph-mon[32247]: pgmap v455: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 8023/131901 objects misplaced (6.083%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:14:17.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:17 smithi190 ceph-mon[32247]: osdmap e151: 12 total, 12 up, 12 in 2024-06-27T21:14:17.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:17 smithi192 ceph-mon[32956]: pgmap v455: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 15 active+clean; 2.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 8023/131901 objects misplaced (6.083%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:14:17.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:17 smithi192 ceph-mon[32956]: osdmap e151: 12 total, 12 up, 12 in 2024-06-27T21:14:17.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:17.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:17.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:18.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:18.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:18.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:18.168 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:18.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:18.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:14:18.204 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:14:18.221 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:18.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:14:18.247 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:14:18.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:14:18.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:14:18.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:18.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:14:18.297 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:14:18.302 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:18.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:14:18.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:14:18.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:14:18.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:14:18.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:14:18.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:14:18.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:18.421 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:14:18.421 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:14:18.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:18.454 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:14:18.455 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:14:18.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:14:18.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:14:18.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:18 smithi192 ceph-mon[32956]: 2.3 scrub starts 2024-06-27T21:14:18.511 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2435815059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:18.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:14:18.520 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:14:18.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:18.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:14:18.549 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:14:18.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:18 smithi151 ceph-mon[28965]: 2.3 scrub starts 2024-06-27T21:14:18.549 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2435815059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:18.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:18.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:18.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:18.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:18 smithi190 ceph-mon[32247]: 2.3 scrub starts 2024-06-27T21:14:18.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2435815059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:18.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:18.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:18.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:18.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:19.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:19.018 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:19.018 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:14:19.018 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:14:19.019 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:14:19.019 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:19.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:19.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:19.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:19.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:19.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:19.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:19.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:19.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:19.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:19.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:19 smithi151 ceph-mon[28965]: pgmap v457: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 15 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 8023/128670 objects misplaced (6.235%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:14:19.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:19 smithi151 ceph-mon[28965]: 2.3 scrub ok 2024-06-27T21:14:19.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:19.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:19 smithi192 ceph-mon[32956]: pgmap v457: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 15 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 8023/128670 objects misplaced (6.235%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:14:19.661 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:19 smithi192 ceph-mon[32956]: 2.3 scrub ok 2024-06-27T21:14:19.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:14:19.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:19.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:19 smithi190 ceph-mon[32247]: pgmap v457: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 15 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 774 op/s; 8023/128670 objects misplaced (6.235%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:14:19.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:19 smithi190 ceph-mon[32247]: 2.3 scrub ok 2024-06-27T21:14:19.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:19.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:19.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:19.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:20.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:20.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:20.043 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:20.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:20.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:20.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:20.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:20.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:20.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:20.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:20.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:20.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:20.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:20.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:20.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:20.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:20.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:20.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:21.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:21.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:21.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:21.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:21.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:21.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:21.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:21.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:21.589 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:21 smithi192 ceph-mon[32956]: pgmap v458: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 7293/122544 objects misplaced (5.951%); 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:14:21.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:21.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:14:21.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:21.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:21 smithi151 ceph-mon[28965]: pgmap v458: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 7293/122544 objects misplaced (5.951%); 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:14:21.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:21 smithi190 ceph-mon[32247]: pgmap v458: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 7293/122544 objects misplaced (5.951%); 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:14:21.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:21.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:21.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:21.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:21.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:21.995 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:21.995 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:14:21.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:915 2.b 2:d2e05b7a:::benchmark_data_smithi192_59818_object914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:46.543532+0000", 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 155.45727025299999, 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.210178481, 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:21.996 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 915 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543532+0000", 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543532+0000", 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:21.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543533+0000", 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2399999999999996e-07 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543550+0000", 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6626999999999999e-05 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543562+0000", 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2191000000000001e-05 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543570+0000", 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7179999999999996e-06 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543621+0000", 2024-06-27T21:14:21.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1082000000000003e-05 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543740+0000", 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119135 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543840+0000", 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9821999999999994e-05 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.544356+0000", 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051664300000000001 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.544567+0000", 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021065999999999999 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753592+0000", 2024-06-27T21:14:22.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20902500399999999 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753653+0000", 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0615000000000002e-05 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753711+0000", 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8060000000000003e-05 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:953 2.b 2:dbebfa1b:::benchmark_data_smithi192_59818_object952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:46.554690+0000", 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 155.44611210400001, 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19913958200000001, 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 953 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554690+0000", 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554690+0000", 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554691+0000", 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1909999999999999e-06 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554715+0000", 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3414999999999999e-05 2024-06-27T21:14:22.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554727+0000", 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2149e-05 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554735+0000", 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2009999999999999e-06 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554796+0000", 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0346000000000001e-05 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.554913+0000", 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011780700000000001 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.555046+0000", 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013249099999999999 2024-06-27T21:14:22.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.555429+0000", 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000383538 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.556024+0000", 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000594901 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753745+0000", 2024-06-27T21:14:22.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19772094300000001 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753794+0000", 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8782000000000002e-05 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753830+0000", 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5818000000000002e-05 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:995 2.b 2:d08418d5:::benchmark_data_smithi192_59818_object994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:46.573491+0000", 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 155.42731107700001, 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18068727400000001, 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.010 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 995 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573491+0000", 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573491+0000", 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573492+0000", 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4399999999999999e-07 2024-06-27T21:14:22.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573514+0000", 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1704000000000001e-05 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573523+0000", 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4769999999999992e-06 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573528+0000", 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9289999999999997e-06 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573573+0000", 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4801e-05 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573685+0000", 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011209600000000001 2024-06-27T21:14:22.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.573818+0000", 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132812 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.574170+0000", 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035199999999999999 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.574494+0000", 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032437500000000001 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.754071+0000", 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17957663900000001 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.754121+0000", 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9898000000000002e-05 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.754179+0000", 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7698999999999998e-05 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:2533 2.b 2:dc2a02a9:::benchmark_data_smithi192_59818_object2532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:47.447069+0000", 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 154.55373307299999, 2024-06-27T21:14:22.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20038641800000001, 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2533 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447069+0000", 2024-06-27T21:14:22.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447069+0000", 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447070+0000", 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1820000000000001e-06 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447090+0000", 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9312000000000001e-05 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447100+0000", 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0674e-05 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447108+0000", 2024-06-27T21:14:22.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3069999999999997e-06 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447159+0000", 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0862000000000002e-05 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447237+0000", 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8464e-05 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447322+0000", 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4458999999999999e-05 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447735+0000", 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000412979 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447939+0000", 2024-06-27T21:14:22.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020400999999999999 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647366+0000", 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19942702600000001 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647418+0000", 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2723999999999999e-05 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647456+0000", 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7419000000000002e-05 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:2533 2.b e108/104 ondisk, result = 0)", 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:47.447918+0000", 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 154.55288391299999, 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19954237899999999, 2024-06-27T21:14:22.024 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447918+0000", 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447918+0000", 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447919+0000", 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8999999999999996e-07 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.447922+0000", 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9579999999999999e-06 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647277+0000", 2024-06-27T21:14:22.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19935534399999999 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647284+0000", 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8399999999999997e-06 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647355+0000", 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0772999999999995e-05 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647360+0000", 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.242e-06 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.647461+0000", 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010063199999999999 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7681 2.6 e108/104 2:6be1fbc1:::benchmark_data_smithi192_59818_object7680:head v 108'492, mlcod=108'488)", 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.217816+0000", 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 151.78298599499999, 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22328922400000001, 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217816+0000", 2024-06-27T21:14:22.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217816+0000", 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217817+0000", 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3699999999999999e-07 2024-06-27T21:14:22.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217853+0000", 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5932000000000001e-05 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440547+0000", 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.222693483 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440552+0000", 2024-06-27T21:14:22.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.507e-06 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440602+0000", 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9755000000000001e-05 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440624+0000", 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2209999999999999e-05 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.441073+0000", 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044889600000000001 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.441106+0000", 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2503999999999999e-05 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:7692 2.b 2:de479498:::benchmark_data_smithi192_59818_object7691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.225536+0000", 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 151.77526626100001, 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21296707500000001, 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.034 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7692 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225536+0000", 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225536+0000", 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225537+0000", 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7499999999999999e-07 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225554+0000", 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7331000000000001e-05 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225565+0000", 2024-06-27T21:14:22.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0642000000000001e-05 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225572+0000", 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9970000000000001e-06 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225613+0000", 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0567e-05 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225689+0000", 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6981e-05 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225772+0000", 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2256999999999995e-05 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226219+0000", 2024-06-27T21:14:22.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044744499999999998 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226663+0000", 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044416399999999998 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438441+0000", 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211778035 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438477+0000", 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5577999999999997e-05 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438503+0000", 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6203e-05 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:7692 2.b e108/104 ondisk, result = 0)", 2024-06-27T21:14:22.040 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.226423+0000", 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 151.774379012, 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21208665900000001, 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226423+0000", 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226423+0000", 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226424+0000", 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226427+0000", 2024-06-27T21:14:22.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.225e-06 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438395+0000", 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21196757899999999 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438399+0000", 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7319999999999998e-06 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438433+0000", 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3351e-05 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438437+0000", 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438510+0000", 2024-06-27T21:14:22.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2614999999999995e-05 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7698 2.6 e108/104 2:6ac723ee:::benchmark_data_smithi192_59818_object7697:head v 108'493, mlcod=108'489)", 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.289682+0000", 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 151.711120736, 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18005770900000001, 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.045 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.289682+0000", 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.289682+0000", 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.289684+0000", 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6570000000000001e-06 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.289746+0000", 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1608999999999999e-05 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.469229+0000", 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17948325200000001 2024-06-27T21:14:22.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.469235+0000", 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4569999999999996e-06 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.469272+0000", 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7444000000000003e-05 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.469285+0000", 2024-06-27T21:14:22.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2577e-05 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.469714+0000", 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000429825 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.469739+0000", 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4888e-05 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.049 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:12970 2.f 2:f05cefe0:::benchmark_data_smithi192_59818_object12969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:53.758551+0000", 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 148.24225171500001, 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.180974038, 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 12970 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758551+0000", 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758551+0000", 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758551+0000", 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7800000000000001e-07 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758569+0000", 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7926000000000001e-05 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758579+0000", 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0093e-05 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758584+0000", 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8330000000000001e-06 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758623+0000", 2024-06-27T21:14:22.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8472e-05 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758703+0000", 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0420000000000006e-05 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.758789+0000", 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5840999999999993e-05 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759312+0000", 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052288400000000004 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759738+0000", 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00042602500000000001 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.939431+0000", 2024-06-27T21:14:22.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.179692611 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.939486+0000", 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5485999999999997e-05 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.939525+0000", 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8668999999999999e-05 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:12971 2.b 2:d2678537:::benchmark_data_smithi192_59818_object12970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-06-27T21:14:22.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:53.759812+0000", 2024-06-27T21:14:22.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 148.240990474, 2024-06-27T21:14:22.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17953502599999999, 2024-06-27T21:14:22.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 12971 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759812+0000", 2024-06-27T21:14:22.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759812+0000", 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759813+0000", 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6499999999999999e-07 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759830+0000", 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7520000000000002e-05 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759838+0000", 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1759999999999993e-06 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759843+0000", 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3420000000000001e-06 2024-06-27T21:14:22.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759882+0000", 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9062e-05 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.759956+0000", 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4796999999999996e-05 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.760049+0000", 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2268999999999994e-05 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.760634+0000", 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00058548699999999999 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.760929+0000", 2024-06-27T21:14:22.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029500899999999999 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.939260+0000", 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.178330773 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.939301+0000", 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0575999999999999e-05 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:53.939347+0000", 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6350000000000002e-05 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:15439 2.5 2:aca3d0f0:::benchmark_data_smithi192_59818_object15438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:55.565217+0000", 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 146.43558516499999, 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.189402612, 2024-06-27T21:14:22.064 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15439 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565217+0000", 2024-06-27T21:14:22.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565217+0000", 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565218+0000", 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2389999999999999e-06 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565242+0000", 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3370000000000002e-05 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565254+0000", 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2264e-05 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565262+0000", 2024-06-27T21:14:22.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.452e-06 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565311+0000", 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9727e-05 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565387+0000", 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5617000000000005e-05 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:14:22.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.565467+0000", 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0047000000000001e-05 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.566136+0000", 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066918900000000005 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.566536+0000", 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039948099999999998 2024-06-27T21:14:22.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.754535+0000", 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18799958999999999 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.754589+0000", 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4110999999999997e-05 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.754620+0000", 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0525000000000003e-05 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:15465 2.f 2:f871ccb0:::benchmark_data_smithi192_59818_object15464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:55.576351+0000", 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 146.42445103599999, 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21259978600000001, 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.071 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:22.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:22.073 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15465 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576351+0000", 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576351+0000", 2024-06-27T21:14:22.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576353+0000", 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5510000000000001e-06 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576377+0000", 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4511e-05 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576396+0000", 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8802999999999999e-05 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576405+0000", 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3569999999999998e-06 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576470+0000", 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5220000000000002e-05 2024-06-27T21:14:22.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576582+0000", 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112493 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576717+0000", 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134583 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.577463+0000", 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000745998 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.577877+0000", 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041452699999999998 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.788844+0000", 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21096653700000001 2024-06-27T21:14:22.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.788893+0000", 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9224999999999997e-05 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.788951+0000", 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7980999999999999e-05 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:79089 2.6 e128/104 2:61ffccd4:::benchmark_data_smithi192_59818_object79088:head v 128'5050, mlcod=128'5047)", 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:01.911357+0000", 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 80.089445326000003, 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.219698061, 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911357+0000", 2024-06-27T21:14:22.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911357+0000", 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911358+0000", 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.75e-07 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911409+0000", 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999999e-05 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128590+0000", 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21718090700000001 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128597+0000", 2024-06-27T21:14:22.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6340000000000003e-06 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128659+0000", 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1767999999999995e-05 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128678+0000", 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8901000000000001e-05 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.131007+0000", 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023295070000000002 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.131055+0000", 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7969000000000001e-05 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:107417 2.b 2:d5dd94fd:::benchmark_data_smithi192_59818_object15342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:31.694933+0000", 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.305868971000002, 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22405386099999999, 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 107417 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.085 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694933+0000", 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694933+0000", 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694934+0000", 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8899999999999998e-07 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694938+0000", 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7019999999999999e-06 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694946+0000", 2024-06-27T21:14:22.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4679999999999992e-06 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694951+0000", 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6319999999999999e-06 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.698026+0000", 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030754419999999998 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.698115+0000", 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8578999999999993e-05 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.698192+0000", 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7249000000000005e-05 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.701121+0000", 2024-06-27T21:14:22.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002928833 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.701939+0000", 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00081798400000000001 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.918912+0000", 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.216973104 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.918948+0000", 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6171999999999999e-05 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.918987+0000", 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8906999999999999e-05 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.091 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:109801 2.6 e138/104 2:6baf6b64:::benchmark_data_smithi192_59818_object17726:head v 138'6978, mlcod=138'6974)", 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:34.064690+0000", 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.936111973999999, 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18552032099999999, 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.064690+0000", 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.064690+0000", 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.064691+0000", 2024-06-27T21:14:22.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4300000000000003e-07 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.064697+0000", 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6169999999999998e-06 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.247628+0000", 2024-06-27T21:14:22.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.182931128 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.247633+0000", 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2900000000000002e-06 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.247675+0000", 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2014000000000001e-05 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.247694+0000", 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9309000000000001e-05 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.250169+0000", 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024748560000000001 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:34.250211+0000", 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1563999999999997e-05 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:121356 2.6 e141/104 2:672e567d:::benchmark_data_smithi192_59818_object29281:head v 141'7736, mlcod=140'7733)", 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.359513536000001, 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.249253161, 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.097 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999996e-07 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641294+0000", 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2629999999999997e-06 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888046+0000", 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24675217899999999 2024-06-27T21:14:22.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888051+0000", 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8280000000000002e-06 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888089+0000", 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8661000000000001e-05 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888103+0000", 2024-06-27T21:14:22.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3981e-05 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.890517+0000", 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024139869999999998 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.890542+0000", 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4658e-05 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.101 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121372 2.5 2:a1740290:::benchmark_data_smithi192_59818_object29297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.672471+0000", 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.328331730999999, 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20977246999999999, 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:22.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121372 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672471+0000", 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672471+0000", 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672472+0000", 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5000000000000001e-07 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672476+0000", 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3959999999999999e-06 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672487+0000", 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0558e-05 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672492+0000", 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4609999999999999e-06 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672537+0000", 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5380999999999998e-05 2024-06-27T21:14:22.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672617+0000", 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9810999999999999e-05 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672697+0000", 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9793e-05 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.675988+0000", 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003291165 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.685499+0000", 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0095106500000000007 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882089+0000", 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19659014699999999 2024-06-27T21:14:22.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882183+0000", 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3621000000000004e-05 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882243+0000", 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0537e-05 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:121372 2.5 e141/104 ondisk, result = 0)", 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.676588+0000", 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.324214810000001, 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20566014699999999, 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.676588+0000", 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.676588+0000", 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.676588+0000", 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3900000000000004e-07 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.676594+0000", 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6670000000000001e-06 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.881887+0000", 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20529330500000001 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.881904+0000", 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6878999999999999e-05 2024-06-27T21:14:22.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882062+0000", 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015776399999999999 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882077+0000", 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5503e-05 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882248+0000", 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017039 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:121411 2.6 e141/104 2:678a3e62:::benchmark_data_smithi192_59818_object29336:head v 141'7739, mlcod=141'7736)", 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:46.011735+0000", 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 35.989067085000002, 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47895283999999999, 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011735+0000", 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011735+0000", 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011736+0000", 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6099999999999999e-07 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011742+0000", 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8109999999999999e-06 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.487949+0000", 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47620712700000001 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.487954+0000", 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9590000000000003e-06 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.488011+0000", 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6526000000000001e-05 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.488035+0000", 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4139999999999999e-05 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.490659+0000", 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026242729999999999 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.490688+0000", 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9142999999999999e-05 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:22.117 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:22.119 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:14:22.119 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:14:22.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:22.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:22.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:22.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:22.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:22.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:22.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:22.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:22.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:22.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:22.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:22.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:22.935 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:22.938 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-06-27T21:14:21.859771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2649883192,"num_objects":40428,"num_object_clones":0,"num_object_copies":121284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7293,"num_objects_unfound":0,"num_objects_dirty":40428,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":143843,"num_write_kb":5894759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1423,"num_bytes_recovered":95644864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146204,"ondisk_log_size":146204,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8780764,"kb_used_data":7990180,"kb_used_omap":18,"kb_used_meta":790445,"kb_avail":1116161060,"statfs":{"total":1151940427776,"available":1142948925440,"internally_reserved":0,"allocated":8181944320,"data_stored":8178636656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":809415994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-317128704,"num_objects":-4839,"num_object_clones":0,"num_object_copies":-14517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1344,"num_objects_unfound":0,"num_objects_dirty":-4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":29360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003507"},"pg_stats":[{"pgid":"2.f","version":"151'9047","reported_seq":9195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:17.205337+0000","last_change":"2024-06-27T21:12:42.522399+0000","last_active":"2024-06-27T21:14:17.205337+0000","last_peered":"2024-06-27T21:14:17.205337+0000","last_clean":"2024-06-27T21:14:17.205337+0000","last_became_active":"2024-06-27T21:11:31.339208+0000","last_became_peered":"2024-06-27T21:11:31.339208+0000","last_unstale":"2024-06-27T21:14:17.205337+0000","last_undegraded":"2024-06-27T21:14:17.205337+0000","last_fullsized":"2024-06-27T21:14:17.205337+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3650","last_scrub_stamp":"2024-06-27T21:12:42.522239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.522239+0000","objects_scrubbed":3627,"log_size":9047,"log_dups_size":0,"ondisk_log_size":9047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173080576,"num_objects":2641,"num_object_clones":0,"num_object_copies":7923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9047,"num_write_kb":374016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"151'8725","reported_seq":8912,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:18.329756+0000","last_change":"2024-06-27T21:14:13.809056+0000","last_active":"2024-06-27T21:14:18.329756+0000","last_peered":"2024-06-27T21:14:18.329756+0000","last_clean":"2024-06-27T21:14:18.329756+0000","last_became_active":"2024-06-27T21:11:31.337998+0000","last_became_peered":"2024-06-27T21:11:31.337998+0000","last_unstale":"2024-06-27T21:14:18.329756+0000","last_undegraded":"2024-06-27T21:14:18.329756+0000","last_fullsized":"2024-06-27T21:14:18.329756+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'8513","last_scrub_stamp":"2024-06-27T21:14:13.808718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:13.808718+0000","objects_scrubbed":2640,"log_size":8725,"log_dups_size":0,"ondisk_log_size":8725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:23.426457+0000","scrub_duration":485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158531584,"num_objects":2419,"num_object_clones":0,"num_object_copies":7257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8725,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"151'9064","reported_seq":9254,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:20.228239+0000","last_change":"2024-06-27T21:14:08.491684+0000","last_active":"2024-06-27T21:14:20.228239+0000","last_peered":"2024-06-27T21:14:20.228239+0000","last_clean":"2024-06-27T21:14:20.228239+0000","last_became_active":"2024-06-27T21:11:31.339229+0000","last_became_peered":"2024-06-27T21:11:31.339229+0000","last_unstale":"2024-06-27T21:14:20.228239+0000","last_undegraded":"2024-06-27T21:14:20.228239+0000","last_fullsized":"2024-06-27T21:14:20.228239+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'8536","last_scrub_stamp":"2024-06-27T21:14:08.491546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:08.491546+0000","objects_scrubbed":3007,"log_size":9064,"log_dups_size":0,"ondisk_log_size":9064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:37.194608+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161415192,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9064,"num_write_kb":368833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"151'9084","reported_seq":9232,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:19.329172+0000","last_change":"2024-06-27T21:12:45.902924+0000","last_active":"2024-06-27T21:14:19.329172+0000","last_peered":"2024-06-27T21:14:19.329172+0000","last_clean":"2024-06-27T21:14:19.329172+0000","last_became_active":"2024-06-27T21:11:31.556192+0000","last_became_peered":"2024-06-27T21:11:31.556192+0000","last_unstale":"2024-06-27T21:14:19.329172+0000","last_undegraded":"2024-06-27T21:14:19.329172+0000","last_fullsized":"2024-06-27T21:14:19.329172+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3891","last_scrub_stamp":"2024-06-27T21:12:45.902837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:45.902837+0000","objects_scrubbed":3854,"log_size":9084,"log_dups_size":0,"ondisk_log_size":9084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161480704,"num_objects":2464,"num_object_clones":0,"num_object_copies":7392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":369536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":262,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:16.456943+0000","last_change":"2024-06-27T21:12:50.081819+0000","last_active":"2024-06-27T21:14:16.456943+0000","last_peered":"2024-06-27T21:14:16.456943+0000","last_clean":"2024-06-27T21:14:16.456943+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:14:16.456943+0000","last_undegraded":"2024-06-27T21:14:16.456943+0000","last_fullsized":"2024-06-27T21:14:16.456943+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:12:50.081718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:12:50.081718+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"151'8918","reported_seq":9066,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:17.258974+0000","last_change":"2024-06-27T21:12:42.932200+0000","last_active":"2024-06-27T21:14:17.258974+0000","last_peered":"2024-06-27T21:14:17.258974+0000","last_clean":"2024-06-27T21:14:17.258974+0000","last_became_active":"2024-06-27T21:11:31.337613+0000","last_became_peered":"2024-06-27T21:11:31.337613+0000","last_unstale":"2024-06-27T21:14:17.258974+0000","last_undegraded":"2024-06-27T21:14:17.258974+0000","last_fullsized":"2024-06-27T21:14:17.258974+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3620","last_scrub_stamp":"2024-06-27T21:12:42.932079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.932079+0000","objects_scrubbed":3600,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8918,"num_write_kb":370368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"151'8868","reported_seq":14663,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:14:19.525633+0000","last_change":"2024-06-27T21:13:47.955243+0000","last_active":"2024-06-27T21:14:19.525633+0000","last_peered":"2024-06-27T21:14:19.525633+0000","last_clean":"2024-06-27T21:13:45.604588+0000","last_became_active":"2024-06-27T21:13:47.901845+0000","last_became_peered":"2024-06-27T21:13:47.901845+0000","last_unstale":"2024-06-27T21:14:19.525633+0000","last_undegraded":"2024-06-27T21:14:19.525633+0000","last_fullsized":"2024-06-27T21:13:47.640498+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3530","last_scrub_stamp":"2024-06-27T21:12:41.700386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:41.700386+0000","objects_scrubbed":3520,"log_size":8868,"log_dups_size":0,"ondisk_log_size":8868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159383552,"num_objects":2432,"num_object_clones":0,"num_object_copies":7296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7293,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8868,"num_write_kb":361600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":92471296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[7,6],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"6,7","objects":1},{"shards":"7,9,10","objects":2431}],"blocked_by":[],"up_primary":11,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"151'9315","reported_seq":9508,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:18.360028+0000","last_change":"2024-06-27T21:14:11.783997+0000","last_active":"2024-06-27T21:14:18.360028+0000","last_peered":"2024-06-27T21:14:18.360028+0000","last_clean":"2024-06-27T21:14:18.360028+0000","last_became_active":"2024-06-27T21:11:31.556127+0000","last_became_peered":"2024-06-27T21:11:31.556127+0000","last_unstale":"2024-06-27T21:14:18.360028+0000","last_undegraded":"2024-06-27T21:14:18.360028+0000","last_fullsized":"2024-06-27T21:14:18.360028+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'9012","last_scrub_stamp":"2024-06-27T21:14:11.783045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:11.783045+0000","objects_scrubbed":2901,"log_size":9315,"log_dups_size":0,"ondisk_log_size":9315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:21.423826+0000","scrub_duration":1351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167313408,"num_objects":2553,"num_object_clones":0,"num_object_copies":7659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9315,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"151'8966","reported_seq":9157,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:17.272057+0000","last_change":"2024-06-27T21:14:08.219342+0000","last_active":"2024-06-27T21:14:17.272057+0000","last_peered":"2024-06-27T21:14:17.272057+0000","last_clean":"2024-06-27T21:14:17.272057+0000","last_became_active":"2024-06-27T21:11:31.337786+0000","last_became_peered":"2024-06-27T21:11:31.337786+0000","last_unstale":"2024-06-27T21:14:17.272057+0000","last_undegraded":"2024-06-27T21:14:17.272057+0000","last_fullsized":"2024-06-27T21:14:17.272057+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'8555","last_scrub_stamp":"2024-06-27T21:14:08.219152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:08.219152+0000","objects_scrubbed":3106,"log_size":8966,"log_dups_size":0,"ondisk_log_size":8966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:18.227207+0000","scrub_duration":679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":372416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"151'8962","reported_seq":9148,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:18.722232+0000","last_change":"2024-06-27T21:14:13.085036+0000","last_active":"2024-06-27T21:14:18.722232+0000","last_peered":"2024-06-27T21:14:18.722232+0000","last_clean":"2024-06-27T21:14:18.722232+0000","last_became_active":"2024-06-27T21:11:31.556366+0000","last_became_peered":"2024-06-27T21:11:31.556366+0000","last_unstale":"2024-06-27T21:14:18.722232+0000","last_undegraded":"2024-06-27T21:14:18.722232+0000","last_fullsized":"2024-06-27T21:14:18.722232+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'8722","last_scrub_stamp":"2024-06-27T21:14:13.084898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:13.084898+0000","objects_scrubbed":2761,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:21.377489+0000","scrub_duration":748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163971072,"num_objects":2502,"num_object_clones":0,"num_object_copies":7506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":366848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'11338","reported_seq":11642,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:17.185256+0000","last_change":"2024-06-27T21:13:08.494332+0000","last_active":"2024-06-27T21:14:17.185256+0000","last_peered":"2024-06-27T21:14:17.185256+0000","last_clean":"2024-06-27T21:14:17.185256+0000","last_became_active":"2024-06-27T21:11:31.556322+0000","last_became_peered":"2024-06-27T21:11:31.556322+0000","last_unstale":"2024-06-27T21:14:17.185256+0000","last_undegraded":"2024-06-27T21:14:17.185256+0000","last_fullsized":"2024-06-27T21:14:17.185256+0000","mapping_epoch":104,"log_start":"111'1300","ondisk_log_start":"111'1300","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7832","last_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_deep_scrub":"129'7832","last_deep_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_clean_scrub_stamp":"2024-06-27T21:13:08.494150+0000","objects_scrubbed":4784,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"queued for scrub","scrub_duration":14657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":363456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"151'9170","reported_seq":9329,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:18.738983+0000","last_change":"2024-06-27T21:12:56.263993+0000","last_active":"2024-06-27T21:14:18.738983+0000","last_peered":"2024-06-27T21:14:18.738983+0000","last_clean":"2024-06-27T21:14:18.738983+0000","last_became_active":"2024-06-27T21:11:31.338960+0000","last_became_peered":"2024-06-27T21:11:31.338960+0000","last_unstale":"2024-06-27T21:14:18.738983+0000","last_undegraded":"2024-06-27T21:14:18.738983+0000","last_fullsized":"2024-06-27T21:14:18.738983+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4740","last_scrub_stamp":"2024-06-27T21:12:56.263856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:56.263856+0000","objects_scrubbed":4707,"log_size":9170,"log_dups_size":0,"ondisk_log_size":9170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:19.793246+0000","scrub_duration":1215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9170,"num_write_kb":376320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"151'8971","reported_seq":9122,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:20.086560+0000","last_change":"2024-06-27T21:12:44.319499+0000","last_active":"2024-06-27T21:14:20.086560+0000","last_peered":"2024-06-27T21:14:20.086560+0000","last_clean":"2024-06-27T21:14:20.086560+0000","last_became_active":"2024-06-27T21:11:31.943105+0000","last_became_peered":"2024-06-27T21:11:31.943105+0000","last_unstale":"2024-06-27T21:14:20.086560+0000","last_undegraded":"2024-06-27T21:14:20.086560+0000","last_fullsized":"2024-06-27T21:14:20.086560+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3747","last_scrub_stamp":"2024-06-27T21:12:44.319331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.319331+0000","objects_scrubbed":3704,"log_size":8971,"log_dups_size":0,"ondisk_log_size":8971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156303360,"num_objects":2385,"num_object_clones":0,"num_object_copies":7155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8971,"num_write_kb":363392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"151'8916","reported_seq":9074,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:19.507052+0000","last_change":"2024-06-27T21:12:55.073348+0000","last_active":"2024-06-27T21:14:19.507052+0000","last_peered":"2024-06-27T21:14:19.507052+0000","last_clean":"2024-06-27T21:14:19.507052+0000","last_became_active":"2024-06-27T21:11:31.556186+0000","last_became_peered":"2024-06-27T21:11:31.556186+0000","last_unstale":"2024-06-27T21:14:19.507052+0000","last_undegraded":"2024-06-27T21:14:19.507052+0000","last_fullsized":"2024-06-27T21:14:19.507052+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4512","last_scrub_stamp":"2024-06-27T21:12:55.073215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:55.073215+0000","objects_scrubbed":4477,"log_size":8916,"log_dups_size":0,"ondisk_log_size":8916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162136064,"num_objects":2474,"num_object_clones":0,"num_object_copies":7422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":364480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"151'9023","reported_seq":9235,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:18.691908+0000","last_change":"2024-06-27T21:14:12.955909+0000","last_active":"2024-06-27T21:14:18.691908+0000","last_peered":"2024-06-27T21:14:18.691908+0000","last_clean":"2024-06-27T21:14:18.691908+0000","last_became_active":"2024-06-27T21:11:31.337888+0000","last_became_peered":"2024-06-27T21:11:31.337888+0000","last_unstale":"2024-06-27T21:14:18.691908+0000","last_undegraded":"2024-06-27T21:14:18.691908+0000","last_fullsized":"2024-06-27T21:14:18.691908+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'8786","last_scrub_stamp":"2024-06-27T21:14:12.955686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:12.955686+0000","objects_scrubbed":2839,"log_size":9023,"log_dups_size":0,"ondisk_log_size":9023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:33.163489+0000","scrub_duration":1661,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575552,"num_objects":2557,"num_object_clones":0,"num_object_copies":7671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":370560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"151'8959","reported_seq":9107,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:19.297439+0000","last_change":"2024-06-27T21:12:42.976967+0000","last_active":"2024-06-27T21:14:19.297439+0000","last_peered":"2024-06-27T21:14:19.297439+0000","last_clean":"2024-06-27T21:14:19.297439+0000","last_became_active":"2024-06-27T21:11:31.340369+0000","last_became_peered":"2024-06-27T21:11:31.340369+0000","last_unstale":"2024-06-27T21:14:19.297439+0000","last_undegraded":"2024-06-27T21:14:19.297439+0000","last_fullsized":"2024-06-27T21:14:19.297439+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3653","last_scrub_stamp":"2024-06-27T21:12:42.976769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.976769+0000","objects_scrubbed":3623,"log_size":8959,"log_dups_size":0,"ondisk_log_size":8959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160497664,"num_objects":2449,"num_object_clones":0,"num_object_copies":7347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8959,"num_write_kb":365056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"151'11386","reported_seq":11708,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-06-27T21:14:17.178488+0000","last_change":"2024-06-27T21:13:29.274450+0000","last_active":"2024-06-27T21:14:17.178488+0000","last_peered":"2024-06-27T21:14:17.178488+0000","last_clean":"2024-06-27T21:14:17.178488+0000","last_became_active":"2024-06-27T21:11:31.337915+0000","last_became_peered":"2024-06-27T21:11:31.337915+0000","last_unstale":"2024-06-27T21:14:17.178488+0000","last_undegraded":"2024-06-27T21:14:17.178488+0000","last_fullsized":"2024-06-27T21:14:17.178488+0000","mapping_epoch":104,"log_start":"111'1300","ondisk_log_start":"111'1300","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9000","last_scrub_stamp":"2024-06-27T21:13:29.274287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:13:29.274287+0000","objects_scrubbed":5002,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:59.000982+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168034304,"num_objects":2564,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2649292824,"num_objects":40426,"num_object_clones":0,"num_object_copies":121278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7293,"num_objects_unfound":0,"num_objects_dirty":40426,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143720,"num_write_kb":5892609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":92471296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8101244928,"data_stored":8101232712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146112,"ondisk_log_size":146112,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252924,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563792,"kb_used_data":484040,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93181360,"statfs":{"total":95995035648,"available":95417712640,"internally_reserved":0,"allocated":495656960,"data_stored":495379657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514561,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":608648,"kb_used_data":559424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93136504,"statfs":{"total":95995035648,"available":95371780096,"internally_reserved":0,"allocated":572850176,"data_stored":572576445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1108748,"kb_used_data":1033156,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92636404,"statfs":{"total":95995035648,"available":94859677696,"internally_reserved":0,"allocated":1057951744,"data_stored":1057673941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972425,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745996,"kb_used_data":674560,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92999156,"statfs":{"total":95995035648,"available":95231135744,"internally_reserved":0,"allocated":690749440,"data_stored":690475709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266766,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880464,"kb_used_data":804872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92864688,"statfs":{"total":95995035648,"available":95093440512,"internally_reserved":0,"allocated":824188928,"data_stored":823911645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528402,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383568,"kb_used_data":320456,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93361584,"statfs":{"total":95995035648,"available":95602262016,"internally_reserved":0,"allocated":328146944,"data_stored":327869661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398096,"kb_used_data":344704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93347056,"statfs":{"total":95995035648,"available":95587385344,"internally_reserved":0,"allocated":352976896,"data_stored":352703165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":556116,"kb_used_data":506884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93189036,"statfs":{"total":95995035648,"available":95425572864,"internally_reserved":0,"allocated":519049216,"data_stored":518771413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063820,"kb_used_data":1000708,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92681332,"statfs":{"total":95995035648,"available":94905683968,"internally_reserved":0,"allocated":1024724992,"data_stored":1024447189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411430,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719944,"kb_used_data":644352,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93025208,"statfs":{"total":95995035648,"available":95257812992,"internally_reserved":0,"allocated":659816448,"data_stored":659542717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888780,"kb_used_data":825664,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92856372,"statfs":{"total":95995035648,"available":95084924928,"internally_reserved":0,"allocated":845479936,"data_stored":845206205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862792,"kb_used_data":791360,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92882360,"statfs":{"total":95995035648,"available":95111536640,"internally_reserved":0,"allocated":810352640,"data_stored":810078909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":798490624,"data_stored":798490624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":835059712,"data_stored":835059712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":658112512,"data_stored":658112512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1017647104,"data_stored":1017643032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":511840256,"data_stored":511836184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":349175808,"data_stored":349175808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":323289088,"data_stored":323289088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":815464448,"data_stored":815464448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":686882816,"data_stored":686882816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1049628672,"data_stored":1049624600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":570884096,"data_stored":570884096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":484769792,"data_stored":484769792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:14:22.939 INFO:tasks.thrashosds.thrasher:Setting 2.8 to [8, 4, 9] 2024-06-27T21:14:22.939 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.8', '8', '4', '9'] 2024-06-27T21:14:22.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap 2.8 8 4 9 2024-06-27T21:14:22.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:22.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:22.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:23.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:23.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:23.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:23.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:23.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:14:23.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:23.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:23.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:23.458 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:14:23.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:23.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:23 smithi192 ceph-mon[32956]: pgmap v459: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 7293/121284 objects misplaced (6.013%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:14:23.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:23 smithi192 ceph-mon[32956]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:14:23.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2410074173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:14:23.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:23.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:23.626 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:23.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:23.734 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:23.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:23 smithi190 ceph-mon[32247]: pgmap v459: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 7293/121284 objects misplaced (6.013%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:14:23.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:23 smithi190 ceph-mon[32247]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:14:23.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2410074173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:14:23.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:23.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:23 smithi151 ceph-mon[28965]: pgmap v459: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 7293/121284 objects misplaced (6.013%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:14:23.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:23 smithi151 ceph-mon[28965]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:14:23.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2410074173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:14:23.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:23.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:23.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:24.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:24.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:24.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:24.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:24.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:24.377 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:24.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:24.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:24.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:24.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:24.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:24 smithi190 ceph-mon[32247]: 2.6 scrub starts 2024-06-27T21:14:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:24 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2410074173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:14:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:24 smithi190 ceph-mon[32247]: osdmap e152: 12 total, 12 up, 12 in 2024-06-27T21:14:24.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:24.773 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:24.773 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:24.773 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:24 smithi192 ceph-mon[32956]: 2.6 scrub starts 2024-06-27T21:14:24.773 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:24 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:24.773 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2410074173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:14:24.773 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:24 smithi192 ceph-mon[32956]: osdmap e152: 12 total, 12 up, 12 in 2024-06-27T21:14:24.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:24.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:24.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:24.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:24.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:24.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:24 smithi151 ceph-mon[28965]: 2.6 scrub starts 2024-06-27T21:14:24.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:24 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:24.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2410074173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:14:24.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:24 smithi151 ceph-mon[28965]: osdmap e152: 12 total, 12 up, 12 in 2024-06-27T21:14:25.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:25.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:25.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:25.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:25.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:25.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:25.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:25.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:25.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:25.488 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:25.488 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:148976 2.d 2:b0723029:::benchmark_data_smithi192_59818_object56901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:25.481127+0000", 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013756045, 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013785504000000001, 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:25.489 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:14:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148976 2024-06-27T21:14:25.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481127+0000", 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:25.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481127+0000", 2024-06-27T21:14:25.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:25.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:25.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481128+0000", 2024-06-27T21:14:25.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5230000000000001e-06 2024-06-27T21:14:25.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:25.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481134+0000", 2024-06-27T21:14:25.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0889999999999996e-06 2024-06-27T21:14:25.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:25.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481151+0000", 2024-06-27T21:14:25.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6750000000000001e-05 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481158+0000", 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1160000000000001e-06 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481445+0000", 2024-06-27T21:14:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028698999999999999 2024-06-27T21:14:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481571+0000", 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012587199999999999 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.481645+0000", 2024-06-27T21:14:25.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4035000000000005e-05 2024-06-27T21:14:25.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:25.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.484820+0000", 2024-06-27T21:14:25.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031755339999999998 2024-06-27T21:14:25.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:25.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.485348+0000", 2024-06-27T21:14:25.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052747899999999997 2024-06-27T21:14:25.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:148986 2.f e152/104 2:f8796a43:::benchmark_data_smithi192_59818_object56911:head v 152'9412, mlcod=152'9410)", 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:25.491364+0000", 2024-06-27T21:14:25.500 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0035182550000000001, 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036403479999999998, 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491364+0000", 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491364+0000", 2024-06-27T21:14:25.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:25.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:25.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491365+0000", 2024-06-27T21:14:25.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8899999999999999e-07 2024-06-27T21:14:25.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491371+0000", 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2929999999999997e-06 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491401+0000", 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9865e-05 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491407+0000", 2024-06-27T21:14:25.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1940000000000003e-06 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491452+0000", 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4406999999999997e-05 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:25.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:25.491480+0000", 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8167999999999999e-05 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:25.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:25.508 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:14:25.508 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-06-27T21:14:25.508 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:25.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:25.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:25.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:25.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:25.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:25 smithi151 ceph-mon[28965]: pgmap v461: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 7293/114264 objects misplaced (6.383%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:25.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:25 smithi190 ceph-mon[32247]: pgmap v461: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 7293/114264 objects misplaced (6.383%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:25.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:25.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:25 smithi192 ceph-mon[32956]: pgmap v461: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 7293/114264 objects misplaced (6.383%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:25.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:25.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:25.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:25.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:26.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:26.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:26.130 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:26.137 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:14:26.137 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:14:26.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:26.169 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:14:26.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:14:26.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:26.195 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:14:26.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:14:26.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:14:26.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:14:26.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:14:26.247 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:14:26.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:26.277 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:14:26.277 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:14:26.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:14:26.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:14:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:26.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:26.338 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:14:26.339 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:14:26.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:14:26.365 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:14:26.365 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:14:26.392 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:14:26.392 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:14:26.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:26.421 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:14:26.421 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:14:26.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:26.451 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:14:26.451 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:14:26.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/58320132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:26.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:26.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:26.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:26.685 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:26.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:26.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/58320132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:26.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/58320132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:26.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:26.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:26.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:26.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:26.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:27.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:27.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:27.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:14:27.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:27.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:27.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:27.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:27.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:27.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:27.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:27.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:27.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:27.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:27.744 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:27 smithi151 ceph-mon[28965]: pgmap v462: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 6567/109983 objects misplaced (5.971%); 3.2 MiB/s, 51 objects/s recovering 2024-06-27T21:14:27.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:27 smithi190 ceph-mon[32247]: pgmap v462: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 6567/109983 objects misplaced (5.971%); 3.2 MiB/s, 51 objects/s recovering 2024-06-27T21:14:27.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:27 smithi192 ceph-mon[32956]: pgmap v462: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 6567/109983 objects misplaced (5.971%); 3.2 MiB/s, 51 objects/s recovering 2024-06-27T21:14:27.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:27.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:27.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:27.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:27.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:28.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:28.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:28.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:28.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:28.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:28.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:28.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:28.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:28.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:28.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:28.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:28.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:28.706 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:28.706 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:28.757 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:28.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:28.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:28.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:28.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:28.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:29.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:29.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:29.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:29.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:29.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:29.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:29.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:29.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:29.476 INFO:teuthology.orchestra.run.smithi151.stderr:set 2.8 pg_upmap mapping to [8,4,9] 2024-06-27T21:14:29.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:14:29.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:29.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:29.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:29.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:14:29.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:29 smithi190 ceph-mon[32247]: pgmap v463: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 6567/106473 objects misplaced (6.168%); 3.0 MiB/s, 48 objects/s recovering 2024-06-27T21:14:29.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/738764870' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [8, 4, 9]}]: dispatch 2024-06-27T21:14:29.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:29 smithi151 ceph-mon[28965]: pgmap v463: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 6567/106473 objects misplaced (6.168%); 3.0 MiB/s, 48 objects/s recovering 2024-06-27T21:14:29.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/738764870' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [8, 4, 9]}]: dispatch 2024-06-27T21:14:29.776 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:29.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:29 smithi192 ceph-mon[32956]: pgmap v463: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.2 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 6567/106473 objects misplaced (6.168%); 3.0 MiB/s, 48 objects/s recovering 2024-06-27T21:14:29.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/738764870' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [8, 4, 9]}]: dispatch 2024-06-27T21:14:29.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:29.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:29.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:29.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:29.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:30.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:30.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:30.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:30.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:30.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:30.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:30.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:30.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:30.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:30.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:30.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:30.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:30.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/738764870' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [8, 4, 9]}]': finished 2024-06-27T21:14:30.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:30 smithi151 ceph-mon[28965]: osdmap e153: 12 total, 12 up, 12 in 2024-06-27T21:14:30.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:30.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:30.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:30.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/738764870' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [8, 4, 9]}]': finished 2024-06-27T21:14:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:30 smithi190 ceph-mon[32247]: osdmap e153: 12 total, 12 up, 12 in 2024-06-27T21:14:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:30.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:30.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:30.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:30.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/738764870' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.8", "id": [8, 4, 9]}]': finished 2024-06-27T21:14:30.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:30 smithi192 ceph-mon[32956]: osdmap e153: 12 total, 12 up, 12 in 2024-06-27T21:14:30.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:30.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:30.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:30.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:30.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:30.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:30.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:31.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:31.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:31.113 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:31.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:31.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:31.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:31.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:31.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:31.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:14:31.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:31.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:31.605 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:31.605 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:31.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:31.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:31 smithi151 ceph-mon[28965]: pgmap v465: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 6567/102111 objects misplaced (6.431%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:31.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:31 smithi151 ceph-mon[28965]: osdmap e154: 12 total, 12 up, 12 in 2024-06-27T21:14:31.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:31.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:31.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1913609890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:31.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:31.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:31 smithi190 ceph-mon[32247]: pgmap v465: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 6567/102111 objects misplaced (6.431%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:31.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:31 smithi190 ceph-mon[32247]: osdmap e154: 12 total, 12 up, 12 in 2024-06-27T21:14:31.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:31.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:31.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1913609890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:31.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:31 smithi192 ceph-mon[32956]: pgmap v465: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 6567/102111 objects misplaced (6.431%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:14:31.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:31 smithi192 ceph-mon[32956]: osdmap e154: 12 total, 12 up, 12 in 2024-06-27T21:14:31.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:31.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:31.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1913609890' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:31.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:31.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:31.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:31.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:31.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:32.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:32.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:32.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:32.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:32.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:32.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:32.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:32.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:32.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:32.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:32.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:14:32.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:32.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:32 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:32.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1913609890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:32.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:32 smithi151 ceph-mon[28965]: osdmap e155: 12 total, 12 up, 12 in 2024-06-27T21:14:32.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:32.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:32.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:32 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:32.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1913609890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:32.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:32 smithi190 ceph-mon[32247]: osdmap e155: 12 total, 12 up, 12 in 2024-06-27T21:14:32.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:32 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:32.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1913609890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:32.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:32 smithi192 ceph-mon[32956]: osdmap e155: 12 total, 12 up, 12 in 2024-06-27T21:14:32.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:32.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:32.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:32.945 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:32.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:33.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:33.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:33.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:33.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:33.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:33.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:33.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:33.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:33.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:33.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:33.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:33.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:33.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:33 smithi151 ceph-mon[28965]: pgmap v468: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 5903/99804 objects misplaced (5.915%); 2.3 MiB/s, 36 objects/s recovering 2024-06-27T21:14:33.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:33.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:33.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:33 smithi190 ceph-mon[32247]: pgmap v468: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 5903/99804 objects misplaced (5.915%); 2.3 MiB/s, 36 objects/s recovering 2024-06-27T21:14:33.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:33.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:33.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:33.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:33 smithi192 ceph-mon[32956]: pgmap v468: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 5903/99804 objects misplaced (5.915%); 2.3 MiB/s, 36 objects/s recovering 2024-06-27T21:14:33.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:33.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:33.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:33.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:33.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:33.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:34.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:34.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:34.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:34.202 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:14:34.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:14:34.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:34.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:14:34.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:14:34.264 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:14:34.264 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:14:34.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:14:34.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:14:34.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:34.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:14:34.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:14:34.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:34.360 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:14:34.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:14:34.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:14:34.389 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:14:34.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:14:34.420 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:14:34.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:34.452 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:14:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:14:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:34.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:34.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:14:34.482 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:14:34.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:14:34.511 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:14:34.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:14:34.544 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:14:34.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:34 smithi151 ceph-mon[28965]: osdmap e156: 12 total, 12 up, 12 in 2024-06-27T21:14:34.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1740061686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:34.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:14:34.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:14:34.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:34.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:34.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:14:34.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:34 smithi192 ceph-mon[32956]: osdmap e156: 12 total, 12 up, 12 in 2024-06-27T21:14:34.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1740061686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:34.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:14:34.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:14:34.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:34.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:34.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:154052 2.4 e157/104 2:2b343cf2:::benchmark_data_smithi192_59818_object61977:head v 157'9593, mlcod=157'9590)", 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:34.841321+0000", 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0020585310000000002, 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020881979999999999, 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:34.838 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841321+0000", 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841321+0000", 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841322+0000", 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.122e-06 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841328+0000", 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6540000000000004e-06 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841357+0000", 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9082e-05 2024-06-27T21:14:34.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841364+0000", 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9110000000000004e-06 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841423+0000", 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9290999999999999e-05 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:34.841444+0000", 2024-06-27T21:14:34.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1219000000000001e-05 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-06-27T21:14:34.842 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:34.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:34.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:34.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:34.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:35.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:34 smithi190 ceph-mon[32247]: osdmap e156: 12 total, 12 up, 12 in 2024-06-27T21:14:35.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1740061686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:35.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:14:35.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:14:35.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:35.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:35.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:35.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:35.330 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:35.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:35.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:35.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:35.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:35.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:35.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:35.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:35.731 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:35 smithi151 ceph-mon[28965]: pgmap v470: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 5903/95904 objects misplaced (6.155%); 3.2 MiB/s, 50 objects/s recovering 2024-06-27T21:14:35.731 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:35 smithi151 ceph-mon[28965]: osdmap e157: 12 total, 12 up, 12 in 2024-06-27T21:14:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:35.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:35 smithi192 ceph-mon[32956]: pgmap v470: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 5903/95904 objects misplaced (6.155%); 3.2 MiB/s, 50 objects/s recovering 2024-06-27T21:14:35.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:35 smithi192 ceph-mon[32956]: osdmap e157: 12 total, 12 up, 12 in 2024-06-27T21:14:35.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:14:35.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:35.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:35.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:35.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:35 smithi190 ceph-mon[32247]: pgmap v470: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 472 op/s; 5903/95904 objects misplaced (6.155%); 3.2 MiB/s, 50 objects/s recovering 2024-06-27T21:14:35.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:35 smithi190 ceph-mon[32247]: osdmap e157: 12 total, 12 up, 12 in 2024-06-27T21:14:35.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:36.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:36.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:36.111 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:14:36.111 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:14:36.111 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-27T21:14:36.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:14:36.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:36.139 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:36.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:36.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:36.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:36.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:36.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:36.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:36.584 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:36.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:36.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:36.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:36.770 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:36 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 122/93006 objects degraded (0.131%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:14:36.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:36.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:36 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 122/93006 objects degraded (0.131%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:14:36.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:36.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:36.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:37.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:37.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:36 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 122/93006 objects degraded (0.131%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:14:37.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:37.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:37.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:37.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:14:37.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:37.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:37.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:37.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:37.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:37.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:37.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:37.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:37.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:37.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:37.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:37.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:37.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:37.936 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:37 smithi192 ceph-mon[32956]: pgmap v472: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.9 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 459 op/s; 122/93006 objects degraded (0.131%); 9316/93006 objects misplaced (10.017%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:14:37.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:37 smithi151 ceph-mon[28965]: pgmap v472: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.9 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 459 op/s; 122/93006 objects degraded (0.131%); 9316/93006 objects misplaced (10.017%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:14:38.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:37 smithi190 ceph-mon[32247]: pgmap v472: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.9 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 459 op/s; 122/93006 objects degraded (0.131%); 9316/93006 objects misplaced (10.017%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:14:38.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:38.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:38.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:38.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:38.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:38.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:38.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:38.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:38.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:38.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:38.577 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:38.578 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:14:38.578 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:14:38.578 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:14:38.578 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:38.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:38.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:38.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:38.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:38.913 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3355244693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:38.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:38.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3355244693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:38.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:38.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:38.989 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:14:39.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3355244693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:14:39.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:39.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:39.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:39.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:39.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:39.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:39.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:39.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:14:39.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:39.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:39.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:39.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:39.669 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:39.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:39.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:39.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:39.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:39 smithi192 ceph-mon[32956]: pgmap v473: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 122/89694 objects degraded (0.136%); 9316/89694 objects misplaced (10.386%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:14:39.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:39 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:39.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3355244693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:14:39.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:39 smithi192 ceph-mon[32956]: osdmap e158: 12 total, 12 up, 12 in 2024-06-27T21:14:39.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:39.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:39 smithi151 ceph-mon[28965]: pgmap v473: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 122/89694 objects degraded (0.136%); 9316/89694 objects misplaced (10.386%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:14:39.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:39 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:39.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3355244693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:14:39.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:39 smithi151 ceph-mon[28965]: osdmap e158: 12 total, 12 up, 12 in 2024-06-27T21:14:39.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:39.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:39.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:39 smithi190 ceph-mon[32247]: pgmap v473: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 122/89694 objects degraded (0.136%); 9316/89694 objects misplaced (10.386%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:39 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3355244693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:14:40.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:39 smithi190 ceph-mon[32247]: osdmap e158: 12 total, 12 up, 12 in 2024-06-27T21:14:40.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:40.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:40.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:40.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:40.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:40.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:40.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:40.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:40.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:40.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:40.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:40.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:40.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:40.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:40.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:40.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:40.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:41.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:41.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:41.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:41.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:41.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:41.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:41.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:993 2.7 e108/104 2:e2f2e664:::benchmark_data_smithi192_59818_object992:head v 108'68, mlcod=108'65)", 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:46.572605+0000", 2024-06-27T21:14:41.496 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 174.92674943200001, 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17253723200000001, 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.572605+0000", 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.572605+0000", 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.572606+0000", 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0300000000000001e-06 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.572645+0000", 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8865999999999997e-05 2024-06-27T21:14:41.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.744669+0000", 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17202366799999999 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.744675+0000", 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0669999999999997e-06 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.744730+0000", 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5238000000000002e-05 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.744753+0000", 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3054999999999999e-05 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.745106+0000", 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035320300000000001 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.745143+0000", 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6105000000000001e-05 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:1050 2.4 e108/104 2:2b4ec2d5:::benchmark_data_smithi192_59818_object1049:head v 108'69, mlcod=108'65)", 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:46.640249+0000", 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 174.85910535299999, 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12339088300000001, 2024-06-27T21:14:41.501 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.640249+0000", 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.640249+0000", 2024-06-27T21:14:41.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.640251+0000", 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.401e-06 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.640290+0000", 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9709000000000002e-05 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.763137+0000", 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12284682600000001 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.763145+0000", 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6769999999999995e-06 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.763198+0000", 2024-06-27T21:14:41.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3059999999999997e-05 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.763217+0000", 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8742000000000001e-05 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.763599+0000", 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038199300000000001 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.763640+0000", 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1474999999999998e-05 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:2525 2.4 e108/104 2:2f9bf993:::benchmark_data_smithi192_59818_object2524:head v 108'162, mlcod=108'159)", 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:47.440162+0000", 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 174.059192481, 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11706657600000001, 2024-06-27T21:14:41.506 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.440162+0000", 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.440162+0000", 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.440163+0000", 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1680000000000001e-06 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.440214+0000", 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1070999999999999e-05 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.556680+0000", 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.116465414 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.556691+0000", 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1406000000000001e-05 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.556806+0000", 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114963 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.556827+0000", 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0270000000000001e-05 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.557195+0000", 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036872100000000002 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:47.557229+0000", 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3562999999999997e-05 2024-06-27T21:14:41.510 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7671 2.4 e108/104 2:2cd1ed4b:::benchmark_data_smithi192_59818_object7670:head v 108'500, mlcod=108'497)", 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.207872+0000", 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.29148276399999, 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22572289500000001, 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.207872+0000", 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.207872+0000", 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.207872+0000", 2024-06-27T21:14:41.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7899999999999999e-07 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.207926+0000", 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3556e-05 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.432869+0000", 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22494341600000001 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.432882+0000", 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2632999999999999e-05 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.433058+0000", 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017629000000000001 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.433098+0000", 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9434000000000002e-05 2024-06-27T21:14:41.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.433560+0000", 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00046231700000000001 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.433595+0000", 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4770000000000001e-05 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:15444 2.7 e110/104 2:e9beafb2:::benchmark_data_smithi192_59818_object15443:head v 110'966, mlcod=110'962)", 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:55.567475+0000", 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 165.931879772, 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16896518899999999, 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.567475+0000", 2024-06-27T21:14:41.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.567475+0000", 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.567476+0000", 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0869999999999999e-06 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.567510+0000", 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4115000000000001e-05 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.735805+0000", 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.168294679 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.735809+0000", 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3470000000000001e-06 2024-06-27T21:14:41.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.735880+0000", 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1153000000000001e-05 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.735895+0000", 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4630999999999999e-05 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.736413+0000", 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051811299999999995 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.736440+0000", 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7064e-05 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:43083 2.7 e123/104 2:e690b9ab:::benchmark_data_smithi192_59818_object43082:head v 123'2689, mlcod=123'2685)", 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.597605+0000", 2024-06-27T21:14:41.520 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.90174927199999, 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.107915047, 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.597605+0000", 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.597605+0000", 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.597606+0000", 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2999999999999999e-07 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.597630+0000", 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3393000000000001e-05 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.704408+0000", 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.106778419 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.704416+0000", 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4159999999999998e-06 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.704461+0000", 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5791000000000003e-05 2024-06-27T21:14:41.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.704477+0000", 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6106000000000001e-05 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.705477+0000", 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00099938400000000008 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.705521+0000", 2024-06-27T21:14:41.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3708000000000002e-05 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:43157 2.7 e123/104 2:e7d51da8:::benchmark_data_smithi192_59818_object43156:head v 123'2695, mlcod=123'2691)", 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.778971+0000", 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.720383527, 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19388234400000001, 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.778971+0000", 2024-06-27T21:14:41.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:41.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.778971+0000", 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.778972+0000", 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5899999999999995e-07 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.779022+0000", 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0269000000000003e-05 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.971646+0000", 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19262373699999999 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.971653+0000", 2024-06-27T21:14:41.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5289999999999997e-06 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.971708+0000", 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5889e-05 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.971731+0000", 2024-06-27T21:14:41.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2385e-05 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.972823+0000", 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001091755 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.972854+0000", 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0920999999999998e-05 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:43158 2.7 e123/104 2:ee159f29:::benchmark_data_smithi192_59818_object43157:head v 123'2696, mlcod=123'2691)", 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.780509+0000", 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.718846227, 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21788435, 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.780509+0000", 2024-06-27T21:14:41.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.780509+0000", 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.780509+0000", 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4299999999999998e-07 2024-06-27T21:14:41.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.780581+0000", 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1926000000000002e-05 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.997255+0000", 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21667425300000001 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.997260+0000", 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6240000000000001e-06 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.997305+0000", 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4798e-05 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.997318+0000", 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3886000000000001e-05 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.998358+0000", 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010398320000000001 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.998393+0000", 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4588000000000002e-05 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.536 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:48540 2.0 e123/104 2:0d1a5795:::benchmark_data_smithi192_59818_object48539:head v 123'2965, mlcod=123'2962)", 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:34.566597+0000", 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 126.932757466, 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12732064400000001, 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.566597+0000", 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.566597+0000", 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.566599+0000", 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.567e-06 2024-06-27T21:14:41.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.566632+0000", 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3325999999999999e-05 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.692295+0000", 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.125662515 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.692308+0000", 2024-06-27T21:14:41.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3317e-05 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.692464+0000", 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000155554 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.692501+0000", 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7611999999999998e-05 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.693883+0000", 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013816849999999999 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:34.693918+0000", 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5067999999999997e-05 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:51515 2.0 e123/104 2:071b7a3b:::benchmark_data_smithi192_59818_object51514:head v 123'3179, mlcod=123'3175)", 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:37.003395+0000", 2024-06-27T21:14:41.541 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 124.495959802, 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.094590327000000002, 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.003395+0000", 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.003395+0000", 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.003396+0000", 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0759999999999999e-06 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.003454+0000", 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8276999999999998e-05 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.096688+0000", 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.093233288999999997 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.096697+0000", 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9679999999999995e-06 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.096747+0000", 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0637999999999999e-05 2024-06-27T21:14:41.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.096763+0000", 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5614e-05 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.097941+0000", 2024-06-27T21:14:41.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011780740000000001 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.097985+0000", 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4391000000000002e-05 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:51718 2.0 e123/104 2:054edf2d:::benchmark_data_smithi192_59818_object51717:head v 123'3192, mlcod=123'3188)", 2024-06-27T21:14:41.546 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:37.298818+0000", 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 124.200536802, 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.177570745, 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.298818+0000", 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.298818+0000", 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.298819+0000", 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2699999999999998e-07 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.298892+0000", 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2679e-05 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.474646+0000", 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17575405499999999 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.474654+0000", 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0909999999999993e-06 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.474751+0000", 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7288999999999996e-05 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.474773+0000", 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2175999999999999e-05 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.476342+0000", 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001569141 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:37.476389+0000", 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6387000000000002e-05 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:56596 2.4 e124/104 2:2917c06d:::benchmark_data_smithi192_59818_object56595:head v 124'3533, mlcod=124'3530)", 2024-06-27T21:14:41.551 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:41.096777+0000", 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 120.402578132, 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.098159837, 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.096777+0000", 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.096777+0000", 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.096780+0000", 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6830000000000001e-06 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.096864+0000", 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3641999999999996e-05 2024-06-27T21:14:41.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.193565+0000", 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.096700942999999998 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.193580+0000", 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5565000000000001e-05 2024-06-27T21:14:41.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.193628+0000", 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7815000000000002e-05 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.193645+0000", 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6620999999999999e-05 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.194905+0000", 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00126063 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.194936+0000", 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0938000000000001e-05 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:83981 2.4 e129/104 2:26b5e778:::benchmark_data_smithi192_59818_object83980:head v 129'5225, mlcod=129'5222)", 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:07.345577+0000", 2024-06-27T21:14:41.556 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 94.153778149999994, 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.102932914, 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.345577+0000", 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.345577+0000", 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.345578+0000", 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.094e-06 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.345620+0000", 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2629000000000001e-05 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.446021+0000", 2024-06-27T21:14:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10040081300000001 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.446029+0000", 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7230000000000004e-06 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.446088+0000", 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9120999999999999e-05 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.446109+0000", 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1016999999999999e-05 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.448459+0000", 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023503920000000002 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.448509+0000", 2024-06-27T21:14:41.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0124999999999999e-05 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:84061 2.4 e129/104 2:237c8434:::benchmark_data_smithi192_59818_object84060:head v 129'5228, mlcod=129'5225)", 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:07.466116+0000", 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 94.033239223999999, 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.095402262000000002, 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.561 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.466116+0000", 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.466116+0000", 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.466117+0000", 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2139999999999999e-06 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.466183+0000", 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6616000000000001e-05 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.558879+0000", 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.092696157000000001 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.558899+0000", 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9256e-05 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.558968+0000", 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9237000000000001e-05 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.558995+0000", 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6757000000000001e-05 2024-06-27T21:14:41.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.561447+0000", 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002452578 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:07.561518+0000", 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0446999999999998e-05 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:107501 2.0 e138/104 2:0a04523b:::benchmark_data_smithi192_59818_object15426:head v 138'6599, mlcod=138'6596)", 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:31.785505+0000", 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.713849577999994, 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14941196200000001, 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.785505+0000", 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.785505+0000", 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.785506+0000", 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.785511+0000", 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.741e-06 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.929658+0000", 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14414717499999999 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.929665+0000", 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6980000000000001e-06 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.929716+0000", 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1097999999999998e-05 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.929745+0000", 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9068999999999999e-05 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.934877+0000", 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0051322640000000001 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.934917+0000", 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0034000000000003e-05 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.570 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:131261 2.4 e145/104 2:27d6bd09:::benchmark_data_smithi192_59818_object39186:head v 145'8183, mlcod=145'8180)", 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:00.805909+0000", 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.693445660999998, 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.103633603, 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.805909+0000", 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.805909+0000", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.805910+0000", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4300000000000001e-07 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.805917+0000", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2400000000000001e-06 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.906711+0000", 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.100793521 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.906718+0000", 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3599999999999998e-06 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.906779+0000", 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0550999999999997e-05 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.906807+0000", 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8705e-05 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.909511+0000", 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027037459999999999 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:00.909543+0000", 2024-06-27T21:14:41.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1537e-05 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:144569 2.7 e151/104 2:e846cb82:::benchmark_data_smithi192_59818_object52494:head v 151'8959, mlcod=151'8956)", 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:19.756036+0000", 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.743318489, 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.087721346000000006, 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.756036+0000", 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.756036+0000", 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.756038+0000", 2024-06-27T21:14:41.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.376e-06 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.756045+0000", 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2880000000000004e-06 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.840736+0000", 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.084691543999999994 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.840745+0000", 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6729999999999998e-06 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.840797+0000", 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1779e-05 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.840816+0000", 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8668000000000002e-05 2024-06-27T21:14:41.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.843708+0000", 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028929139999999999 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:19.843758+0000", 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9104000000000003e-05 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:145023 2.7 e151/104 2:e1601446:::benchmark_data_smithi192_59818_object52948:head v 151'8988, mlcod=151'8985)", 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:20.358531+0000", 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.140823455, 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10038219900000001, 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.358531+0000", 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.358531+0000", 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.358533+0000", 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3349999999999999e-06 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.358542+0000", 2024-06-27T21:14:41.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5189999999999998e-06 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.455646+0000", 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.097104130999999996 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.455655+0000", 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8640000000000002e-06 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.455730+0000", 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4648999999999996e-05 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.455754+0000", 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3808e-05 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.458868+0000", 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031139660000000001 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:20.458913+0000", 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5927000000000003e-05 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:154860 2.4 e157/104 2:2effb58d:::benchmark_data_smithi192_59818_object62785:head v 157'9647, mlcod=157'9644)", 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:36.255615+0000", 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.2437395379999998, 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.096734686, 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.584 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.255615+0000", 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.255615+0000", 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.255616+0000", 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1599999999999999e-06 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.255627+0000", 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0213e-05 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.348492+0000", 2024-06-27T21:14:41.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.092865612 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.348502+0000", 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7140000000000005e-06 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.348576+0000", 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3836000000000003e-05 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.348602+0000", 2024-06-27T21:14:41.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6298000000000001e-05 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.352296+0000", 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036942149999999998 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.352350+0000", 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3637999999999997e-05 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:155050 2.7 e157/104 2:eb6f86c5:::benchmark_data_smithi192_59818_object62975:head v 157'9616, mlcod=157'9613)", 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:36.579723+0000", 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.9196312850000004, 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.104836992, 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.579723+0000", 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.579723+0000", 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.579725+0000", 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.933e-06 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.579738+0000", 2024-06-27T21:14:41.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.259e-05 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.681054+0000", 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.101315716 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.681061+0000", 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9739999999999996e-06 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.681113+0000", 2024-06-27T21:14:41.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2444999999999997e-05 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.681128+0000", 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4983999999999999e-05 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.684511+0000", 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033834030000000001 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:14:41.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:36.684560+0000", 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8946999999999997e-05 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:41.593 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:41.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:41.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:41 smithi151 ceph-mon[28965]: pgmap v475: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 122/84783 objects degraded (0.144%); 8704/84783 objects misplaced (10.266%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:14:41.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:41.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:41.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:41 smithi192 ceph-mon[32956]: pgmap v475: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 122/84783 objects degraded (0.144%); 8704/84783 objects misplaced (10.266%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:14:41.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:41.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:41.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:41.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:41.980 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:42.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:42.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:41 smithi190 ceph-mon[32247]: pgmap v475: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 122/84783 objects degraded (0.144%); 8704/84783 objects misplaced (10.266%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:14:42.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:42.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:42.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:42.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:42.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:42.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:42.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:42.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:42.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:42.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:42.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:42.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:42.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:42.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:42.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:42.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:42 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 202/83007 objects degraded (0.243%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:14:43.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:43.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:42 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 202/83007 objects degraded (0.243%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:14:43.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:43.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:42 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 202/83007 objects degraded (0.243%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:14:43.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:14:43.146 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:14:43.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:43.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:43.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:43.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:43.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:43.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:14:43.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:14:43.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:43.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:43.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:43.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:43.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:43.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:43 smithi192 ceph-mon[32956]: pgmap v476: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 202/83007 objects degraded (0.243%); 8380/83007 objects misplaced (10.096%); 4.2 MiB/s, 66 objects/s recovering 2024-06-27T21:14:43.795 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:43.807 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:43.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:43.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:43.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:43.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:43 smithi151 ceph-mon[28965]: pgmap v476: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 202/83007 objects degraded (0.243%); 8380/83007 objects misplaced (10.096%); 4.2 MiB/s, 66 objects/s recovering 2024-06-27T21:14:44.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:43 smithi190 ceph-mon[32247]: pgmap v476: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 202/83007 objects degraded (0.243%); 8380/83007 objects misplaced (10.096%); 4.2 MiB/s, 66 objects/s recovering 2024-06-27T21:14:44.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:44.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:44.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:44.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:44.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:44.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:44.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:44.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:44.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:44.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:44.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=1142698920,config=2353167246,health=4178928455,kv=3272918494,logm=1089918350}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721547812}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719843747}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20164813}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2415172773}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370828766}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3148753053}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3517245298}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939143921}) 2024-06-27T21:14:44.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4009963805}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3593390754}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468033917}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263761593}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826292668}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208779058}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=444698153}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426649310}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4177272990}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1056092077}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2880629562}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3958991874}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053081535}) 2024-06-27T21:14:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136464142}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237909158}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=99044809,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=1176695316}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=4056331786,mon_config_key=2094130750}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1864463496}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=98,monmap=2} crc {mon_config_key=2382388094,monmap=3552787606}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {monmap=3,osd_metadata=12,osd_pg_creating=1,osdmap=84} crc {monmap=1469187849,osd_metadata=992691294,osd_pg_creating=868418439,osdmap=633618804}) 2024-06-27T21:14:44.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=25} crc {osdmap=3440785326}) 2024-06-27T21:14:44.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:44.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:44.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:44.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:44.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:44.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:44.989 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:14:44.989 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=1142698920,config=2353167246,health=4178928455,kv=3272918494,logm=1089918350}) 2024-06-27T21:14:44.989 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721547812}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719843747}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20164813}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2415172773}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370828766}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3148753053}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3517245298}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939143921}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4009963805}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3593390754}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468033917}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263761593}) 2024-06-27T21:14:44.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826292668}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208779058}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=444698153}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426649310}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4177272990}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1056092077}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2880629562}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3958991874}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053081535}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136464142}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237909158}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=99044809,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=1176695316}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=4056331786,mon_config_key=2094130750}) 2024-06-27T21:14:44.991 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1864463496}) 2024-06-27T21:14:44.992 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=98,monmap=2} crc {mon_config_key=2382388094,monmap=3552787606}) 2024-06-27T21:14:44.992 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {monmap=3,osd_metadata=12,osd_pg_creating=1,osdmap=84} crc {monmap=1469187849,osd_metadata=992691294,osd_pg_creating=868418439,osdmap=633618804}) 2024-06-27T21:14:44.992 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=25} crc {osdmap=3440785326}) 2024-06-27T21:14:44.999 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:14:45.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=1142698920,config=2353167246,health=4178928455,kv=3272918494,logm=1089918350}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721547812}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3719843747}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20164813}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2415172773}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3370828766}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3148753053}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3517245298}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939143921}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4009963805}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3593390754}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1468033917}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3263761593}) 2024-06-27T21:14:45.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826292668}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208779058}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=444698153}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2426649310}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4177272990}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1056092077}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2880629562}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3958991874}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053081535}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1136464142}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237909158}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=99044809,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=1176695316}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=4056331786,mon_config_key=2094130750}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1864463496}) 2024-06-27T21:14:45.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=98,monmap=2} crc {mon_config_key=2382388094,monmap=3552787606}) 2024-06-27T21:14:45.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {monmap=3,osd_metadata=12,osd_pg_creating=1,osdmap=84} crc {monmap=1469187849,osd_metadata=992691294,osd_pg_creating=868418439,osdmap=633618804}) 2024-06-27T21:14:45.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=25} crc {osdmap=3440785326}) 2024-06-27T21:14:45.018 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-06-27T21:14:41.863663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1813709368,"num_objects":27669,"num_object_clones":0,"num_object_copies":83007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":8380,"num_objects_unfound":0,"num_objects_dirty":27669,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":156602,"num_write_kb":5894759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2422,"num_bytes_recovered":160918720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157263,"ondisk_log_size":157263,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6529908,"kb_used_data":5672576,"kb_used_omap":18,"kb_used_meta":857069,"kb_avail":1118411916,"statfs":{"total":1151940427776,"available":1145253801984,"internally_reserved":0,"allocated":5808717824,"data_stored":5805764204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":877638970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,13,4,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-281739264,"num_objects":-4299,"num_object_clones":0,"num_object_copies":-12897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":2477,"num_objects_unfound":0,"num_objects_dirty":-4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4299,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":34930688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001675"},"pg_stats":[{"pgid":"2.f","version":"157'9831","reported_seq":9992,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:37.223873+0000","last_change":"2024-06-27T21:12:42.522399+0000","last_active":"2024-06-27T21:14:37.223873+0000","last_peered":"2024-06-27T21:14:37.223873+0000","last_clean":"2024-06-27T21:14:37.223873+0000","last_became_active":"2024-06-27T21:11:31.339208+0000","last_became_peered":"2024-06-27T21:11:31.339208+0000","last_unstale":"2024-06-27T21:14:37.223873+0000","last_undegraded":"2024-06-27T21:14:37.223873+0000","last_fullsized":"2024-06-27T21:14:37.223873+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3650","last_scrub_stamp":"2024-06-27T21:12:42.522239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.522239+0000","objects_scrubbed":3627,"log_size":9831,"log_dups_size":0,"ondisk_log_size":9831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121700352,"num_objects":1857,"num_object_clones":0,"num_object_copies":5571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":374016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"157'9461","reported_seq":9660,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:38.286428+0000","last_change":"2024-06-27T21:14:13.809056+0000","last_active":"2024-06-27T21:14:38.286428+0000","last_peered":"2024-06-27T21:14:38.286428+0000","last_clean":"2024-06-27T21:14:38.286428+0000","last_became_active":"2024-06-27T21:11:31.337998+0000","last_became_peered":"2024-06-27T21:11:31.337998+0000","last_unstale":"2024-06-27T21:14:38.286428+0000","last_undegraded":"2024-06-27T21:14:38.286428+0000","last_fullsized":"2024-06-27T21:14:38.286428+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'8513","last_scrub_stamp":"2024-06-27T21:14:13.808718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:13.808718+0000","objects_scrubbed":2640,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:23.426457+0000","scrub_duration":485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110297088,"num_objects":1683,"num_object_clones":0,"num_object_copies":5049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"158'9864","reported_seq":10066,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:40.242001+0000","last_change":"2024-06-27T21:14:08.491684+0000","last_active":"2024-06-27T21:14:40.242001+0000","last_peered":"2024-06-27T21:14:40.242001+0000","last_clean":"2024-06-27T21:14:40.242001+0000","last_became_active":"2024-06-27T21:11:31.339229+0000","last_became_peered":"2024-06-27T21:11:31.339229+0000","last_unstale":"2024-06-27T21:14:40.242001+0000","last_undegraded":"2024-06-27T21:14:40.242001+0000","last_fullsized":"2024-06-27T21:14:40.242001+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'8536","last_scrub_stamp":"2024-06-27T21:14:08.491546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:08.491546+0000","objects_scrubbed":3007,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:37.194608+0000","scrub_duration":811,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986392,"num_objects":1664,"num_object_clones":0,"num_object_copies":4992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9864,"num_write_kb":368833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"158'9820","reported_seq":9984,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:39.339175+0000","last_change":"2024-06-27T21:12:45.902924+0000","last_active":"2024-06-27T21:14:39.339175+0000","last_peered":"2024-06-27T21:14:39.339175+0000","last_clean":"2024-06-27T21:14:39.339175+0000","last_became_active":"2024-06-27T21:11:31.556192+0000","last_became_peered":"2024-06-27T21:11:31.556192+0000","last_unstale":"2024-06-27T21:14:39.339175+0000","last_undegraded":"2024-06-27T21:14:39.339175+0000","last_fullsized":"2024-06-27T21:14:39.339175+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3891","last_scrub_stamp":"2024-06-27T21:12:45.902837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:45.902837+0000","objects_scrubbed":3854,"log_size":9820,"log_dups_size":0,"ondisk_log_size":9820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":369536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":276,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:38.650304+0000","last_change":"2024-06-27T21:12:50.081819+0000","last_active":"2024-06-27T21:14:38.650304+0000","last_peered":"2024-06-27T21:14:38.650304+0000","last_clean":"2024-06-27T21:14:38.650304+0000","last_became_active":"2024-06-27T21:09:39.669723+0000","last_became_peered":"2024-06-27T21:09:39.669723+0000","last_unstale":"2024-06-27T21:14:38.650304+0000","last_undegraded":"2024-06-27T21:14:38.650304+0000","last_fullsized":"2024-06-27T21:14:38.650304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:12:50.081718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:12:50.081718+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"157'9714","reported_seq":9909,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:37.356335+0000","last_change":"2024-06-27T21:14:18.262904+0000","last_active":"2024-06-27T21:14:37.356335+0000","last_peered":"2024-06-27T21:14:37.356335+0000","last_clean":"2024-06-27T21:14:37.356335+0000","last_became_active":"2024-06-27T21:11:31.337613+0000","last_became_peered":"2024-06-27T21:11:31.337613+0000","last_unstale":"2024-06-27T21:14:37.356335+0000","last_undegraded":"2024-06-27T21:14:37.356335+0000","last_fullsized":"2024-06-27T21:14:37.356335+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'8961","last_scrub_stamp":"2024-06-27T21:14:18.262761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:18.262761+0000","objects_scrubbed":2637,"log_size":9714,"log_dups_size":0,"ondisk_log_size":9714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:39.364162+0000","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121896960,"num_objects":1860,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9714,"num_write_kb":370368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"158'9663","reported_seq":18650,"reported_epoch":158,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:14:39.570036+0000","last_change":"2024-06-27T21:13:47.955243+0000","last_active":"2024-06-27T21:14:39.570036+0000","last_peered":"2024-06-27T21:14:39.570036+0000","last_clean":"2024-06-27T21:13:45.604588+0000","last_became_active":"2024-06-27T21:13:47.901845+0000","last_became_peered":"2024-06-27T21:13:47.901845+0000","last_unstale":"2024-06-27T21:14:39.570036+0000","last_undegraded":"2024-06-27T21:14:39.570036+0000","last_fullsized":"2024-06-27T21:13:47.640498+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3530","last_scrub_stamp":"2024-06-27T21:12:41.700386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:41.700386+0000","objects_scrubbed":3520,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":4911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4910,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":361600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2205,"num_bytes_recovered":144506880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[7,6],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"6,7,9,10","objects":1},{"shards":"7,9,10","objects":1636}],"blocked_by":[],"up_primary":11,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"157'10114","reported_seq":10320,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:38.248023+0000","last_change":"2024-06-27T21:14:11.783997+0000","last_active":"2024-06-27T21:14:38.248023+0000","last_peered":"2024-06-27T21:14:38.248023+0000","last_clean":"2024-06-27T21:14:38.248023+0000","last_became_active":"2024-06-27T21:11:31.556127+0000","last_became_peered":"2024-06-27T21:11:31.556127+0000","last_unstale":"2024-06-27T21:14:38.248023+0000","last_undegraded":"2024-06-27T21:14:38.248023+0000","last_fullsized":"2024-06-27T21:14:38.248023+0000","mapping_epoch":104,"log_start":"108'100","ondisk_log_start":"108'100","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'9012","last_scrub_stamp":"2024-06-27T21:14:11.783045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:11.783045+0000","objects_scrubbed":2901,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:21.423826+0000","scrub_duration":1351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":379776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"157'9770","reported_seq":9972,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:37.334769+0000","last_change":"2024-06-27T21:14:08.219342+0000","last_active":"2024-06-27T21:14:37.334769+0000","last_peered":"2024-06-27T21:14:37.334769+0000","last_clean":"2024-06-27T21:14:37.334769+0000","last_became_active":"2024-06-27T21:11:31.337786+0000","last_became_peered":"2024-06-27T21:11:31.337786+0000","last_unstale":"2024-06-27T21:14:37.334769+0000","last_undegraded":"2024-06-27T21:14:37.334769+0000","last_fullsized":"2024-06-27T21:14:37.334769+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'8555","last_scrub_stamp":"2024-06-27T21:14:08.219152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:08.219152+0000","objects_scrubbed":3106,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:18.227207+0000","scrub_duration":679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122421248,"num_objects":1868,"num_object_clones":0,"num_object_copies":5604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":372416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"158'9761","reported_seq":9961,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:38.763468+0000","last_change":"2024-06-27T21:14:13.085036+0000","last_active":"2024-06-27T21:14:38.763468+0000","last_peered":"2024-06-27T21:14:38.763468+0000","last_clean":"2024-06-27T21:14:38.763468+0000","last_became_active":"2024-06-27T21:11:31.556366+0000","last_became_peered":"2024-06-27T21:11:31.556366+0000","last_unstale":"2024-06-27T21:14:38.763468+0000","last_undegraded":"2024-06-27T21:14:38.763468+0000","last_fullsized":"2024-06-27T21:14:38.763468+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'8722","last_scrub_stamp":"2024-06-27T21:14:13.084898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:13.084898+0000","objects_scrubbed":2761,"log_size":9761,"log_dups_size":0,"ondisk_log_size":9761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:21.377489+0000","scrub_duration":748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9761,"num_write_kb":366848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'12151","reported_seq":12467,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:37.227627+0000","last_change":"2024-06-27T21:13:08.494332+0000","last_active":"2024-06-27T21:14:37.227627+0000","last_peered":"2024-06-27T21:14:37.227627+0000","last_clean":"2024-06-27T21:14:37.227627+0000","last_became_active":"2024-06-27T21:11:31.556322+0000","last_became_peered":"2024-06-27T21:11:31.556322+0000","last_unstale":"2024-06-27T21:14:37.227627+0000","last_undegraded":"2024-06-27T21:14:37.227627+0000","last_fullsized":"2024-06-27T21:14:37.227627+0000","mapping_epoch":104,"log_start":"121'2100","ondisk_log_start":"121'2100","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7832","last_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_deep_scrub":"129'7832","last_deep_scrub_stamp":"2024-06-27T21:13:08.494150+0000","last_clean_scrub_stamp":"2024-06-27T21:13:08.494150+0000","objects_scrubbed":4784,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"queued for scrub","scrub_duration":14657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117374976,"num_objects":1791,"num_object_clones":0,"num_object_copies":5373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9567,"num_write_kb":363456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"158'10028","reported_seq":10210,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:38.746331+0000","last_change":"2024-06-27T21:14:23.522439+0000","last_active":"2024-06-27T21:14:38.746331+0000","last_peered":"2024-06-27T21:14:38.746331+0000","last_clean":"2024-06-27T21:14:38.746331+0000","last_became_active":"2024-06-27T21:11:31.338960+0000","last_became_peered":"2024-06-27T21:11:31.338960+0000","last_unstale":"2024-06-27T21:14:38.746331+0000","last_undegraded":"2024-06-27T21:14:38.746331+0000","last_fullsized":"2024-06-27T21:14:38.746331+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4740","last_scrub_stamp":"2024-06-27T21:12:56.263856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:56.263856+0000","objects_scrubbed":549,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:43.225584+0000","scrub_duration":1215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113508352,"num_objects":1732,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10028,"num_write_kb":376320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"158'9770","reported_seq":9935,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:40.000794+0000","last_change":"2024-06-27T21:12:44.319499+0000","last_active":"2024-06-27T21:14:40.000794+0000","last_peered":"2024-06-27T21:14:40.000794+0000","last_clean":"2024-06-27T21:14:40.000794+0000","last_became_active":"2024-06-27T21:11:31.943105+0000","last_became_peered":"2024-06-27T21:11:31.943105+0000","last_unstale":"2024-06-27T21:14:40.000794+0000","last_undegraded":"2024-06-27T21:14:40.000794+0000","last_fullsized":"2024-06-27T21:14:40.000794+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3747","last_scrub_stamp":"2024-06-27T21:12:44.319331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:44.319331+0000","objects_scrubbed":3704,"log_size":9770,"log_dups_size":0,"ondisk_log_size":9770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103940096,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9770,"num_write_kb":363392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"158'9756","reported_seq":10546,"reported_epoch":158,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:14:40.530470+0000","last_change":"2024-06-27T21:14:31.777478+0000","last_active":"2024-06-27T21:14:40.530470+0000","last_peered":"2024-06-27T21:14:40.530470+0000","last_clean":"2024-06-27T21:14:29.467366+0000","last_became_active":"2024-06-27T21:14:31.694604+0000","last_became_peered":"2024-06-27T21:14:31.694604+0000","last_unstale":"2024-06-27T21:14:40.530470+0000","last_undegraded":"2024-06-27T21:14:31.629494+0000","last_fullsized":"2024-06-27T21:14:31.493641+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'4512","last_scrub_stamp":"2024-06-27T21:12:55.073215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:55.073215+0000","objects_scrubbed":4477,"log_size":9756,"log_dups_size":0,"ondisk_log_size":9756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107085824,"num_objects":1634,"num_object_clones":0,"num_object_copies":4902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":3470,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9756,"num_write_kb":364480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":13238272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,9],"acting":[0,9],"avail_no_missing":["0","7","9"],"object_location_counts":[{"shards":"0,7,9","objects":1634},{"shards":"0,9","objects":-202}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"158'9821","reported_seq":10047,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:38.763113+0000","last_change":"2024-06-27T21:14:12.955909+0000","last_active":"2024-06-27T21:14:38.763113+0000","last_peered":"2024-06-27T21:14:38.763113+0000","last_clean":"2024-06-27T21:14:38.763113+0000","last_became_active":"2024-06-27T21:11:31.337888+0000","last_became_peered":"2024-06-27T21:11:31.337888+0000","last_unstale":"2024-06-27T21:14:38.763113+0000","last_undegraded":"2024-06-27T21:14:38.763113+0000","last_fullsized":"2024-06-27T21:14:38.763113+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'8786","last_scrub_stamp":"2024-06-27T21:14:12.955686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:14:12.955686+0000","objects_scrubbed":2839,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:15:33.163489+0000","scrub_duration":1661,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115277824,"num_objects":1759,"num_object_clones":0,"num_object_copies":5277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":370560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"158'9768","reported_seq":9930,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-06-27T21:14:39.246865+0000","last_change":"2024-06-27T21:12:42.976967+0000","last_active":"2024-06-27T21:14:39.246865+0000","last_peered":"2024-06-27T21:14:39.246865+0000","last_clean":"2024-06-27T21:14:39.246865+0000","last_became_active":"2024-06-27T21:11:31.340369+0000","last_became_peered":"2024-06-27T21:11:31.340369+0000","last_unstale":"2024-06-27T21:14:39.246865+0000","last_undegraded":"2024-06-27T21:14:39.246865+0000","last_fullsized":"2024-06-27T21:14:39.246865+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'3653","last_scrub_stamp":"2024-06-27T21:12:42.976769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:12:42.976769+0000","objects_scrubbed":3623,"log_size":9768,"log_dups_size":0,"ondisk_log_size":9768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":770,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107479040,"num_objects":1640,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9768,"num_write_kb":365056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"157'12179","reported_seq":12512,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-06-27T21:14:37.207892+0000","last_change":"2024-06-27T21:13:29.274450+0000","last_active":"2024-06-27T21:14:37.207892+0000","last_peered":"2024-06-27T21:14:37.207892+0000","last_clean":"2024-06-27T21:14:37.207892+0000","last_became_active":"2024-06-27T21:11:31.337915+0000","last_became_peered":"2024-06-27T21:11:31.337915+0000","last_unstale":"2024-06-27T21:14:37.207892+0000","last_undegraded":"2024-06-27T21:14:37.207892+0000","last_fullsized":"2024-06-27T21:14:37.207892+0000","mapping_epoch":104,"log_start":"121'2100","ondisk_log_start":"121'2100","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'9000","last_scrub_stamp":"2024-06-27T21:13:29.274287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:11:30.322172+0000","last_clean_scrub_stamp":"2024-06-27T21:13:29.274287+0000","objects_scrubbed":5002,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:14:59.000982+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116064256,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1813119000,"num_objects":27667,"num_object_clones":0,"num_object_copies":83001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":8380,"num_objects_unfound":0,"num_objects_dirty":27667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156479,"num_write_kb":5892609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2410,"num_bytes_recovered":157745152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5722812416,"data_stored":5722800200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157171,"ondisk_log_size":157171,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3173568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252928,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407944,"kb_used_data":332352,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93337208,"statfs":{"total":95995035648,"available":95577300992,"internally_reserved":0,"allocated":340328448,"data_stored":340082512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515088,"kb_used_data":457540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93230064,"statfs":{"total":95995035648,"available":95467585536,"internally_reserved":0,"allocated":468520960,"data_stored":468274124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743497,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853268,"kb_used_data":781832,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92891884,"statfs":{"total":95995035648,"available":95121289216,"internally_reserved":0,"allocated":800595968,"data_stored":800345060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972429,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579520,"kb_used_data":467832,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93165632,"statfs":{"total":95995035648,"available":95401607168,"internally_reserved":0,"allocated":479059968,"data_stored":478817604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266770,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":644696,"kb_used_data":573260,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93100456,"statfs":{"total":95995035648,"available":95334866944,"internally_reserved":0,"allocated":587018240,"data_stored":586767852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528407,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":294552,"kb_used_data":218956,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93450600,"statfs":{"total":95995035648,"available":95693414400,"internally_reserved":0,"allocated":224210944,"data_stored":223960556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300160,"kb_used_data":242616,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93444992,"statfs":{"total":95995035648,"available":95687671808,"internally_reserved":0,"allocated":248438784,"data_stored":248196420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408324,"kb_used_data":345212,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93336828,"statfs":{"total":95995035648,"available":95576911872,"internally_reserved":0,"allocated":353497088,"data_stored":353250652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755332,"kb_used_data":687932,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92989820,"statfs":{"total":95995035648,"available":95221575680,"internally_reserved":0,"allocated":704442368,"data_stored":704195932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":524928,"kb_used_data":453496,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93220224,"statfs":{"total":95995035648,"available":95457509376,"internally_reserved":0,"allocated":464379904,"data_stored":464137540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":632384,"kb_used_data":569272,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93112768,"statfs":{"total":95995035648,"available":95347474432,"internally_reserved":0,"allocated":582934528,"data_stored":582692164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":613712,"kb_used_data":542276,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93131440,"statfs":{"total":95995035648,"available":95366594560,"internally_reserved":0,"allocated":555290624,"data_stored":555043788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":540672000,"data_stored":540672000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":574226432,"data_stored":574226432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":453246976,"data_stored":453246976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":696782848,"data_stored":696778776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":350752768,"data_stored":350748696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":244252672,"data_stored":244252672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":219283456,"data_stored":219283456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":579207168,"data_stored":579207168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":473563136,"data_stored":473563136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":792858624,"data_stored":792854552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":466485248,"data_stored":466485248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":331481088,"data_stored":331481088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:14:45.020 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-27T21:14:45.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-06-27T21:14:45.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:45.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:45.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:45.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:45.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:45.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:45.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:45.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:45.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:45.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:45.723 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:14:45.723 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:161172 2.c 2:3807061f:::benchmark_data_smithi192_59818_object69097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:45.715142+0000", 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014335907, 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014367708, 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:45.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161172 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.715142+0000", 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.715142+0000", 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.715144+0000", 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.562e-06 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.715150+0000", 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6849999999999998e-06 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:45.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.715165+0000", 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5369000000000001e-05 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.715175+0000", 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.516e-06 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.717507+0000", 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023328979999999999 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.717673+0000", 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000165985 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.717794+0000", 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120364 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.722188+0000", 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043946139999999998 2024-06-27T21:14:45.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.726713+0000", 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004524943 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:161167 2.c 2:316a223d:::benchmark_data_smithi192_59818_object69092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:45.712006+0000", 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017472219000000001, 2024-06-27T21:14:45.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017552247, 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161167 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:14:45.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712006+0000", 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712006+0000", 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:14:45.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712008+0000", 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.655e-06 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712014+0000", 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0440000000000001e-06 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:14:45.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712028+0000", 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4124000000000001e-05 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712036+0000", 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3250000000000008e-06 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712242+0000", 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000205362 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712632+0000", 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038998899999999998 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.712741+0000", 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109472 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.722044+0000", 2024-06-27T21:14:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093033190000000009 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:45.722608+0000", 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056408000000000001 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-06-27T21:14:45.735 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:14:45.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:45.768 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:45.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:45.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:45.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:45.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:45 smithi151 ceph-mon[28965]: pgmap v477: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 202/76506 objects degraded (0.264%); 8380/76506 objects misplaced (10.953%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:14:45.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:45 smithi151 ceph-mon[28965]: 2.6 scrub starts 2024-06-27T21:14:45.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:45 smithi151 ceph-mon[28965]: 2.6 scrub ok 2024-06-27T21:14:45.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:46.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:45 smithi190 ceph-mon[32247]: pgmap v477: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 202/76506 objects degraded (0.264%); 8380/76506 objects misplaced (10.953%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:14:46.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:45 smithi190 ceph-mon[32247]: 2.6 scrub starts 2024-06-27T21:14:46.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:45 smithi190 ceph-mon[32247]: 2.6 scrub ok 2024-06-27T21:14:46.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:45 smithi192 ceph-mon[32956]: pgmap v477: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 202/76506 objects degraded (0.264%); 8380/76506 objects misplaced (10.953%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:14:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:45 smithi192 ceph-mon[32956]: 2.6 scrub starts 2024-06-27T21:14:46.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:45 smithi192 ceph-mon[32956]: 2.6 scrub ok 2024-06-27T21:14:46.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:46.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:46.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:46.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:46.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:46.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:46.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:46.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:14:46.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:46.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:46.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:46.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:46.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:46.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:46.930 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:46.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:46.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:47.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:47.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:47.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:47.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:47.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:47.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:47.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:47.428 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:47.461 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:47.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:47.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:47.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:47.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:47 smithi151 ceph-mon[28965]: pgmap v478: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 310/72927 objects degraded (0.425%); 7464/72927 objects misplaced (10.235%); 3.9 MiB/s, 63 objects/s recovering 2024-06-27T21:14:47.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2714010237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:47.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:47.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:47.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:47.925 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:47 smithi192 ceph-mon[32956]: pgmap v478: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 310/72927 objects degraded (0.425%); 7464/72927 objects misplaced (10.235%); 3.9 MiB/s, 63 objects/s recovering 2024-06-27T21:14:47.925 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2714010237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:47.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:47.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:47.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:48.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:47 smithi190 ceph-mon[32247]: pgmap v478: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 310/72927 objects degraded (0.425%); 7464/72927 objects misplaced (10.235%); 3.9 MiB/s, 63 objects/s recovering 2024-06-27T21:14:48.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2714010237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:14:48.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:48.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:48.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:48.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:48.331 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:14:48.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:14:48.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:48.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:48.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:14:48.368 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:14:48.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:14:48.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:14:48.455 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:14:48.455 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:14:48.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:48.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:48.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:14:48.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:14:48.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:14:48.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:14:48.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:14:48.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:14:48.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:48.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:14:48.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:14:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:48.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:48.628 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:14:48.628 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:14:48.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:14:48.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:14:48.665 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:14:48.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:14:48.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:14:48.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:48.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:14:48.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:14:48.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.756 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:48.847 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:48.847 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:48.847 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3057959928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:48.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:48.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:48.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:48.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3057959928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:48.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:49.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3057959928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:14:49.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:49.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:49.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:49.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:49.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:49.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:49.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:49.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:49.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:49.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:14:49.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:49.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:49.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:49 smithi151 ceph-mon[28965]: pgmap v479: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 310/69849 objects degraded (0.444%); 7464/69849 objects misplaced (10.686%); 3.9 MiB/s, 63 objects/s recovering 2024-06-27T21:14:49.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 310/69849 objects degraded (0.444%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:14:49.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:49 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:14:49.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3057959928' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:49.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:49 smithi151 ceph-mon[28965]: osdmap e159: 12 total, 12 up, 12 in 2024-06-27T21:14:49.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:49.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:49 smithi192 ceph-mon[32956]: pgmap v479: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 310/69849 objects degraded (0.444%); 7464/69849 objects misplaced (10.686%); 3.9 MiB/s, 63 objects/s recovering 2024-06-27T21:14:49.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 310/69849 objects degraded (0.444%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:14:49.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:49 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:14:49.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3057959928' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:49.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:49 smithi192 ceph-mon[32956]: osdmap e159: 12 total, 12 up, 12 in 2024-06-27T21:14:49.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:49.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:49.917 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:14:49.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:49.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:49 smithi190 ceph-mon[32247]: pgmap v479: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 310/69849 objects degraded (0.444%); 7464/69849 objects misplaced (10.686%); 3.9 MiB/s, 63 objects/s recovering 2024-06-27T21:14:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 310/69849 objects degraded (0.444%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:14:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:49 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:14:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3057959928' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:14:50.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:49 smithi190 ceph-mon[32247]: osdmap e159: 12 total, 12 up, 12 in 2024-06-27T21:14:50.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:50.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:50.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:50.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:50.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:50.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:50.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:50.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:50.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:50.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:50.661 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 11 2024-06-27T21:14:50.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:50.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:50.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:50.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:50.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:50.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/440068041' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-27T21:14:50.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-27T21:14:50.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:50.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:51.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/440068041' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-27T21:14:51.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-27T21:14:51.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/440068041' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-27T21:14:51.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-06-27T21:14:51.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:51.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:51.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:51.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:51.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:51.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:51.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:51.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:14:51.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:51.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:51.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:51.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:51.786 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:51.786 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:51.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:51 smithi192 ceph-mon[32956]: pgmap v481: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 310/64773 objects degraded (0.479%); 6834/64773 objects misplaced (10.551%); 4.0 MiB/s, 63 objects/s recovering 2024-06-27T21:14:51.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-27T21:14:51.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:51 smithi192 ceph-mon[32956]: osdmap e160: 12 total, 12 up, 12 in 2024-06-27T21:14:51.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:51.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:51.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:51.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:51 smithi151 ceph-mon[28965]: pgmap v481: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 310/64773 objects degraded (0.479%); 6834/64773 objects misplaced (10.551%); 4.0 MiB/s, 63 objects/s recovering 2024-06-27T21:14:51.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-27T21:14:51.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:51 smithi151 ceph-mon[28965]: osdmap e160: 12 total, 12 up, 12 in 2024-06-27T21:14:51.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:51.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:52.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:51 smithi190 ceph-mon[32247]: pgmap v481: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 310/64773 objects degraded (0.479%); 6834/64773 objects misplaced (10.551%); 4.0 MiB/s, 63 objects/s recovering 2024-06-27T21:14:52.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-06-27T21:14:52.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:51 smithi190 ceph-mon[32247]: osdmap e160: 12 total, 12 up, 12 in 2024-06-27T21:14:52.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:52.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:52.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:52.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:52.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:14:52.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:52.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:52.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:52.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:52.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:14:52.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:52.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:52.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:52.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:52.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:52.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:52.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:52.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-27T21:14:52.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:14:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:52.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-27T21:14:53.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:14:53.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-06-27T21:14:53.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:14:53.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:53.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:53.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:53.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:53.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:53.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:53.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:53.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:53.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:53.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:53.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:53.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:53.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:53.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:53.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: pgmap v483: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 394/62973 objects degraded (0.626%); 6527/62973 objects misplaced (10.365%); 4.9 MiB/s, 78 objects/s recovering 2024-06-27T21:14:53.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-27T21:14:53.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: osdmap e161: 12 total, 12 up, 12 in 2024-06-27T21:14:53.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:53.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:53.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:53.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:53 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:53.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:53.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:53.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:54.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: pgmap v483: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 394/62973 objects degraded (0.626%); 6527/62973 objects misplaced (10.365%); 4.9 MiB/s, 78 objects/s recovering 2024-06-27T21:14:54.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-27T21:14:54.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: osdmap e161: 12 total, 12 up, 12 in 2024-06-27T21:14:54.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:53 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: pgmap v483: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 394/62973 objects degraded (0.626%); 6527/62973 objects misplaced (10.365%); 4.9 MiB/s, 78 objects/s recovering 2024-06-27T21:14:54.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-06-27T21:14:54.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: osdmap e161: 12 total, 12 up, 12 in 2024-06-27T21:14:54.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:53 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:54.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:54.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:54.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:54.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:54.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:54.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:54.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:54.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:54.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:54.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:54.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:54.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:54.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:54.881 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:54.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:54.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:54 smithi151 ceph-mon[28965]: osdmap e162: 12 total, 12 up, 12 in 2024-06-27T21:14:54.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-27T21:14:54.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:54 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:14:54.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:54 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:14:54.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 394/62973 objects degraded (0.626%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:14:54.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:55.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:54 smithi190 ceph-mon[32247]: osdmap e162: 12 total, 12 up, 12 in 2024-06-27T21:14:55.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-27T21:14:55.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:54 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:14:55.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:54 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:14:55.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 394/62973 objects degraded (0.626%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:14:55.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:55.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:54 smithi192 ceph-mon[32956]: osdmap e162: 12 total, 12 up, 12 in 2024-06-27T21:14:55.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-06-27T21:14:55.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:54 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:14:55.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:54 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:14:55.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 394/62973 objects degraded (0.626%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:14:55.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:55.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:55.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:55.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:55.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:55.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:55.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:55.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:55.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:55.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:55.744 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:55.752 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:14:55.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:55.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:55.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: pgmap v486: 27 pgs: 10 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.2 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 394/56652 objects degraded (0.695%); 6527/56652 objects misplaced (11.521%); 3.9 MiB/s, 61 objects/s recovering 2024-06-27T21:14:55.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-27T21:14:55.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: osdmap e163: 12 total, 12 up, 12 in 2024-06-27T21:14:55.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:14:55.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:14:55.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:14:55.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:14:55.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:55.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: pgmap v486: 27 pgs: 10 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.2 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 394/56652 objects degraded (0.695%); 6527/56652 objects misplaced (11.521%); 3.9 MiB/s, 61 objects/s recovering 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: osdmap e163: 12 total, 12 up, 12 in 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:14:55.939 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:14:55.940 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:55.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:55.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:56.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: pgmap v486: 27 pgs: 10 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 1.2 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 763 op/s; 394/56652 objects degraded (0.695%); 6527/56652 objects misplaced (11.521%); 3.9 MiB/s, 61 objects/s recovering 2024-06-27T21:14:56.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-06-27T21:14:56.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: osdmap e163: 12 total, 12 up, 12 in 2024-06-27T21:14:56.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:14:56.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:14:56.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:14:56.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:14:56.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:14:56.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:56.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:56.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:56.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:56.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:14:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:56.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:14:56.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:56.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:56.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:56.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:56.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:56.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:56.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:56.836 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:56.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:14:56.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:56.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:14:57.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:56 smithi190 ceph-mon[32247]: osdmap e164: 12 total, 12 up, 12 in 2024-06-27T21:14:57.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:56 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:14:57.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:56 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:14:57.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:56 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:14:57.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:56 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:14:57.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:56 smithi192 ceph-mon[32956]: osdmap e164: 12 total, 12 up, 12 in 2024-06-27T21:14:57.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:56 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:14:57.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:56 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:14:57.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:56 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:14:57.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:56 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:14:57.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:14:57.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:57.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:56 smithi151 ceph-mon[28965]: osdmap e164: 12 total, 12 up, 12 in 2024-06-27T21:14:57.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:56 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:14:57.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:56 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:14:57.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:56 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:14:57.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:56 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:14:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:57.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:57.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:57.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:57.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:57.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:57.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:57.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:57.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:57.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:57.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:57.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:57.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:14:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:57.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:57.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:14:57.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: pgmap v489: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+remapped, 23 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s; 473/52623 objects degraded (0.899%); 5483/52623 objects misplaced (10.419%); 5.7 MiB/s, 92 objects/s recovering 2024-06-27T21:14:57.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:14:57.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:14:57.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:14:57.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.a deep-scrub starts 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.a deep-scrub ok 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:14:57.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:14:57.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:57 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:14:57.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:58.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: pgmap v489: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+remapped, 23 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s; 473/52623 objects degraded (0.899%); 5483/52623 objects misplaced (10.419%); 5.7 MiB/s, 92 objects/s recovering 2024-06-27T21:14:58.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:14:58.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.a deep-scrub starts 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.a deep-scrub ok 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:14:58.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:57 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:14:58.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:58.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: pgmap v489: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+remapped, 23 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s; 473/52623 objects degraded (0.899%); 5483/52623 objects misplaced (10.419%); 5.7 MiB/s, 92 objects/s recovering 2024-06-27T21:14:58.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:14:58.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:14:58.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.a deep-scrub starts 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.a deep-scrub ok 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:14:58.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:57 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:14:58.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:58.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:58.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:58.270 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:14:58.270 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:14:58.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:58.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:14:58.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:14:58.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:14:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:58.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:14:58.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:14:58.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:14:58.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:14:58.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:14:58.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:58.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:58.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:58.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:58 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:14:58.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:58 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:14:58.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:58 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:14:59.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:58 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:14:59.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:58 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:14:59.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:58 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:14:59.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:59.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:58 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:14:59.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:58 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:14:59.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:58 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:14:59.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:59.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:59.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:59.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:59.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:59.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:14:59.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:14:59.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:14:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:59.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:14:59.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:14:59.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:14:59.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:14:59.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:14:59.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:14:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:14:59.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:14:59.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:14:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:14:59.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:00.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:59 smithi190 ceph-mon[32247]: pgmap v490: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+remapped, 23 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 473/50721 objects degraded (0.933%); 5483/50721 objects misplaced (10.810%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:15:00.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:59 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:15:00.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:14:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 473/50721 objects degraded (0.933%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:00.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:59 smithi151 ceph-mon[28965]: pgmap v490: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+remapped, 23 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 473/50721 objects degraded (0.933%); 5483/50721 objects misplaced (10.810%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:15:00.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:59 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:15:00.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:14:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 473/50721 objects degraded (0.933%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:00.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:59 smithi192 ceph-mon[32956]: pgmap v490: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+remapped, 23 active+clean; 1.0 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 473/50721 objects degraded (0.933%); 5483/50721 objects misplaced (10.810%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:15:00.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:59 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:15:00.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:14:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 473/50721 objects degraded (0.933%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:00.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:15:00.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:00.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:00.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:00.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:00.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:00.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:00.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:00.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:00.625 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:15:00.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 0 1 2024-06-27T21:15:00.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:00.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:00.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:00.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:00.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:00.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:00.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:01.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:01.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:01.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:01.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:01.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:01.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:01.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:01.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:01.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:01.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:01.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:01.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:01.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:01.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:01.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:01 smithi151 ceph-mon[28965]: pgmap v491: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 23 active+clean; 918 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 473/44058 objects degraded (1.074%); 4847/44058 objects misplaced (11.001%); 5.5 MiB/s, 92 objects/s recovering 2024-06-27T21:15:01.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3665474693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:01.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:01.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:01.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:02.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:01 smithi190 ceph-mon[32247]: pgmap v491: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 23 active+clean; 918 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 473/44058 objects degraded (1.074%); 4847/44058 objects misplaced (11.001%); 5.5 MiB/s, 92 objects/s recovering 2024-06-27T21:15:02.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3665474693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:02.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:01 smithi192 ceph-mon[32956]: pgmap v491: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 23 active+clean; 918 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 473/44058 objects degraded (1.074%); 4847/44058 objects misplaced (11.001%); 5.5 MiB/s, 92 objects/s recovering 2024-06-27T21:15:02.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3665474693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:02.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:02.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:02.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:02.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:02.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:02.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:02.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:02.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:02.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:15:02.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:15:02.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:15:02.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:15:02.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:15:02.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:15:02.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:15:02.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:15:02.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:02.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:02.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:02.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:02.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:02.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:02.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:02 smithi151 ceph-mon[28965]: pgmap v492: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 877 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 512/42087 objects degraded (1.217%); 4490/42087 objects misplaced (10.668%); 5.4 MiB/s, 91 objects/s recovering 2024-06-27T21:15:02.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:02 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-27T21:15:02.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:02.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:02.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:03.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:02 smithi192 ceph-mon[32956]: pgmap v492: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 877 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 512/42087 objects degraded (1.217%); 4490/42087 objects misplaced (10.668%); 5.4 MiB/s, 91 objects/s recovering 2024-06-27T21:15:03.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:02 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-27T21:15:03.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:03.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:03.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:03.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:03.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:02 smithi190 ceph-mon[32247]: pgmap v492: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 877 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 512/42087 objects degraded (1.217%); 4490/42087 objects misplaced (10.668%); 5.4 MiB/s, 91 objects/s recovering 2024-06-27T21:15:03.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:02 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-06-27T21:15:03.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:03.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:03.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:03.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:03.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:03.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:03.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:03.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:03.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:03.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:03.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:03.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:03.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:03.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:03 smithi151 ceph-mon[28965]: osdmap e165: 12 total, 12 up, 12 in 2024-06-27T21:15:03.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:04.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:04.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:04.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:04.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:03 smithi192 ceph-mon[32956]: osdmap e165: 12 total, 12 up, 12 in 2024-06-27T21:15:04.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:04.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:04.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:04.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:04.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:04.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:03 smithi190 ceph-mon[32247]: osdmap e165: 12 total, 12 up, 12 in 2024-06-27T21:15:04.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:04.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:04.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:04.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:04.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:04.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:04.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:04.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:04.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:04.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:04.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:04.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:04.977 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:05.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:05.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:04 smithi192 ceph-mon[32956]: pgmap v494: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 691 op/s; 512/35334 objects degraded (1.449%); 4490/35334 objects misplaced (12.707%); 2.5 MiB/s, 44 objects/s recovering 2024-06-27T21:15:05.038 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 512/42087 objects degraded (1.217%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:05.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:05.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:04 smithi151 ceph-mon[28965]: pgmap v494: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 691 op/s; 512/35334 objects degraded (1.449%); 4490/35334 objects misplaced (12.707%); 2.5 MiB/s, 44 objects/s recovering 2024-06-27T21:15:05.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 512/42087 objects degraded (1.217%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:05.158 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:04 smithi190 ceph-mon[32247]: pgmap v494: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 737 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 691 op/s; 512/35334 objects degraded (1.449%); 4490/35334 objects misplaced (12.707%); 2.5 MiB/s, 44 objects/s recovering 2024-06-27T21:15:05.158 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 512/42087 objects degraded (1.217%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:05.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:05.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:05.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:05.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:05.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:05.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:05.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:05.560 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:05.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:05.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:05.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:05.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:05.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:05.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:05.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:05.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:06.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:06.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:06.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:06.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:06.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:06.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:06.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:06.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:06.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:06.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:06.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:06.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:06.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:06.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:06.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:06.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:15:06.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:07.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:06 smithi190 ceph-mon[32247]: pgmap v495: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 661 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 594/31695 objects degraded (1.874%); 3569/31695 objects misplaced (11.260%); 4.0 MiB/s, 68 objects/s recovering 2024-06-27T21:15:07.014 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:06 smithi151 ceph-mon[28965]: pgmap v495: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 661 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 594/31695 objects degraded (1.874%); 3569/31695 objects misplaced (11.260%); 4.0 MiB/s, 68 objects/s recovering 2024-06-27T21:15:07.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:07.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:07.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:07.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:07.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:06 smithi192 ceph-mon[32956]: pgmap v495: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 661 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 594/31695 objects degraded (1.874%); 3569/31695 objects misplaced (11.260%); 4.0 MiB/s, 68 objects/s recovering 2024-06-27T21:15:07.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:07.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:07.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:07.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:07.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:07.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:07.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:07.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:07.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:07.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:07.932 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:15:07.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:07.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:08.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:08.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:08.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:08.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3927848522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:15:08.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:07 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:15:08.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:08.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3927848522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:15:08.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:07 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:15:08.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3927848522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:15:08.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:07 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:15:08.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:08.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:08.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:08.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:08.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:08.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:08.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:08.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:08.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:08.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:08.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:08.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:09.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:09.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:09.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:09.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:08 smithi151 ceph-mon[28965]: pgmap v496: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 591 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 594/28359 objects degraded (2.095%); 3569/28359 objects misplaced (12.585%); 4.0 MiB/s, 68 objects/s recovering 2024-06-27T21:15:09.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:08 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:09.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:08 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:15:09.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:08 smithi151 ceph-mon[28965]: osdmap e166: 12 total, 12 up, 12 in 2024-06-27T21:15:09.222 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:09.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:09.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:08 smithi190 ceph-mon[32247]: pgmap v496: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 591 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 594/28359 objects degraded (2.095%); 3569/28359 objects misplaced (12.585%); 4.0 MiB/s, 68 objects/s recovering 2024-06-27T21:15:09.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:08 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:09.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:08 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:15:09.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:08 smithi190 ceph-mon[32247]: osdmap e166: 12 total, 12 up, 12 in 2024-06-27T21:15:09.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:09.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:08 smithi192 ceph-mon[32956]: pgmap v496: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 591 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 594/28359 objects degraded (2.095%); 3569/28359 objects misplaced (12.585%); 4.0 MiB/s, 68 objects/s recovering 2024-06-27T21:15:09.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:08 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:09.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:08 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:15:09.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:08 smithi192 ceph-mon[32956]: osdmap e166: 12 total, 12 up, 12 in 2024-06-27T21:15:09.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:09.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:09.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:09.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:09.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:09.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:09.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:09.783 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:09.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:09.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:09.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:10.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:10.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:10.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:10.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:10.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 594/28359 objects degraded (2.095%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:10.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 594/28359 objects degraded (2.095%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:10.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:10.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 594/28359 objects degraded (2.095%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:10.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:10.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:10.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:10.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:10.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:10.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.681 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:10.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:10.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:10.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:10.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:10.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:10.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:11.039 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:15:11.039 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:15:11.040 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:15:11.040 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:15:11.040 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:15:11.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:11.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:11.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:11.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:11.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:11.328 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:11.360 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-06-27T21:15:11.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:11.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:11.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:11.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:11.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:11.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:11 smithi151 ceph-mon[28965]: pgmap v498: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 481 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 594/23046 objects degraded (2.577%); 2960/23046 objects misplaced (12.844%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:15:11.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2350965478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:15:11.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:11 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:15:11.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:11.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:11 smithi190 ceph-mon[32247]: pgmap v498: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 481 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 594/23046 objects degraded (2.577%); 2960/23046 objects misplaced (12.844%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:15:11.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2350965478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:15:11.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:11 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:15:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:11 smithi192 ceph-mon[32956]: pgmap v498: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 481 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 594/23046 objects degraded (2.577%); 2960/23046 objects misplaced (12.844%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:15:11.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2350965478' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:15:11.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:11 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:15:11.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:11.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:11.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:11.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:11.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:12.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:15:12.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:12.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:12.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:12.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:12.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.354 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:12.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:12.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:12.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:12.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:12.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:12.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:12 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:15:12.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:12 smithi151 ceph-mon[28965]: osdmap e167: 12 total, 12 up, 12 in 2024-06-27T21:15:12.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:12.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:12.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:12 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:15:12.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:12 smithi190 ceph-mon[32247]: osdmap e167: 12 total, 12 up, 12 in 2024-06-27T21:15:12.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:12 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:15:12.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:12 smithi192 ceph-mon[32956]: osdmap e167: 12 total, 12 up, 12 in 2024-06-27T21:15:12.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:12.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:12.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:12.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:15:12.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:12.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:13.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:13.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:13.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:13.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:13.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:13.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:13.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:13.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:13.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:13.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:13.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:13 smithi151 ceph-mon[28965]: pgmap v500: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 690/21057 objects degraded (3.277%); 2607/21057 objects misplaced (12.381%); 5.1 MiB/s, 81 objects/s recovering 2024-06-27T21:15:13.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:13.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:13 smithi190 ceph-mon[32247]: pgmap v500: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 690/21057 objects degraded (3.277%); 2607/21057 objects misplaced (12.381%); 5.1 MiB/s, 81 objects/s recovering 2024-06-27T21:15:13.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:13 smithi192 ceph-mon[32956]: pgmap v500: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 439 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 690/21057 objects degraded (3.277%); 2607/21057 objects misplaced (12.381%); 5.1 MiB/s, 81 objects/s recovering 2024-06-27T21:15:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:13.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:13.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 1 1 2024-06-27T21:15:13.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:13.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:13.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:14.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:14.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:14.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:14.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:14.182 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:14.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:15:14.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:15:14.194 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:15:14.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:15:14.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:15:14.265 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:15:14.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:14.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:14.299 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:15:14.299 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:15:14.328 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:15:14.328 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:15:14.360 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:15:14.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:15:14.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:15:14.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:15:14.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:14.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:15:14.419 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:15:14.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:14.450 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:15:14.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:15:14.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:15:14.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:15:14.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:15:14.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:15:14.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:14.545 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:15:14.545 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:15:14.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:14.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:14.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:14.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/216126383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:14.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 690/21057 objects degraded (3.277%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/216126383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:14.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 690/21057 objects degraded (3.277%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:14.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/216126383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:14.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 690/21057 objects degraded (3.277%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:14.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:14.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:14.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:14.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:15.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:15.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:15.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:15.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:15.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:15.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:15.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:15.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:15.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:15.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:15.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:15 smithi151 ceph-mon[28965]: pgmap v501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 298 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 690/14292 objects degraded (4.828%); 2607/14292 objects misplaced (18.241%); 2.6 MiB/s, 41 objects/s recovering 2024-06-27T21:15:15.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:15.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:15.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:15 smithi190 ceph-mon[32247]: pgmap v501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 298 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 690/14292 objects degraded (4.828%); 2607/14292 objects misplaced (18.241%); 2.6 MiB/s, 41 objects/s recovering 2024-06-27T21:15:15.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:15 smithi192 ceph-mon[32956]: pgmap v501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 298 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 690/14292 objects degraded (4.828%); 2607/14292 objects misplaced (18.241%); 2.6 MiB/s, 41 objects/s recovering 2024-06-27T21:15:15.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:15.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:15.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:15.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:15.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:16.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:16.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:16.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:16.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:16.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:16.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:16.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:16.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:16.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:16.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:16 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:16.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:16 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:16.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:16 smithi151 ceph-mon[28965]: osdmap e168: 12 total, 12 up, 12 in 2024-06-27T21:15:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:16.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:16.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:16.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:16.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:16 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:16.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:16 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:16.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:16 smithi190 ceph-mon[32247]: osdmap e168: 12 total, 12 up, 12 in 2024-06-27T21:15:16.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:16 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:16.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:16 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:16.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:16 smithi192 ceph-mon[32956]: osdmap e168: 12 total, 12 up, 12 in 2024-06-27T21:15:16.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:16.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:16.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:16.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:16.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:17.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:17.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:17.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:17.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:17.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:17.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:17.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:17.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:17.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:17.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:17.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:17.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:17 smithi151 ceph-mon[28965]: pgmap v503: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 218 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 752 op/s; 744/10443 objects degraded (7.124%); 1576/10443 objects misplaced (15.091%); 5.5 MiB/s, 88 objects/s recovering 2024-06-27T21:15:17.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:17 smithi190 ceph-mon[32247]: pgmap v503: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 218 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 752 op/s; 744/10443 objects degraded (7.124%); 1576/10443 objects misplaced (15.091%); 5.5 MiB/s, 88 objects/s recovering 2024-06-27T21:15:17.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:17.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:17 smithi192 ceph-mon[32956]: pgmap v503: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 218 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 752 op/s; 744/10443 objects degraded (7.124%); 1576/10443 objects misplaced (15.091%); 5.5 MiB/s, 88 objects/s recovering 2024-06-27T21:15:17.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:17.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:17.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:17.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:18.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:18.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:18.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:18.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:18.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:18.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:18.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:18.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:18.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:18.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:18.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:18.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:18.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:18.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:18.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:18.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:18.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:19.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:19.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:19.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:19.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:19.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:19.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:19.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:19.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:19.441 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 92072 objects 2024-06-27T21:15:19.441 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :122.184 2024-06-27T21:15:19.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:19.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:19.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:19.636 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:15:19.636 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:15:19.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:19 smithi151 ceph-mon[28965]: pgmap v504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 744/7146 objects degraded (10.411%); 1576/7146 objects misplaced (22.054%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:15:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 744/7146 objects degraded (10.411%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:19.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:19.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:15:19.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:19 smithi190 ceph-mon[32247]: pgmap v504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 744/7146 objects degraded (10.411%); 1576/7146 objects misplaced (22.054%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:15:19.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 744/7146 objects degraded (10.411%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:19.768 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-06-27T21:15:19.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-06-27T21:15:19.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:19 smithi192 ceph-mon[32956]: pgmap v504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 149 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 744/7146 objects degraded (10.411%); 1576/7146 objects misplaced (22.054%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:15:19.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 744/7146 objects degraded (10.411%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:19.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:19.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:19.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:19.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:19.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:20.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:20.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:20.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:20.190 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:20.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:20.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:20.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:20.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:20.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:20.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:20.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:20.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:20.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:20.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:20.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:20.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:21.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:21.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:21.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:15:21.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:21.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:21.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:21.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:21.364 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:21.374 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:21.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:21.418 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:15:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:21.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:21.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:21.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:21.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:21.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:21.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:21 smithi151 ceph-mon[28965]: pgmap v505: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 48 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 744/2298 objects degraded (32.376%); 1072/2298 objects misplaced (46.649%); 4.0 MiB/s, 69 objects/s recovering 2024-06-27T21:15:21.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:21.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:21.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:21 smithi151 ceph-mon[28965]: osdmap e169: 12 total, 12 up, 12 in 2024-06-27T21:15:21.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3008520414' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:15:21.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:21 smithi190 ceph-mon[32247]: pgmap v505: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 48 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 744/2298 objects degraded (32.376%); 1072/2298 objects misplaced (46.649%); 4.0 MiB/s, 69 objects/s recovering 2024-06-27T21:15:21.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:21.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:21.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:21 smithi190 ceph-mon[32247]: osdmap e169: 12 total, 12 up, 12 in 2024-06-27T21:15:21.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3008520414' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:15:21.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:21.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:21 smithi192 ceph-mon[32956]: pgmap v505: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 48 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 744/2298 objects degraded (32.376%); 1072/2298 objects misplaced (46.649%); 4.0 MiB/s, 69 objects/s recovering 2024-06-27T21:15:21.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:21.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:21.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:21 smithi192 ceph-mon[32956]: osdmap e169: 12 total, 12 up, 12 in 2024-06-27T21:15:21.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3008520414' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:15:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:21.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:21.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:21.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:21.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:21.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:22.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:22.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:22.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:22.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:22.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:22.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:22.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:22.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:22.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:22.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:22.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:22.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:22.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:22 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:22.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3008520414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:15:22.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:22 smithi151 ceph-mon[28965]: osdmap e170: 12 total, 12 up, 12 in 2024-06-27T21:15:22.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:22 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:22.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3008520414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:15:22.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:22 smithi190 ceph-mon[32247]: osdmap e170: 12 total, 12 up, 12 in 2024-06-27T21:15:22.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:22.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:22 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:22.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3008520414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:15:22.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:22 smithi192 ceph-mon[32956]: osdmap e170: 12 total, 12 up, 12 in 2024-06-27T21:15:22.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:22.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:22.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:22.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:23.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:23.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:23.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:23.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:23.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:23.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:23.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:23.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:23.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:23.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:23.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:23.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:23.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:23 smithi151 ceph-mon[28965]: pgmap v508: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 21 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 822/972 objects degraded (84.568%); 822/972 objects misplaced (84.568%); 3.8 MiB/s, 79 objects/s recovering 2024-06-27T21:15:23.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:23 smithi190 ceph-mon[32247]: pgmap v508: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 21 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 822/972 objects degraded (84.568%); 822/972 objects misplaced (84.568%); 3.8 MiB/s, 79 objects/s recovering 2024-06-27T21:15:23.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:23.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:23 smithi192 ceph-mon[32956]: pgmap v508: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 21 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 822/972 objects degraded (84.568%); 822/972 objects misplaced (84.568%); 3.8 MiB/s, 79 objects/s recovering 2024-06-27T21:15:23.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:23.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:23.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:23.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:24.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:24.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:24.087 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:24.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:24.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:24.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:24.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:24.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:24.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:24.429 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-06-27T21:15:24.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:24.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:24.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:24.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:24.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3048525815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:15:24.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 822/972 objects degraded (84.568%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:24.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3048525815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:15:24.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 822/972 objects degraded (84.568%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:24.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:24.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3048525815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:15:24.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 822/972 objects degraded (84.568%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:24.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:15:24.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:24.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:24.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:25.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:25.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:25.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:25.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:25.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:25.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:25.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:25.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:25.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:25.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:25.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:25.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:25.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:25 smithi192 ceph-mon[32956]: pgmap v509: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 434 op/s; 822/6 objects degraded (13700.000%); 822/6 objects misplaced (13700.000%); 2.0 MiB/s, 46 objects/s recovering 2024-06-27T21:15:25.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3048525815' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:15:25.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:25 smithi192 ceph-mon[32956]: osdmap e171: 12 total, 12 up, 12 in 2024-06-27T21:15:25.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:25 smithi151 ceph-mon[28965]: pgmap v509: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 434 op/s; 822/6 objects degraded (13700.000%); 822/6 objects misplaced (13700.000%); 2.0 MiB/s, 46 objects/s recovering 2024-06-27T21:15:25.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3048525815' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:15:25.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:25 smithi151 ceph-mon[28965]: osdmap e171: 12 total, 12 up, 12 in 2024-06-27T21:15:25.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:25 smithi190 ceph-mon[32247]: pgmap v509: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 434 op/s; 822/6 objects degraded (13700.000%); 822/6 objects misplaced (13700.000%); 2.0 MiB/s, 46 objects/s recovering 2024-06-27T21:15:25.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3048525815' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:15:25.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:25 smithi190 ceph-mon[32247]: osdmap e171: 12 total, 12 up, 12 in 2024-06-27T21:15:25.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:25.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:25.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:25.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:25.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:26.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:26.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:26.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:26.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:26.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:26.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:26.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:26.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:26.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:26.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:26.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:26.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:26.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:26.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:26.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:26.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:26.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:26.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:27.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:27.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:27.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:27.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:27.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:27.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:27.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:27.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:27.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:27.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:27.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:27.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:27.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:27.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:27 smithi151 ceph-mon[28965]: pgmap v511: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 823/6 objects degraded (13716.667%); 822/6 objects misplaced (13700.000%); 917 KiB/s, 163 objects/s recovering 2024-06-27T21:15:27.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:27.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:27 smithi190 ceph-mon[32247]: pgmap v511: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 823/6 objects degraded (13716.667%); 822/6 objects misplaced (13700.000%); 917 KiB/s, 163 objects/s recovering 2024-06-27T21:15:27.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:27.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:27.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:27 smithi192 ceph-mon[32956]: pgmap v511: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 127 op/s; 823/6 objects degraded (13716.667%); 822/6 objects misplaced (13700.000%); 917 KiB/s, 163 objects/s recovering 2024-06-27T21:15:27.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:27.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:27.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:28.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:28.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:28.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:28.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:28.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:28.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:28.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:28.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:28.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:28.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:28.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:28.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:28.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:28.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:28.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:28.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:28.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:28.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:29.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:29.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:29.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:29.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:29.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:29.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:29.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:29.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:29.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:29.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:29.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:29.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:29.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:29 smithi151 ceph-mon[28965]: pgmap v512: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 65 op/s; 823/6 objects degraded (13716.667%); 822/6 objects misplaced (13700.000%); 738 KiB/s, 131 objects/s recovering 2024-06-27T21:15:29.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 823/6 objects degraded (13716.667%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:29.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:29 smithi151 ceph-mon[28965]: osdmap e172: 12 total, 12 up, 12 in 2024-06-27T21:15:29.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:29.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:29 smithi190 ceph-mon[32247]: pgmap v512: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 65 op/s; 823/6 objects degraded (13716.667%); 822/6 objects misplaced (13700.000%); 738 KiB/s, 131 objects/s recovering 2024-06-27T21:15:29.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 823/6 objects degraded (13716.667%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:29.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:29 smithi190 ceph-mon[32247]: osdmap e172: 12 total, 12 up, 12 in 2024-06-27T21:15:29.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:29.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:29 smithi192 ceph-mon[32956]: pgmap v512: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 65 op/s; 823/6 objects degraded (13716.667%); 822/6 objects misplaced (13700.000%); 738 KiB/s, 131 objects/s recovering 2024-06-27T21:15:29.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 823/6 objects degraded (13716.667%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:15:29.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:29 smithi192 ceph-mon[32956]: osdmap e172: 12 total, 12 up, 12 in 2024-06-27T21:15:29.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:29.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:29.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:29.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:30.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:30.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:30.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:30.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:30.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:30.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:30.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:30.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:30.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:30.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:30.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:30.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:30.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:30.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:30.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:30.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:30.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:30.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:30.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:30.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:30.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:30.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:31.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:31.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:31.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:31.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:31.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:31.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:31.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:31.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:31.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:31.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:31 smithi151 ceph-mon[28965]: pgmap v514: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 40 op/s; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 130 objects/s recovering 2024-06-27T21:15:31.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:31.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:31.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:31 smithi151 ceph-mon[28965]: osdmap e173: 12 total, 12 up, 12 in 2024-06-27T21:15:31.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:31.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:31.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:31 smithi190 ceph-mon[32247]: pgmap v514: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 40 op/s; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 130 objects/s recovering 2024-06-27T21:15:31.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:31.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:31.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:31 smithi190 ceph-mon[32247]: osdmap e173: 12 total, 12 up, 12 in 2024-06-27T21:15:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:31 smithi192 ceph-mon[32956]: pgmap v514: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 40 op/s; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 130 objects/s recovering 2024-06-27T21:15:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:31 smithi192 ceph-mon[32956]: osdmap e173: 12 total, 12 up, 12 in 2024-06-27T21:15:31.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:31.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:31.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:31.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:31.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:32.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:32.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:32.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:32.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:32.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:32.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:32.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:32.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:32.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:32.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:32.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:32.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:32.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:32.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:32.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:33.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:33.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:33.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:33.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:33.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:33.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:33.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:33.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:33.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:33.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:33.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:33.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:33 smithi190 ceph-mon[32247]: pgmap v516: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 194 objects/s recovering 2024-06-27T21:15:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:33 smithi190 ceph-mon[32247]: osdmap e174: 12 total, 12 up, 12 in 2024-06-27T21:15:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:33 smithi192 ceph-mon[32956]: pgmap v516: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 194 objects/s recovering 2024-06-27T21:15:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:33 smithi192 ceph-mon[32956]: osdmap e174: 12 total, 12 up, 12 in 2024-06-27T21:15:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:33.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:33.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:33.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:33 smithi151 ceph-mon[28965]: pgmap v516: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 194 objects/s recovering 2024-06-27T21:15:33.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:33.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:33.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:33 smithi151 ceph-mon[28965]: osdmap e174: 12 total, 12 up, 12 in 2024-06-27T21:15:33.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:33.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:33.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:33.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:33.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:34.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:34.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:34.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:34.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:34.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:34.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:34.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:34.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:34.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:34.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:34.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:34.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:34 smithi151 ceph-mon[28965]: osdmap e175: 12 total, 12 up, 12 in 2024-06-27T21:15:34.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 824/6 objects degraded (13733.333%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:15:34.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:34 smithi151 ceph-mon[28965]: osdmap e176: 12 total, 12 up, 12 in 2024-06-27T21:15:34.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:34.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:34 smithi190 ceph-mon[32247]: osdmap e175: 12 total, 12 up, 12 in 2024-06-27T21:15:34.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 824/6 objects degraded (13733.333%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:15:34.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:34 smithi190 ceph-mon[32247]: osdmap e176: 12 total, 12 up, 12 in 2024-06-27T21:15:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:34 smithi192 ceph-mon[32956]: osdmap e175: 12 total, 12 up, 12 in 2024-06-27T21:15:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 824/6 objects degraded (13733.333%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:15:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:34 smithi192 ceph-mon[32956]: osdmap e176: 12 total, 12 up, 12 in 2024-06-27T21:15:34.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:34.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:34.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:34.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:34.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:35.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:35.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:35.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:35.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:35.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:35.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:35.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:35.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:35.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:35.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:35.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:35.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:35.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:35.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:35.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:35 smithi151 ceph-mon[28965]: pgmap v519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 138 objects/s recovering 2024-06-27T21:15:35.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:35 smithi190 ceph-mon[32247]: pgmap v519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 138 objects/s recovering 2024-06-27T21:15:35.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:35 smithi192 ceph-mon[32956]: pgmap v519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 824/6 objects degraded (13733.333%); 822/6 objects misplaced (13700.000%); 0 B/s, 138 objects/s recovering 2024-06-27T21:15:35.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:35.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:35.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:35.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:35.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:35.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:36.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:36.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:36.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:36.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:36.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:36.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:36.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:36.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:36.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:36.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:36.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:36.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:36.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:36.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:36.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:36.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:36.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:36.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:37.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:37.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:37.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:37.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:37.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:37.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:37.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:37.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:37.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:37.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:37.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:37.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:37.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:37 smithi151 ceph-mon[28965]: pgmap v521: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825/6 objects degraded (13750.000%); 822/6 objects misplaced (13700.000%); 0 B/s, 261 objects/s recovering 2024-06-27T21:15:37.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:37 smithi190 ceph-mon[32247]: pgmap v521: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825/6 objects degraded (13750.000%); 822/6 objects misplaced (13700.000%); 0 B/s, 261 objects/s recovering 2024-06-27T21:15:37.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:37.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:37 smithi192 ceph-mon[32956]: pgmap v521: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825/6 objects degraded (13750.000%); 822/6 objects misplaced (13700.000%); 0 B/s, 261 objects/s recovering 2024-06-27T21:15:37.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:37.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:37.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:37.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:38.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:38.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:38.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:38.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:38.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:38.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:38.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:38.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:38.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:38.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:38.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 2 1 2024-06-27T21:15:38.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:38.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:38.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:38.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:38.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:38.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:38.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:39.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:39.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:39.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:39.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:39.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:39.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:39.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:39.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:39.512 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:15:39.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:15:39.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:39.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:15:39.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:15:39.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:39.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:15:39.569 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:15:39.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:39 smithi151 ceph-mon[28965]: pgmap v522: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825/6 objects degraded (13750.000%); 822/6 objects misplaced (13700.000%); 0 B/s, 170 objects/s recovering 2024-06-27T21:15:39.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:39.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:39.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2979177054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:39.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 825/6 objects degraded (13750.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:15:39.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:39 smithi151 ceph-mon[28965]: osdmap e177: 12 total, 12 up, 12 in 2024-06-27T21:15:39.595 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:15:39.595 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:15:39.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:15:39.632 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:15:39.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:39.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:15:39.656 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:15:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:39.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:39.681 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:15:39.681 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:15:39.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:15:39.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:15:39.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:15:39.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:15:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:39 smithi190 ceph-mon[32247]: pgmap v522: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825/6 objects degraded (13750.000%); 822/6 objects misplaced (13700.000%); 0 B/s, 170 objects/s recovering 2024-06-27T21:15:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2979177054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 825/6 objects degraded (13750.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:15:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:39 smithi190 ceph-mon[32247]: osdmap e177: 12 total, 12 up, 12 in 2024-06-27T21:15:39.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:39.778 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:15:39.779 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:15:39.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:39 smithi192 ceph-mon[32956]: pgmap v522: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825/6 objects degraded (13750.000%); 822/6 objects misplaced (13700.000%); 0 B/s, 170 objects/s recovering 2024-06-27T21:15:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2979177054' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 825/6 objects degraded (13750.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:15:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:39 smithi192 ceph-mon[32956]: osdmap e177: 12 total, 12 up, 12 in 2024-06-27T21:15:39.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:39.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:15:39.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:15:39.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:15:39.839 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:15:39.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:39.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:39.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:40.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:40.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:40.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:40.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:40.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:40.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:40.328 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:15:40.329 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:15:40.329 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:15:40.329 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:15:40.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:40.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:40.469 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_0' removed 2024-06-27T21:15:40.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:40.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:40.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:40.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:40.725 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3768434323' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:15:40.725 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:40 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:15:40.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3768434323' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:15:40.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:40 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:15:40.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:40.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3768434323' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:15:40.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:40 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:15:40.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:40.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:40.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:40.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:41.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:41.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:41.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:41.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:41.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:41.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:41.292 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:41.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:41.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:41.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:41.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:41.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:41.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:41.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:41.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:41.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:41 smithi151 ceph-mon[28965]: pgmap v524: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 826/6 objects degraded (13766.667%); 822/6 objects misplaced (13700.000%); 0 B/s, 212 objects/s recovering 2024-06-27T21:15:41.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:41 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:15:41.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:41 smithi151 ceph-mon[28965]: osdmap e178: 12 total, 12 up, 12 in 2024-06-27T21:15:41.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:41 smithi190 ceph-mon[32247]: pgmap v524: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 826/6 objects degraded (13766.667%); 822/6 objects misplaced (13700.000%); 0 B/s, 212 objects/s recovering 2024-06-27T21:15:41.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:41 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:15:41.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:41 smithi190 ceph-mon[32247]: osdmap e178: 12 total, 12 up, 12 in 2024-06-27T21:15:41.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:41.782 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:15:41.783 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:15:41.783 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-06-27T21:15:41.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-06-27T21:15:41.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:41 smithi192 ceph-mon[32956]: pgmap v524: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 826/6 objects degraded (13766.667%); 822/6 objects misplaced (13700.000%); 0 B/s, 212 objects/s recovering 2024-06-27T21:15:41.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:41 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:15:41.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:41 smithi192 ceph-mon[32956]: osdmap e178: 12 total, 12 up, 12 in 2024-06-27T21:15:41.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:41.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:41.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:41.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:42.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:42.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:42.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:42.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:42.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:15:42.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:42.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:42.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:42.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:42.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:42.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:42.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:42.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:42.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:15:42.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:42.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:42 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 826/6 objects degraded (13766.667%), 1 pg degraded, 2 pgs undersized) 2024-06-27T21:15:42.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:42 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 826/6 objects degraded (13766.667%), 1 pg degraded, 2 pgs undersized) 2024-06-27T21:15:42.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:42.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:42.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:42.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:43.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:42 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 826/6 objects degraded (13766.667%), 1 pg degraded, 2 pgs undersized) 2024-06-27T21:15:43.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:43.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:43.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:43.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:43.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:43.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:43.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:43.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:43.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:43.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:43.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:43.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:43.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:43.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:43 smithi151 ceph-mon[28965]: pgmap v526: 11 pgs: 11 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:43.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:43 smithi192 ceph-mon[32956]: pgmap v526: 11 pgs: 11 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:43.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:43.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:43.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:43.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:43.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:44.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:43 smithi190 ceph-mon[32247]: pgmap v526: 11 pgs: 11 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:44.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:44.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:44.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:44.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:44.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:44.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:44.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:44.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:44.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:44.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:44.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:44.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:44.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:44.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:44.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:44.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:44.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:44.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:45.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:45.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:45.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:45.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:45.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:45.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:45.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:45.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:45.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:45.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:45 smithi151 ceph-mon[28965]: pgmap v527: 11 pgs: 11 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:45.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:45 smithi151 ceph-mon[28965]: osdmap e179: 12 total, 12 up, 12 in 2024-06-27T21:15:45.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:45.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:45.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:45 smithi190 ceph-mon[32247]: pgmap v527: 11 pgs: 11 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:45.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:45 smithi190 ceph-mon[32247]: osdmap e179: 12 total, 12 up, 12 in 2024-06-27T21:15:45.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:45 smithi192 ceph-mon[32956]: pgmap v527: 11 pgs: 11 active+clean; 577 KiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:45.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:45 smithi192 ceph-mon[32956]: osdmap e179: 12 total, 12 up, 12 in 2024-06-27T21:15:45.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:45.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:45.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:45.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:45.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:46.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:46.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:46.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:46.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:46.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:46.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:46.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:46.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:46.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:46.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:46.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:46.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:46.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:46.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:46.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:46.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:46.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:46.788 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:46.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:46.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:46.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:46.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:46.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:47.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:47.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:47.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:47.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:47.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:47.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:47.400 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:47.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:47.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:47.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:47.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:47.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:47 smithi151 ceph-mon[28965]: pgmap v529: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:47.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:47 smithi151 ceph-mon[28965]: osdmap e180: 12 total, 12 up, 12 in 2024-06-27T21:15:47.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:47.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:47 smithi190 ceph-mon[32247]: pgmap v529: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:47.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:47 smithi190 ceph-mon[32247]: osdmap e180: 12 total, 12 up, 12 in 2024-06-27T21:15:47.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:47.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:47 smithi192 ceph-mon[32956]: pgmap v529: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:47.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:47 smithi192 ceph-mon[32956]: osdmap e180: 12 total, 12 up, 12 in 2024-06-27T21:15:47.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:47.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:48.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:48.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:48.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:48.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:48.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:48.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:48.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:48.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:48.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:48.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:48.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:48.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:48.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:48.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:48.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:48.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:48.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:49.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:49.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:49.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:49.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:49.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:49.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:49.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:49.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:49.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:49.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:49.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:49.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:49 smithi151 ceph-mon[28965]: pgmap v531: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:49.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:49.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:49.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:49 smithi151 ceph-mon[28965]: osdmap e181: 12 total, 12 up, 12 in 2024-06-27T21:15:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:49.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:49.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:49 smithi190 ceph-mon[32247]: pgmap v531: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:49.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:49.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:49.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:49 smithi190 ceph-mon[32247]: osdmap e181: 12 total, 12 up, 12 in 2024-06-27T21:15:49.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:49 smithi192 ceph-mon[32956]: pgmap v531: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:49.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:49.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:49.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:49 smithi192 ceph-mon[32956]: osdmap e181: 12 total, 12 up, 12 in 2024-06-27T21:15:49.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:49.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:49.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:49.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:50.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:50.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:50.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:50.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:50.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:50.311 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:50.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:50.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:50.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:50.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:50.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:50.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:50.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:50.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:50.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:50.882 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:50.883 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:50.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:50.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:50.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:51.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:51.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:51.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:51.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:51.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:51.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:51.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:51.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:51.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:51.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:51.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:51 smithi151 ceph-mon[28965]: pgmap v533: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:51.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:51.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:51.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:51 smithi190 ceph-mon[32247]: pgmap v533: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:51.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:51 smithi192 ceph-mon[32956]: pgmap v533: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:51.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:51.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:51.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:51.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:51.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:52.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:52.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:52.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:52.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:52.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:52.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-06-27T21:15:52.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:52.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:52.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:52.622 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:52.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3592745998' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:15:52.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:52.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:52.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3592745998' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:15:52.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3592745998' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:15:52.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:52.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:52.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:52.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:53.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:53.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:53.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:53.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:53.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:53.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:15:53.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:15:53.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:53.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:53.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:53 smithi151 ceph-mon[28965]: pgmap v534: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:53.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3592745998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:15:53.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:53 smithi151 ceph-mon[28965]: osdmap e182: 12 total, 12 up, 12 in 2024-06-27T21:15:53.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:53.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:53.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:53.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:53 smithi190 ceph-mon[32247]: pgmap v534: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:53.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3592745998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:15:53.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:53 smithi190 ceph-mon[32247]: osdmap e182: 12 total, 12 up, 12 in 2024-06-27T21:15:53.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:53.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:53 smithi192 ceph-mon[32956]: pgmap v534: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:53.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3592745998' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:15:53.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:53 smithi192 ceph-mon[32956]: osdmap e182: 12 total, 12 up, 12 in 2024-06-27T21:15:53.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:53.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:53.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:53.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:54.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:54.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:54.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:54.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:54.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:54.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:54.368 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:15:54.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:54.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:54.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:54.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:54.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:54.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:54.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/236127378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:15:54.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:54 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:54.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/236127378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:15:54.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:54 smithi151 ceph-mon[28965]: osdmap e183: 12 total, 12 up, 12 in 2024-06-27T21:15:54.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:54.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/236127378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:15:54.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:54 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:54.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/236127378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:15:54.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:54 smithi190 ceph-mon[32247]: osdmap e183: 12 total, 12 up, 12 in 2024-06-27T21:15:54.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:54.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/236127378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:15:54.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:54 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:15:54.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/236127378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:15:54.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:54 smithi192 ceph-mon[32956]: osdmap e183: 12 total, 12 up, 12 in 2024-06-27T21:15:54.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:54.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:54.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:55.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:55.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:55.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:55.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:55.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:55.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:55.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:55.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:55.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:55.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:55.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:55.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:55.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:55.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:55.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:55 smithi192 ceph-mon[32956]: pgmap v536: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:55.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:15:55.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:55.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:55.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:55 smithi151 ceph-mon[28965]: pgmap v536: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:55.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:15:55.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 3 1 2024-06-27T21:15:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:55 smithi190 ceph-mon[32247]: pgmap v536: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:15:55.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:55.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:55.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:55.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:55.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:56.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:56.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:56.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:56.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:56.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:56.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:56.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:56.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:56.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:15:56.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:15:56.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:56.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:56.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:15:56.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:15:56.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:15:56.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:15:56.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:15:56.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:15:56.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:15:56.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:15:56.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:15:56.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:56.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:56.844 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:15:56.844 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:15:56.844 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:15:56.844 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:15:56.844 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:15:56.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:56.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:56.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:57.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:57.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:57.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:57.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:57.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:57.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:15:57.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:57.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:15:57.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:15:57.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:57.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:15:57.453 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:15:57.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:15:57.475 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:15:57.498 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:15:57.498 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:15:57.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:15:57.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:15:57.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:57.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:15:57.541 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:15:57.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:15:57.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:15:57.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:57.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:57.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:15:57.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:15:57.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:15:57.610 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:15:57.611 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:15:57.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:15:57.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:15:57.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:15:57.656 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:15:57.656 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:57 smithi151 ceph-mon[28965]: pgmap v538: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:57.656 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2575490319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:57.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:15:57.678 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:15:57.678 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:15:57.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:15:57.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:57 smithi190 ceph-mon[32247]: pgmap v538: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:57.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2575490319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:57.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:57 smithi192 ceph-mon[32956]: pgmap v538: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:57.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2575490319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:15:57.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:57.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:57.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:57.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:57.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:58.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:58.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:58.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:15:58.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:58.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:58.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:58.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:58.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:58.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:15:58.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:15:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:58.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:58.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:15:58.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:15:58.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:15:58.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:15:58.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:15:58.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:15:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:58.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:15:58.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:15:58.791 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:15:58.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:15:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:58.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:15:58.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:15:58.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:58.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:15:59.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:59.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:59.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:59.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:59.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:15:59.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:15:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:15:59.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:59.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:59.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:15:59.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:15:59.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:59 smithi151 ceph-mon[28965]: pgmap v539: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:59.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:59 smithi151 ceph-mon[28965]: osdmap e184: 12 total, 12 up, 12 in 2024-06-27T21:15:59.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:15:59 smithi151 ceph-mon[28965]: osdmap e185: 12 total, 12 up, 12 in 2024-06-27T21:15:59.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:15:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:59.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:15:59.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:59 smithi190 ceph-mon[32247]: pgmap v539: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:59.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:59 smithi190 ceph-mon[32247]: osdmap e184: 12 total, 12 up, 12 in 2024-06-27T21:15:59.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:15:59 smithi190 ceph-mon[32247]: osdmap e185: 12 total, 12 up, 12 in 2024-06-27T21:15:59.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:59 smithi192 ceph-mon[32956]: pgmap v539: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:15:59.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:59 smithi192 ceph-mon[32956]: osdmap e184: 12 total, 12 up, 12 in 2024-06-27T21:15:59.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:15:59 smithi192 ceph-mon[32956]: osdmap e185: 12 total, 12 up, 12 in 2024-06-27T21:15:59.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:15:59.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:15:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:15:59.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:15:59.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:15:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:15:59.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:00.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:00.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:00.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:00.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:00.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:00.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:00.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:00.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:00.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:00.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:00.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:00.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:16:00.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:00.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:00.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:00.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:00.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:00.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:00.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:01.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:01.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:01.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:01.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:01.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:01.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:01.415 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_1' created 2024-06-27T21:16:01.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:01.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:01.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:01.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:01.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:01 smithi151 ceph-mon[28965]: pgmap v542: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:01.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:01 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:16:01.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:01 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:16:01.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1631962909' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-27T21:16:01.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:01.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:01.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:01 smithi190 ceph-mon[32247]: pgmap v542: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:01 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:16:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:01 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:16:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1631962909' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-27T21:16:01.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:01 smithi192 ceph-mon[32956]: pgmap v542: 11 pgs: 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:01.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:01 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:16:01.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:01 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:16:01.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1631962909' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-06-27T21:16:01.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:01.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:01.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:01.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:01.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:02.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:02.113 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:02.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:02.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:02.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:02.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:02.420 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-06-27T21:16:02.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:02.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:02.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-06-27T21:16:02.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:02.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:02.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1631962909' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-27T21:16:02.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:02 smithi151 ceph-mon[28965]: osdmap e186: 12 total, 12 up, 12 in 2024-06-27T21:16:02.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/324141193' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:16:02.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:02.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:02.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1631962909' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-27T21:16:02.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:02 smithi190 ceph-mon[32247]: osdmap e186: 12 total, 12 up, 12 in 2024-06-27T21:16:02.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/324141193' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:16:02.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1631962909' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-06-27T21:16:02.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:02 smithi192 ceph-mon[32956]: osdmap e186: 12 total, 12 up, 12 in 2024-06-27T21:16:02.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/324141193' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:16:02.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:02.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:02.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:02.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:02.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:03.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 4 1 2024-06-27T21:16:03.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:03.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:03.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:03.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:03.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:03.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:03.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:03.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:03.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:03.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:03.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:03 smithi151 ceph-mon[28965]: pgmap v544: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:03.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:03 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:16:03.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/324141193' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:16:03.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:03 smithi151 ceph-mon[28965]: osdmap e187: 12 total, 12 up, 12 in 2024-06-27T21:16:03.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:03.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:03.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:03 smithi190 ceph-mon[32247]: pgmap v544: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:03.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:03 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:16:03.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/324141193' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:16:03.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:03 smithi190 ceph-mon[32247]: osdmap e187: 12 total, 12 up, 12 in 2024-06-27T21:16:03.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:03 smithi192 ceph-mon[32956]: pgmap v544: 27 pgs: 16 unknown, 11 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:03.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:03 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:16:03.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/324141193' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:16:03.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:03 smithi192 ceph-mon[32956]: osdmap e187: 12 total, 12 up, 12 in 2024-06-27T21:16:03.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:03.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:03.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:03.944 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:16:03.945 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:16:03.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:04.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:04.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:04.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:04.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:04.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:04.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:04.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:04.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:04.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:04.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:04.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:04.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:04 smithi151 ceph-mon[28965]: osdmap e188: 12 total, 12 up, 12 in 2024-06-27T21:16:04.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:04 smithi151 ceph-mon[28965]: osdmap e189: 12 total, 12 up, 12 in 2024-06-27T21:16:04.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:04.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:16:04.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:04 smithi190 ceph-mon[32247]: osdmap e188: 12 total, 12 up, 12 in 2024-06-27T21:16:04.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:04 smithi190 ceph-mon[32247]: osdmap e189: 12 total, 12 up, 12 in 2024-06-27T21:16:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:04.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:04.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:04 smithi192 ceph-mon[32956]: osdmap e188: 12 total, 12 up, 12 in 2024-06-27T21:16:04.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:04 smithi192 ceph-mon[32956]: osdmap e189: 12 total, 12 up, 12 in 2024-06-27T21:16:04.789 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:04.790 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:04.830 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:04.833 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:04.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:04.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:04.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:04.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:05.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:05.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:05.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:05.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:05.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:05.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:05.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:05.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:05.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:05.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:05.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:05.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:05.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:05 smithi151 ceph-mon[28965]: pgmap v547: 27 pgs: 6 creating+peering, 8 unknown, 13 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:05.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:05.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:05 smithi190 ceph-mon[32247]: pgmap v547: 27 pgs: 6 creating+peering, 8 unknown, 13 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:05.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:05.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:05 smithi192 ceph-mon[32956]: pgmap v547: 27 pgs: 6 creating+peering, 8 unknown, 13 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:05.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:05.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:05.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:05.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:06.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:06.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:06.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:06.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:06.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:06.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:06.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:06.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:06.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:06.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:06.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:06.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:06.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:06.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:06.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:06.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:06.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:07.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:07.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:07.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:07.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:07.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:07.285 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:07.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:07.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:07.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:07.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:07.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:07.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:07.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:07 smithi151 ceph-mon[28965]: pgmap v549: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:07.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:07 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:16:07.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:07 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:16:07.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:07 smithi190 ceph-mon[32247]: pgmap v549: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:07.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:07 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:16:07.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:07 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:16:07.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:07.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:07.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:07 smithi192 ceph-mon[32956]: pgmap v549: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:07.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:07 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:16:07.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:07 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:16:07.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:07.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:07.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:08.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:08.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:08.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:08.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:08.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:08.282 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:08.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:08.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:08.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:08.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:08.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:08.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:08.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:08.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:08.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:08.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:08.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:08.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:08.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:09.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:09.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:09.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:09.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:09.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:09.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:09.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:09.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:09.444 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:16:09.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:09.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:09.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:09.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:09.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:09 smithi190 ceph-mon[32247]: pgmap v550: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:09.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/268194741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:09.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:09.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:09 smithi192 ceph-mon[32956]: pgmap v550: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:09.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/268194741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:09.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:09.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:09.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:09.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:09.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:09 smithi151 ceph-mon[28965]: pgmap v550: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:09.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/268194741' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:10.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:10.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:10.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:10.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:10.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:10.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:10.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:10.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:10.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:10.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:10.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:10.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:10.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:10 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:16:10.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/268194741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:10.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:10 smithi151 ceph-mon[28965]: osdmap e190: 12 total, 12 up, 12 in 2024-06-27T21:16:10.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:10 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:16:10.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/268194741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:10.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:10 smithi190 ceph-mon[32247]: osdmap e190: 12 total, 12 up, 12 in 2024-06-27T21:16:10.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:10.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:10 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:16:10.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/268194741' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:10.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:10 smithi192 ceph-mon[32956]: osdmap e190: 12 total, 12 up, 12 in 2024-06-27T21:16:10.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:10.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:10.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:10.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:10.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:11.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:11.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:11.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:11.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:11.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:11.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:11.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:11.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:11.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:11.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:11.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:11.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:11.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:11 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:16:11.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:11 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:16:11.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:11 smithi151 ceph-mon[28965]: pgmap v552: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:11.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:11 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:16:11.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:11 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:16:11.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:11.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:11 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:16:11.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:11 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:16:11.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:11 smithi190 ceph-mon[32247]: pgmap v552: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:11.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:11 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:16:11.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:11 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:16:11.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:11 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:16:11.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:11 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:16:11.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:11 smithi192 ceph-mon[32956]: pgmap v552: 27 pgs: 27 active+clean; 577 KiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:11.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:11 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:16:11.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:11 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:16:11.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:11.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:11.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:11.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:11.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:12.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:12.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:12.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:12.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:16:12.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:12.255 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:16:12.255 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:16:12.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:12.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:12.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:12.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:12.475 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-06-27T21:16:12.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:12.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:12.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:12.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/279535250' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:16:12.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:12.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/279535250' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:16:12.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/279535250' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:16:12.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:12.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:12.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:12.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:12.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:12.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:13.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:13.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:13.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:13.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:13.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:13.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:13.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:13.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:13.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:13.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:16:13.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:16:13.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:13.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:16:13.709 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:16:13.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:13.730 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:16:13.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:16:13.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:13 smithi151 ceph-mon[28965]: pgmap v553: 27 pgs: 27 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:13.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/279535250' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:16:13.730 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:13 smithi151 ceph-mon[28965]: osdmap e191: 12 total, 12 up, 12 in 2024-06-27T21:16:13.751 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:16:13.751 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:16:13.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:13 smithi190 ceph-mon[32247]: pgmap v553: 27 pgs: 27 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:13.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/279535250' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:16:13.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:13 smithi190 ceph-mon[32247]: osdmap e191: 12 total, 12 up, 12 in 2024-06-27T21:16:13.773 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:16:13.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:16:13.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:13 smithi192 ceph-mon[32956]: pgmap v553: 27 pgs: 27 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:13.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/279535250' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:16:13.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:13 smithi192 ceph-mon[32956]: osdmap e191: 12 total, 12 up, 12 in 2024-06-27T21:16:13.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:16:13.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:16:13.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:16:13.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:16:13.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:13.841 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:16:13.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:16:13.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:13.868 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:16:13.868 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:16:13.888 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:16:13.889 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:16:13.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:16:13.911 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:16:13.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:16:13.933 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:16:13.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:13.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:13.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:14.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:14.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:14.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:14.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:14.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:14.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:16:14.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:14.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:14.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:14.499 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-06-27T21:16:14.530 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:16:14.530 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:16:14.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:14.590 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:16:14.590 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:16:14.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_74314 2024-06-27T21:16:14.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:16:14.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:16:14.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:14.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:14 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:16:14.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:14 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:16:14.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1024607428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:14.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:14.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:14.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:14 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:16:14.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:14 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:16:14.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1024607428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:14.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:14 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:16:14.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:14 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:16:14.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1024607428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:14.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:14.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:14.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:14.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:15.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:15.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:15.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:15.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:15.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:15.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:15.376 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:15.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:15.484 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-06-27T21:16:15.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:15.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:15.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 1652 1636 102.244 102.25 0.0287681 0.00963572 2024-06-27T21:16:15.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:15.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:15.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:15 smithi151 ceph-mon[28965]: pgmap v555: 27 pgs: 27 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:15.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:15 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:16:15.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:15 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:16:15.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/341354911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:16:15.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:15.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:15 smithi190 ceph-mon[32247]: pgmap v555: 27 pgs: 27 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:15.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:15 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:16:15.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:15 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:16:15.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/341354911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:16:15.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:15.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:15 smithi192 ceph-mon[32956]: pgmap v555: 27 pgs: 27 active+clean; 577 KiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:16:15.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:15 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:16:15.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:15 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:16:15.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/341354911' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:16:15.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:15.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:15.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:15.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:16.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:16.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:16.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:16.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:16.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:16.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:16.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:16.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:16.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:16.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 3077 3061 95.6502 89.0625 0.00192548 0.0103841 2024-06-27T21:16:16.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:16.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:16.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:16.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:16 smithi190 ceph-mon[32247]: 1.3 deep-scrub starts 2024-06-27T21:16:16.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:16 smithi190 ceph-mon[32247]: 1.3 deep-scrub ok 2024-06-27T21:16:16.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/341354911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:16:16.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:16 smithi190 ceph-mon[32247]: osdmap e192: 12 total, 12 up, 12 in 2024-06-27T21:16:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:16.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:16.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:16 smithi192 ceph-mon[32956]: 1.3 deep-scrub starts 2024-06-27T21:16:16.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:16 smithi192 ceph-mon[32956]: 1.3 deep-scrub ok 2024-06-27T21:16:16.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/341354911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:16:16.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:16 smithi192 ceph-mon[32956]: osdmap e192: 12 total, 12 up, 12 in 2024-06-27T21:16:16.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:16 smithi151 ceph-mon[28965]: 1.3 deep-scrub starts 2024-06-27T21:16:16.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:16 smithi151 ceph-mon[28965]: 1.3 deep-scrub ok 2024-06-27T21:16:16.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/341354911' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:16:16.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:16 smithi151 ceph-mon[28965]: osdmap e192: 12 total, 12 up, 12 in 2024-06-27T21:16:16.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 5 1 2024-06-27T21:16:16.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:16.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:16.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:17.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:17.028 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:17.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:17.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:17.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:17.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:17.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:16:17.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:17.401 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:17.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:17.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:17.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:17.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 16 4630 4614 96.1182 97.0625 0.00484586 0.0103755 2024-06-27T21:16:17.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:17.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:17.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:17 smithi151 ceph-mon[28965]: pgmap v557: 27 pgs: 27 active+clean; 21 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 51 op/s 2024-06-27T21:16:17.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:17 smithi190 ceph-mon[32247]: pgmap v557: 27 pgs: 27 active+clean; 21 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 51 op/s 2024-06-27T21:16:17.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:17.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:17 smithi192 ceph-mon[32956]: pgmap v557: 27 pgs: 27 active+clean; 21 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 51 op/s 2024-06-27T21:16:17.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:17.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:17.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:17.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:18.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:18.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:18.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:18.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:18.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:18.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:18.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:18.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:18.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:18.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:18.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 6597 6581 102.821 122.938 0.00729546 0.00970713 2024-06-27T21:16:18.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:18.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:18.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:18.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:18.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:18.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:18.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:19.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:19.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:19.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:19.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:19.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:19.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:19.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:19.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:19.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:19.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:19.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 8042 8026 100.318 90.3125 0.00396522 0.00995619 2024-06-27T21:16:19.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:19.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:19.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:19 smithi190 ceph-mon[32247]: pgmap v558: 27 pgs: 27 active+clean; 140 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 279 op/s 2024-06-27T21:16:19.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:19 smithi190 ceph-mon[32247]: osdmap e193: 12 total, 12 up, 12 in 2024-06-27T21:16:19.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:19 smithi192 ceph-mon[32956]: pgmap v558: 27 pgs: 27 active+clean; 140 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 279 op/s 2024-06-27T21:16:19.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:19 smithi192 ceph-mon[32956]: osdmap e193: 12 total, 12 up, 12 in 2024-06-27T21:16:19.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:19.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:19.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:19 smithi151 ceph-mon[28965]: pgmap v558: 27 pgs: 27 active+clean; 140 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s wr, 279 op/s 2024-06-27T21:16:19.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:19 smithi151 ceph-mon[28965]: osdmap e193: 12 total, 12 up, 12 in 2024-06-27T21:16:19.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:19.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:19.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:20.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:20.074 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:16:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:16:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:16:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:16:20.075 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:16:20.080 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:20.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:20.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:20.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:20.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:20.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:20.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:20.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:20.511 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:16:20.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:20.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 9442 9426 98.1804 87.5 0.00627627 0.0101649 2024-06-27T21:16:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:20.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:20.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:20.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:20 smithi192 ceph-mon[32956]: 1.6 deep-scrub starts 2024-06-27T21:16:20.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:20 smithi192 ceph-mon[32956]: 1.6 deep-scrub ok 2024-06-27T21:16:20.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2430150949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:16:20.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:20.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:20.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:20.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:20 smithi151 ceph-mon[28965]: 1.6 deep-scrub starts 2024-06-27T21:16:20.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:20 smithi151 ceph-mon[28965]: 1.6 deep-scrub ok 2024-06-27T21:16:20.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2430150949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:16:20.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:20.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:20.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:21.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:20 smithi190 ceph-mon[32247]: 1.6 deep-scrub starts 2024-06-27T21:16:21.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:20 smithi190 ceph-mon[32247]: 1.6 deep-scrub ok 2024-06-27T21:16:21.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2430150949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:16:21.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:21.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:21.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:21.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:21.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:21.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:21.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:21.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:21.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:21.548 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:21.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:16:21.591 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 11375 11359 101.412 120.812 0.0217023 0.0098425 2024-06-27T21:16:21.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:21.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:21.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:21.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:21.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:21 smithi192 ceph-mon[32956]: pgmap v560: 27 pgs: 27 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 692 op/s 2024-06-27T21:16:21.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:21 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:21.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2430150949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:16:21.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:21 smithi192 ceph-mon[32956]: osdmap e194: 12 total, 12 up, 12 in 2024-06-27T21:16:21.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:21.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:21.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:21.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:21 smithi151 ceph-mon[28965]: pgmap v560: 27 pgs: 27 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 692 op/s 2024-06-27T21:16:21.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:21 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:21.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2430150949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:16:21.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:21 smithi151 ceph-mon[28965]: osdmap e194: 12 total, 12 up, 12 in 2024-06-27T21:16:21.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:22.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:22.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:21 smithi190 ceph-mon[32247]: pgmap v560: 27 pgs: 27 active+clean; 321 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 692 op/s 2024-06-27T21:16:22.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:21 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:22.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2430150949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:16:22.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:21 smithi190 ceph-mon[32247]: osdmap e194: 12 total, 12 up, 12 in 2024-06-27T21:16:22.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:22.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:22.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:22.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:22.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:22.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:22.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:22.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:22.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 12499 12483 97.5164 70.25 0.00323709 0.0102365 2024-06-27T21:16:22.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:22.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:22.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:22.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:22.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:22.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:22.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:22.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:23.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:23.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:23.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:23.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:23.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:23.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:23.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:23.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:23.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:23.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 13714 13698 95.1181 75.9375 0.00834287 0.0104776 2024-06-27T21:16:23.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:23.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:23.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:23.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:23.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:23 smithi192 ceph-mon[32956]: pgmap v562: 27 pgs: 27 active+clean; 411 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:16:23.813 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:23 smithi151 ceph-mon[28965]: pgmap v562: 27 pgs: 27 active+clean; 411 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:16:23.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:23.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:23.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:23.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:24.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:23 smithi190 ceph-mon[32247]: pgmap v562: 27 pgs: 27 active+clean; 411 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:16:24.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:24.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:24.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:24.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:16:24.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:24.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:24.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:24.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:24.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:24.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:24.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 14756 14740 92.1182 65.125 0.00361852 0.0108403 2024-06-27T21:16:24.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:24.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:24.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:24.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:24.806 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:24.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:24.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:24.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:25.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:25.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:25.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:25.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:25.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:25.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:25.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:25.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:25.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:25.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:25.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 16175 16159 91.8057 88.6875 0.00841988 0.0108758 2024-06-27T21:16:25.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:25.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:25.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:25 smithi151 ceph-mon[28965]: pgmap v563: 27 pgs: 27 active+clean; 646 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:16:25.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:25 smithi151 ceph-mon[28965]: osdmap e195: 12 total, 12 up, 12 in 2024-06-27T21:16:25.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:25.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:25 smithi190 ceph-mon[32247]: pgmap v563: 27 pgs: 27 active+clean; 646 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:16:25.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:25 smithi190 ceph-mon[32247]: osdmap e195: 12 total, 12 up, 12 in 2024-06-27T21:16:25.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:25.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:25 smithi192 ceph-mon[32956]: pgmap v563: 27 pgs: 27 active+clean; 646 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:16:25.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:25 smithi192 ceph-mon[32956]: osdmap e195: 12 total, 12 up, 12 in 2024-06-27T21:16:25.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:25.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:25.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:26.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:26.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:26.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:26.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:26.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:26.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:26.420 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:26.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:26.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:26.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 18068 18052 94.0139 118.312 0.0117083 0.0105895 2024-06-27T21:16:26.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:26.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:26.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:26.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:26.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:26.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:26.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:27.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:27.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:27.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:27.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:27.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:27.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:27.391 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-06-27T21:16:27.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:27.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:27.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:27.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:27.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 19513 19497 93.7286 90.3125 0.0126242 0.0106556 2024-06-27T21:16:27.640 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:27 smithi151 ceph-mon[28965]: pgmap v565: 27 pgs: 27 active+clean; 859 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-06-27T21:16:27.640 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2337810481' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:16:27.640 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:27 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:16:27.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:27.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:27.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:27 smithi190 ceph-mon[32247]: pgmap v565: 27 pgs: 27 active+clean; 859 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-06-27T21:16:27.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2337810481' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:16:27.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:27 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:16:27.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:27 smithi192 ceph-mon[32956]: pgmap v565: 27 pgs: 27 active+clean; 859 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-06-27T21:16:27.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2337810481' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:16:27.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:27 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:16:27.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:27.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:27.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:27.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:27.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:28.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:28.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:28.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:28.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:28.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:28.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:28.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 6 1 2024-06-27T21:16:28.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:28.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:28.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:28.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:28.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 20644 20628 92.0825 70.6875 0.0389757 0.01079 2024-06-27T21:16:28.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:28 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:16:28.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:28 smithi151 ceph-mon[28965]: osdmap e196: 12 total, 12 up, 12 in 2024-06-27T21:16:28.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:28.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:28.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:28.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:28.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:28 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:16:28.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:28 smithi190 ceph-mon[32247]: osdmap e196: 12 total, 12 up, 12 in 2024-06-27T21:16:28.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:28.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:28.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:28 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:16:28.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:28 smithi192 ceph-mon[32956]: osdmap e196: 12 total, 12 up, 12 in 2024-06-27T21:16:28.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:28.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:28.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:28.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:28.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:28.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:16:28.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:16:28.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:16:28.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:16:28.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:28.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:16:28.964 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:16:28.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:28.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:28.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:16:28.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:16:29.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:16:29.007 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:16:29.029 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:16:29.029 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:16:29.053 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:16:29.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:16:29.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:29.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:16:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:16:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:29.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:29.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:16:29.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:16:29.120 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:16:29.120 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:16:29.144 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:16:29.144 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:16:29.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:16:29.167 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:16:29.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:29.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:29.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:29.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:29.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:29.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:29 smithi151 ceph-mon[28965]: pgmap v567: 27 pgs: 27 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-27T21:16:29.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:29 smithi151 ceph-mon[28965]: osdmap e197: 12 total, 12 up, 12 in 2024-06-27T21:16:29.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2445483726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:29.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:29.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:29.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:29.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 20948 20932 87.2102 19 0.00149219 0.0111043 2024-06-27T21:16:29.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:29.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:29.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:29 smithi190 ceph-mon[32247]: pgmap v567: 27 pgs: 27 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-27T21:16:29.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:29 smithi190 ceph-mon[32247]: osdmap e197: 12 total, 12 up, 12 in 2024-06-27T21:16:29.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2445483726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:29.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:29 smithi192 ceph-mon[32956]: pgmap v567: 27 pgs: 27 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-06-27T21:16:29.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:29 smithi192 ceph-mon[32956]: osdmap e197: 12 total, 12 up, 12 in 2024-06-27T21:16:29.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2445483726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:29.791 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:16:29.791 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:16:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:29.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:29.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:29.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:29.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:30.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:30.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:30.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:30.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:30.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:30.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:30.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:30.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:30.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:30.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:30.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 21048 21032 82.1501 6.25 0.147819 0.0111879 2024-06-27T21:16:30.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:30.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:30.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:30.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:30.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:30.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:30.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:30.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:31.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:31.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:31.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:31.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:31.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:31.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:31.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:31.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:31.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:31.577 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:31.582 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:31 smithi151 ceph-mon[28965]: pgmap v569: 27 pgs: 27 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-27T21:16:31.582 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:31.582 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:31.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 21058 21042 77.3543 0.625 0.00194264 0.0112196 2024-06-27T21:16:31.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:31.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:31.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:31 smithi190 ceph-mon[32247]: pgmap v569: 27 pgs: 27 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-27T21:16:31.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:31.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:31 smithi192 ceph-mon[32956]: pgmap v569: 27 pgs: 27 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-06-27T21:16:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:31.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:31.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:31.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:31.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:31.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:32.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:32.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:32.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:32.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:32.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:32.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:32.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:32.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:32.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:16:32.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:32.592 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 16 22697 22681 78.7474 102.438 0.0290063 0.0126891 2024-06-27T21:16:32.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:32.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:32 smithi151 ceph-mon[28965]: osdmap e198: 12 total, 12 up, 12 in 2024-06-27T21:16:32.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:32.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:32.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:32 smithi190 ceph-mon[32247]: osdmap e198: 12 total, 12 up, 12 in 2024-06-27T21:16:32.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:32 smithi192 ceph-mon[32956]: osdmap e198: 12 total, 12 up, 12 in 2024-06-27T21:16:32.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:32.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:32.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:32.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:32.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:33.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:33.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:33.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:16:33.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:33.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:33.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:33.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:33.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:33.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:33.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 15 24130 24115 79.3195 89.625 0.00160493 0.0125995 2024-06-27T21:16:33.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:33.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:33.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:33 smithi151 ceph-mon[28965]: pgmap v571: 27 pgs: 27 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-27T21:16:33.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:33.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:33.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:33 smithi190 ceph-mon[32247]: pgmap v571: 27 pgs: 27 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-27T21:16:33.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:33 smithi192 ceph-mon[32956]: pgmap v571: 27 pgs: 27 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-06-27T21:16:33.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:33.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:33.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:33.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:34.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:34.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:34.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:34.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:34.251 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:34.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:34.284 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:34.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:16:34.382 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-06-27T21:16:34.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:34.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:34.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:34.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:16:34.598889+0000 min lat: 0.00107574 max lat: 3.0199 avg lat: 0.0124628 2024-06-27T21:16:34.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:16:34.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 25682 25666 80.2 96.9375 0.00224406 0.0124628 2024-06-27T21:16:34.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:34.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3236026888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1648956435' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:16:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:34 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3236026888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1648956435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:16:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:34 smithi151 ceph-mon[28965]: osdmap e199: 12 total, 12 up, 12 in 2024-06-27T21:16:34.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3236026888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:34.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1648956435' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:16:34.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:34 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:34.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3236026888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:34.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1648956435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:16:34.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:34 smithi190 ceph-mon[32247]: osdmap e199: 12 total, 12 up, 12 in 2024-06-27T21:16:34.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:34.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3236026888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1648956435' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:16:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:34 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:34.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3236026888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:34.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1648956435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:16:34.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:34 smithi192 ceph-mon[32956]: osdmap e199: 12 total, 12 up, 12 in 2024-06-27T21:16:34.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:34.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:34.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:35.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:35.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:35.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:35.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:35.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:35.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:35.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.425 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:35.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:35.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:35.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:35.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 16 27285 27269 81.1513 100.188 0.00383204 0.0123161 2024-06-27T21:16:35.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:35.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:35.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:35 smithi151 ceph-mon[28965]: pgmap v572: 27 pgs: 27 active+clean; 1.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:16:35.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:35 smithi190 ceph-mon[32247]: pgmap v572: 27 pgs: 27 active+clean; 1.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:16:35.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:35.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:35 smithi192 ceph-mon[32956]: pgmap v572: 27 pgs: 27 active+clean; 1.3 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:16:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:35.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:35.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:35.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:35.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:36.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 7 1 2024-06-27T21:16:36.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:36.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:36.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:36.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:36.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:36.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:36.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:36.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:36.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:36.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:36.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 28816 28800 81.8117 95.6875 0.00189034 0.0122094 2024-06-27T21:16:36.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:36.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:36.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:36.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:36.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:36.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:36.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:37.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:37.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:37.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:37.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:16:37.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:37.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:37.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:37.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:37.466 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:37.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:37 smithi151 ceph-mon[28965]: pgmap v574: 27 pgs: 27 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-06-27T21:16:37.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:37.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 30334 30318 82.3794 94.875 0.0126427 0.0121318 2024-06-27T21:16:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:37.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:37.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:37.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:37.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:37 smithi190 ceph-mon[32247]: pgmap v574: 27 pgs: 27 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-06-27T21:16:37.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:37 smithi192 ceph-mon[32956]: pgmap v574: 27 pgs: 27 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 993 op/s 2024-06-27T21:16:37.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:37.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:37.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:37.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:37.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:38.015 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:38.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:38.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:38.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:38.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:38.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:38.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:38.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:38.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:38.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:38.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:16:38.571 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:16:38.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:38.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 16 31611 31595 82.2722 79.8125 0.00586296 0.0121471 2024-06-27T21:16:38.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:16:38.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:16:38.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:38.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:16:38.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:16:38.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:16:38.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:16:38.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:16:38.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:16:38.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:16:38.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:16:38.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:38.721 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:16:38.722 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:16:38.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:38.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:16:38.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:16:38.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:16:38.771 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:16:38.794 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:16:38.794 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:16:38.818 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:16:38.818 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:16:38.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:38.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:16:38.839 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:16:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:38.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:38.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:38.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:39.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:39.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:39.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:39.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:39.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:39.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:39.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:39.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:39.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:39.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 32558 32542 81.3486 59.1875 0.00310114 0.0122869 2024-06-27T21:16:39.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:39.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:39 smithi151 ceph-mon[28965]: pgmap v575: 27 pgs: 27 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:16:39.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/132776328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:39.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:39.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:39 smithi190 ceph-mon[32247]: pgmap v575: 27 pgs: 27 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:16:39.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/132776328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:39 smithi192 ceph-mon[32956]: pgmap v575: 27 pgs: 27 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:16:39.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/132776328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:39.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:39.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:39.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:39.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:39.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:40.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:40.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:40.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:40.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:40.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:40.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:40.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:40.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:40.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:40.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 33780 33764 81.1572 76.375 0.0110835 0.0123157 2024-06-27T21:16:40.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:40.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:40.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:40.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:40.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:40.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:40.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:40.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:41.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:41.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:34291 3.b 3:d1a748f3:::benchmark_data_smithi192_74314_object34290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:41.139075+0000", 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.025297891999999999, 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025324454, 2024-06-27T21:16:41.160 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 34291 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139075+0000", 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:16:41.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139075+0000", 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139080+0000", 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9420000000000003e-06 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139124+0000", 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3785000000000002e-05 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139140+0000", 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6466000000000001e-05 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139148+0000", 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1960000000000008e-06 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139201+0000", 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2367000000000002e-05 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139351+0000", 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150066 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.139509+0000", 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015833500000000001 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.140696+0000", 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011873680000000001 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:34309 3.7 3:ed92aa56:::benchmark_data_smithi192_74314_object34308:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:41.152360+0000", 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.012012803000000001, 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012091207, 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 34309 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152360+0000", 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152360+0000", 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152361+0000", 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5e-06 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152411+0000", 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9277000000000003e-05 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152423+0000", 2024-06-27T21:16:41.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2717999999999999e-05 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152431+0000", 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.464e-06 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152499+0000", 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8539000000000002e-05 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152619+0000", 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011911100000000001 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.152747+0000", 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128699 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.154068+0000", 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001320363 2024-06-27T21:16:41.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:34267 3.7 3:eee4601b:::benchmark_data_smithi192_74314_object34266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:41.121494+0000", 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.042878887999999997, 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042998591000000003, 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:16:41.171 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 34267 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121494+0000", 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121494+0000", 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121495+0000", 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3480000000000001e-06 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121557+0000", 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1898999999999998e-05 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121576+0000", 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8929000000000001e-05 2024-06-27T21:16:41.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121585+0000", 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5750000000000001e-06 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121635+0000", 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0683999999999998e-05 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:16:41.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121762+0000", 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012629500000000001 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.121904+0000", 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014211200000000001 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.123396+0000", 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001492548 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:16:41.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.154546+0000", 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.031150188999999998 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-06-27T21:16:41.176 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:16:41.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:41.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:41.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:41.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:41.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:41.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:41.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 16 34723 34707 80.334 58.9375 0.00443408 0.0124381 2024-06-27T21:16:41.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:41.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:41.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:41 smithi151 ceph-mon[28965]: pgmap v576: 27 pgs: 27 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:16:41.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:41.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:16:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:41.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:41.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:41 smithi190 ceph-mon[32247]: pgmap v576: 27 pgs: 27 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:16:41.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:41 smithi192 ceph-mon[32956]: pgmap v576: 27 pgs: 27 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:16:41.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:41.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:41.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:41.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:41.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:42.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:42.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:42.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:42.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:42.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:42.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:42.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:42.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:42.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:42.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 36326 36310 81.0428 100.188 0.00610987 0.0123329 2024-06-27T21:16:42.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:42.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:42.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:42.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:42.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:42.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:42.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:42.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:42.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:42.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:42.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:42.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:42.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:43.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:43.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:43.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:43.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:43.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:43.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:43.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:43.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:43.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:43.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:43.593 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 37707 37691 81.2242 86.3125 0.123735 0.0122553 2024-06-27T21:16:43.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:43.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:43.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:16:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:43 smithi151 ceph-mon[28965]: pgmap v577: 27 pgs: 27 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:16:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:43 smithi151 ceph-mon[28965]: osdmap e200: 12 total, 12 up, 12 in 2024-06-27T21:16:43.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:43 smithi190 ceph-mon[32247]: pgmap v577: 27 pgs: 27 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:16:43.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:43 smithi190 ceph-mon[32247]: osdmap e200: 12 total, 12 up, 12 in 2024-06-27T21:16:43.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:43.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:43 smithi192 ceph-mon[32956]: pgmap v577: 27 pgs: 27 active+clean; 1.9 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:16:43.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:43 smithi192 ceph-mon[32956]: osdmap e200: 12 total, 12 up, 12 in 2024-06-27T21:16:43.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:43.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:43.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:43.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:44.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:44.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:44.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:44.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:44.209 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:44.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:44.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:44.385 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:16:44.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:44.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:44.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:44.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:44.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 39173 39157 81.5706 91.625 0.012381 0.0122461 2024-06-27T21:16:44.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1301150632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:16:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:44 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1301150632' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:16:44.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:44 smithi151 ceph-mon[28965]: osdmap e201: 12 total, 12 up, 12 in 2024-06-27T21:16:44.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:44.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1301150632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:16:44.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:44 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:44.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1301150632' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:16:44.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:44 smithi190 ceph-mon[32247]: osdmap e201: 12 total, 12 up, 12 in 2024-06-27T21:16:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1301150632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:16:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:44 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:16:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1301150632' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:16:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:44 smithi192 ceph-mon[32956]: osdmap e201: 12 total, 12 up, 12 in 2024-06-27T21:16:44.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:44.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:44.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:44.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:44.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:45.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:45.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:45.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:45.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:45.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:45.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:45.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:45.453 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-06-27T21:16:45.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:45.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:45.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 40040 40024 80.6872 54.1875 0.0497135 0.0123857 2024-06-27T21:16:45.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:45.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:45.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:45.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:45 smithi190 ceph-mon[32247]: pgmap v579: 27 pgs: 27 active+clean; 2.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:16:45.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3332839315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:16:45.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:45 smithi192 ceph-mon[32956]: pgmap v579: 27 pgs: 27 active+clean; 2.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:16:45.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3332839315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:16:45.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:45.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:45.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:45 smithi151 ceph-mon[28965]: pgmap v579: 27 pgs: 27 active+clean; 2.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:16:45.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3332839315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:16:45.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:45.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:45.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:46.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:46.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:46.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:46.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:46.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:46.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:46.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:46.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:46.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 16 41354 41338 80.7319 82.125 0.00271162 0.0123502 2024-06-27T21:16:46.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:46.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:46.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:46.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:46.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:46.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:46.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3332839315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:16:46.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:46 smithi190 ceph-mon[32247]: osdmap e202: 12 total, 12 up, 12 in 2024-06-27T21:16:46.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:46 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:16:46.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:46.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:46.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3332839315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:16:46.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:46 smithi192 ceph-mon[32956]: osdmap e202: 12 total, 12 up, 12 in 2024-06-27T21:16:46.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:46 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:16:46.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:46.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:46.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3332839315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:16:46.795 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:46 smithi151 ceph-mon[28965]: osdmap e202: 12 total, 12 up, 12 in 2024-06-27T21:16:46.795 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:46 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:16:46.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:46.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:46.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:46.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:47.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:47.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:47.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:47.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:47.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:47.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 8 1 2024-06-27T21:16:47.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:47.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:47.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:47.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:47.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 42769 42753 80.9652 88.4375 0.00759612 0.0123456 2024-06-27T21:16:47.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:47.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:47.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:47 smithi151 ceph-mon[28965]: pgmap v582: 27 pgs: 27 active+clean; 2.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:16:47.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:47 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:16:47.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:47.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:47 smithi190 ceph-mon[32247]: pgmap v582: 27 pgs: 27 active+clean; 2.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:16:47.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:47 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:16:47.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:47.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:47 smithi192 ceph-mon[32956]: pgmap v582: 27 pgs: 27 active+clean; 2.3 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:16:47.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:47 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:16:47.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:47.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:47.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:48.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:48.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:48.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:48.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:48.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:48.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:48.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:48.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:48.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:48.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:16:48.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:48.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:48.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 44695 44679 82.1241 120.375 0.0135988 0.0121711 2024-06-27T21:16:48.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:48.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:48.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:48.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:48.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:48 smithi190 ceph-mon[32247]: osdmap e203: 12 total, 12 up, 12 in 2024-06-27T21:16:48.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:48.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:48.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:48 smithi151 ceph-mon[28965]: osdmap e203: 12 total, 12 up, 12 in 2024-06-27T21:16:48.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:48.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:48.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:48 smithi192 ceph-mon[32956]: osdmap e203: 12 total, 12 up, 12 in 2024-06-27T21:16:48.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:48.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:48.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:48.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:48.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:49.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:49.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:49.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:49.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:49.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:49.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:49.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:49.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:49.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:49.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:49.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 16 46004 45988 82.115 81.8125 0.00725361 0.0121693 2024-06-27T21:16:49.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:49.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:49 smithi151 ceph-mon[28965]: pgmap v584: 27 pgs: 27 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-27T21:16:49.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:49.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:49 smithi190 ceph-mon[32247]: pgmap v584: 27 pgs: 27 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-27T21:16:49.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:49 smithi192 ceph-mon[32956]: pgmap v584: 27 pgs: 27 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-27T21:16:49.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:49.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:49.881 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:49.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:49.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:49.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:49.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:50.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:50.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:50.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:50.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:50.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:50.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:50.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:50.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:50.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:50.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:50.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:50.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:50.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 47083 47067 81.7072 67.4375 0.0557829 0.0122264 2024-06-27T21:16:50.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:50.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:50.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:50.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:50.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:50.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:50.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:51.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:51.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:51.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:51.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:51.276 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:16:51.276 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:16:51.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:16:51.306 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:16:51.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:51.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:16:51.333 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:16:51.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:51.354 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:16:51.355 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:16:51.376 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:16:51.377 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:16:51.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:16:51.398 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:16:51.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:16:51.420 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:16:51.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:51.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:51.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:16:51.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:16:51.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:51.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:16:51.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:16:51.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:51 smithi151 ceph-mon[28965]: pgmap v585: 27 pgs: 27 active+clean; 2.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-27T21:16:51.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:51 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:16:51.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:51 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:16:51.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:51 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:16:51.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:51 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:16:51.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/821679982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:51.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 15 48159 48144 81.318 67.3125 0.00565876 0.0122924 2024-06-27T21:16:51.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:51.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:16:51.604 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:16:51.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:16:51.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:16:51.675 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:16:51.675 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:16:51.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:51.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:51 smithi190 ceph-mon[32247]: pgmap v585: 27 pgs: 27 active+clean; 2.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-27T21:16:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:51 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:16:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:51 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:16:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:51 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:16:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:51 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:16:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/821679982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:51.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:51 smithi192 ceph-mon[32956]: pgmap v585: 27 pgs: 27 active+clean; 2.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-27T21:16:51.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:51 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:16:51.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:51 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:16:51.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:51 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:16:51.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:51 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:16:51.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/821679982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:16:51.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:51.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:51.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:51.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:52.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:52.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:52.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:52.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:52.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:52.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:52.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:52.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:52.516 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:16:52.516 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:16:52.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:52.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 49225 49209 80.9296 66.5625 0.0132835 0.0123478 2024-06-27T21:16:52.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:52.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:52.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:52.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:52 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:16:52.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:52 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:16:52.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:52 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:16:52.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:52 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:16:52.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:52.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:52.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:52.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:52 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:16:52.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:52 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:16:52.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:52.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:52.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:53.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:53.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:53.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:53.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:53.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:53.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:53.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:53.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:53.594 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 50778 50762 81.343 97.0625 0.00271054 0.0122875 2024-06-27T21:16:53.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:53.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:53.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:53.754 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:53.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:53 smithi190 ceph-mon[32247]: pgmap v586: 27 pgs: 27 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:16:53.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:53 smithi192 ceph-mon[32956]: pgmap v586: 27 pgs: 27 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:16:53.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:53.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:53.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:53.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:53 smithi151 ceph-mon[28965]: pgmap v586: 27 pgs: 27 active+clean; 2.7 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:16:53.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:54.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:54.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:54.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:54.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:54.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:54.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:54.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:54.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:16:54.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:54.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:54.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:16:54.600457+0000 min lat: 0.00107574 max lat: 3.0199 avg lat: 0.0123333 2024-06-27T21:16:54.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:16:54.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 51874 51858 81.0218 68.5 0.00896583 0.0123333 2024-06-27T21:16:54.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:54.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:54.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:16:54.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:16:54.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:54.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:54.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:54.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:55.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:55.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:55.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:55.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:55.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:55.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:55.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:55.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:55.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:55.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 15 53171 53156 81.0241 81.125 0.0216403 0.0123356 2024-06-27T21:16:55.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:55.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:55 smithi190 ceph-mon[32247]: pgmap v587: 27 pgs: 27 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:16:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:55.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:55.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:55 smithi192 ceph-mon[32956]: pgmap v587: 27 pgs: 27 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:16:55.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:55.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:55.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:55.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:55.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:55.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:55 smithi151 ceph-mon[28965]: pgmap v587: 27 pgs: 27 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:16:55.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:55.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:56.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:56.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:56.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:56.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:56.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:56.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:56.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:56.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:56.512 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:56.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:56.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:56.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 54584 54568 81.196 88.25 0.0108641 0.0123088 2024-06-27T21:16:56.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:56.681 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:56.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:56 smithi190 ceph-mon[32247]: osdmap e204: 12 total, 12 up, 12 in 2024-06-27T21:16:56.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:56 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:16:56.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:16:56.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:56 smithi192 ceph-mon[32956]: osdmap e204: 12 total, 12 up, 12 in 2024-06-27T21:16:56.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:56 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:16:56.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:56 smithi151 ceph-mon[28965]: osdmap e204: 12 total, 12 up, 12 in 2024-06-27T21:16:56.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:56 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:16:56.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:16:56.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:56.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:56.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:57.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:57.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:57.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:57.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:57.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:57.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:57.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:57.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:57.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:16:57.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:57.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:57.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 55753 55737 81.0067 73.0625 0.00337345 0.012339 2024-06-27T21:16:57.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:57.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:57.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:57 smithi190 ceph-mon[32247]: pgmap v589: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:16:57.789 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:57 smithi192 ceph-mon[32956]: pgmap v589: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:16:57.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:57.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:57.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:57 smithi151 ceph-mon[28965]: pgmap v589: 27 pgs: 27 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:16:57.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:57.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:57.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:58.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:58.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:58.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:16:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:58.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:16:58.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:16:58.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:16:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:16:58.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:58.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:58.502 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-06-27T21:16:58.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 57293 57277 81.353 96.25 0.0175366 0.0122838 2024-06-27T21:16:58.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:16:58.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:16:58.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:58.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:58.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:58.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:58.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: osdmap e205: 12 total, 12 up, 12 in 2024-06-27T21:16:58.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/654775951' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:16:58.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:16:58.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:16:58.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:58.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:58.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:58.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:58.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:58.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:58.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: osdmap e205: 12 total, 12 up, 12 in 2024-06-27T21:16:58.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/654775951' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:16:58.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:16:58.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:16:58.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:16:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:58.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:16:58.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:58.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:58.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: osdmap e205: 12 total, 12 up, 12 in 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/654775951' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:16:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:16:59.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:16:59.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:16:59.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:16:59.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:16:59.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:59.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:59.389 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:16:59.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:16:59.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:16:59.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 57973 57957 80.4895 42.5 0.0101859 0.012418 2024-06-27T21:16:59.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:16:59.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:16:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: pgmap v591: 27 pgs: 27 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/654775951' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: osdmap e206: 12 total, 12 up, 12 in 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1205703852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:16:59.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1205703852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:59.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:16:59 smithi151 ceph-mon[28965]: osdmap e207: 12 total, 12 up, 12 in 2024-06-27T21:16:59.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:16:59.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:16:59.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: pgmap v591: 27 pgs: 27 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/654775951' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: osdmap e206: 12 total, 12 up, 12 in 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1205703852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1205703852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:16:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:16:59 smithi192 ceph-mon[32956]: osdmap e207: 12 total, 12 up, 12 in 2024-06-27T21:16:59.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:16:59.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:16:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:16:59.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:16:59.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:00.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: pgmap v591: 27 pgs: 27 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/654775951' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: osdmap e206: 12 total, 12 up, 12 in 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1205703852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1205703852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:16:59 smithi190 ceph-mon[32247]: osdmap e207: 12 total, 12 up, 12 in 2024-06-27T21:17:00.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:00.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:00.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:00.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:00.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:00.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:00.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:00.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:00.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 9 1 2024-06-27T21:17:00.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 59063 59047 80.2206 68.125 0.00253665 0.0124575 2024-06-27T21:17:00.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:00.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:00.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:00.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:00.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:00.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:00.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:00.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:01.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:01.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:01.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:01.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:01.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:01.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:01.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:01.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:01.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:01.516 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:01.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 60147 60131 79.9552 67.75 0.00310989 0.0125 2024-06-27T21:17:01.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:01.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:01.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:01.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:01.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:01 smithi192 ceph-mon[32956]: pgmap v594: 27 pgs: 27 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:17:01.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:01.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:01.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:01.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:01 smithi151 ceph-mon[28965]: pgmap v594: 27 pgs: 27 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:17:02.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:02.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:01 smithi190 ceph-mon[32247]: pgmap v594: 27 pgs: 27 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:17:02.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:02.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:02.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:02.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:02.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:02.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:02.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:02.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:02.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:02.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 61429 61413 79.9586 80.125 0.0059121 0.0124987 2024-06-27T21:17:02.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:02.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:02.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:02.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:02.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:02.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:02.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:03.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:03.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:03.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:03.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:03.244 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:03.244 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:03.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:03.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:03.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:03.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:03.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:03.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:03.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 16 62674 62658 79.9147 77.8125 0.0071704 0.0125076 2024-06-27T21:17:03.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:03.695 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:03.724 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:17:03.724 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:17:03.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:17:03.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:17:03.767 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:17:03.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:17:03.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:03 smithi151 ceph-mon[28965]: pgmap v595: 27 pgs: 27 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:17:03.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:03 smithi151 ceph-mon[28965]: 3.b scrub starts 2024-06-27T21:17:03.788 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:17:03.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:17:03.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:03 smithi192 ceph-mon[32956]: pgmap v595: 27 pgs: 27 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:17:03.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:03 smithi192 ceph-mon[32956]: 3.b scrub starts 2024-06-27T21:17:03.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:03.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:03.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:17:03.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:17:03.859 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:17:03.860 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:17:03.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:17:03.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:17:03.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:03.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:17:03.926 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:17:03.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:03.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:03.959 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:17:03.960 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:17:03.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:17:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:17:04.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:03 smithi190 ceph-mon[32247]: pgmap v595: 27 pgs: 27 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:17:04.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:03 smithi190 ceph-mon[32247]: 3.b scrub starts 2024-06-27T21:17:04.029 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:17:04.029 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:17:04.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:04.057 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:17:04.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:17:04.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:04.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:04.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:17:04.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:04.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:04.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:04.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:04.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:04.542 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:04.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:04.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:04.595 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 63867 63851 79.8075 74.5625 0.0374063 0.0124954 2024-06-27T21:17:04.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:04.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:17:04.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:04.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: 3.1 scrub starts 2024-06-27T21:17:04.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:04.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:04.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: 3.1 scrub ok 2024-06-27T21:17:04.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: osdmap e208: 12 total, 12 up, 12 in 2024-06-27T21:17:04.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3247249074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: osdmap e209: 12 total, 12 up, 12 in 2024-06-27T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:17:04.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:04 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:17:04.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:04.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: 3.1 scrub starts 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: 3.1 scrub ok 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: osdmap e208: 12 total, 12 up, 12 in 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3247249074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: osdmap e209: 12 total, 12 up, 12 in 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:17:04.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:17:04.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:17:04.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:17:04.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:04 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:17:04.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: 3.1 scrub starts 2024-06-27T21:17:04.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:04.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:04.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: 3.1 scrub ok 2024-06-27T21:17:04.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: osdmap e208: 12 total, 12 up, 12 in 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3247249074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: osdmap e209: 12 total, 12 up, 12 in 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:17:04.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:04 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:17:04.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:04.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:04.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:05.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:05.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:05.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:05.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:05.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:05.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:05.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:05.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:05.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:05.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 65069 65053 79.7155 75.125 0.00951734 0.0125401 2024-06-27T21:17:05.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:05.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:05.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:05.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:05.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: 3.b scrub ok 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:17:05.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:05 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: 3.b scrub ok 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:17:05.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:05 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:17:05.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: 3.b scrub ok 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:17:05.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:17:05.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:05 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:17:05.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:05.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:05.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:06.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:06.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:06.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:06.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:06.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:06.358 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:06.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:06.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:06.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:06.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:06.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 15 66273 66258 79.6307 75.3125 0.00333785 0.0125521 2024-06-27T21:17:06.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:06.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:06.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:06.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:06.882 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:17:06.883 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:17:06.883 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:17:06.883 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:17:06.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:06.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:06.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:07.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:07.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:07.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:07.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:07.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:07.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:07.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:07.425 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:07.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:07.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:17:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:07.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:07.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 67508 67492 79.5834 77.125 0.0129732 0.0125588 2024-06-27T21:17:07.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:17:07.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:07.663 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:07 smithi151 ceph-mon[28965]: pgmap v599: 27 pgs: 27 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:17:07.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:07.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:07.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:07 smithi192 ceph-mon[32956]: pgmap v599: 27 pgs: 27 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:17:07.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:07.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:07.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:07.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:07.951 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:07.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:07 smithi190 ceph-mon[32247]: pgmap v599: 27 pgs: 27 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:17:08.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:08.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:08.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:08.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:08.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:08.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:08.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:08.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:08.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:08.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:08.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 68577 68561 79.3468 66.8125 0.00388181 0.0125963 2024-06-27T21:17:08.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:08 smithi151 ceph-mon[28965]: 3.8 scrub starts 2024-06-27T21:17:08.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:08.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:08.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:08.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:08.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:08 smithi192 ceph-mon[32956]: 3.8 scrub starts 2024-06-27T21:17:08.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:08.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:08.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:08.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:08.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:08.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:08.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:09.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:08 smithi190 ceph-mon[32247]: 3.8 scrub starts 2024-06-27T21:17:09.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:17:09.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:17:09.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:09.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:09.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:09.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:09.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:09.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:09.445 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:09.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:09.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:09.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 69704 69688 79.1847 70.4375 0.00489304 0.0126225 2024-06-27T21:17:09.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: pgmap v600: 27 pgs: 27 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:17:09.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: osdmap e210: 12 total, 12 up, 12 in 2024-06-27T21:17:09.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: 3.8 scrub ok 2024-06-27T21:17:09.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: 1.a deep-scrub starts 2024-06-27T21:17:09.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: 1.a deep-scrub ok 2024-06-27T21:17:09.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: osdmap e211: 12 total, 12 up, 12 in 2024-06-27T21:17:09.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1753669330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:09.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:09 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:09.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:09.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:09.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: pgmap v600: 27 pgs: 27 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: osdmap e210: 12 total, 12 up, 12 in 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: 3.8 scrub ok 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: 1.a deep-scrub starts 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: 1.a deep-scrub ok 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: osdmap e211: 12 total, 12 up, 12 in 2024-06-27T21:17:09.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1753669330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:09.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:09 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:09.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:09.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:09.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:09.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: pgmap v600: 27 pgs: 27 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: osdmap e210: 12 total, 12 up, 12 in 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: 3.8 scrub ok 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: 1.a deep-scrub starts 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: 1.a deep-scrub ok 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: osdmap e211: 12 total, 12 up, 12 in 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1753669330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:09.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:09 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:10.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:10.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:10.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:10.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:10.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:10.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:10.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:10.401 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-06-27T21:17:10.408 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:17:10.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:10.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:10.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1468789680' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:17:10.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:10 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:10.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:10 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:17:10.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1468789680' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:17:10.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:10 smithi190 ceph-mon[32247]: osdmap e212: 12 total, 12 up, 12 in 2024-06-27T21:17:10.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 70807 70791 79.0016 68.9375 0.00310388 0.0126528 2024-06-27T21:17:10.603 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:17:10.603 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:17:10.603 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:17:10.603 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:17:10.604 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:17:10.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:10.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:10.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1468789680' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:17:10.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:10 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:10.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:10 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:17:10.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1468789680' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:17:10.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:10 smithi151 ceph-mon[28965]: osdmap e212: 12 total, 12 up, 12 in 2024-06-27T21:17:10.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:10.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:10.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1468789680' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:17:10.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:10 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:10.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:10 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:17:10.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1468789680' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:17:10.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:10 smithi192 ceph-mon[32956]: osdmap e212: 12 total, 12 up, 12 in 2024-06-27T21:17:10.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:10.905 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:10.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:11.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:11.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:11.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:11.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:11.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:11.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:11.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:11.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:11.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:11.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:11.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 72200 72184 79.1429 87.0625 0.00708818 0.0126304 2024-06-27T21:17:11.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:11.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:11.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:11 smithi151 ceph-mon[28965]: pgmap v603: 27 pgs: 27 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-27T21:17:11.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:11 smithi151 ceph-mon[28965]: 3.4 scrub starts 2024-06-27T21:17:11.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 10 1 2024-06-27T21:17:11.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:11.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:11 smithi190 ceph-mon[32247]: pgmap v603: 27 pgs: 27 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-27T21:17:11.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:11 smithi190 ceph-mon[32247]: 3.4 scrub starts 2024-06-27T21:17:11.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:11.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:17:11.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:11.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:11.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:12.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:11 smithi192 ceph-mon[32956]: pgmap v603: 27 pgs: 27 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-06-27T21:17:12.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:11 smithi192 ceph-mon[32956]: 3.4 scrub starts 2024-06-27T21:17:12.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:12.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:12.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:12.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:12.263 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:12.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:12.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:12.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:12.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:12.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:12.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:12.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 73648 73632 79.3386 90.5 0.00499889 0.0125993 2024-06-27T21:17:12.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:12.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:12.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:12.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:12.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:12.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:12.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:13.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:13.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:13.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:13.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:13.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:13.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:13.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:13.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:13.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:13.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 74716 74700 79.1252 66.75 0.00434456 0.0126328 2024-06-27T21:17:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:13.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:13.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:13.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:13.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:13.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:13 smithi190 ceph-mon[32247]: pgmap v605: 27 pgs: 27 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 974 op/s 2024-06-27T21:17:13.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:13.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:13.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:13.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:13 smithi151 ceph-mon[28965]: pgmap v605: 27 pgs: 27 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 974 op/s 2024-06-27T21:17:13.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:14.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:14.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:13 smithi192 ceph-mon[32956]: pgmap v605: 27 pgs: 27 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 974 op/s 2024-06-27T21:17:14.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:14.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:14.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:14.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:14.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:14.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:14.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:14.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:14.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:17:14.601983+0000 min lat: 0.00107574 max lat: 3.0199 avg lat: 0.0126492 2024-06-27T21:17:14.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:17:14.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 75885 75869 79.0241 73.0625 0.00526454 0.0126492 2024-06-27T21:17:14.627 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:17:14.627 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:17:14.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:17:14.627 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:12970 2.f e109/104 2:f05cefe0:::benchmark_data_smithi192_59818_object12969:head v 109'790, mlcod=109'787)", 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:53.758876+0000", 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 320.870556663, 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18035446499999999, 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.758876+0000", 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.758876+0000", 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.758877+0000", 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8999999999999996e-07 2024-06-27T21:17:14.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.758913+0000", 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6081999999999998e-05 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.938618+0000", 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.179705324 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.938625+0000", 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9110000000000004e-06 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.938670+0000", 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4490999999999998e-05 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.938684+0000", 2024-06-27T21:17:14.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3848e-05 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.939189+0000", 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050494699999999995 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:53.939231+0000", 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2271999999999998e-05 2024-06-27T21:17:14.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:15439 2.5 e110/104 2:aca3d0f0:::benchmark_data_smithi192_59818_object15438:head v 110'900, mlcod=110'898)", 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:55.565577+0000", 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 319.06385619500003, 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18853719999999999, 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.633 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565577+0000", 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565577+0000", 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565578+0000", 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.071e-06 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565648+0000", 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9678000000000006e-05 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753307+0000", 2024-06-27T21:17:14.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18765964600000001 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753322+0000", 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4707e-05 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753441+0000", 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011891 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753458+0000", 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7104000000000001e-05 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.754083+0000", 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00062465600000000002 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.754114+0000", 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1427999999999998e-05 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:15465 2.f e110/104 2:f871ccb0:::benchmark_data_smithi192_59818_object15464:head v 110'946, mlcod=110'944)", 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 319.05257720499998, 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.211785213, 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.638 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4700000000000002e-07 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576894+0000", 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7351999999999997e-05 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787747+0000", 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.210853076 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787752+0000", 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3489999999999999e-06 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787815+0000", 2024-06-27T21:17:14.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2385000000000007e-05 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787848+0000", 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3575999999999998e-05 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.788593+0000", 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074521400000000001 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.788641+0000", 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7814e-05 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:121203 2.f e140/104 2:f332e2ed:::benchmark_data_smithi192_59818_object29128:head v 140'7678, mlcod=140'7675)", 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.258481+0000", 2024-06-27T21:17:14.643 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 209.37095261499999, 2024-06-27T21:17:14.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.17272217500000001, 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.258481+0000", 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.258481+0000", 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.258481+0000", 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9299999999999998e-07 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.258486+0000", 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4689999999999999e-06 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.428598+0000", 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.170112613 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.428605+0000", 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3829999999999997e-06 2024-06-27T21:17:14.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.428683+0000", 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8065000000000006e-05 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.428699+0000", 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.615e-05 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.431170+0000", 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024712660000000002 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.431203+0000", 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2635999999999997e-05 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.649 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(2.8 158/154 [PushOp(2:1ef68a07:::benchmark_data_smithi192_59818_object66542:head, version: 125'4118, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1ef68a07:::benchmark_data_smithi192_59818_object66542:head@125'4118, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:14:41.642894+0000", 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 152.98653928300001, 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.233094159, 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642894+0000", 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642894+0000", 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642895+0000", 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0979999999999999e-06 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642940+0000", 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4817000000000001e-05 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.873921+0000", 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.230981196 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.873936+0000", 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5299999999999999e-05 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.874003+0000", 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7003999999999997e-05 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.874011+0000", 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5979999999999999e-06 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.875988+0000", 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019771459999999999 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:2786 3.b e192/186 3:d5b8f08f:::benchmark_data_smithi192_74314_object2785:head v 192'170, mlcod=192'167)", 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:16.249262+0000", 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 58.38017095, 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.212590011, 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.654 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249262+0000", 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249262+0000", 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249263+0000", 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249315+0000", 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2621000000000003e-05 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461452+0000", 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.212136667 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461457+0000", 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5020000000000003e-06 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461502+0000", 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5300999999999999e-05 2024-06-27T21:17:14.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461515+0000", 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2909e-05 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461821+0000", 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030602 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461852+0000", 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1319000000000003e-05 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:6040 3.8 e192/186 3:18c08d67:::benchmark_data_smithi192_74314_object6039:head v 192'392, mlcod=192'389)", 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.444313121999997, 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201679577, 2024-06-27T21:17:14.659 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:17:14.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9399999999999995e-07 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185217+0000", 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6491999999999997e-05 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386204+0000", 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20098743599999999 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386212+0000", 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5730000000000002e-06 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386285+0000", 2024-06-27T21:17:14.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3151999999999994e-05 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386313+0000", 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8017999999999998e-05 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386769+0000", 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045611700000000002 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386800+0000", 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0295e-05 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:6041 3.8 e192/186 3:1fe618f9:::benchmark_data_smithi192_74314_object6040:head v 192'393, mlcod=192'389)", 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:18.185644+0000", 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.443788646000002, 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201844833, 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.664 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185644+0000", 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185644+0000", 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185645+0000", 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1899999999999998e-07 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185700+0000", 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5392000000000001e-05 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386986+0000", 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201285557 2024-06-27T21:17:14.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386990+0000", 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2830000000000003e-06 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387035+0000", 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4728000000000003e-05 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387056+0000", 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0859000000000001e-05 2024-06-27T21:17:14.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387454+0000", 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039844799999999998 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387489+0000", 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5147000000000001e-05 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:6939 3.6 e192/186 3:684a123b:::benchmark_data_smithi192_74314_object6938:head v 192'460, mlcod=192'458)", 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:18.820458+0000", 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 55.808975056999998, 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.159661831, 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.820458+0000", 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.820458+0000", 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.820459+0000", 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0999999999999998e-07 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.820482+0000", 2024-06-27T21:17:14.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3621999999999998e-05 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.979574+0000", 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15909126900000001 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.979580+0000", 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4579999999999998e-06 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.979633+0000", 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2546000000000001e-05 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.979646+0000", 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3665e-05 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.980093+0000", 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044628499999999999 2024-06-27T21:17:14.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.980120+0000", 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7376e-05 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:11628 3.b e194/186 3:d81385b5:::benchmark_data_smithi192_74314_object11627:head v 194'698, mlcod=194'695)", 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 52.604605308000004, 2024-06-27T21:17:14.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18984046600000001, 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4799999999999998e-07 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024890+0000", 2024-06-27T21:17:14.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1645999999999999e-05 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214049+0000", 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.1891593 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214053+0000", 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.738e-06 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214095+0000", 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2193000000000001e-05 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214117+0000", 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1214999999999999e-05 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214634+0000", 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051760599999999999 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214668+0000", 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4119999999999999e-05 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.678 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:18061 3.3 3:c79bbfab:::benchmark_data_smithi192_74314_object18060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:26.554109+0000", 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.075324530000003, 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18618305199999999, 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 18061 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554109+0000", 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554109+0000", 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554111+0000", 2024-06-27T21:17:14.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4109999999999999e-06 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554147+0000", 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6078000000000003e-05 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554163+0000", 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5753e-05 2024-06-27T21:17:14.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554171+0000", 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.858e-06 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554227+0000", 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6242000000000001e-05 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554347+0000", 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120308 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.554483+0000", 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000135945 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.555338+0000", 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085482600000000005 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.561236+0000", 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058976499999999999 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.740226+0000", 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.17899061499999999 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.740262+0000", 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6040000000000001e-05 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.740292+0000", 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9326e-05 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:18059 3.6 e195/186 3:65e67f00:::benchmark_data_smithi192_74314_object18058:head v 195'1140, mlcod=195'1137)", 2024-06-27T21:17:14.685 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:26.560267+0000", 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.069166482999997, 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.178031418, 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.560267+0000", 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.560267+0000", 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.560267+0000", 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.06e-07 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.560291+0000", 2024-06-27T21:17:14.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4116000000000001e-05 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.737433+0000", 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.177141621 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.737440+0000", 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7689999999999999e-06 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.737493+0000", 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3386e-05 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.737518+0000", 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.516e-05 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.738256+0000", 2024-06-27T21:17:14.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000737852 2024-06-27T21:17:14.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:26.738298+0000", 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1907999999999999e-05 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.691 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:31879 3.b e199/186 3:d2f3bbbc:::benchmark_data_smithi192_74314_object31878:head v 199'1975, mlcod=199'1970)", 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:38.913878+0000", 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.715555502999997, 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15571423200000001, 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:38.913878+0000", 2024-06-27T21:17:14.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:38.913878+0000", 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:38.913879+0000", 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7999999999999999e-06 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:38.913926+0000", 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6548999999999998e-05 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:39.068556+0000", 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.154629881 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:39.068562+0000", 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8289999999999996e-06 2024-06-27T21:17:14.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:39.068614+0000", 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2225000000000002e-05 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:39.068629+0000", 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4785e-05 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:39.069561+0000", 2024-06-27T21:17:14.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000932524 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:39.069592+0000", 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0639000000000003e-05 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:41330 3.3 3:c3cab74f:::benchmark_data_smithi192_74314_object41329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:46.500500+0000", 2024-06-27T21:17:14.696 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.128932818999999, 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20752989099999999, 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 41330 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500500+0000", 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500500+0000", 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500502+0000", 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.206e-06 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500561+0000", 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.948e-05 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500572+0000", 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0818999999999999e-05 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500577+0000", 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3120000000000001e-06 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500626+0000", 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8449000000000003e-05 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500732+0000", 2024-06-27T21:17:14.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106128 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500895+0000", 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000163282 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.502196+0000", 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001300867 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.502914+0000", 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071834000000000004 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707895+0000", 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20498121599999999 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707998+0000", 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010217099999999999 2024-06-27T21:17:14.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.708030+0000", 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2620999999999998e-05 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.15132.0:41330 3.3 e202/186 ondisk, result = 0)", 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:46.514144+0000", 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.115289177000001, 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19389235699999999, 2024-06-27T21:17:14.703 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514144+0000", 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514144+0000", 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514145+0000", 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3499999999999996e-07 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514148+0000", 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.19e-06 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707715+0000", 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19356746599999999 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707729+0000", 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.384e-05 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707868+0000", 2024-06-27T21:17:14.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013926300000000001 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707886+0000", 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7753000000000001e-05 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.708036+0000", 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015020999999999999 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.707 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:42498 3.b e202/186 3:d2874a3a:::benchmark_data_smithi192_74314_object42497:head v 202'2668, mlcod=202'2665)", 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:47.300452+0000", 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.328981598999999, 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15317024400000001, 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.300452+0000", 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.300452+0000", 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.300453+0000", 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7409999999999999e-06 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.300489+0000", 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5413000000000001e-05 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.452121+0000", 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.151631924 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.452133+0000", 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2173e-05 2024-06-27T21:17:14.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.452226+0000", 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2992999999999995e-05 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.452249+0000", 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3455000000000002e-05 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.453587+0000", 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001337719 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:47.453622+0000", 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4826000000000002e-05 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:63857 3.3 3:ccd25bbb:::benchmark_data_smithi192_74314_object63856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:17:14.712 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:04.475553+0000", 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.153880344999999, 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21602158199999999, 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 63857 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475553+0000", 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475553+0000", 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475554+0000", 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4950000000000001e-06 2024-06-27T21:17:14.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475608+0000", 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3335999999999999e-05 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475626+0000", 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8119000000000001e-05 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475633+0000", 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0690000000000004e-06 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475693+0000", 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0194e-05 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475847+0000", 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015441599999999999 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.476076+0000", 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022837399999999999 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.478815+0000", 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027391020000000002 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.478856+0000", 2024-06-27T21:17:14.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1196000000000003e-05 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691507+0000", 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21265120200000001 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691545+0000", 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7818000000000003e-05 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691574+0000", 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9261e-05 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:63864 3.3 3:cdb1142a:::benchmark_data_smithi192_74314_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:04.485645+0000", 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.143787909, 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20605995899999999, 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.719 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 63864 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485645+0000", 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485645+0000", 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485646+0000", 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1880000000000001e-06 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485700+0000", 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3418999999999998e-05 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485712+0000", 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1904e-05 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485718+0000", 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0560000000000001e-06 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485756+0000", 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8611e-05 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485858+0000", 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101432 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485953+0000", 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5674999999999995e-05 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.487617+0000", 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001663159 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.488072+0000", 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045525600000000002 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691621+0000", 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20354951299999999 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691670+0000", 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8455999999999998e-05 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691705+0000", 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5290000000000003e-05 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.725 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:74089 3.3 3:c8a9f374:::benchmark_data_smithi192_74314_object74088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:12.970999+0000", 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.658434379, 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16573149000000001, 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:14.726 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 74089 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.970999+0000", 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.970999+0000", 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971000+0000", 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.452e-06 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971042+0000", 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1857000000000003e-05 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971055+0000", 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3379e-05 2024-06-27T21:17:14.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971063+0000", 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2479999999999999e-06 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971118+0000", 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5568000000000001e-05 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971243+0000", 2024-06-27T21:17:14.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125196 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.971382+0000", 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013823700000000001 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.976242+0000", 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048606860000000003 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.976268+0000", 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5486999999999998e-05 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.136625+0000", 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16035718299999999 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.136684+0000", 2024-06-27T21:17:14.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8539000000000003e-05 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.136730+0000", 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6657999999999999e-05 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:74090 3.3 3:cbf6fca6:::benchmark_data_smithi192_74314_object74089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-27T21:17:14.732 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:12.972294+0000", 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.6571396109999998, 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16945737999999999, 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 74090 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.972294+0000", 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.972294+0000", 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.972294+0000", 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7000000000000003e-07 2024-06-27T21:17:14.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.972358+0000", 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3423000000000006e-05 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.972369+0000", 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0937999999999999e-05 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:14.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.972377+0000", 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.763e-06 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.973748+0000", 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013715190000000001 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.973886+0000", 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013792200000000001 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.974036+0000", 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014974699999999999 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.976313+0000", 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022773989999999998 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.024816+0000", 2024-06-27T21:17:14.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.048502793000000002 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.141648+0000", 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.116831779 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.141704+0000", 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5992000000000002e-05 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:14.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.141751+0000", 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7135000000000002e-05 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:14.739 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:17:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:14.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:14.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:14.849 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:14.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:14.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:14.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:15.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:15.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:15.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:15.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:15.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:15.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:15.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:17:15.378 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:17:15.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:15.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:17:15.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:17:15.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:17:15.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:17:15.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:17:15.469 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:17:15.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:15 smithi190 ceph-mon[32247]: pgmap v606: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:17:15.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:15 smithi190 ceph-mon[32247]: osdmap e213: 12 total, 12 up, 12 in 2024-06-27T21:17:15.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2026121808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:15.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:17:15.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:17:15.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:15.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:17:15.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:17:15.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:15.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:15.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:17:15.543 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:17:15.543 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:17:15.564 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:17:15.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:17:15.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:17:15.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:17:15.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 77121 77105 78.9949 77.25 0.071458 0.0126357 2024-06-27T21:17:15.608 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:17:15.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:17:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:15 smithi151 ceph-mon[28965]: pgmap v606: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:17:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:15 smithi151 ceph-mon[28965]: osdmap e213: 12 total, 12 up, 12 in 2024-06-27T21:17:15.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2026121808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:15.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:15.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:17:15.631 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:17:15.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:15.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:15.653 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:17:15.653 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:17:15.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:15.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:15.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:15 smithi192 ceph-mon[32956]: pgmap v606: 27 pgs: 27 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:17:15.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:15 smithi192 ceph-mon[32956]: osdmap e213: 12 total, 12 up, 12 in 2024-06-27T21:17:15.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2026121808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:15.834 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:15.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:15.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:15.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:16.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:16.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:16.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:16.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:16.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:16.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:16.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:16.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:16.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:16.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:16.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 78362 78346 78.9717 77.5625 0.026927 0.0126579 2024-06-27T21:17:16.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:16.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:16.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:17:16.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:16.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:16.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:16.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:16.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:17.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:17.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:17.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:17.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:17.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:17.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:17.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:17.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:17.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:17.545 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:17 smithi151 ceph-mon[28965]: pgmap v608: 27 pgs: 27 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:17:17.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:17.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 79333 79317 78.6814 60.6875 0.0113465 0.0127048 2024-06-27T21:17:17.632 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:17.632 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:17.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:17 smithi190 ceph-mon[32247]: pgmap v608: 27 pgs: 27 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:17:17.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:17.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:17.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:17.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:17 smithi192 ceph-mon[32956]: pgmap v608: 27 pgs: 27 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:17:17.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:17.842 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:17.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:17.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:17.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:18.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:18.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:18.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:18.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:18.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:18.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:18.409 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-06-27T21:17:18.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:18.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:18.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:18 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:17:18.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:18 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:17:18.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3042053766' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:17:18.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:18 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:17:18.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:18.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:17:18.596 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 80723 80707 78.8093 86.875 0.00379299 0.0126838 2024-06-27T21:17:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:18.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:18.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:18 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:18.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:18 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:18.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3042053766' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:17:18.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:18 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:17:18.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:18.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:18.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:18 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:18.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:18 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:18.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3042053766' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:17:18.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:18 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:17:18.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:18.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:17:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:18.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:18.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:18.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:18.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:19.007 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:19.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:19.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:19.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:19.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:19.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 11 1 2024-06-27T21:17:19.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:19.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:19.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:19.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:19.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 82000 81984 78.8246 79.8125 0.0409345 0.0126803 2024-06-27T21:17:19.625 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:19.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:19.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:19 smithi151 ceph-mon[28965]: pgmap v609: 27 pgs: 27 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:17:19.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:19 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:17:19.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:19 smithi151 ceph-mon[28965]: osdmap e214: 12 total, 12 up, 12 in 2024-06-27T21:17:19.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:19 smithi151 ceph-mon[28965]: osdmap e215: 12 total, 12 up, 12 in 2024-06-27T21:17:19.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:19.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:19 smithi190 ceph-mon[32247]: pgmap v609: 27 pgs: 27 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:17:19.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:19 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:17:19.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:19 smithi190 ceph-mon[32247]: osdmap e214: 12 total, 12 up, 12 in 2024-06-27T21:17:19.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:19 smithi190 ceph-mon[32247]: osdmap e215: 12 total, 12 up, 12 in 2024-06-27T21:17:19.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:19.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:19 smithi192 ceph-mon[32956]: pgmap v609: 27 pgs: 27 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:17:19.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:19 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:17:19.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:19 smithi192 ceph-mon[32956]: osdmap e214: 12 total, 12 up, 12 in 2024-06-27T21:17:19.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:19 smithi192 ceph-mon[32956]: osdmap e215: 12 total, 12 up, 12 in 2024-06-27T21:17:19.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:19.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:19.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:20.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:20.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:20.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:20.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:20.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:20.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:20.347 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:20.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:20.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:20.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:20.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:20.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 82895 82879 78.4778 55.9375 0.0105133 0.0127375 2024-06-27T21:17:20.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:20.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:20.694 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:20.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:20.777 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:20.777 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:20.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:20.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:20.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:20.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:21.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:21.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:21.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:21.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:21.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:21.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:21.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:21 smithi190 ceph-mon[32247]: pgmap v612: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:17:21.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:21.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:17:21.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:21.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:21.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 15 84126 84111 78.4556 77 0.00686417 0.0127414 2024-06-27T21:17:21.660 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:21 smithi151 ceph-mon[28965]: pgmap v612: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:17:21.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:21.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:21.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:21 smithi192 ceph-mon[32956]: pgmap v612: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:17:21.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:21.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:21.865 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:21.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:21.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:21.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:22.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:22.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:22.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:22.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:22.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:22.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:22.412 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:17:22.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:22.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:22.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:22.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 15 85266 85251 78.3496 71.25 0.0196394 0.0127587 2024-06-27T21:17:22.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:22.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2545302249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:22.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2545302249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:22.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:22.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:22.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2545302249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:22.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:22.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:22.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:22.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:22.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:23.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:23.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:23.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:23.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:23.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:23.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:23.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:23.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:23.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:23 smithi151 ceph-mon[28965]: pgmap v613: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:17:23.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:23 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:23.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2545302249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:23.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:23 smithi151 ceph-mon[28965]: osdmap e216: 12 total, 12 up, 12 in 2024-06-27T21:17:23.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 86428 86412 78.2656 72.5625 0.00355583 0.0127715 2024-06-27T21:17:23.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:23.662 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:23.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:23 smithi190 ceph-mon[32247]: pgmap v613: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:17:23.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:23 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:23.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2545302249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:23.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:23 smithi190 ceph-mon[32247]: osdmap e216: 12 total, 12 up, 12 in 2024-06-27T21:17:23.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:23.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:23.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:23 smithi192 ceph-mon[32956]: pgmap v613: 27 pgs: 27 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:17:23.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:23 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:23.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2545302249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:23.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:23 smithi192 ceph-mon[32956]: osdmap e216: 12 total, 12 up, 12 in 2024-06-27T21:17:23.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:23.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:23.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:24.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:24.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:24.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:24.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:24.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:24.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:24.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:24.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:24.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:24.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:24.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 87525 87509 78.1269 68.5625 0.00313052 0.0127949 2024-06-27T21:17:24.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:24.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:24.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:24.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:24.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:24.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:24.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:25.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:25.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:25.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:25.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:25.259 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:25.259 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:25.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:25.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:25.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:25.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:25.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:25.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:25.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 88498 88482 77.883 60.8125 0.00825814 0.0128356 2024-06-27T21:17:25.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:25 smithi190 ceph-mon[32247]: pgmap v615: 27 pgs: 27 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:17:25.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:25 smithi190 ceph-mon[32247]: osdmap e217: 12 total, 12 up, 12 in 2024-06-27T21:17:25.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:25.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:25 smithi151 ceph-mon[28965]: pgmap v615: 27 pgs: 27 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:17:25.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:25 smithi151 ceph-mon[28965]: osdmap e217: 12 total, 12 up, 12 in 2024-06-27T21:17:25.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:25.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:25 smithi192 ceph-mon[32956]: pgmap v615: 27 pgs: 27 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:17:25.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:25 smithi192 ceph-mon[32956]: osdmap e217: 12 total, 12 up, 12 in 2024-06-27T21:17:25.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:25.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:25.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:17:25.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:25.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:25.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:26.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:26.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:26.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:26.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:26.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:26.275 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:26.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:26.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:26.413 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-06-27T21:17:26.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2112122793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:17:26.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:26 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:17:26.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:26.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:17:26.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:17:26.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:26.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:17:26.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:17:26.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:17:26.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:17:26.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:17:26.596 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:17:26.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 89619 89603 77.7743 70.0625 0.0106339 0.0128531 2024-06-27T21:17:26.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:26.631 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:17:26.631 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:17:26.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:26.665 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:17:26.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:17:26.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2112122793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:17:26.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:26 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:17:26.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:17:26.700 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:17:26.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:26.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:17:26.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:17:26.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:26.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:17:26.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:17:26.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2112122793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:17:26.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:26 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:17:26.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:17:26.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:17:26.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:17:26.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:17:26.858 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:17:26.858 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:17:26.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:26.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:26.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:26.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:27.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:27.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:17:27.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:27.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:27.172 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-27T21:17:27.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:17:27.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:27.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:27.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:27.410 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:27 smithi151 ceph-mon[28965]: pgmap v617: 27 pgs: 27 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:17:27.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:27 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:17:27.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:27 smithi151 ceph-mon[28965]: osdmap e218: 12 total, 12 up, 12 in 2024-06-27T21:17:27.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1192247131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:27.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:27.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:27.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 90835 90819 77.75 76 0.0106914 0.0128554 2024-06-27T21:17:27.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:27.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:27.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:27 smithi190 ceph-mon[32247]: pgmap v617: 27 pgs: 27 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:17:27.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:27 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:17:27.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:27 smithi190 ceph-mon[32247]: osdmap e218: 12 total, 12 up, 12 in 2024-06-27T21:17:27.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1192247131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:27.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:27.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:27 smithi192 ceph-mon[32956]: pgmap v617: 27 pgs: 27 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:17:27.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:27 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:17:27.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:27 smithi192 ceph-mon[32956]: osdmap e218: 12 total, 12 up, 12 in 2024-06-27T21:17:27.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1192247131' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:27.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:27.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:27.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:27.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:28.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:28.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:28.071 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:28.122 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:28.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:28.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:28.204 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:17:28.204 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:17:28.204 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:17:28.205 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:17:28.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:28.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:28.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:28.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:28.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:28.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:28.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 91595 91579 77.3411 47.5 0.010276 0.0129248 2024-06-27T21:17:28.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:28.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:28.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:28.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:28.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:28.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:17:28.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:28.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:29.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:29.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:29.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:29.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:29.318 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:29.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:29.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:29.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:29.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:29.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 92632 92616 77.174 64.8125 0.00377891 0.0129521 2024-06-27T21:17:29.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:29.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:29 smithi151 ceph-mon[28965]: pgmap v619: 27 pgs: 27 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-27T21:17:29.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:29 smithi151 ceph-mon[28965]: osdmap e219: 12 total, 12 up, 12 in 2024-06-27T21:17:29.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:29.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:29 smithi190 ceph-mon[32247]: pgmap v619: 27 pgs: 27 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-27T21:17:29.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:29 smithi190 ceph-mon[32247]: osdmap e219: 12 total, 12 up, 12 in 2024-06-27T21:17:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:29.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:29.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:29 smithi192 ceph-mon[32956]: pgmap v619: 27 pgs: 27 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-27T21:17:29.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:29 smithi192 ceph-mon[32956]: osdmap e219: 12 total, 12 up, 12 in 2024-06-27T21:17:29.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:29.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:29.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:29.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:30.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:30.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:30.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:17:30.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:30.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:30.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:30.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:30.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:30.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:30.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:30.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:30.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 93663 93647 77.0064 64.4375 0.0155986 0.0129814 2024-06-27T21:17:30.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:30.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:30.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:30.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:30.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:30.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:30.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:31.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:31.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:31.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:31.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:31.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:31.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:31.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:31.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:31.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:31 smithi190 ceph-mon[32247]: pgmap v621: 27 pgs: 27 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:17:31.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:31 smithi151 ceph-mon[28965]: pgmap v621: 27 pgs: 27 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:17:31.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:31.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:31.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 94703 94687 76.8504 65 0.0107688 0.0130074 2024-06-27T21:17:31.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:31.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:31.740 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:17:31.740 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:17:31.740 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:17:31.740 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:17:31.741 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:17:31.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:31.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:31 smithi192 ceph-mon[32956]: pgmap v621: 27 pgs: 27 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:17:31.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:31.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:31.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:31.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:32.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:32.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:32.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:32.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:32.233 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:17:32.234 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:17:32.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:32.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:32.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:32.421 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:17:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/189455042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:17:32.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:32 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:17:32.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:32.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:17:32.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/189455042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:17:32.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:32 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:17:32.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:32.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:32.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 95799 95783 76.7433 68.5 0.0119832 0.0130258 2024-06-27T21:17:32.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:32.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:32.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/189455042' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:17:32.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:32 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:17:32.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:32.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:32.855 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:32.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:32.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:32.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:33.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:33.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:33.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:33.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:33.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:33.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:33.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:33.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:33.568 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2024-06-27T21:17:31.899449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6020268576,"num_objects":91855,"num_object_clones":0,"num_object_copies":275565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91853,"num_write_kb":5878592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92865,"ondisk_log_size":92865,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19023548,"kb_used_data":17619268,"kb_used_omap":18,"kb_used_meta":1404077,"kb_avail":1105918276,"statfs":{"total":1151940427776,"available":1132460314624,"internally_reserved":0,"allocated":18042130432,"data_stored":18035835768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":1437775162},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":460718080,"num_objects":7030,"num_object_clones":0,"num_object_copies":21090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7030,"num_write_kb":449920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.496310"},"pg_stats":[{"pgid":"3.f","version":"219'5729","reported_seq":5805,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.585427+0000","last_change":"2024-06-27T21:16:02.437545+0000","last_active":"2024-06-27T21:17:29.585427+0000","last_peered":"2024-06-27T21:17:29.585427+0000","last_clean":"2024-06-27T21:17:29.585427+0000","last_became_active":"2024-06-27T21:16:02.437136+0000","last_became_peered":"2024-06-27T21:16:02.437136+0000","last_unstale":"2024-06-27T21:17:29.585427+0000","last_undegraded":"2024-06-27T21:17:29.585427+0000","last_fullsized":"2024-06-27T21:17:29.585427+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5729,"log_dups_size":0,"ondisk_log_size":5729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375455744,"num_objects":5729,"num_object_clones":0,"num_object_copies":17187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5729,"num_write_kb":366656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"218'5898","reported_seq":5972,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.343326+0000","last_change":"2024-06-27T21:16:02.437149+0000","last_active":"2024-06-27T21:17:29.343326+0000","last_peered":"2024-06-27T21:17:29.343326+0000","last_clean":"2024-06-27T21:17:29.343326+0000","last_became_active":"2024-06-27T21:16:02.436824+0000","last_became_peered":"2024-06-27T21:16:02.436824+0000","last_unstale":"2024-06-27T21:17:29.343326+0000","last_undegraded":"2024-06-27T21:17:29.343326+0000","last_fullsized":"2024-06-27T21:17:29.343326+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.783559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386531328,"num_objects":5898,"num_object_clones":0,"num_object_copies":17694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":377472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"219'5869","reported_seq":5945,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.591587+0000","last_change":"2024-06-27T21:16:02.856966+0000","last_active":"2024-06-27T21:17:29.591587+0000","last_peered":"2024-06-27T21:17:29.591587+0000","last_clean":"2024-06-27T21:17:29.591587+0000","last_became_active":"2024-06-27T21:16:02.856526+0000","last_became_peered":"2024-06-27T21:16:02.856526+0000","last_unstale":"2024-06-27T21:17:29.591587+0000","last_undegraded":"2024-06-27T21:17:29.591587+0000","last_fullsized":"2024-06-27T21:17:29.591587+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:30.364259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384630784,"num_objects":5869,"num_object_clones":0,"num_object_copies":17607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":375616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"218'5626","reported_seq":5700,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.406384+0000","last_change":"2024-06-27T21:16:02.439608+0000","last_active":"2024-06-27T21:17:27.406384+0000","last_peered":"2024-06-27T21:17:27.406384+0000","last_clean":"2024-06-27T21:17:27.406384+0000","last_became_active":"2024-06-27T21:16:02.439289+0000","last_became_peered":"2024-06-27T21:16:02.439289+0000","last_unstale":"2024-06-27T21:17:27.406384+0000","last_undegraded":"2024-06-27T21:17:27.406384+0000","last_fullsized":"2024-06-27T21:17:27.406384+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.267320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368705536,"num_objects":5626,"num_object_clones":0,"num_object_copies":16878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5626,"num_write_kb":360064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"218'5589","reported_seq":5662,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.260699+0000","last_change":"2024-06-27T21:16:02.436089+0000","last_active":"2024-06-27T21:17:27.260699+0000","last_peered":"2024-06-27T21:17:27.260699+0000","last_clean":"2024-06-27T21:17:27.260699+0000","last_became_active":"2024-06-27T21:16:02.435740+0000","last_became_peered":"2024-06-27T21:16:02.435740+0000","last_unstale":"2024-06-27T21:17:27.260699+0000","last_undegraded":"2024-06-27T21:17:27.260699+0000","last_fullsized":"2024-06-27T21:17:27.260699+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:28.360593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366280704,"num_objects":5589,"num_object_clones":0,"num_object_copies":16767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5589,"num_write_kb":357696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"218'5632","reported_seq":5706,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.408681+0000","last_change":"2024-06-27T21:16:02.856709+0000","last_active":"2024-06-27T21:17:27.408681+0000","last_peered":"2024-06-27T21:17:27.408681+0000","last_clean":"2024-06-27T21:17:27.408681+0000","last_became_active":"2024-06-27T21:16:02.856475+0000","last_became_peered":"2024-06-27T21:16:02.856475+0000","last_unstale":"2024-06-27T21:17:27.408681+0000","last_undegraded":"2024-06-27T21:17:27.408681+0000","last_fullsized":"2024-06-27T21:17:27.408681+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":5632,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5632,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":424,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.408604+0000","last_change":"2024-06-27T21:16:50.057759+0000","last_active":"2024-06-27T21:17:29.408604+0000","last_peered":"2024-06-27T21:17:29.408604+0000","last_clean":"2024-06-27T21:17:29.408604+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:17:29.408604+0000","last_undegraded":"2024-06-27T21:17:29.408604+0000","last_fullsized":"2024-06-27T21:17:29.408604+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:50.057656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:50.057656+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:57.842257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"219'5931","reported_seq":6052,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:30.615280+0000","last_change":"2024-06-27T21:17:03.391790+0000","last_active":"2024-06-27T21:17:30.615280+0000","last_peered":"2024-06-27T21:17:30.615280+0000","last_clean":"2024-06-27T21:17:30.615280+0000","last_became_active":"2024-06-27T21:16:03.380506+0000","last_became_peered":"2024-06-27T21:16:03.380506+0000","last_unstale":"2024-06-27T21:17:30.615280+0000","last_undegraded":"2024-06-27T21:17:30.615280+0000","last_fullsized":"2024-06-27T21:17:30.615280+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'3949","last_scrub_stamp":"2024-06-27T21:17:03.391664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:03.391664+0000","objects_scrubbed":3922,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:11.607426+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388694016,"num_objects":5931,"num_object_clones":0,"num_object_copies":17793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5931,"num_write_kb":379584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":135,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:26.416548+0000","last_change":"2024-06-27T21:16:15.304573+0000","last_active":"2024-06-27T21:17:26.416548+0000","last_peered":"2024-06-27T21:17:26.416548+0000","last_clean":"2024-06-27T21:17:26.416548+0000","last_became_active":"2024-06-27T21:14:55.861161+0000","last_became_peered":"2024-06-27T21:14:55.861161+0000","last_unstale":"2024-06-27T21:17:26.416548+0000","last_undegraded":"2024-06-27T21:17:26.416548+0000","last_fullsized":"2024-06-27T21:17:26.416548+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:16:15.304450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:36.576722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"218'5809","reported_seq":5882,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.353312+0000","last_change":"2024-06-27T21:16:02.856992+0000","last_active":"2024-06-27T21:17:29.353312+0000","last_peered":"2024-06-27T21:17:29.353312+0000","last_clean":"2024-06-27T21:17:29.353312+0000","last_became_active":"2024-06-27T21:16:02.856410+0000","last_became_peered":"2024-06-27T21:16:02.856410+0000","last_unstale":"2024-06-27T21:17:29.353312+0000","last_undegraded":"2024-06-27T21:17:29.353312+0000","last_fullsized":"2024-06-27T21:17:29.353312+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5809,"log_dups_size":0,"ondisk_log_size":5809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.783535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380698624,"num_objects":5809,"num_object_clones":0,"num_object_copies":17427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5809,"num_write_kb":371776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":429,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.408676+0000","last_change":"2024-06-27T21:16:10.176659+0000","last_active":"2024-06-27T21:17:29.408676+0000","last_peered":"2024-06-27T21:17:29.408676+0000","last_clean":"2024-06-27T21:17:29.408676+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:17:29.408676+0000","last_undegraded":"2024-06-27T21:17:29.408676+0000","last_fullsized":"2024-06-27T21:17:29.408676+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:10.176566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:10.176566+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"218'5557","reported_seq":5631,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.694613+0000","last_change":"2024-06-27T21:16:02.437762+0000","last_active":"2024-06-27T21:17:27.694613+0000","last_peered":"2024-06-27T21:17:27.694613+0000","last_clean":"2024-06-27T21:17:27.694613+0000","last_became_active":"2024-06-27T21:16:02.437264+0000","last_became_peered":"2024-06-27T21:16:02.437264+0000","last_unstale":"2024-06-27T21:17:27.694613+0000","last_undegraded":"2024-06-27T21:17:27.694613+0000","last_fullsized":"2024-06-27T21:17:27.694613+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:27.987659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364183552,"num_objects":5557,"num_object_clones":0,"num_object_copies":16671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":355648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":136,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.434448+0000","last_change":"2024-06-27T21:16:00.337996+0000","last_active":"2024-06-27T21:17:29.434448+0000","last_peered":"2024-06-27T21:17:29.434448+0000","last_clean":"2024-06-27T21:17:29.434448+0000","last_became_active":"2024-06-27T21:14:55.718653+0000","last_became_peered":"2024-06-27T21:14:55.718653+0000","last_unstale":"2024-06-27T21:17:29.434448+0000","last_undegraded":"2024-06-27T21:17:29.434448+0000","last_fullsized":"2024-06-27T21:17:29.434448+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:00.337874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:00.337874+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:29.535089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"218'5765","reported_seq":5839,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.350361+0000","last_change":"2024-06-27T21:16:02.438849+0000","last_active":"2024-06-27T21:17:27.350361+0000","last_peered":"2024-06-27T21:17:27.350361+0000","last_clean":"2024-06-27T21:17:27.350361+0000","last_became_active":"2024-06-27T21:16:02.438343+0000","last_became_peered":"2024-06-27T21:16:02.438343+0000","last_unstale":"2024-06-27T21:17:27.350361+0000","last_undegraded":"2024-06-27T21:17:27.350361+0000","last_fullsized":"2024-06-27T21:17:27.350361+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5765,"log_dups_size":0,"ondisk_log_size":5765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377815040,"num_objects":5765,"num_object_clones":0,"num_object_copies":17295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5765,"num_write_kb":368960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":162,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.412347+0000","last_change":"2024-06-27T21:16:46.599756+0000","last_active":"2024-06-27T21:17:29.412347+0000","last_peered":"2024-06-27T21:17:29.412347+0000","last_clean":"2024-06-27T21:17:29.412347+0000","last_became_active":"2024-06-27T21:14:55.861874+0000","last_became_peered":"2024-06-27T21:14:55.861874+0000","last_unstale":"2024-06-27T21:17:29.412347+0000","last_undegraded":"2024-06-27T21:17:29.412347+0000","last_fullsized":"2024-06-27T21:17:29.412347+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:46.599672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:46.599672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:07.778960+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"218'5822","reported_seq":5909,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:28.838523+0000","last_change":"2024-06-27T21:17:10.434112+0000","last_active":"2024-06-27T21:17:28.838523+0000","last_peered":"2024-06-27T21:17:28.838523+0000","last_clean":"2024-06-27T21:17:28.838523+0000","last_became_active":"2024-06-27T21:16:02.437389+0000","last_became_peered":"2024-06-27T21:16:02.437389+0000","last_unstale":"2024-06-27T21:17:28.838523+0000","last_undegraded":"2024-06-27T21:17:28.838523+0000","last_fullsized":"2024-06-27T21:17:28.838523+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":1199,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.315792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381550592,"num_objects":5822,"num_object_clones":0,"num_object_copies":17466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":372608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":136,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.432642+0000","last_change":"2024-06-27T21:16:19.344653+0000","last_active":"2024-06-27T21:17:29.432642+0000","last_peered":"2024-06-27T21:17:29.432642+0000","last_clean":"2024-06-27T21:17:29.432642+0000","last_became_active":"2024-06-27T21:14:55.711705+0000","last_became_peered":"2024-06-27T21:14:55.711705+0000","last_unstale":"2024-06-27T21:17:29.432642+0000","last_undegraded":"2024-06-27T21:17:29.432642+0000","last_fullsized":"2024-06-27T21:17:29.432642+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:16:19.344541+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:48.473574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"218'5648","reported_seq":5722,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.393501+0000","last_change":"2024-06-27T21:16:02.436377+0000","last_active":"2024-06-27T21:17:27.393501+0000","last_peered":"2024-06-27T21:17:27.393501+0000","last_clean":"2024-06-27T21:17:27.393501+0000","last_became_active":"2024-06-27T21:16:02.436035+0000","last_became_peered":"2024-06-27T21:16:02.436035+0000","last_unstale":"2024-06-27T21:17:27.393501+0000","last_undegraded":"2024-06-27T21:17:27.393501+0000","last_fullsized":"2024-06-27T21:17:27.393501+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370147328,"num_objects":5648,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":136,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.432579+0000","last_change":"2024-06-27T21:16:51.300220+0000","last_active":"2024-06-27T21:17:29.432579+0000","last_peered":"2024-06-27T21:17:29.432579+0000","last_clean":"2024-06-27T21:17:29.432579+0000","last_became_active":"2024-06-27T21:14:55.718043+0000","last_became_peered":"2024-06-27T21:14:55.718043+0000","last_unstale":"2024-06-27T21:17:29.432579+0000","last_undegraded":"2024-06-27T21:17:29.432579+0000","last_fullsized":"2024-06-27T21:17:29.432579+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:51.300103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:51.300103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:05.137291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.6","version":"219'5791","reported_seq":5867,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:30.586430+0000","last_change":"2024-06-27T21:16:02.437398+0000","last_active":"2024-06-27T21:17:30.586430+0000","last_peered":"2024-06-27T21:17:30.586430+0000","last_clean":"2024-06-27T21:17:30.586430+0000","last_became_active":"2024-06-27T21:16:02.436785+0000","last_became_peered":"2024-06-27T21:16:02.436785+0000","last_unstale":"2024-06-27T21:17:30.586430+0000","last_undegraded":"2024-06-27T21:17:30.586430+0000","last_fullsized":"2024-06-27T21:17:30.586430+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:35.466412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379518976,"num_objects":5791,"num_object_clones":0,"num_object_copies":17373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5791,"num_write_kb":370624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":231,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:26.420054+0000","last_change":"2024-06-27T21:16:50.793468+0000","last_active":"2024-06-27T21:17:26.420054+0000","last_peered":"2024-06-27T21:17:26.420054+0000","last_clean":"2024-06-27T21:17:26.420054+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:17:26.420054+0000","last_undegraded":"2024-06-27T21:17:26.420054+0000","last_fullsized":"2024-06-27T21:17:26.420054+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:50.793390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:50.793390+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:16.614251+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"218'5708","reported_seq":5832,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.384763+0000","last_change":"2024-06-27T21:17:08.576309+0000","last_active":"2024-06-27T21:17:29.384763+0000","last_peered":"2024-06-27T21:17:29.384763+0000","last_clean":"2024-06-27T21:17:29.384763+0000","last_became_active":"2024-06-27T21:16:02.436507+0000","last_became_peered":"2024-06-27T21:16:02.436507+0000","last_unstale":"2024-06-27T21:17:29.384763+0000","last_undegraded":"2024-06-27T21:17:29.384763+0000","last_fullsized":"2024-06-27T21:17:29.384763+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'4223","last_scrub_stamp":"2024-06-27T21:17:08.576188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:08.576188+0000","objects_scrubbed":4199,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:25.088833+0000","scrub_duration":725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":365312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":145,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.410479+0000","last_change":"2024-06-27T21:17:08.765002+0000","last_active":"2024-06-27T21:17:29.410479+0000","last_peered":"2024-06-27T21:17:29.410479+0000","last_clean":"2024-06-27T21:17:29.410479+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:17:29.410479+0000","last_undegraded":"2024-06-27T21:17:29.410479+0000","last_fullsized":"2024-06-27T21:17:29.410479+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:17:08.764871+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:27.034904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.b","version":"218'5645","reported_seq":5766,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:27.424056+0000","last_change":"2024-06-27T21:17:04.020522+0000","last_active":"2024-06-27T21:17:27.424056+0000","last_peered":"2024-06-27T21:17:27.424056+0000","last_clean":"2024-06-27T21:17:27.424056+0000","last_became_active":"2024-06-27T21:16:02.435710+0000","last_became_peered":"2024-06-27T21:16:02.435710+0000","last_unstale":"2024-06-27T21:17:27.424056+0000","last_undegraded":"2024-06-27T21:17:27.424056+0000","last_fullsized":"2024-06-27T21:17:27.424056+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'3907","last_scrub_stamp":"2024-06-27T21:17:04.020273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:04.020273+0000","objects_scrubbed":3842,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:09.284507+0000","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369950720,"num_objects":5645,"num_object_clones":0,"num_object_copies":16935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":361280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":429,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.408676+0000","last_change":"2024-06-27T21:16:13.173642+0000","last_active":"2024-06-27T21:17:29.408676+0000","last_peered":"2024-06-27T21:17:29.408676+0000","last_clean":"2024-06-27T21:17:29.408676+0000","last_became_active":"2024-06-27T21:14:55.722247+0000","last_became_peered":"2024-06-27T21:14:55.722247+0000","last_unstale":"2024-06-27T21:17:29.408676+0000","last_undegraded":"2024-06-27T21:17:29.408676+0000","last_fullsized":"2024-06-27T21:17:29.408676+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:13.173550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:13.173550+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"219'5834","reported_seq":5909,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:30.181619+0000","last_change":"2024-06-27T21:16:02.436989+0000","last_active":"2024-06-27T21:17:30.181619+0000","last_peered":"2024-06-27T21:17:30.181619+0000","last_clean":"2024-06-27T21:17:30.181619+0000","last_became_active":"2024-06-27T21:16:02.436617+0000","last_became_peered":"2024-06-27T21:16:02.436617+0000","last_unstale":"2024-06-27T21:17:30.181619+0000","last_undegraded":"2024-06-27T21:17:30.181619+0000","last_fullsized":"2024-06-27T21:17:30.181619+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382337024,"num_objects":5834,"num_object_clones":0,"num_object_copies":17502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":373376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":141,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-06-27T21:17:26.418169+0000","last_change":"2024-06-27T21:16:09.844344+0000","last_active":"2024-06-27T21:17:26.418169+0000","last_peered":"2024-06-27T21:17:26.418169+0000","last_clean":"2024-06-27T21:17:26.418169+0000","last_became_active":"2024-06-27T21:14:55.715529+0000","last_became_peered":"2024-06-27T21:14:55.715529+0000","last_unstale":"2024-06-27T21:17:26.418169+0000","last_undegraded":"2024-06-27T21:17:26.418169+0000","last_fullsized":"2024-06-27T21:17:26.418169+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:09.844224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:09.844224+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6019678208,"num_objects":91853,"num_object_clones":0,"num_object_copies":275559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91853,"num_write_kb":5878592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18091933696,"data_stored":18091933696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91853,"ondisk_log_size":91853,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252962,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2736036,"kb_used_data":2574360,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91009116,"statfs":{"total":95995035648,"available":93193334784,"internally_reserved":0,"allocated":2636144640,"data_stored":2635621349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514599,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438880,"kb_used_data":375768,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93306272,"statfs":{"total":95995035648,"available":95545622528,"internally_reserved":0,"allocated":384786432,"data_stored":384263149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1272880,"kb_used_data":1155620,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92472272,"statfs":{"total":95995035648,"available":94691606528,"internally_reserved":0,"allocated":1183354880,"data_stored":1182827001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972463,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852052,"kb_used_data":708428,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92893100,"statfs":{"total":95995035648,"available":95122534400,"internally_reserved":0,"allocated":725430272,"data_stored":724911577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1603360,"kb_used_data":1473624,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92141792,"statfs":{"total":95995035648,"available":94353195008,"internally_reserved":0,"allocated":1508990976,"data_stored":1508467693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528441,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1958112,"kb_used_data":1836696,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":91787040,"statfs":{"total":95995035648,"available":93989928960,"internally_reserved":0,"allocated":1880776704,"data_stored":1880253421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757374,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1530796,"kb_used_data":1463520,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92214356,"statfs":{"total":95995035648,"available":94427500544,"internally_reserved":0,"allocated":1498644480,"data_stored":1498117645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986307,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2666080,"kb_used_data":2540440,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91079072,"statfs":{"total":95995035648,"available":93264969728,"internally_reserved":0,"allocated":2601410560,"data_stored":2600887277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182535,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1935280,"kb_used_data":1791648,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91809872,"statfs":{"total":95995035648,"available":94013308928,"internally_reserved":0,"allocated":1834647552,"data_stored":1834120717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411468,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":442920,"kb_used_data":371488,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93302232,"statfs":{"total":95995035648,"available":95541485568,"internally_reserved":0,"allocated":380403712,"data_stored":379876877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574993,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1569572,"kb_used_data":1457880,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92175580,"statfs":{"total":95995035648,"available":94387793920,"internally_reserved":0,"allocated":1492869120,"data_stored":1492345837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738518,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2017580,"kb_used_data":1869796,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91727572,"statfs":{"total":95995035648,"available":93929033728,"internally_reserved":0,"allocated":1914671104,"data_stored":1914143225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1917255680,"data_stored":1917255680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1501691904,"data_stored":1501691904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380960768,"data_stored":380960768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1855979520,"data_stored":1855979520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2603483136,"data_stored":2603483136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1499725824,"data_stored":1499725824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1899036672,"data_stored":1899036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1513881600,"data_stored":1513881600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":732626944,"data_stored":732626944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1151139840,"data_stored":1151139840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":383909888,"data_stored":383909888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2652241920,"data_stored":2652241920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:17:33.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:17:33.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:33.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 96906 96890 76.6475 69.1875 0.0150912 0.0130391 2024-06-27T21:17:33.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:33.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:33 smithi151 ceph-mon[28965]: pgmap v622: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s 2024-06-27T21:17:33.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:33 smithi151 ceph-mon[28965]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:17:33.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:33 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:33.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:33 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:17:33.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:33 smithi151 ceph-mon[28965]: osdmap e220: 12 total, 12 up, 12 in 2024-06-27T21:17:33.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:33.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:33.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:33 smithi190 ceph-mon[32247]: pgmap v622: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s 2024-06-27T21:17:33.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:33 smithi190 ceph-mon[32247]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:17:33.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:33 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:33.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:33 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:17:33.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:33 smithi190 ceph-mon[32247]: osdmap e220: 12 total, 12 up, 12 in 2024-06-27T21:17:33.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:33 smithi192 ceph-mon[32956]: pgmap v622: 27 pgs: 27 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s 2024-06-27T21:17:33.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:33 smithi192 ceph-mon[32956]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:17:33.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:33 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:33.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:33 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:17:33.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:33 smithi192 ceph-mon[32956]: osdmap e220: 12 total, 12 up, 12 in 2024-06-27T21:17:33.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:33.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:33.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:33.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:33.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:34.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:34.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:34.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:34.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:34.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:34.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:34.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:34.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:34 smithi190 ceph-mon[32247]: 3.5 scrub starts 2024-06-27T21:17:34.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:34 smithi190 ceph-mon[32247]: osdmap e221: 12 total, 12 up, 12 in 2024-06-27T21:17:34.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:34.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:17:34.603549+0000 min lat: 0.00107574 max lat: 3.0199 avg lat: 0.0130922 2024-06-27T21:17:34.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:17:34.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 97752 97736 76.3503 52.875 0.0134712 0.0130922 2024-06-27T21:17:34.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:34.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:34 smithi151 ceph-mon[28965]: 3.5 scrub starts 2024-06-27T21:17:34.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:34 smithi151 ceph-mon[28965]: osdmap e221: 12 total, 12 up, 12 in 2024-06-27T21:17:34.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:34.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:34.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:34 smithi192 ceph-mon[32956]: 3.5 scrub starts 2024-06-27T21:17:34.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:34 smithi192 ceph-mon[32956]: osdmap e221: 12 total, 12 up, 12 in 2024-06-27T21:17:34.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:34.888 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:17:34.888 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:17:34.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:15444 2.7 2:e9beafb2:::benchmark_data_smithi192_59818_object15443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:55.566933+0000", 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 339.32235519099999, 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22234326200000001, 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.889 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 15444 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566933+0000", 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566933+0000", 2024-06-27T21:17:34.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566935+0000", 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4589999999999999e-06 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566983+0000", 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8597999999999998e-05 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566994+0000", 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0373e-05 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566999+0000", 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4389999999999999e-06 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567045+0000", 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5927000000000003e-05 2024-06-27T21:17:34.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567159+0000", 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113916 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567276+0000", 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011696699999999999 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567884+0000", 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000608005 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.568622+0000", 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073796400000000002 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.789206+0000", 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22058386599999999 2024-06-27T21:17:34.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.789239+0000", 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3337000000000003e-05 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.789277+0000", 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7410999999999998e-05 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.895 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:43157 2.7 2:e7d51da8:::benchmark_data_smithi192_59818_object43156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:30.778192+0000", 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 304.11109685700001, 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22017993299999999, 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.896 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 43157 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778192+0000", 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778192+0000", 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778193+0000", 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4139999999999999e-06 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778253+0000", 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9471e-05 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778264+0000", 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0968000000000001e-05 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778269+0000", 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4609999999999999e-06 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778321+0000", 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2071999999999999e-05 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778442+0000", 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012131200000000001 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778586+0000", 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014394200000000001 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779754+0000", 2024-06-27T21:17:34.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011675749999999999 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780861+0000", 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001107219 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.998270+0000", 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21740916099999999 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.998326+0000", 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5892e-05 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.998372+0000", 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5445999999999998e-05 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:43158 2.7 2:ee159f29:::benchmark_data_smithi192_59818_object43157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:17:34.902 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:17:34.903 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 304.10939665000001, 2024-06-27T21:17:34.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22168315799999999, 2024-06-27T21:17:34.903 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.903 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.903 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 43158 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.44e-07 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779952+0000", 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9070000000000002e-05 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779960+0000", 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4570000000000005e-06 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779964+0000", 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0770000000000001e-06 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779995+0000", 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1319000000000003e-05 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780079+0000", 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3338999999999996e-05 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780162+0000", 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3011999999999997e-05 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780961+0000", 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079940899999999995 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.781132+0000", 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017053899999999999 2024-06-27T21:17:34.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:31.001476+0000", 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22034437300000001 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:31.001530+0000", 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3934000000000002e-05 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:31.001575+0000", 2024-06-27T21:17:34.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5084999999999999e-05 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:48511 2.7 2:e603054b:::benchmark_data_smithi192_59818_object48510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.481199+0000", 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 300.40808950399997, 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.234965234, 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.910 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 48511 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481199+0000", 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481199+0000", 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481201+0000", 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2840000000000001e-06 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481250+0000", 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8384999999999998e-05 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481279+0000", 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9283000000000001e-05 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481295+0000", 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5472e-05 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481396+0000", 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101228 2024-06-27T21:17:34.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481513+0000", 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117568 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481600+0000", 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6884000000000004e-05 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.482792+0000", 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011914639999999999 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483325+0000", 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053293100000000005 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716091+0000", 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23276621 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716124+0000", 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2898999999999998e-05 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716164+0000", 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0626e-05 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:48511 2.7 e123/104 ondisk, result = 0)", 2024-06-27T21:17:34.916 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.483126+0000", 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 300.40616261999998, 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23304767200000001, 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483126+0000", 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483126+0000", 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483127+0000", 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483130+0000", 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0520000000000002e-06 2024-06-27T21:17:34.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.714854+0000", 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23172449000000001 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.714858+0000", 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0090000000000001e-06 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716081+0000", 2024-06-27T21:17:34.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001222929 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716086+0000", 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1329999999999998e-06 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716174+0000", 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7328000000000001e-05 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.920 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:79085 2.7 2:e5effe75:::benchmark_data_smithi192_59818_object79084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:01.904400+0000", 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 272.98488834699998, 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21667033599999999, 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.921 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 79085 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904400+0000", 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904400+0000", 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904402+0000", 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.545e-06 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904448+0000", 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5906999999999999e-05 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904458+0000", 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0796e-05 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904464+0000", 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2140000000000004e-06 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904520+0000", 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6617999999999999e-05 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904643+0000", 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012265799999999999 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904785+0000", 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014182600000000001 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.906755+0000", 2024-06-27T21:17:34.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019700809999999998 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.907117+0000", 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000361689 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.120970+0000", 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21385347399999999 2024-06-27T21:17:34.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.121026+0000", 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6329999999999998e-05 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.121071+0000", 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4197999999999999e-05 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.927 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:79085 2.7 e128/104 ondisk, result = 0)", 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:01.907087+0000", 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 272.98220199500003, 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21399069600000001, 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.907087+0000", 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.907087+0000", 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.907087+0000", 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.907094+0000", 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5780000000000004e-06 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.120914+0000", 2024-06-27T21:17:34.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21382010400000001 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.120919+0000", 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4129999999999997e-06 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.120958+0000", 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8688e-05 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.120965+0000", 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3830000000000003e-06 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.121077+0000", 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011193599999999999 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.932 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121181 2.7 2:edd475b0:::benchmark_data_smithi192_59818_object29106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.209941+0000", 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 229.67934775699999, 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.248862888, 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121181 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209941+0000", 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209941+0000", 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209943+0000", 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.747e-06 2024-06-27T21:17:34.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209949+0000", 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3570000000000003e-06 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209966+0000", 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6786000000000002e-05 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209976+0000", 2024-06-27T21:17:34.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7969999999999995e-06 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.210053+0000", 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7533999999999993e-05 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.210179+0000", 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125926 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.210304+0000", 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012508699999999999 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.213723+0000", 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003419395 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.227334+0000", 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013610224000000001 2024-06-27T21:17:34.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.458736+0000", 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.231402099 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.458776+0000", 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0691000000000001e-05 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.458804+0000", 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7245000000000001e-05 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121194 2.7 2:ede15273:::benchmark_data_smithi192_59818_object29119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:17:34.939 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.235274+0000", 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 229.65401491099999, 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.233618453, 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121194 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235274+0000", 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235274+0000", 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235275+0000", 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5400000000000001e-06 2024-06-27T21:17:34.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235298+0000", 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2521e-05 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235310+0000", 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2680000000000001e-05 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235316+0000", 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999998e-06 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235363+0000", 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6100999999999998e-05 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235443+0000", 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0623000000000004e-05 2024-06-27T21:17:34.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235508+0000", 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5108000000000001e-05 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.238187+0000", 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026788820000000001 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.238196+0000", 2024-06-27T21:17:34.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.251e-06 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.468818+0000", 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.230621254 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.468863+0000", 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5043999999999998e-05 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.468892+0000", 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9408999999999999e-05 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121370 2.7 2:e447b26c:::benchmark_data_smithi192_59818_object29295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.665171+0000", 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 229.22411754699999, 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31690380899999998, 2024-06-27T21:17:34.946 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121370 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665171+0000", 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665171+0000", 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665172+0000", 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.302e-06 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665177+0000", 2024-06-27T21:17:34.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9309999999999999e-06 2024-06-27T21:17:34.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:34.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:34.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665189+0000", 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1612e-05 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665195+0000", 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.755e-06 2024-06-27T21:17:34.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665252+0000", 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7067000000000001e-05 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665383+0000", 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131765 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:17:34.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665504+0000", 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120136 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.668112+0000", 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002607945 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.668584+0000", 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047197599999999998 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.982000+0000", 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31341653200000003 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.982040+0000", 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0159e-05 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.982075+0000", 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4629000000000003e-05 2024-06-27T21:17:34.955 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(2.0 e176,142 2:096fcc4d:::benchmark_data_smithi192_59818_object22725:head,139'7076)", 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:15:39.096407+0000", 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 115.79288176999999, 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62392472799999998, 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096407+0000", 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096407+0000", 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096408+0000", 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0500000000000003e-07 2024-06-27T21:17:34.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096411+0000", 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8990000000000003e-06 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720154+0000", 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62374265200000001 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720176+0000", 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2195999999999999e-05 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720266+0000", 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0172000000000003e-05 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720278+0000", 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1715e-05 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720332+0000", 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3389e-05 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:2780 3.c e192/186 3:3093fbba:::benchmark_data_smithi192_74314_object2779:head v 192'173, mlcod=192'169)", 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:16.246116+0000", 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 78.643172633000006, 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21023797799999999, 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.960 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.246116+0000", 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.246116+0000", 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.246116+0000", 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4400000000000001e-07 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.246134+0000", 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7231000000000002e-05 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.455802+0000", 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.209668193 2024-06-27T21:17:34.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.455808+0000", 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4779999999999996e-06 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.455869+0000", 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1173999999999994e-05 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.455890+0000", 2024-06-27T21:17:34.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1084e-05 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.456310+0000", 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041939199999999999 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.456354+0000", 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4082000000000002e-05 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.964 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:34085 3.c e199/186 3:31ab7127:::benchmark_data_smithi192_74314_object34084:head v 199'2184, mlcod=199'2181)", 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:40.797794+0000", 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 54.091494544, 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23472686700000001, 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797794+0000", 2024-06-27T21:17:34.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797794+0000", 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797795+0000", 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4e-07 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797819+0000", 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4051999999999999e-05 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030409+0000", 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23259012300000001 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030414+0000", 2024-06-27T21:17:34.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7160000000000002e-06 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030486+0000", 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2717000000000002e-05 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030507+0000", 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0649000000000002e-05 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032473+0000", 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019658929999999998 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032521+0000", 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7877000000000003e-05 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:63865 3.0 e209/186 3:0a199132:::benchmark_data_smithi192_74314_object63864:head v 209'4000, mlcod=209'3996)", 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:04.486048+0000", 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.403240282999999, 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21823163800000001, 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486048+0000", 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486048+0000", 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486049+0000", 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486103+0000", 2024-06-27T21:17:34.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4200999999999998e-05 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702091+0000", 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21598763300000001 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702097+0000", 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1580000000000002e-06 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702148+0000", 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.083e-05 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702168+0000", 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9839000000000001e-05 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.704149+0000", 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001981029 2024-06-27T21:17:34.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.704280+0000", 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013115800000000001 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:74026 3.a 3:5e5f5656:::benchmark_data_smithi192_74314_object74025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-27T21:17:34.974 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:12.910360+0000", 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.978928796000002, 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22282300999999999, 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 74026 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910360+0000", 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910360+0000", 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910361+0000", 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3149999999999999e-06 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910406+0000", 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4926999999999999e-05 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910418+0000", 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1685000000000001e-05 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910425+0000", 2024-06-27T21:17:34.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.306e-06 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910473+0000", 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8204999999999997e-05 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910574+0000", 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100437 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:17:34.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910663+0000", 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8849000000000003e-05 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.912821+0000", 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002158872 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.946702+0000", 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.033880217999999997 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.133064+0000", 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18636245500000001 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.133130+0000", 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5468000000000004e-05 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.133183+0000", 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3273000000000003e-05 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.980 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91001 3.a 3:5cd187b6:::benchmark_data_smithi192_74314_object91000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.755993+0000", 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 7.133295436, 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21809930599999999, 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:34.981 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91001 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.755993+0000", 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.755993+0000", 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.755994+0000", 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.299e-06 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756041+0000", 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6492999999999997e-05 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756054+0000", 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2577e-05 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756061+0000", 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3479999999999998e-06 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756120+0000", 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8684999999999998e-05 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756259+0000", 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000139376 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756404+0000", 2024-06-27T21:17:34.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014487199999999999 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.758788+0000", 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023840179999999999 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.758856+0000", 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8298000000000002e-05 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974025+0000", 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.215169042 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974065+0000", 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0036000000000001e-05 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974092+0000", 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7262000000000001e-05 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.989 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91005 3.a 3:5b53ffe7:::benchmark_data_smithi192_74314_object91004:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.762103+0000", 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 7.1271858469999998, 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23945181400000001, 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:34.990 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91005 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762103+0000", 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762103+0000", 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762104+0000", 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.172e-06 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762160+0000", 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5951e-05 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762173+0000", 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3087e-05 2024-06-27T21:17:34.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762179+0000", 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7629999999999997e-06 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762221+0000", 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2635000000000001e-05 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762319+0000", 2024-06-27T21:17:34.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7162000000000002e-05 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762406+0000", 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7294999999999997e-05 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.765175+0000", 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027689920000000001 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.765304+0000", 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128787 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001466+0000", 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23616261299999999 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001507+0000", 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.121e-05 2024-06-27T21:17:34.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001555+0000", 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7147000000000001e-05 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.15132.0:91001 3.a e218/186 ondisk, result = 0)", 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.763972+0000", 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 7.1253164819999997, 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21012465599999999, 2024-06-27T21:17:34.996 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763972+0000", 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763972+0000", 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763973+0000", 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8999999999999996e-07 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763977+0000", 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0890000000000002e-06 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.973962+0000", 2024-06-27T21:17:34.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20998470299999999 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.973968+0000", 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2609999999999999e-06 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974009+0000", 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1372999999999998e-05 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:34.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974018+0000", 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2220000000000008e-06 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974097+0000", 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8318000000000005e-05 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.000 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91008 3.a 3:579cf028:::benchmark_data_smithi192_74314_object91007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.768226+0000", 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 7.1210627190000002, 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.236103593, 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91008 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:35.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768226+0000", 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768226+0000", 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:35.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768227+0000", 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1850000000000001e-06 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768287+0000", 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0040000000000001e-05 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768299+0000", 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1762e-05 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768305+0000", 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6969999999999998e-06 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768342+0000", 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7653e-05 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:35.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768434+0000", 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1506000000000001e-05 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768532+0000", 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7875999999999996e-05 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.770803+0000", 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022713009999999999 2024-06-27T21:17:35.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.770820+0000", 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6983999999999999e-05 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.004218+0000", 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23339822399999999 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.004283+0000", 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4589000000000001e-05 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.004329+0000", 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6776000000000001e-05 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.007 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.15132.0:91005 3.a e218/186 ondisk, result = 0)", 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.768794+0000", 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 7.1204943890000001, 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.232767417, 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768794+0000", 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:35.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768794+0000", 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768795+0000", 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7400000000000002e-07 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768798+0000", 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9699999999999999e-06 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.998115+0000", 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22931665300000001 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.998119+0000", 2024-06-27T21:17:35.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7369999999999997e-06 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001457+0000", 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033374720000000002 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001463+0000", 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7339999999999996e-06 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:17:35.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001562+0000", 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9077e-05 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:17:35.012 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:17:35.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:35.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:35.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:35.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:35.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:35.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:35.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:35.463 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:35 smithi151 ceph-mon[28965]: pgmap v624: 27 pgs: 27 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:17:35.463 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:35 smithi151 ceph-mon[28965]: 3.a scrub starts 2024-06-27T21:17:35.463 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:35 smithi151 ceph-mon[28965]: 3.d scrub starts 2024-06-27T21:17:35.463 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:35 smithi151 ceph-mon[28965]: 3.4 scrub starts 2024-06-27T21:17:35.463 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:35 smithi151 ceph-mon[28965]: 3.2 scrub starts 2024-06-27T21:17:35.463 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:35 smithi151 ceph-mon[28965]: 3.0 scrub starts 2024-06-27T21:17:35.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:35.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:35.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:35.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 98538 98522 76.0141 49.125 0.0234824 0.0131499 2024-06-27T21:17:35.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:35.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:35.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:35 smithi190 ceph-mon[32247]: pgmap v624: 27 pgs: 27 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:17:35.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:35 smithi190 ceph-mon[32247]: 3.a scrub starts 2024-06-27T21:17:35.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:35 smithi190 ceph-mon[32247]: 3.d scrub starts 2024-06-27T21:17:35.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:35 smithi190 ceph-mon[32247]: 3.4 scrub starts 2024-06-27T21:17:35.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:35 smithi190 ceph-mon[32247]: 3.2 scrub starts 2024-06-27T21:17:35.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:35 smithi190 ceph-mon[32247]: 3.0 scrub starts 2024-06-27T21:17:35.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:35 smithi192 ceph-mon[32956]: pgmap v624: 27 pgs: 27 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:17:35.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:35 smithi192 ceph-mon[32956]: 3.a scrub starts 2024-06-27T21:17:35.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:35 smithi192 ceph-mon[32956]: 3.d scrub starts 2024-06-27T21:17:35.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:35 smithi192 ceph-mon[32956]: 3.4 scrub starts 2024-06-27T21:17:35.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:35 smithi192 ceph-mon[32956]: 3.2 scrub starts 2024-06-27T21:17:35.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:35 smithi192 ceph-mon[32956]: 3.0 scrub starts 2024-06-27T21:17:35.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:35.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:35.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:35.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:17:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:35.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:35.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:36.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:36.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:36.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:17:36.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:36.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:36.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:36.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:36.468 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:36.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:36.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:36.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 99221 99205 75.6077 42.6875 0.00731127 0.0132208 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.f scrub starts 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.7 scrub starts 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.3 scrub starts 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.4 scrub ok 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.e scrub starts 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.5 scrub ok 2024-06-27T21:17:36.630 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:36 smithi151 ceph-mon[28965]: 3.a scrub ok 2024-06-27T21:17:36.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.f scrub starts 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.7 scrub starts 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.3 scrub starts 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.4 scrub ok 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.e scrub starts 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.5 scrub ok 2024-06-27T21:17:36.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:36 smithi190 ceph-mon[32247]: 3.a scrub ok 2024-06-27T21:17:36.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:36.680 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:17:36.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:17:36.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:17:36.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:17:36.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:17:36.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:17:36.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:17:36.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:17:36.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:36.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.f scrub starts 2024-06-27T21:17:36.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.7 scrub starts 2024-06-27T21:17:36.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.3 scrub starts 2024-06-27T21:17:36.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.4 scrub ok 2024-06-27T21:17:36.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.e scrub starts 2024-06-27T21:17:36.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.5 scrub ok 2024-06-27T21:17:36.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:36 smithi192 ceph-mon[32956]: 3.a scrub ok 2024-06-27T21:17:36.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:17:36.794 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:17:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:36.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:36.822 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:17:36.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:17:36.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:17:36.854 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:17:36.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:36.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:36.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:37.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:37.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:37.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:37.188 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:37.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:37.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:37.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:37.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:37.443 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:17:37.443 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:17:37.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:37.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:37.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 100077 100061 75.3413 53.5 0.0160265 0.0132671 2024-06-27T21:17:37.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:37.690 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:17:37.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: pgmap v626: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3819261977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: 3.2 scrub ok 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:37.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:37 smithi151 ceph-mon[28965]: 3.3 scrub ok 2024-06-27T21:17:37.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:37.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:17:37.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:17:37.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: pgmap v626: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:17:37.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:17:37.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:17:37.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3819261977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:37.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: 3.2 scrub ok 2024-06-27T21:17:37.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:17:37.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:17:37.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:37 smithi190 ceph-mon[32247]: 3.3 scrub ok 2024-06-27T21:17:37.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:37.748 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:17:37.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:17:37.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:17:37.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:17:37.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: pgmap v626: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:17:37.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:17:37.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:17:37.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3819261977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:37.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: 3.2 scrub ok 2024-06-27T21:17:37.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:37.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:37.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:37 smithi192 ceph-mon[32956]: 3.3 scrub ok 2024-06-27T21:17:37.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:17:37.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:17:37.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:37.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:37.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:37.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:38.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:38.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:38.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:38.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:38.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:38.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:38.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:38.405 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:38.405 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:38.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:38.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:38.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 101058 101042 75.1742 61.3125 0.0222927 0.0132976 2024-06-27T21:17:38.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:38.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:38.723 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 3.d scrub ok 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 3.e scrub ok 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 3.0 scrub ok 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 3.c scrub starts 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: 3.f scrub ok 2024-06-27T21:17:38.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:38 smithi190 ceph-mon[32247]: osdmap e222: 12 total, 12 up, 12 in 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 3.d scrub ok 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 3.e scrub ok 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 3.0 scrub ok 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 3.c scrub starts 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:17:38.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: 3.f scrub ok 2024-06-27T21:17:38.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:38 smithi151 ceph-mon[28965]: osdmap e222: 12 total, 12 up, 12 in 2024-06-27T21:17:38.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:38.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:38.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 3.d scrub ok 2024-06-27T21:17:38.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 3.e scrub ok 2024-06-27T21:17:38.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 3.0 scrub ok 2024-06-27T21:17:38.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:17:38.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:17:38.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 3.c scrub starts 2024-06-27T21:17:38.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: from='client.25007 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:17:38.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: 3.f scrub ok 2024-06-27T21:17:38.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:38 smithi192 ceph-mon[32956]: osdmap e222: 12 total, 12 up, 12 in 2024-06-27T21:17:38.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:38.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:38.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:39.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:39.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:39.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:17:39.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:39.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:39.211 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2024-06-27T21:17:35.900207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6346834464,"num_objects":96838,"num_object_clones":0,"num_object_copies":290514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96836,"num_write_kb":6197504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97848,"ondisk_log_size":97848,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19985032,"kb_used_data":18591820,"kb_used_omap":18,"kb_used_meta":1393005,"kb_avail":1104956792,"statfs":{"total":1151940427776,"available":1131475755008,"internally_reserved":0,"allocated":19038023680,"data_stored":19031627912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":1426437434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,6,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":433651712,"num_objects":6617,"num_object_clones":0,"num_object_copies":19851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.495794"},"pg_stats":[{"pgid":"3.f","version":"221'6066","reported_seq":6146,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.603170+0000","last_change":"2024-06-27T21:16:02.437545+0000","last_active":"2024-06-27T21:17:34.603170+0000","last_peered":"2024-06-27T21:17:34.603170+0000","last_clean":"2024-06-27T21:17:34.603170+0000","last_became_active":"2024-06-27T21:16:02.437136+0000","last_became_peered":"2024-06-27T21:16:02.437136+0000","last_unstale":"2024-06-27T21:17:34.603170+0000","last_undegraded":"2024-06-27T21:17:34.603170+0000","last_fullsized":"2024-06-27T21:17:34.603170+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:35.149932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397541376,"num_objects":6066,"num_object_clones":0,"num_object_copies":18198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":388224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"220'6192","reported_seq":6270,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.384721+0000","last_change":"2024-06-27T21:16:02.437149+0000","last_active":"2024-06-27T21:17:34.384721+0000","last_peered":"2024-06-27T21:17:34.384721+0000","last_clean":"2024-06-27T21:17:34.384721+0000","last_became_active":"2024-06-27T21:16:02.436824+0000","last_became_peered":"2024-06-27T21:16:02.436824+0000","last_unstale":"2024-06-27T21:17:34.384721+0000","last_undegraded":"2024-06-27T21:17:34.384721+0000","last_fullsized":"2024-06-27T21:17:34.384721+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405798912,"num_objects":6192,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6192,"num_write_kb":396288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"221'6186","reported_seq":6281,"reported_epoch":221,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:17:34.621563+0000","last_change":"2024-06-27T21:17:34.245550+0000","last_active":"2024-06-27T21:17:34.621563+0000","last_peered":"2024-06-27T21:17:34.621563+0000","last_clean":"2024-06-27T21:17:34.621563+0000","last_became_active":"2024-06-27T21:16:02.856526+0000","last_became_peered":"2024-06-27T21:16:02.856526+0000","last_unstale":"2024-06-27T21:17:34.621563+0000","last_undegraded":"2024-06-27T21:17:34.621563+0000","last_fullsized":"2024-06-27T21:17:34.621563+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":1199,"log_size":6186,"log_dups_size":0,"ondisk_log_size":6186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405405696,"num_objects":6186,"num_object_clones":0,"num_object_copies":18558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":395904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"219'5928","reported_seq":6004,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.404616+0000","last_change":"2024-06-27T21:16:02.439608+0000","last_active":"2024-06-27T21:17:32.404616+0000","last_peered":"2024-06-27T21:17:32.404616+0000","last_clean":"2024-06-27T21:17:32.404616+0000","last_became_active":"2024-06-27T21:16:02.439289+0000","last_became_peered":"2024-06-27T21:16:02.439289+0000","last_unstale":"2024-06-27T21:17:32.404616+0000","last_undegraded":"2024-06-27T21:17:32.404616+0000","last_fullsized":"2024-06-27T21:17:32.404616+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388497408,"num_objects":5928,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5928,"num_write_kb":379392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"219'5912","reported_seq":5988,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.265905+0000","last_change":"2024-06-27T21:16:02.436089+0000","last_active":"2024-06-27T21:17:32.265905+0000","last_peered":"2024-06-27T21:17:32.265905+0000","last_clean":"2024-06-27T21:17:32.265905+0000","last_became_active":"2024-06-27T21:16:02.435740+0000","last_became_peered":"2024-06-27T21:16:02.435740+0000","last_unstale":"2024-06-27T21:17:32.265905+0000","last_undegraded":"2024-06-27T21:17:32.265905+0000","last_fullsized":"2024-06-27T21:17:32.265905+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:35.299567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387448832,"num_objects":5912,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5912,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"219'5924","reported_seq":6002,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.427118+0000","last_change":"2024-06-27T21:16:02.856709+0000","last_active":"2024-06-27T21:17:32.427118+0000","last_peered":"2024-06-27T21:17:32.427118+0000","last_clean":"2024-06-27T21:17:32.427118+0000","last_became_active":"2024-06-27T21:16:02.856475+0000","last_became_peered":"2024-06-27T21:16:02.856475+0000","last_unstale":"2024-06-27T21:17:32.427118+0000","last_undegraded":"2024-06-27T21:17:32.427118+0000","last_fullsized":"2024-06-27T21:17:32.427118+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5924,"log_dups_size":0,"ondisk_log_size":5924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.508935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388235264,"num_objects":5924,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":379136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":428,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.413373+0000","last_change":"2024-06-27T21:16:50.057759+0000","last_active":"2024-06-27T21:17:34.413373+0000","last_peered":"2024-06-27T21:17:34.413373+0000","last_clean":"2024-06-27T21:17:34.413373+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:17:34.413373+0000","last_undegraded":"2024-06-27T21:17:34.413373+0000","last_fullsized":"2024-06-27T21:17:34.413373+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:50.057656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:50.057656+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:57.842257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"221'6235","reported_seq":6362,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:35.580985+0000","last_change":"2024-06-27T21:17:03.391790+0000","last_active":"2024-06-27T21:17:35.580985+0000","last_peered":"2024-06-27T21:17:35.580985+0000","last_clean":"2024-06-27T21:17:35.580985+0000","last_became_active":"2024-06-27T21:16:03.380506+0000","last_became_peered":"2024-06-27T21:16:03.380506+0000","last_unstale":"2024-06-27T21:17:35.580985+0000","last_undegraded":"2024-06-27T21:17:35.580985+0000","last_fullsized":"2024-06-27T21:17:35.580985+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'3949","last_scrub_stamp":"2024-06-27T21:17:03.391664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:03.391664+0000","objects_scrubbed":3922,"log_size":6235,"log_dups_size":0,"ondisk_log_size":6235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:11.607426+0000","scrub_duration":819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408616960,"num_objects":6235,"num_object_clones":0,"num_object_copies":18705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":399040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":139,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.434413+0000","last_change":"2024-06-27T21:16:15.304573+0000","last_active":"2024-06-27T21:17:32.434413+0000","last_peered":"2024-06-27T21:17:32.434413+0000","last_clean":"2024-06-27T21:17:32.434413+0000","last_became_active":"2024-06-27T21:14:55.861161+0000","last_became_peered":"2024-06-27T21:14:55.861161+0000","last_unstale":"2024-06-27T21:17:32.434413+0000","last_undegraded":"2024-06-27T21:17:32.434413+0000","last_fullsized":"2024-06-27T21:17:32.434413+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:16:15.304450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:36.576722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"220'6137","reported_seq":6215,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.351214+0000","last_change":"2024-06-27T21:16:02.856992+0000","last_active":"2024-06-27T21:17:34.351214+0000","last_peered":"2024-06-27T21:17:34.351214+0000","last_clean":"2024-06-27T21:17:34.351214+0000","last_became_active":"2024-06-27T21:16:02.856410+0000","last_became_peered":"2024-06-27T21:16:02.856410+0000","last_unstale":"2024-06-27T21:17:34.351214+0000","last_undegraded":"2024-06-27T21:17:34.351214+0000","last_fullsized":"2024-06-27T21:17:34.351214+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402194432,"num_objects":6137,"num_object_clones":0,"num_object_copies":18411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":392768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":433,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.413175+0000","last_change":"2024-06-27T21:16:10.176659+0000","last_active":"2024-06-27T21:17:34.413175+0000","last_peered":"2024-06-27T21:17:34.413175+0000","last_clean":"2024-06-27T21:17:34.413175+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:17:34.413175+0000","last_undegraded":"2024-06-27T21:17:34.413175+0000","last_fullsized":"2024-06-27T21:17:34.413175+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:10.176566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:10.176566+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:35.149974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"220'5871","reported_seq":5949,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.696792+0000","last_change":"2024-06-27T21:16:02.437762+0000","last_active":"2024-06-27T21:17:32.696792+0000","last_peered":"2024-06-27T21:17:32.696792+0000","last_clean":"2024-06-27T21:17:32.696792+0000","last_became_active":"2024-06-27T21:16:02.437264+0000","last_became_peered":"2024-06-27T21:16:02.437264+0000","last_unstale":"2024-06-27T21:17:32.696792+0000","last_undegraded":"2024-06-27T21:17:32.696792+0000","last_fullsized":"2024-06-27T21:17:32.696792+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5871,"log_dups_size":0,"ondisk_log_size":5871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:34.576640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384761856,"num_objects":5871,"num_object_clones":0,"num_object_copies":17613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5871,"num_write_kb":375744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":140,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.477037+0000","last_change":"2024-06-27T21:16:00.337996+0000","last_active":"2024-06-27T21:17:34.477037+0000","last_peered":"2024-06-27T21:17:34.477037+0000","last_clean":"2024-06-27T21:17:34.477037+0000","last_became_active":"2024-06-27T21:14:55.718653+0000","last_became_peered":"2024-06-27T21:14:55.718653+0000","last_unstale":"2024-06-27T21:17:34.477037+0000","last_undegraded":"2024-06-27T21:17:34.477037+0000","last_fullsized":"2024-06-27T21:17:34.477037+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:00.337874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:00.337874+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:37.270398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"219'6076","reported_seq":6152,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.393460+0000","last_change":"2024-06-27T21:16:02.438849+0000","last_active":"2024-06-27T21:17:32.393460+0000","last_peered":"2024-06-27T21:17:32.393460+0000","last_clean":"2024-06-27T21:17:32.393460+0000","last_became_active":"2024-06-27T21:16:02.438343+0000","last_became_peered":"2024-06-27T21:16:02.438343+0000","last_unstale":"2024-06-27T21:17:32.393460+0000","last_undegraded":"2024-06-27T21:17:32.393460+0000","last_fullsized":"2024-06-27T21:17:32.393460+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":6076,"log_dups_size":0,"ondisk_log_size":6076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398196736,"num_objects":6076,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6076,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":166,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.420348+0000","last_change":"2024-06-27T21:16:46.599756+0000","last_active":"2024-06-27T21:17:34.420348+0000","last_peered":"2024-06-27T21:17:34.420348+0000","last_clean":"2024-06-27T21:17:34.420348+0000","last_became_active":"2024-06-27T21:14:55.861874+0000","last_became_peered":"2024-06-27T21:14:55.861874+0000","last_unstale":"2024-06-27T21:17:34.420348+0000","last_undegraded":"2024-06-27T21:17:34.420348+0000","last_fullsized":"2024-06-27T21:17:34.420348+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:46.599672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:46.599672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:07.778960+0000","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"220'6150","reported_seq":6243,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:33.820926+0000","last_change":"2024-06-27T21:17:10.434112+0000","last_active":"2024-06-27T21:17:33.820926+0000","last_peered":"2024-06-27T21:17:33.820926+0000","last_clean":"2024-06-27T21:17:33.820926+0000","last_became_active":"2024-06-27T21:16:02.437389+0000","last_became_peered":"2024-06-27T21:16:02.437389+0000","last_unstale":"2024-06-27T21:17:33.820926+0000","last_undegraded":"2024-06-27T21:17:33.820926+0000","last_fullsized":"2024-06-27T21:17:33.820926+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":1199,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403046400,"num_objects":6150,"num_object_clones":0,"num_object_copies":18450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":393600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":140,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.475211+0000","last_change":"2024-06-27T21:16:19.344653+0000","last_active":"2024-06-27T21:17:34.475211+0000","last_peered":"2024-06-27T21:17:34.475211+0000","last_clean":"2024-06-27T21:17:34.475211+0000","last_became_active":"2024-06-27T21:14:55.711705+0000","last_became_peered":"2024-06-27T21:14:55.711705+0000","last_unstale":"2024-06-27T21:17:34.475211+0000","last_undegraded":"2024-06-27T21:17:34.475211+0000","last_fullsized":"2024-06-27T21:17:34.475211+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:16:19.344541+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:48.473574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"219'5957","reported_seq":6034,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.427271+0000","last_change":"2024-06-27T21:16:02.436377+0000","last_active":"2024-06-27T21:17:32.427271+0000","last_peered":"2024-06-27T21:17:32.427271+0000","last_clean":"2024-06-27T21:17:32.427271+0000","last_became_active":"2024-06-27T21:16:02.436035+0000","last_became_peered":"2024-06-27T21:16:02.436035+0000","last_unstale":"2024-06-27T21:17:32.427271+0000","last_undegraded":"2024-06-27T21:17:32.427271+0000","last_fullsized":"2024-06-27T21:17:32.427271+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.508964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":140,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.475202+0000","last_change":"2024-06-27T21:16:51.300220+0000","last_active":"2024-06-27T21:17:34.475202+0000","last_peered":"2024-06-27T21:17:34.475202+0000","last_clean":"2024-06-27T21:17:34.475202+0000","last_became_active":"2024-06-27T21:14:55.718043+0000","last_became_peered":"2024-06-27T21:14:55.718043+0000","last_unstale":"2024-06-27T21:17:34.475202+0000","last_undegraded":"2024-06-27T21:17:34.475202+0000","last_fullsized":"2024-06-27T21:17:34.475202+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:51.300103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:51.300103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:05.137291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.6","version":"221'6066","reported_seq":6146,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:35.569151+0000","last_change":"2024-06-27T21:16:02.437398+0000","last_active":"2024-06-27T21:17:35.569151+0000","last_peered":"2024-06-27T21:17:35.569151+0000","last_clean":"2024-06-27T21:17:35.569151+0000","last_became_active":"2024-06-27T21:16:02.436785+0000","last_became_peered":"2024-06-27T21:16:02.436785+0000","last_unstale":"2024-06-27T21:17:35.569151+0000","last_undegraded":"2024-06-27T21:17:35.569151+0000","last_fullsized":"2024-06-27T21:17:35.569151+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":0,"log_size":6066,"log_dups_size":0,"ondisk_log_size":6066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397541376,"num_objects":6066,"num_object_clones":0,"num_object_copies":18198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":388224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":235,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.426989+0000","last_change":"2024-06-27T21:16:50.793468+0000","last_active":"2024-06-27T21:17:32.426989+0000","last_peered":"2024-06-27T21:17:32.426989+0000","last_clean":"2024-06-27T21:17:32.426989+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:17:32.426989+0000","last_undegraded":"2024-06-27T21:17:32.426989+0000","last_fullsized":"2024-06-27T21:17:32.426989+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:50.793390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:50.793390+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:16.614251+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"220'6040","reported_seq":6167,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.394143+0000","last_change":"2024-06-27T21:17:08.576309+0000","last_active":"2024-06-27T21:17:34.394143+0000","last_peered":"2024-06-27T21:17:34.394143+0000","last_clean":"2024-06-27T21:17:34.394143+0000","last_became_active":"2024-06-27T21:16:02.436507+0000","last_became_peered":"2024-06-27T21:16:02.436507+0000","last_unstale":"2024-06-27T21:17:34.394143+0000","last_undegraded":"2024-06-27T21:17:34.394143+0000","last_fullsized":"2024-06-27T21:17:34.394143+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'4223","last_scrub_stamp":"2024-06-27T21:17:08.576188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:08.576188+0000","objects_scrubbed":4199,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:25.088833+0000","scrub_duration":725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395837440,"num_objects":6040,"num_object_clones":0,"num_object_copies":18120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":386560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":149,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.415441+0000","last_change":"2024-06-27T21:17:08.765002+0000","last_active":"2024-06-27T21:17:34.415441+0000","last_peered":"2024-06-27T21:17:34.415441+0000","last_clean":"2024-06-27T21:17:34.415441+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:17:34.415441+0000","last_undegraded":"2024-06-27T21:17:34.415441+0000","last_fullsized":"2024-06-27T21:17:34.415441+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:17:08.764871+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:27.034904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.b","version":"219'5957","reported_seq":6082,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-06-27T21:17:32.427188+0000","last_change":"2024-06-27T21:17:04.020522+0000","last_active":"2024-06-27T21:17:32.427188+0000","last_peered":"2024-06-27T21:17:32.427188+0000","last_clean":"2024-06-27T21:17:32.427188+0000","last_became_active":"2024-06-27T21:16:02.435710+0000","last_became_peered":"2024-06-27T21:16:02.435710+0000","last_unstale":"2024-06-27T21:17:32.427188+0000","last_undegraded":"2024-06-27T21:17:32.427188+0000","last_fullsized":"2024-06-27T21:17:32.427188+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'3907","last_scrub_stamp":"2024-06-27T21:17:04.020273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:04.020273+0000","objects_scrubbed":3842,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:18:09.284507+0000","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390397952,"num_objects":5957,"num_object_clones":0,"num_object_copies":17871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":381248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":433,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-06-27T21:17:34.414568+0000","last_change":"2024-06-27T21:16:13.173642+0000","last_active":"2024-06-27T21:17:34.414568+0000","last_peered":"2024-06-27T21:17:34.414568+0000","last_clean":"2024-06-27T21:17:34.414568+0000","last_became_active":"2024-06-27T21:14:55.722247+0000","last_became_peered":"2024-06-27T21:14:55.722247+0000","last_unstale":"2024-06-27T21:17:34.414568+0000","last_undegraded":"2024-06-27T21:17:34.414568+0000","last_fullsized":"2024-06-27T21:17:34.414568+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:13.173550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:13.173550+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:35.149955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"221'6139","reported_seq":6248,"reported_epoch":221,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:17:35.188471+0000","last_change":"2024-06-27T21:17:34.234296+0000","last_active":"2024-06-27T21:17:35.188471+0000","last_peered":"2024-06-27T21:17:35.188471+0000","last_clean":"2024-06-27T21:17:35.188471+0000","last_became_active":"2024-06-27T21:16:02.436617+0000","last_became_peered":"2024-06-27T21:16:02.436617+0000","last_unstale":"2024-06-27T21:17:35.188471+0000","last_undegraded":"2024-06-27T21:17:35.188471+0000","last_fullsized":"2024-06-27T21:17:35.188471+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:16:01.415884+0000","objects_scrubbed":2599,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402325504,"num_objects":6139,"num_object_clones":0,"num_object_copies":18417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6139,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":143,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-06-27T21:17:29.410097+0000","last_change":"2024-06-27T21:16:09.844344+0000","last_active":"2024-06-27T21:17:29.410097+0000","last_peered":"2024-06-27T21:17:29.410097+0000","last_clean":"2024-06-27T21:17:29.410097+0000","last_became_active":"2024-06-27T21:14:55.715529+0000","last_became_peered":"2024-06-27T21:14:55.715529+0000","last_unstale":"2024-06-27T21:17:29.410097+0000","last_undegraded":"2024-06-27T21:17:29.410097+0000","last_fullsized":"2024-06-27T21:17:29.410097+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:09.844224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:09.844224+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:17:32.267262+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6346244096,"num_objects":96836,"num_object_clones":0,"num_object_copies":290508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96836,"num_write_kb":6197504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19080347648,"data_stored":19080347648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96836,"ondisk_log_size":96836,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252963,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2880764,"kb_used_data":2723248,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":90864388,"statfs":{"total":95995035648,"available":93045133312,"internally_reserved":0,"allocated":2788605952,"data_stored":2788073469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514600,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458684,"kb_used_data":395568,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93286468,"statfs":{"total":95995035648,"available":95525343232,"internally_reserved":0,"allocated":405061632,"data_stored":404529157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329292,"kb_used_data":1207868,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92415860,"statfs":{"total":95995035648,"available":94633840640,"internally_reserved":0,"allocated":1236856832,"data_stored":1236319761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972464,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905964,"kb_used_data":758180,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92839188,"statfs":{"total":95995035648,"available":95067328512,"internally_reserved":0,"allocated":776376320,"data_stored":775848441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680888,"kb_used_data":1559472,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92064264,"statfs":{"total":95995035648,"available":94273806336,"internally_reserved":0,"allocated":1596899328,"data_stored":1596366853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528442,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062904,"kb_used_data":1937328,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91682248,"statfs":{"total":95995035648,"available":93882621952,"internally_reserved":0,"allocated":1983823872,"data_stored":1983291397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757375,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1637364,"kb_used_data":1543660,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92107788,"statfs":{"total":95995035648,"available":94318374912,"internally_reserved":0,"allocated":1580707840,"data_stored":1580176409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986308,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2792508,"kb_used_data":2675248,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":90952644,"statfs":{"total":95995035648,"available":93135507456,"internally_reserved":0,"allocated":2739453952,"data_stored":2738921477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2025780,"kb_used_data":1890476,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91719372,"statfs":{"total":95995035648,"available":93920636928,"internally_reserved":0,"allocated":1935847424,"data_stored":1935315993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411469,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467776,"kb_used_data":392184,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93277376,"statfs":{"total":95995035648,"available":95516033024,"internally_reserved":0,"allocated":401596416,"data_stored":401060389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574994,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1643388,"kb_used_data":1544176,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92101764,"statfs":{"total":95995035648,"available":94312206336,"internally_reserved":0,"allocated":1581236224,"data_stored":1580703749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738519,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2099720,"kb_used_data":1964412,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91645432,"statfs":{"total":95995035648,"available":93844922368,"internally_reserved":0,"allocated":2011557888,"data_stored":2011020817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2013593600,"data_stored":2013593600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1587609600,"data_stored":1587609600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403439616,"data_stored":403439616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1955921920,"data_stored":1955921920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2744582144,"data_stored":2744582144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1580072960,"data_stored":1580072960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1999175680,"data_stored":1999175680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1600913408,"data_stored":1600913408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":775225344,"data_stored":775225344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1213792256,"data_stored":1213792256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":404291584,"data_stored":404291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2801729536,"data_stored":2801729536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:17:39.212 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-27T21:17:39.213 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:17:39.213 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:17:39.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 2 2024-06-27T21:17:39.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:39.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:39.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:39.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:39.456 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:39.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:39 smithi190 ceph-mon[32247]: 3.7 scrub ok 2024-06-27T21:17:39.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:39 smithi190 ceph-mon[32247]: pgmap v628: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:17:39.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:39 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:17:39.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:39 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:17:39.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:39 smithi190 ceph-mon[32247]: 3.6 scrub starts 2024-06-27T21:17:39.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:39 smithi190 ceph-mon[32247]: osdmap e223: 12 total, 12 up, 12 in 2024-06-27T21:17:39.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:39.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:39.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 102114 102098 75.0662 66 0.00469986 0.0133174 2024-06-27T21:17:39.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:39.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:39.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:39 smithi192 ceph-mon[32956]: 3.7 scrub ok 2024-06-27T21:17:39.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:39 smithi192 ceph-mon[32956]: pgmap v628: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:17:39.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:39 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:17:39.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:39 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:17:39.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:39 smithi192 ceph-mon[32956]: 3.6 scrub starts 2024-06-27T21:17:39.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:39 smithi192 ceph-mon[32956]: osdmap e223: 12 total, 12 up, 12 in 2024-06-27T21:17:39.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:39.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:39 smithi151 ceph-mon[28965]: 3.7 scrub ok 2024-06-27T21:17:39.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:39 smithi151 ceph-mon[28965]: pgmap v628: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:17:39.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:39 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:17:39.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:39 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:17:39.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:39 smithi151 ceph-mon[28965]: 3.6 scrub starts 2024-06-27T21:17:39.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:39 smithi151 ceph-mon[28965]: osdmap e223: 12 total, 12 up, 12 in 2024-06-27T21:17:39.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:39.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:39.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:39.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:40.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:40.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:40.173 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:40.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:40.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:40.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:40.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:40.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:40.475 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:40.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:40.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 103025 103009 74.8554 56.9375 0.00628362 0.0133525 2024-06-27T21:17:40.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:40.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:40 smithi151 ceph-mon[28965]: 3.c scrub ok 2024-06-27T21:17:40.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:40 smithi151 ceph-mon[28965]: 3.6 scrub ok 2024-06-27T21:17:40.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:40.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:40.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:40 smithi190 ceph-mon[32247]: 3.c scrub ok 2024-06-27T21:17:40.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:40 smithi190 ceph-mon[32247]: 3.6 scrub ok 2024-06-27T21:17:40.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:40 smithi192 ceph-mon[32956]: 3.c scrub ok 2024-06-27T21:17:40.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:40 smithi192 ceph-mon[32956]: 3.6 scrub ok 2024-06-27T21:17:40.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:40.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:40.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:40.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:40.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:40.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:41.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:41.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:41.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:41.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:41.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:41.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:41.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:41.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:41.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:41.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 103840 103824 74.5804 50.9375 0.010195 0.0134025 2024-06-27T21:17:41.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:41.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:17:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:41 smithi151 ceph-mon[28965]: pgmap v630: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 998 op/s 2024-06-27T21:17:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:41 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:17:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:41 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:17:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:41.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:41.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:41.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:41 smithi190 ceph-mon[32247]: pgmap v630: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 998 op/s 2024-06-27T21:17:41.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:41 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:17:41.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:41 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:17:41.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:17:41.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:17:41.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:41 smithi192 ceph-mon[32956]: pgmap v630: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 998 op/s 2024-06-27T21:17:41.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:41 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:17:41.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:41 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:17:41.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:41.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:41.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:41.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:41.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:41.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:42.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:42.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:42.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:42.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:42.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:42.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:42.481 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:17:42.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:42.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:42.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 104762 104746 74.3877 57.625 0.0345092 0.0134368 2024-06-27T21:17:42.606 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:42 smithi190 ceph-mon[32247]: osdmap e224: 12 total, 12 up, 12 in 2024-06-27T21:17:42.606 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3334658672' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:42.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:42.623 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:42.623 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:42.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:42.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:42.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:42.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:42 smithi192 ceph-mon[32956]: osdmap e224: 12 total, 12 up, 12 in 2024-06-27T21:17:42.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3334658672' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:42.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:42.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:42.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:42.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:42 smithi151 ceph-mon[28965]: osdmap e224: 12 total, 12 up, 12 in 2024-06-27T21:17:42.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3334658672' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:17:42.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:43.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:43.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:43.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:43.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:43.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:17:43.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:43.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:43.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:43.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:43.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 15 105835 105820 74.306 67.125 0.034256 0.013452 2024-06-27T21:17:43.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:43.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: pgmap v632: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 732 op/s 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: 3.9 scrub starts 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3334658672' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:43.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-mon[28965]: osdmap e225: 12 total, 12 up, 12 in 2024-06-27T21:17:43.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:43.732 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: pgmap v632: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 732 op/s 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: 3.9 scrub starts 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3334658672' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:43.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:43 smithi190 ceph-mon[32247]: osdmap e225: 12 total, 12 up, 12 in 2024-06-27T21:17:43.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: pgmap v632: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 732 op/s 2024-06-27T21:17:43.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:17:43.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:17:43.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: 3.9 scrub starts 2024-06-27T21:17:43.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:17:43.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:17:43.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3334658672' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:17:43.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:43 smithi192 ceph-mon[32956]: osdmap e225: 12 total, 12 up, 12 in 2024-06-27T21:17:43.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:43.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:43.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:43.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:43.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:44.042 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:44.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:44.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:44.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:44.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:44.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:44.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:44.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:43 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:17:43.950+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e225 definitely_dead 0 2024-06-27T21:17:44.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:44.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:44.489 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.2. 2024-06-27T21:17:44.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:44.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:44.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 16 106635 106619 74.0352 49.9375 0.10417 0.0134917 2024-06-27T21:17:44.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:44.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:44.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1568149859' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-27T21:17:44.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:44 smithi190 ceph-mon[32247]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-27T21:17:44.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:44 smithi190 ceph-mon[32247]: 3.9 scrub ok 2024-06-27T21:17:44.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1568149859' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-27T21:17:44.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:44 smithi151 ceph-mon[28965]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-27T21:17:44.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:44 smithi151 ceph-mon[28965]: 3.9 scrub ok 2024-06-27T21:17:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1568149859' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-06-27T21:17:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:44 smithi192 ceph-mon[32956]: Client client.admin marked osd.2 out, while it was still marked up 2024-06-27T21:17:44.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:44 smithi192 ceph-mon[32956]: 3.9 scrub ok 2024-06-27T21:17:44.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:44.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:44.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:44.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:44.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:45.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:45.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:45.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:45.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:45.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:45.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:45.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:45.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:17:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:45.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:45.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:45.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:45.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 91 7 106635 106628 73.2278 0.5625 0.892527 0.0135331 2024-06-27T21:17:45.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:45.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:45 smithi151 ceph-mon[28965]: pgmap v634: 27 pgs: 27 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 804 op/s 2024-06-27T21:17:45.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1568149859' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-27T21:17:45.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:45 smithi151 ceph-mon[28965]: osdmap e226: 12 total, 12 up, 11 in 2024-06-27T21:17:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:45.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:45.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:45 smithi190 ceph-mon[32247]: pgmap v634: 27 pgs: 27 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 804 op/s 2024-06-27T21:17:45.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1568149859' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-27T21:17:45.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:45 smithi190 ceph-mon[32247]: osdmap e226: 12 total, 12 up, 11 in 2024-06-27T21:17:45.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:45 smithi192 ceph-mon[32956]: pgmap v634: 27 pgs: 27 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 804 op/s 2024-06-27T21:17:45.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1568149859' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-06-27T21:17:45.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:45 smithi192 ceph-mon[32956]: osdmap e226: 12 total, 12 up, 11 in 2024-06-27T21:17:45.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:45.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:45.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:45.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:45.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:46.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:46.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:46.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:46.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:46.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:46.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:46.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:46.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:46.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:46.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:46.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:46.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 92 7 106635 106628 72.4317 0 - 0.0135331 2024-06-27T21:17:46.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:46.724 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:46.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:46 smithi190 ceph-mon[32247]: osdmap e227: 12 total, 12 up, 11 in 2024-06-27T21:17:46.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:46 smithi192 ceph-mon[32956]: osdmap e227: 12 total, 12 up, 11 in 2024-06-27T21:17:46.813 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:46 smithi151 ceph-mon[28965]: osdmap e227: 12 total, 12 up, 11 in 2024-06-27T21:17:46.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:46.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:46.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:46.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:46.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:47.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:47.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:47.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:47.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:47.313 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:47.313 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:47.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:47.354 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:47.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:17:47.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:17:47.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:17:47.419 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:17:47.437 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:47.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:17:47.442 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:17:47.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:47.465 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:17:47.466 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:17:47.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:47.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:17:47.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:17:47.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:17:47.511 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:17:47.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:17:47.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:17:47.554 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:17:47.554 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:17:47.577 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:17:47.577 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:17:47.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 92.4132 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 106635 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 72.1184 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 23.4478 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 122.938 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 0 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1153 2024-06-27T21:17:47.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 375.165 2024-06-27T21:17:47.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 1967 2024-06-27T21:17:47.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 0 2024-06-27T21:17:47.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0136951 2024-06-27T21:17:47.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0412699 2024-06-27T21:17:47.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 3.0199 2024-06-27T21:17:47.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00107574 2024-06-27T21:17:47.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:47.602 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:17:47.602 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:17:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:47.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:17:47.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:17:47.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:17:47.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:17:47.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:17:47.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:47 smithi151 ceph-mon[28965]: pgmap v637: 27 pgs: 2 peering, 25 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:17:47.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:47 smithi151 ceph-mon[28965]: osdmap e228: 12 total, 12 up, 11 in 2024-06-27T21:17:47.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4004995248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:47.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:47.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:47.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:47.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:47 smithi190 ceph-mon[32247]: pgmap v637: 27 pgs: 2 peering, 25 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:17:47.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:47 smithi190 ceph-mon[32247]: osdmap e228: 12 total, 12 up, 11 in 2024-06-27T21:17:47.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4004995248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:47.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:47 smithi192 ceph-mon[32956]: pgmap v637: 27 pgs: 2 peering, 25 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:17:47.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:47 smithi192 ceph-mon[32956]: osdmap e228: 12 total, 12 up, 11 in 2024-06-27T21:17:47.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4004995248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:47.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:47.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:47.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:47.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:47.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:48.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:48.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:48.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:48.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:48.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:48.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:48.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:48.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:17:48.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:48.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:48.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:48.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:48.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:48.803 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:17:48.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:48.886 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:48.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:48.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:49.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:49.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:49.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:49.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:49.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:49.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:49.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:49.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:49.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:49.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:49.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:49.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:49.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:49 smithi190 ceph-mon[32247]: pgmap v639: 27 pgs: 1 active+recovering+undersized+remapped, 2 peering, 24 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 41 MiB/s wr, 658 op/s; 6774/319683 objects misplaced (2.119%); 379 KiB/s, 5 objects/s recovering 2024-06-27T21:17:49.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:49.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:49.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:49 smithi192 ceph-mon[32956]: pgmap v639: 27 pgs: 1 active+recovering+undersized+remapped, 2 peering, 24 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 41 MiB/s wr, 658 op/s; 6774/319683 objects misplaced (2.119%); 379 KiB/s, 5 objects/s recovering 2024-06-27T21:17:49.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:49.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:49.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:49.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:49 smithi151 ceph-mon[28965]: pgmap v639: 27 pgs: 1 active+recovering+undersized+remapped, 2 peering, 24 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 41 MiB/s wr, 658 op/s; 6774/319683 objects misplaced (2.119%); 379 KiB/s, 5 objects/s recovering 2024-06-27T21:17:49.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:50.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:50.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:50.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:50.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:50.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:50.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:50.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:50.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:50.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:50.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:50.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:50.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:50.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:50.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:50.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:50.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:50.950 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:17:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:17:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:109647 3.9 3:97cb0157:::benchmark_data_smithi192_74314_object3009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:50.924411+0000", 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.026781276, 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026821893999999999, 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 109647 2024-06-27T21:17:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924411+0000", 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924411+0000", 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924414+0000", 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2359999999999999e-06 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924423+0000", 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0489999999999993e-06 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924443+0000", 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0859000000000001e-05 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924452+0000", 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4700000000000002e-06 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924505+0000", 2024-06-27T21:17:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3536999999999999e-05 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924603+0000", 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8024999999999997e-05 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.924678+0000", 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4794000000000003e-05 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.927449+0000", 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027711089999999999 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:50.927940+0000", 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049094899999999996 2024-06-27T21:17:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:17:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:17:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:17:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:17:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:17:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-06-27T21:17:50.957 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:17:51.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:51.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:51.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:17:51.148 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:17:51.148 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-27T21:17:51.148 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.199432 2024-06-27T21:17:51.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 1 0.1994320834545441 2024-06-27T21:17:51.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:51.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:51.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:51.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:51.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:51.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:51.457 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:51.510 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:17:51.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:51.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:51.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:17:51.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:51.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:51.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:51 smithi151 ceph-mon[28965]: pgmap v640: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 170 B/s rd, 32 MiB/s wr, 618 op/s; 6774/318021 objects misplaced (2.130%); 342 KiB/s, 4 objects/s recovering 2024-06-27T21:17:51.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2330991476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:51 smithi190 ceph-mon[32247]: pgmap v640: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 170 B/s rd, 32 MiB/s wr, 618 op/s; 6774/318021 objects misplaced (2.130%); 342 KiB/s, 4 objects/s recovering 2024-06-27T21:17:51.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2330991476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:51.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:51 smithi192 ceph-mon[32956]: pgmap v640: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 6.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 170 B/s rd, 32 MiB/s wr, 618 op/s; 6774/318021 objects misplaced (2.130%); 342 KiB/s, 4 objects/s recovering 2024-06-27T21:17:51.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2330991476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:17:51.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:51.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:51.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:51.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:17:51.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:51.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:52.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:52.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:52.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:52.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:52.291 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:17:52.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:52.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:52.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:52.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:52.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:52.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:52.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:52 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:17:52.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:52 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:17:52.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:52 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:52.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2330991476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:17:52.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:52 smithi192 ceph-mon[32956]: osdmap e229: 12 total, 12 up, 11 in 2024-06-27T21:17:52.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:52.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:52.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:52 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:17:52.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:52 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:17:52.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:52 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:52.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2330991476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:17:52.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:52 smithi190 ceph-mon[32247]: osdmap e229: 12 total, 12 up, 11 in 2024-06-27T21:17:52.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:52.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:52.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:52 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:17:52.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:52 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:17:52.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:52 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:17:52.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2330991476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:17:52.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:52 smithi151 ceph-mon[28965]: osdmap e229: 12 total, 12 up, 11 in 2024-06-27T21:17:52.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:52.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:52.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:53.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:53.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:53.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:53.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:53.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:53.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:53.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:53.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:53.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:53.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:53.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:53.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:53.779 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:53 smithi192 ceph-mon[32956]: pgmap v642: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 159 B/s rd, 14 MiB/s wr, 394 op/s; 6774/316503 objects misplaced (2.140%); 320 KiB/s, 4 objects/s recovering 2024-06-27T21:17:53.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:53.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:53.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:53 smithi151 ceph-mon[28965]: pgmap v642: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 159 B/s rd, 14 MiB/s wr, 394 op/s; 6774/316503 objects misplaced (2.140%); 320 KiB/s, 4 objects/s recovering 2024-06-27T21:17:53.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:53.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:53.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:54.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:53 smithi190 ceph-mon[32247]: pgmap v642: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 6.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 159 B/s rd, 14 MiB/s wr, 394 op/s; 6774/316503 objects misplaced (2.140%); 320 KiB/s, 4 objects/s recovering 2024-06-27T21:17:54.036 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:17:54.036 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:17:54.036 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:17:54.036 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:17:54.036 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:17:54.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:54.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:54.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:54.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:54.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:54.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:54.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:54.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:54.601 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:54.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:54.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:54.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:54.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:54.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:54 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 97/310413 objects degraded (0.031%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:17:54.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:54.886 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:54 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 97/310413 objects degraded (0.031%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:17:54.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:54.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:55.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:55.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:54 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 97/310413 objects degraded (0.031%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:17:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:55.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:55.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:55.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:55.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:55.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:55.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:55.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:55.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:55.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:55.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:55.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:55.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:17:55.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:55.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:55.880 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:55 smithi151 ceph-mon[28965]: pgmap v643: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 7.4 MiB/s wr, 514 op/s; 97/310413 objects degraded (0.031%); 6540/310413 objects misplaced (2.107%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:17:55.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:55.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:55.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:56.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:55 smithi190 ceph-mon[32247]: pgmap v643: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 7.4 MiB/s wr, 514 op/s; 97/310413 objects degraded (0.031%); 6540/310413 objects misplaced (2.107%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:17:56.017 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:17:56.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:56.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:55 smithi192 ceph-mon[32956]: pgmap v643: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 127 B/s rd, 7.4 MiB/s wr, 514 op/s; 97/310413 objects degraded (0.031%); 6540/310413 objects misplaced (2.107%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:17:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:56.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:56.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:17:56.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:56.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:56.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:56.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:56.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:56.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:56.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:56.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:56.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:56.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:56.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:56.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:17:56.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:17:56.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:56.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:56.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:57.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:17:57.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:17:57.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:57.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:57.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:57.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:57.424 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:17:57.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:57.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:17:57.459 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:17:57.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:57.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:17:57.489 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:17:57.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:17:57.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:17:57.543 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:17:57.543 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:17:57.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:17:57.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:17:57.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:57.582 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.1 primary-affinity to 0.199432 (8130692) 2024-06-27T21:17:57.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:17:57.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:17:57.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:17:57.638 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:17:57.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:17:57.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:17:57.660 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:17:57.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:57 smithi151 ceph-mon[28965]: pgmap v644: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 108 B/s rd, 6.3 MiB/s wr, 668 op/s; 97/303903 objects degraded (0.032%); 6540/303903 objects misplaced (2.152%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:17:57.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4162022166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:57.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:57 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:57.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:57 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:57.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/874347301' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]: dispatch 2024-06-27T21:17:57.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:57 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]: dispatch 2024-06-27T21:17:57.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:17:57.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:17:57.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:57.712 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:17:57.712 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:17:57.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:57.739 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:17:57.739 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:17:57.761 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:17:57.762 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:17:57.774 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:57 smithi192 ceph-mon[32956]: pgmap v644: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 108 B/s rd, 6.3 MiB/s wr, 668 op/s; 97/303903 objects degraded (0.032%); 6540/303903 objects misplaced (2.152%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:17:57.774 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4162022166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:57.775 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:57 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:57.775 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:57 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:57.775 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/874347301' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]: dispatch 2024-06-27T21:17:57.775 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:57 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]: dispatch 2024-06-27T21:17:57.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:57.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:57.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:57.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:57.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:58.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:57 smithi190 ceph-mon[32247]: pgmap v644: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 108 B/s rd, 6.3 MiB/s wr, 668 op/s; 97/303903 objects degraded (0.032%); 6540/303903 objects misplaced (2.152%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:17:58.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4162022166' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:17:58.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:57 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:17:58.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:57 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:17:58.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/874347301' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]: dispatch 2024-06-27T21:17:58.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:57 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]: dispatch 2024-06-27T21:17:58.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:58.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:58.137 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:17:58.138 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:17:58.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:58.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:58.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:17:58.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:17:58.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:58.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:58.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:17:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:58.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:17:58.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:58.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:58.722 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:58 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]': finished 2024-06-27T21:17:58.723 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:58 smithi192 ceph-mon[32956]: osdmap e230: 12 total, 12 up, 11 in 2024-06-27T21:17:58.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:17:58.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:17:58.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:17:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:58 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]': finished 2024-06-27T21:17:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:58 smithi151 ceph-mon[28965]: osdmap e230: 12 total, 12 up, 11 in 2024-06-27T21:17:58.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:58.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:58.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:59.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:58 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.1994320834545441}]': finished 2024-06-27T21:17:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:58 smithi190 ceph-mon[32247]: osdmap e230: 12 total, 12 up, 11 in 2024-06-27T21:17:59.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:59.094 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:17:59.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:59.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:17:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:17:59.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:17:59.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:17:59.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:17:59.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:17:59.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:17:59.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:17:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:17:59.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:17:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:17:59.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:17:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:59 smithi192 ceph-mon[32956]: pgmap v646: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 274/298140 objects degraded (0.092%); 6253/298140 objects misplaced (2.097%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:17:59.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:59 smithi192 ceph-mon[32956]: osdmap e231: 12 total, 12 up, 11 in 2024-06-27T21:17:59.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:17:59.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:17:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:17:59.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:17:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:59.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:17:59.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:59 smithi151 ceph-mon[28965]: pgmap v646: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 274/298140 objects degraded (0.092%); 6253/298140 objects misplaced (2.097%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:17:59.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:59 smithi151 ceph-mon[28965]: osdmap e231: 12 total, 12 up, 11 in 2024-06-27T21:17:59.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:17:59.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:17:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:17:59.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:17:59.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:17:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:17:59.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:00.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:59 smithi190 ceph-mon[32247]: pgmap v646: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 274/298140 objects degraded (0.092%); 6253/298140 objects misplaced (2.097%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:18:00.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:59 smithi190 ceph-mon[32247]: osdmap e231: 12 total, 12 up, 11 in 2024-06-27T21:18:00.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:00.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:17:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:00.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:00.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:00.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:00.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:00.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:00.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:00.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:00.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:00.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:00.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:00.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:00.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:00.851 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:00.852 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:00.852 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:00 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:18:00.852 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:00 smithi192 ceph-mon[32956]: osdmap e232: 12 total, 12 up, 11 in 2024-06-27T21:18:00.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:00.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:00.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:00.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:00 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:18:00.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:00 smithi151 ceph-mon[28965]: osdmap e232: 12 total, 12 up, 11 in 2024-06-27T21:18:00.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:01.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:01.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:00 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:18:01.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:00 smithi190 ceph-mon[32247]: osdmap e232: 12 total, 12 up, 11 in 2024-06-27T21:18:01.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:01.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:01.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:01.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:01.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:01.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:01.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:01.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:01.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:18:01.604 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:18:01.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:01.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:01.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:01.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:01.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:01 smithi192 ceph-mon[32956]: pgmap v649: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 924 op/s; 274/293763 objects degraded (0.093%); 6253/293763 objects misplaced (2.129%); 1.1 MiB/s, 18 objects/s recovering 2024-06-27T21:18:01.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:01 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 274/293763 objects degraded (0.093%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:01.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3679172570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:01.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:01.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:01.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:01.873 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:01.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:01.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:01.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:01 smithi151 ceph-mon[28965]: pgmap v649: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 924 op/s; 274/293763 objects degraded (0.093%); 6253/293763 objects misplaced (2.129%); 1.1 MiB/s, 18 objects/s recovering 2024-06-27T21:18:01.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:01 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 274/293763 objects degraded (0.093%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:01.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3679172570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:01.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:01.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:01.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:18:01.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:02.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:01 smithi190 ceph-mon[32247]: pgmap v649: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 6.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 924 op/s; 274/293763 objects degraded (0.093%); 6253/293763 objects misplaced (2.129%); 1.1 MiB/s, 18 objects/s recovering 2024-06-27T21:18:02.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:01 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 274/293763 objects degraded (0.093%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:02.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3679172570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:02.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:02.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:02.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:02.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:02.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:02.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:02.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:02.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:02.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:02.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:02.539 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:02.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:02.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:02.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:02.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:02 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:02.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3679172570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:02.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:02 smithi151 ceph-mon[28965]: osdmap e233: 12 total, 12 up, 11 in 2024-06-27T21:18:02.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:02.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:02 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:02.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3679172570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:02.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:02 smithi192 ceph-mon[32956]: osdmap e233: 12 total, 12 up, 11 in 2024-06-27T21:18:02.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:02.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:02.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:02.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:03.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:02 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:03.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3679172570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:03.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:02 smithi190 ceph-mon[32247]: osdmap e233: 12 total, 12 up, 11 in 2024-06-27T21:18:03.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:03.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:03.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:03.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:03.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:03.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:03.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:03.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:03.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:03.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:03.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:03.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:03.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:03 smithi151 ceph-mon[28965]: pgmap v651: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 274/291690 objects degraded (0.094%); 6253/291690 objects misplaced (2.144%) 2024-06-27T21:18:03.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:03.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:03.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:03 smithi151 ceph-mon[28965]: osdmap e234: 12 total, 12 up, 11 in 2024-06-27T21:18:03.712 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:03.712 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:03.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:03.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:03.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:03.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:03.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:04.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:03 smithi190 ceph-mon[32247]: pgmap v651: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 274/291690 objects degraded (0.094%); 6253/291690 objects misplaced (2.144%) 2024-06-27T21:18:04.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:04.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:04.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:03 smithi190 ceph-mon[32247]: osdmap e234: 12 total, 12 up, 11 in 2024-06-27T21:18:04.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:03 smithi192 ceph-mon[32956]: pgmap v651: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 274/291690 objects degraded (0.094%); 6253/291690 objects misplaced (2.144%) 2024-06-27T21:18:04.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:04.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:04.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:03 smithi192 ceph-mon[32956]: osdmap e234: 12 total, 12 up, 11 in 2024-06-27T21:18:04.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:04.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:04.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:04.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:04.211 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:18:04.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:18:04.212 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-06-27T21:18:04.212 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-27T21:18:04.212 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-27T21:18:04.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-06-27T21:18:04.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:04.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:04.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:18:04.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:04.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:18:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:04.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:04.545 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:04.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.593 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:04.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:04.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:04.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:04.822 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:18:04.822 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:18:04.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:04.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:18:04.853 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:18:04.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:04.874 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:04.883 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:18:04.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:18:04.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:18:04.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:18:04.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:18:04.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:18:04.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:04.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:18:04.962 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:18:04.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:18:04.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:18:04.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:04.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:05.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:18:05.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:18:05.037 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:18:05.038 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:18:05.062 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:18:05.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:18:05.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:18:05.088 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:18:05.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:05.118 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:18:05.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:18:05.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:05.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:05.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:05.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:05.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:05.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:05.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:05.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:05.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:05.703 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:05 smithi192 ceph-mon[32956]: pgmap v653: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 353/286980 objects degraded (0.123%); 6025/286980 objects misplaced (2.099%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:18:05.703 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2702559677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:05.703 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:05 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:18:05.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:05.753 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:05.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:05.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:05.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:05.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:05.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:05 smithi151 ceph-mon[28965]: pgmap v653: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 353/286980 objects degraded (0.123%); 6025/286980 objects misplaced (2.099%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:18:05.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2702559677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:05.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:05 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:18:05.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:06.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:05 smithi190 ceph-mon[32247]: pgmap v653: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 353/286980 objects degraded (0.123%); 6025/286980 objects misplaced (2.099%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:18:06.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2702559677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:06.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:05 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:18:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:06.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:06.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:06.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:06.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:06.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:06.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:06.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:06.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:06.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:06.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:06.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:06.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:06 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:18:06.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:06.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:06.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:06.927 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:06.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:06.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:06 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:18:07.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:07.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:06 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:18:07.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:07.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:07.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:07.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:07.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:07.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:07.429 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:18:07.429 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-27T21:18:07.430 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:18:07.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:07.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:07.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:07.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:07.709 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:07 smithi151 ceph-mon[28965]: pgmap v654: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 353/280155 objects degraded (0.126%); 6025/280155 objects misplaced (2.151%); 1.5 MiB/s, 23 objects/s recovering 2024-06-27T21:18:07.709 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:07 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 353/280155 objects degraded (0.126%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:07.709 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:07 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:07.709 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:07 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:07.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:07.763 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:07 smithi192 ceph-mon[32956]: pgmap v654: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 353/280155 objects degraded (0.126%); 6025/280155 objects misplaced (2.151%); 1.5 MiB/s, 23 objects/s recovering 2024-06-27T21:18:07.763 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:07 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 353/280155 objects degraded (0.126%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:07.763 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:07 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:07.763 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:07 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:07.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:07.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:07.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:07.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:07.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:08.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:07 smithi190 ceph-mon[32247]: pgmap v654: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 353/280155 objects degraded (0.126%); 6025/280155 objects misplaced (2.151%); 1.5 MiB/s, 23 objects/s recovering 2024-06-27T21:18:08.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:07 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 353/280155 objects degraded (0.126%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:08.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:07 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:08.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:07 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:08.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:08.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:08.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:08.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:08.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:08.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:08.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:08.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:08.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:08.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:08.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:18:08.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:18:08.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:08 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:08.651 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1101916907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:7672 2.d 2:bd9a34d0:::benchmark_data_smithi192_59818_object7671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.208270+0000", 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 378.44564172000003, 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.227281121, 2024-06-27T21:18:08.652 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7672 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208270+0000", 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208270+0000", 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208271+0000", 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2729999999999999e-06 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208291+0000", 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9899e-05 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208304+0000", 2024-06-27T21:18:08.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.256e-05 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208311+0000", 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.294e-06 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208351+0000", 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9774000000000002e-05 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208433+0000", 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2402000000000002e-05 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208517+0000", 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3277999999999998e-05 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209005+0000", 2024-06-27T21:18:08.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000487997 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209476+0000", 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047170399999999998 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435492+0000", 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22601585399999999 2024-06-27T21:18:08.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435524+0000", 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2202000000000001e-05 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435551+0000", 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6883999999999999e-05 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:7672 2.d e108/104 ondisk, result = 0)", 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.209290+0000", 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 378.44462153299997, 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.226265196, 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209290+0000", 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209290+0000", 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209291+0000", 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8599999999999998e-07 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209294+0000", 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9979999999999999e-06 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435435+0000", 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22614099600000001 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435439+0000", 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5120000000000002e-06 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435479+0000", 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9264999999999998e-05 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435488+0000", 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3279999999999995e-06 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435556+0000", 2024-06-27T21:18:08.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7510999999999998e-05 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7689 2.0 e108/104 2:05a01aa7:::benchmark_data_smithi192_59818_object7688:head v 108'502, mlcod=108'499)", 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.224260+0000", 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 378.42965193700002, 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22816329499999999, 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224260+0000", 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224260+0000", 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224261+0000", 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6000000000000003e-07 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224357+0000", 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6608999999999996e-05 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451882+0000", 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.227524636 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451887+0000", 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.724e-06 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451931+0000", 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4607000000000002e-05 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451945+0000", 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3582e-05 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.452386+0000", 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044162299999999998 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.452423+0000", 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6854000000000003e-05 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7699 2.f e108/104 2:f296be81:::benchmark_data_smithi192_59818_object7698:head v 108'484, mlcod=108'481)", 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.291358+0000", 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 378.36255433100001, 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.156959709, 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.291358+0000", 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.291358+0000", 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.291359+0000", 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4789999999999999e-06 2024-06-27T21:18:08.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.291390+0000", 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0617000000000002e-05 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.447806+0000", 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.15641630200000001 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.447813+0000", 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5289999999999997e-06 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.447862+0000", 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9678000000000001e-05 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.447874+0000", 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.217e-05 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.448288+0000", 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041407400000000001 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.448317+0000", 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8860000000000002e-05 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:15976 2.d 2:b9cc67ce:::benchmark_data_smithi192_59818_object15975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:56.096393+0000", 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 372.557519044, 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.15893384999999999, 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15976 2024-06-27T21:18:08.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096393+0000", 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096393+0000", 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096394+0000", 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2950000000000001e-06 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096411+0000", 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6444e-05 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096421+0000", 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0448000000000001e-05 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096426+0000", 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3199999999999999e-06 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096466+0000", 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9637e-05 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096551+0000", 2024-06-27T21:18:08.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4934999999999999e-05 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.096635+0000", 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3784999999999998e-05 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.097215+0000", 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00058033300000000004 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.119545+0000", 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022329675 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.255266+0000", 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.135721641 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.255298+0000", 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1888999999999999e-05 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.255327+0000", 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8447999999999999e-05 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:16214 2.d 2:bcd220af:::benchmark_data_smithi192_59818_object16213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:56.283291+0000", 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 372.37062095200002, 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.172051116, 2024-06-27T21:18:08.681 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 16214 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283291+0000", 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283291+0000", 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283292+0000", 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0529999999999999e-06 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283312+0000", 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9700000000000001e-05 2024-06-27T21:18:08.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283323+0000", 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0930999999999999e-05 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283328+0000", 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0490000000000003e-06 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283368+0000", 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0550999999999999e-05 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283450+0000", 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2126999999999995e-05 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.283542+0000", 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1741000000000001e-05 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.284081+0000", 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00053888500000000002 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.311481+0000", 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027400146 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.455221+0000", 2024-06-27T21:18:08.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14373988500000001 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.455291+0000", 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9572000000000004e-05 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:56.455342+0000", 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1476e-05 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:43154 2.d 2:b53185c5:::benchmark_data_smithi192_59818_object43153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.776149+0000", 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 337.87776330700001, 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24192956500000001, 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 43154 2024-06-27T21:18:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776149+0000", 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776149+0000", 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776150+0000", 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.052e-06 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776171+0000", 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1206e-05 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776184+0000", 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2835e-05 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776191+0000", 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1600000000000001e-06 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776226+0000", 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5216000000000003e-05 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776296+0000", 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0240000000000005e-05 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776376+0000", 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9312000000000002e-05 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777045+0000", 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066960100000000003 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777729+0000", 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00068351800000000004 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017986+0000", 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24025681700000001 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.018034+0000", 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8449999999999999e-05 2024-06-27T21:18:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.018078+0000", 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4158e-05 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:43154 2.d e123/104 ondisk, result = 0)", 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:18:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 337.87631685399998, 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24048932100000001, 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777598+0000", 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3640000000000002e-06 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017922+0000", 2024-06-27T21:18:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24032410600000001 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017928+0000", 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8950000000000003e-06 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017969+0000", 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1403999999999999e-05 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017980+0000", 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0722e-05 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.018084+0000", 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104425 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:50215 2.d 2:b84cda7e:::benchmark_data_smithi192_59818_object50214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:18:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:35.962578+0000", 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 332.69133394200003, 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21343746199999999, 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50215 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962578+0000", 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962578+0000", 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962580+0000", 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6220000000000001e-06 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962600+0000", 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9893000000000001e-05 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962611+0000", 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1416999999999999e-05 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962616+0000", 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2290000000000002e-06 2024-06-27T21:18:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962666+0000", 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0210000000000002e-05 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962761+0000", 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4616000000000003e-05 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962851+0000", 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9772999999999996e-05 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.964264+0000", 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001413414 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.964542+0000", 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027784499999999998 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.175922+0000", 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21138020299999999 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.175987+0000", 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4610000000000007e-05 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176015+0000", 2024-06-27T21:18:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8629999999999999e-05 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:50226 2.d 2:bc1dd2b3:::benchmark_data_smithi192_59818_object50225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:35.972508+0000", 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 332.68140438799998, 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20364497100000001, 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50226 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972508+0000", 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972508+0000", 2024-06-27T21:18:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972510+0000", 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.984e-06 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972527+0000", 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7677999999999999e-05 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972539+0000", 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1949e-05 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972545+0000", 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6810000000000003e-06 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972586+0000", 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1520999999999998e-05 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972681+0000", 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4173999999999997e-05 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.972773+0000", 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2491999999999995e-05 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.974309+0000", 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015360129999999999 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.974630+0000", 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032059900000000003 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176070+0000", 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.201440651 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176127+0000", 2024-06-27T21:18:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6289000000000003e-05 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176153+0000", 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5939999999999999e-05 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:50239 2.d 2:befbb5ef:::benchmark_data_smithi192_59818_object50238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:18:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:35.983981+0000", 2024-06-27T21:18:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 332.66993076099999, 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.192299357, 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50239 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.983981+0000", 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.983981+0000", 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.983983+0000", 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9879999999999999e-06 2024-06-27T21:18:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.984004+0000", 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0378e-05 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.984015+0000", 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1043000000000001e-05 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.984020+0000", 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9819999999999999e-06 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.984069+0000", 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9716000000000003e-05 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.984179+0000", 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109376 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.984291+0000", 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011213999999999999 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.985546+0000", 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012556760000000001 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.985860+0000", 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000313584 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176193+0000", 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19033277100000001 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176240+0000", 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7132000000000002e-05 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176281+0000", 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0571000000000002e-05 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:50246 2.a e123/104 2:5f8478cd:::benchmark_data_smithi192_59818_object50245:head v 123'3112, mlcod=123'3109)", 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:35.998994+0000", 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 332.65491811099997, 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18365081, 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.998994+0000", 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.998994+0000", 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.998994+0000", 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2699999999999999e-07 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.999017+0000", 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2050999999999999e-05 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.180874+0000", 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18185704499999999 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.180881+0000", 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2640000000000002e-06 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.180936+0000", 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5115999999999999e-05 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.180954+0000", 2024-06-27T21:18:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8448e-05 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.182579+0000", 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016241910000000001 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.182645+0000", 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6068000000000005e-05 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:78416 2.d 2:b1a8e389:::benchmark_data_smithi192_59818_object78415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:01.009269+0000", 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 307.64464282300003, 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.203406008, 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 78416 2024-06-27T21:18:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009269+0000", 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009269+0000", 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009270+0000", 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3740000000000001e-06 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009290+0000", 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9967e-05 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009300+0000", 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9839999999999996e-06 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009306+0000", 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.101e-06 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009356+0000", 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0463000000000001e-05 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009483+0000", 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000126699 2024-06-27T21:18:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.009634+0000", 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000151452 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.011317+0000", 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001683288 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.063011+0000", 2024-06-27T21:18:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.051693475000000003 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.212599+0000", 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.149587674 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.212644+0000", 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5093999999999999e-05 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.212675+0000", 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1436999999999997e-05 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121198 2.d 2:bd9f4ce1:::benchmark_data_smithi192_59818_object29123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.242800+0000", 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.41111238899998, 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21629783799999999, 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121198 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242800+0000", 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242800+0000", 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242801+0000", 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.068e-06 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242805+0000", 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2930000000000002e-06 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242817+0000", 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1787000000000001e-05 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242822+0000", 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.614e-06 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242875+0000", 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2468e-05 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.243001+0000", 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012661199999999999 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.243118+0000", 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011650999999999999 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.245693+0000", 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025746229999999998 2024-06-27T21:18:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.245906+0000", 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000213122 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.459027+0000", 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21312176299999999 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.459061+0000", 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3868999999999997e-05 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.459097+0000", 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6109000000000003e-05 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:154228 2.e e157/104 2:7405f5a5:::benchmark_data_smithi192_59818_object62153:head v 157'9339, mlcod=157'9335)", 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:35.138482+0000", 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 213.51542947499999, 2024-06-27T21:18:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16320016100000001, 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.138482+0000", 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.138482+0000", 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.138483+0000", 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8999999999999995e-07 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.138491+0000", 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3130000000000001e-06 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.298472+0000", 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.15998192 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.298479+0000", 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1589999999999998e-06 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.298521+0000", 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1925000000000003e-05 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.298535+0000", 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4878000000000001e-05 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.301638+0000", 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003102516 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:35.301683+0000", 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4660000000000003e-05 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:161706 2.d 2:bf6d504a:::benchmark_data_smithi192_59818_object69631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:46.391941+0000", 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 202.26197102500001, 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.219282276, 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161706 2024-06-27T21:18:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391941+0000", 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391941+0000", 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391943+0000", 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.531e-06 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391950+0000", 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7460000000000003e-06 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391969+0000", 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8564999999999999e-05 2024-06-27T21:18:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.392007+0000", 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8473999999999998e-05 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.392986+0000", 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00097909499999999992 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.393087+0000", 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010039000000000001 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.393186+0000", 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9653000000000003e-05 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.399148+0000", 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059618889999999997 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.405716+0000", 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0065678999999999998 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.611040+0000", 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20532411 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.611160+0000", 2024-06-27T21:18:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120061 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.611223+0000", 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2861999999999995e-05 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:161711 2.d 2:bda238be:::benchmark_data_smithi192_59818_object69636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:46.397659+0000", 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 202.25625300999999, 2024-06-27T21:18:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23285805100000001, 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161711 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397659+0000", 2024-06-27T21:18:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397659+0000", 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397660+0000", 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5719999999999999e-06 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397667+0000", 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1240000000000002e-06 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397680+0000", 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3346999999999999e-05 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397686+0000", 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0469999999999999e-06 2024-06-27T21:18:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.399444+0000", 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001757488 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.399717+0000", 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027336100000000001 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.400227+0000", 2024-06-27T21:18:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051034599999999996 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.405741+0000", 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055133339999999999 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.406373+0000", 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063264199999999999 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.630405+0000", 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22403214199999999 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.630472+0000", 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6199999999999996e-05 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.630517+0000", 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5448000000000003e-05 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15132.0:2793 3.0 e192/186 3:02f22ba0:::benchmark_data_smithi192_74314_object2792:head v 192'178, mlcod=192'174)", 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.253973+0000", 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 112.399938662, 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21077112000000001, 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253973+0000", 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253973+0000", 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253979+0000", 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3890000000000004e-06 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.254000+0000", 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1155999999999999e-05 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464256+0000", 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.210255739 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464262+0000", 2024-06-27T21:18:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8809999999999998e-06 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464314+0000", 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1462999999999998e-05 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464335+0000", 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1256000000000001e-05 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464700+0000", 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036450399999999998 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464744+0000", 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4731999999999998e-05 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15132.0:63865 3.0 e209/186 3:0a199132:::benchmark_data_smithi192_74314_object63864:head v 209'4000, mlcod=209'3996)", 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:04.486070+0000", 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 64.167841667999994, 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.203997294, 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.486070+0000", 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.486070+0000", 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.486071+0000", 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9899999999999999e-07 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.486091+0000", 2024-06-27T21:18:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9896999999999999e-05 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.688188+0000", 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20209695799999999 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.688193+0000", 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3159999999999996e-06 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.688271+0000", 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7279000000000003e-05 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.688295+0000", 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4462000000000001e-05 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690033+0000", 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001738084 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690068+0000", 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499000000000003e-05 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15132.0:91262 3.2 e218/186 3:444f31c2:::benchmark_data_smithi192_74314_object91261:head v 218'5669, mlcod=218'5665)", 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:28.196524+0000", 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.457388289999997, 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19741131100000001, 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.196524+0000", 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.196524+0000", 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.196524+0000", 2024-06-27T21:18:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9999999999999996e-07 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.196608+0000", 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3228999999999999e-05 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.390933+0000", 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.194325208 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.390940+0000", 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.571e-06 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.391031+0000", 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0575999999999995e-05 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.391055+0000", 2024-06-27T21:18:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3816999999999999e-05 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.393891+0000", 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028359050000000001 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:28.393935+0000", 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4205000000000001e-05 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:08.768 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:18:08.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:08.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:18:08.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:08.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:08.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:08.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:08.936 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:18:08.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:18:08.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:18:08.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:08 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:08.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1101916907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:18:09.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:18:09.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:18:09.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:08 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:18:09.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1101916907' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:18:09.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:09.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:09.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:09.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:09.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:09.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:09.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:09.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:09.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:09.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:09.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:09.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:18:09.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:09.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:09.774 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:09.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:09.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:09.895 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:09.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:09 smithi151 ceph-mon[28965]: pgmap v655: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 495/274956 objects degraded (0.180%); 5770/274956 objects misplaced (2.099%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:09.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:09 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:09.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1101916907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:18:09.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:09 smithi151 ceph-mon[28965]: osdmap e235: 12 total, 12 up, 11 in 2024-06-27T21:18:09.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:09 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:18:09.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:09 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:18:09.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:09.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:09.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:10.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:09 smithi190 ceph-mon[32247]: pgmap v655: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 495/274956 objects degraded (0.180%); 5770/274956 objects misplaced (2.099%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:10.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:09 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:10.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1101916907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:18:10.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:09 smithi190 ceph-mon[32247]: osdmap e235: 12 total, 12 up, 11 in 2024-06-27T21:18:10.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:09 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:18:10.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:09 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:18:10.021 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:09 smithi192 ceph-mon[32956]: pgmap v655: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 495/274956 objects degraded (0.180%); 5770/274956 objects misplaced (2.099%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:10.021 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:09 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:10.021 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1101916907' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:18:10.021 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:09 smithi192 ceph-mon[32956]: osdmap e235: 12 total, 12 up, 11 in 2024-06-27T21:18:10.021 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:09 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:18:10.021 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:09 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:18:10.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:10.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:10.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:10.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:10.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:10.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:10.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:10.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:10.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:10.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:10.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:10.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:10.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:10.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:10.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:10.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:10.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:10.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:10.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:11.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:11.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:11.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:11.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:11.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:11.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:11.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:11.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:11.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:11.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:11.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:11.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:11.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:11.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:11.679 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:18:11.680 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:18:11.680 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:18:11.680 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:18:11.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:11 smithi192 ceph-mon[32956]: pgmap v657: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 495/270930 objects degraded (0.183%); 5770/270930 objects misplaced (2.130%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:11.680 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:11 smithi192 ceph-mon[32956]: osdmap e236: 12 total, 12 up, 11 in 2024-06-27T21:18:11.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:11.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:11.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:18:11.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:11.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:11.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:11 smithi151 ceph-mon[28965]: pgmap v657: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 495/270930 objects degraded (0.183%); 5770/270930 objects misplaced (2.130%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:11.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:11 smithi151 ceph-mon[28965]: osdmap e236: 12 total, 12 up, 11 in 2024-06-27T21:18:11.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:11.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:11.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:12.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:11 smithi190 ceph-mon[32247]: pgmap v657: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 495/270930 objects degraded (0.183%); 5770/270930 objects misplaced (2.130%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:12.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:11 smithi190 ceph-mon[32247]: osdmap e236: 12 total, 12 up, 11 in 2024-06-27T21:18:12.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:12.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:12.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:12.218 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:18:12.219 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:18:12.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:12.252 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:18:12.253 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:18:12.294 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:18:12.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:18:12.328 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:18:12.328 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:18:12.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:12.331 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:12.366 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:18:12.366 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:18:12.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:12.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:18:12.403 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:18:12.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:18:12.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:18:12.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:18:12.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:18:12.462 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:18:12.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:12.495 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:18:12.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:18:12.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:12.527 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:18:12.528 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:18:12.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:18:12.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:18:12.605 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:18:12.605 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:18:12.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:12.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:12.649 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:12 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:18:12.650 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:12 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:18:12.650 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:12 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:18:12.650 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:12 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:18:12.650 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1918639599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:12.650 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:12 smithi192 ceph-mon[32956]: 3.1 scrub starts 2024-06-27T21:18:12.743 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:12.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:12.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:12.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:12.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:12 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:18:12.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:12 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:18:12.820 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:12 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:18:12.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:12 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:18:12.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1918639599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:12.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:12 smithi151 ceph-mon[28965]: 3.1 scrub starts 2024-06-27T21:18:12.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:12.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:12.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:13.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:12 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:18:13.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:12 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:18:13.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:12 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:18:13.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:12 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:18:13.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1918639599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:13.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:12 smithi190 ceph-mon[32247]: 3.1 scrub starts 2024-06-27T21:18:13.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:13.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:13.113 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:18:13.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 0 1 2024-06-27T21:18:13.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:13.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:13.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:13.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:13.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:13.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:13.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:13.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:13.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:13.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:13.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:13 smithi151 ceph-mon[28965]: pgmap v659: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 495/268932 objects degraded (0.184%); 5770/268932 objects misplaced (2.146%); 896 KiB/s, 14 objects/s recovering 2024-06-27T21:18:13.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:13 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 495/268932 objects degraded (0.184%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:13.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:13 smithi151 ceph-mon[28965]: 3.1 scrub ok 2024-06-27T21:18:13.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:13.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:13.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:13.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:13.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:14.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:13 smithi190 ceph-mon[32247]: pgmap v659: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 495/268932 objects degraded (0.184%); 5770/268932 objects misplaced (2.146%); 896 KiB/s, 14 objects/s recovering 2024-06-27T21:18:14.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:13 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 495/268932 objects degraded (0.184%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:14.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:13 smithi190 ceph-mon[32247]: 3.1 scrub ok 2024-06-27T21:18:14.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:14.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:13 smithi192 ceph-mon[32956]: pgmap v659: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 495/268932 objects degraded (0.184%); 5770/268932 objects misplaced (2.146%); 896 KiB/s, 14 objects/s recovering 2024-06-27T21:18:14.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:13 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 495/268932 objects degraded (0.184%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:14.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:13 smithi192 ceph-mon[32956]: 3.1 scrub ok 2024-06-27T21:18:14.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:14.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:14.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:14.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:14.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:14.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:14.442 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:14.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:14.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:14.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:14.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:14.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:14.772 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:14.772 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:14.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:14.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:14.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:14.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:14.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:15.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:15.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:15.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:15.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:15.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:15.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:15.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:18:15.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:15.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:15.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:15.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:15.652 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:18:15.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:15.742 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:15.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:15.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:18:15.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:15 smithi151 ceph-mon[28965]: pgmap v660: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 567/264069 objects degraded (0.215%); 5545/264069 objects misplaced (2.100%); 2.0 MiB/s, 33 objects/s recovering 2024-06-27T21:18:15.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1217337410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:15.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:15.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:15.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:15.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:15.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:16.002 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:15 smithi192 ceph-mon[32956]: pgmap v660: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 567/264069 objects degraded (0.215%); 5545/264069 objects misplaced (2.100%); 2.0 MiB/s, 33 objects/s recovering 2024-06-27T21:18:16.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1217337410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:16.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:15 smithi190 ceph-mon[32247]: pgmap v660: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 567/264069 objects degraded (0.215%); 5545/264069 objects misplaced (2.100%); 2.0 MiB/s, 33 objects/s recovering 2024-06-27T21:18:16.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1217337410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:16.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:16.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:16.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:16.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:16.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:16.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:16.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:16.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:16.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:16.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:16.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:16.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:16 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:18:16.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1217337410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:16.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:16 smithi151 ceph-mon[28965]: osdmap e237: 12 total, 12 up, 11 in 2024-06-27T21:18:16.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:16.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:16.876 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:16.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:16.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:16.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:16.901 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:16 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:18:16.901 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1217337410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:16.901 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:16 smithi192 ceph-mon[32956]: osdmap e237: 12 total, 12 up, 11 in 2024-06-27T21:18:17.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:17.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:16 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:18:17.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1217337410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:17.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:16 smithi190 ceph-mon[32247]: osdmap e237: 12 total, 12 up, 11 in 2024-06-27T21:18:17.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:17.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:17.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:17.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:17.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:17.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:17.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:17.514 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:17.515 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:17.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:17.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:17.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:17.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:17.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:17.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:17.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:17 smithi192 ceph-mon[32956]: pgmap v662: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 567/256893 objects degraded (0.221%); 5545/256893 objects misplaced (2.158%); 1.3 MiB/s, 21 objects/s recovering 2024-06-27T21:18:17.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:17 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:18:17.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:17 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:18:17.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:17.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:17.933 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:17 smithi190 ceph-mon[32247]: pgmap v662: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 567/256893 objects degraded (0.221%); 5545/256893 objects misplaced (2.158%); 1.3 MiB/s, 21 objects/s recovering 2024-06-27T21:18:17.933 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:17 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:18:17.933 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:17 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:18:17.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:17.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:17.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:17 smithi151 ceph-mon[28965]: pgmap v662: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 864 op/s; 567/256893 objects degraded (0.221%); 5545/256893 objects misplaced (2.158%); 1.3 MiB/s, 21 objects/s recovering 2024-06-27T21:18:17.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:17 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:18:17.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:17 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:18:18.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:18.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:18.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:18.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:18.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:18.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.328 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:18.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:18.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:18:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:18.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:18.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:18.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:18.685 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-06-27T21:18:18.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:18.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:18.760 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:18.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:18.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:18.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:18:18.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1575738492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:18:18.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:18 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:18:18.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:18.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:18.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:18.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:19.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1575738492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:18:19.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:18 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:18:19.020 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1575738492' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:18:19.020 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:18 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:18:19.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:19.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:19.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:19.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:19.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:19.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:19.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:19.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:19.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:19.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:19.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:19.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:19.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:19.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:19 smithi151 ceph-mon[28965]: pgmap v663: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 831 op/s; 751/250980 objects degraded (0.299%); 5242/250980 objects misplaced (2.089%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:18:19.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:19.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:19.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 751/250980 objects degraded (0.299%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:19.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:19 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:18:19.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:19 smithi151 ceph-mon[28965]: osdmap e238: 12 total, 12 up, 11 in 2024-06-27T21:18:19.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:19.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:19.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 1 1 2024-06-27T21:18:19.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:19.979 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:19 smithi190 ceph-mon[32247]: pgmap v663: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 831 op/s; 751/250980 objects degraded (0.299%); 5242/250980 objects misplaced (2.089%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:18:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 751/250980 objects degraded (0.299%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:19 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:18:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:19 smithi190 ceph-mon[32247]: osdmap e238: 12 total, 12 up, 11 in 2024-06-27T21:18:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:19 smithi192 ceph-mon[32956]: pgmap v663: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 831 op/s; 751/250980 objects degraded (0.299%); 5242/250980 objects misplaced (2.089%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:18:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 751/250980 objects degraded (0.299%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:19 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:18:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:19 smithi192 ceph-mon[32956]: osdmap e238: 12 total, 12 up, 11 in 2024-06-27T21:18:20.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:20.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:20.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:20.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:20.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:20.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:20.483 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:20.483 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:20.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:18:20.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:18:20.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:20.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:18:20.547 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:18:20.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:20.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:18:20.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:18:20.612 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:18:20.612 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:18:20.639 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:18:20.640 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:18:20.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:20.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:18:20.667 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:18:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:20.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:20.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:18:20.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:18:20.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:18:20.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:18:20.728 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:20 smithi151 ceph-mon[28965]: osdmap e239: 12 total, 12 up, 11 in 2024-06-27T21:18:20.728 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/611907938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:20.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:18:20.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:18:20.782 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:18:20.783 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:18:20.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:20.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:18:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:18:20.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:20.835 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:18:20.835 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:18:20.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:20.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:20.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:20.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:20.978 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:20 smithi192 ceph-mon[32956]: osdmap e239: 12 total, 12 up, 11 in 2024-06-27T21:18:20.978 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/611907938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:21.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:20 smithi190 ceph-mon[32247]: osdmap e239: 12 total, 12 up, 11 in 2024-06-27T21:18:21.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/611907938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:21.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:21.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:21.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:21.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:21.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:21.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:21.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:18:21.391 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:21.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:21.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:21.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:21.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:21.641 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:21.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:21.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:21.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:21.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:21.889 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:21 smithi151 ceph-mon[28965]: pgmap v666: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s; 751/246015 objects degraded (0.305%); 5242/246015 objects misplaced (2.131%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:21.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:21.949 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:21 smithi192 ceph-mon[32956]: pgmap v666: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s; 751/246015 objects degraded (0.305%); 5242/246015 objects misplaced (2.131%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:21.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:21.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:22.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:21 smithi190 ceph-mon[32247]: pgmap v666: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s; 751/246015 objects degraded (0.305%); 5242/246015 objects misplaced (2.131%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:22.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:22.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:22.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:22.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:22.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:22.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:22.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:22.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:22.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:22.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:22.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:22.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:22.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:22.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:22.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:22 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:22.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:22 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:22.954 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:22 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:22.954 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:22 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:22.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:22.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:22.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:23.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:22 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:23.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:22 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:23.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:23.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:23.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:23.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:23.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:23.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:23.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:23.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:23.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:23.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:23.710 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:18:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:18:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:18:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:18:23.711 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:18:23.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:23.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:23.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:23.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:23.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:23.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:23 smithi151 ceph-mon[28965]: pgmap v667: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 751/243561 objects degraded (0.308%); 5242/243561 objects misplaced (2.152%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:23.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:23 smithi151 ceph-mon[28965]: osdmap e240: 12 total, 12 up, 11 in 2024-06-27T21:18:23.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:23 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:23.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:23 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:23.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:24.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:23 smithi190 ceph-mon[32247]: pgmap v667: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 751/243561 objects degraded (0.308%); 5242/243561 objects misplaced (2.152%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:24.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:23 smithi190 ceph-mon[32247]: osdmap e240: 12 total, 12 up, 11 in 2024-06-27T21:18:24.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:23 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:24.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:23 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:24.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:24.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:23 smithi192 ceph-mon[32956]: pgmap v667: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 751/243561 objects degraded (0.308%); 5242/243561 objects misplaced (2.152%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:24.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:23 smithi192 ceph-mon[32956]: osdmap e240: 12 total, 12 up, 11 in 2024-06-27T21:18:24.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:23 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:24.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:23 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:24.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:24.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:24.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:24.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:24.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:18:24.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:24.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:24.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:24.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:24.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:24.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:18:24.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:24.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:24.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:24 smithi151 ceph-mon[28965]: osdmap e241: 12 total, 12 up, 11 in 2024-06-27T21:18:24.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 751/243561 objects degraded (0.308%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:24.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:24.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:24.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:24.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:24.941 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:18:25.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:24 smithi190 ceph-mon[32247]: osdmap e241: 12 total, 12 up, 11 in 2024-06-27T21:18:25.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 751/243561 objects degraded (0.308%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:25.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:25.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:24 smithi192 ceph-mon[32956]: osdmap e241: 12 total, 12 up, 11 in 2024-06-27T21:18:25.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 751/243561 objects degraded (0.308%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:25.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:25.076 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:25.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:25.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:25.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:25.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:25.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:25.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:25.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:25.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:25.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:25.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:25.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:25.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:25 smithi151 ceph-mon[28965]: pgmap v670: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 892/238008 objects degraded (0.375%); 4998/238008 objects misplaced (2.100%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:25.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1075730355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:18:25.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3860463719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:18:25.735 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:18:25.735 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-06-27T21:18:25.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:25.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:25.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:25.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:25.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:26.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:25 smithi190 ceph-mon[32247]: pgmap v670: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 892/238008 objects degraded (0.375%); 4998/238008 objects misplaced (2.100%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:26.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1075730355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:18:26.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3860463719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:18:26.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:25 smithi192 ceph-mon[32956]: pgmap v670: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 892/238008 objects degraded (0.375%); 4998/238008 objects misplaced (2.100%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:18:26.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1075730355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:18:26.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3860463719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:18:26.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:26.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:26.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:26.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:26.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:26.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:26.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:26.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:26.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:26.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:26.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:26.699 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:7679 2.c 2:39129954:::benchmark_data_smithi192_59818_object7678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.214661+0000", 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 396.486220968, 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.247452693, 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7679 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214661+0000", 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214661+0000", 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214662+0000", 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214680+0000", 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7581999999999999e-05 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214689+0000", 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0489999999999993e-06 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214693+0000", 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7520000000000004e-06 2024-06-27T21:18:26.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214737+0000", 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3136000000000002e-05 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214846+0000", 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109242 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214987+0000", 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014072600000000001 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.215443+0000", 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000456274 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.215988+0000", 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054537599999999998 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.462014+0000", 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24602538800000001 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.462071+0000", 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7448000000000003e-05 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.462114+0000", 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2984e-05 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:57137 2.a 2:5f7ac0a5:::benchmark_data_smithi192_59818_object57136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:41.565099+0000", 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 345.135783731, 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.216924167, 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:26.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 57137 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565099+0000", 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565099+0000", 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565100+0000", 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5e-06 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565123+0000", 2024-06-27T21:18:26.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2543000000000001e-05 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565139+0000", 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6747000000000001e-05 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565146+0000", 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0060000000000003e-06 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565196+0000", 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0123000000000001e-05 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565298+0000", 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101382 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,9", 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.565466+0000", 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016832699999999999 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.566916+0000", 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001449433 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.567282+0000", 2024-06-27T21:18:26.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000366447 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.781916+0000", 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21463433100000001 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.781971+0000", 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4898999999999997e-05 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:41.782023+0000", 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1428999999999998e-05 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121553 2.c 2:388c4988:::benchmark_data_smithi192_59818_object29478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:47.089910+0000", 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 279.61097227599998, 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61414529399999995, 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.715 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.716 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:26.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121553 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089910+0000", 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089910+0000", 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089911+0000", 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1349999999999999e-06 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089916+0000", 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9429999999999999e-06 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089929+0000", 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2551000000000001e-05 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089935+0000", 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6279999999999999e-06 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089990+0000", 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4555000000000002e-05 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.090074+0000", 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4549999999999995e-05 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.090141+0000", 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6808000000000001e-05 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.092673+0000", 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025313480000000001 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093694+0000", 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010215420000000001 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703914+0000", 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61022028100000003 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703985+0000", 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0164000000000001e-05 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.704055+0000", 2024-06-27T21:18:26.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0789000000000003e-05 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:121553 2.c e142/104 ondisk, result = 0)", 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:47.093550+0000", 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 279.60733246000001, 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61051329200000004, 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.725 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093550+0000", 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093550+0000", 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093552+0000", 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9599999999999999e-06 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093560+0000", 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3170000000000002e-06 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703821+0000", 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61026075000000002 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703831+0000", 2024-06-27T21:18:26.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6739999999999992e-06 2024-06-27T21:18:26.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703895+0000", 2024-06-27T21:18:26.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4135999999999997e-05 2024-06-27T21:18:26.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703907+0000", 2024-06-27T21:18:26.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2373e-05 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.704063+0000", 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015608199999999999 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.732 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.733 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121562 2.c 2:356cce12:::benchmark_data_smithi192_59818_object29487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-06-27T21:18:26.733 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:47.763119+0000", 2024-06-27T21:18:26.734 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 278.93776318599998, 2024-06-27T21:18:26.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30556668300000001, 2024-06-27T21:18:26.735 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.735 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.735 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:18:26.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:18:26.736 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121562 2024-06-27T21:18:26.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.736 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763119+0000", 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763119+0000", 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763121+0000", 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.917e-06 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763130+0000", 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1160000000000005e-06 2024-06-27T21:18:26.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763147+0000", 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7024000000000002e-05 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763159+0000", 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1724e-05 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763237+0000", 2024-06-27T21:18:26.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8146999999999996e-05 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763334+0000", 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6927000000000002e-05 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763408+0000", 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4337000000000003e-05 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.766248+0000", 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002839622 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.766887+0000", 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063907000000000005 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:48.068540+0000", 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30165340899999998 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:48.068637+0000", 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6249999999999995e-05 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:48.068686+0000", 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.914e-05 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(2.0 e177,142 2:07b9da72:::benchmark_data_smithi192_59818_object24729:head,139'7201)", 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:15:40.453073+0000", 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 166.24780882600001, 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98256926600000005, 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453073+0000", 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453073+0000", 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453074+0000", 2024-06-27T21:18:26.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7800000000000001e-07 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453080+0000", 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:41.435537+0000", 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98245643100000002 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:41.435549+0000", 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2714000000000001e-05 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:41.435643+0000", 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3451999999999999e-05 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.746 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:2775 3.e 3:70f43acf:::benchmark_data_smithi192_74314_object2774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.240568+0000", 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.460314536, 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22133396799999999, 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2775 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240568+0000", 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240568+0000", 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240569+0000", 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7100000000000001e-07 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240590+0000", 2024-06-27T21:18:26.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1858999999999999e-05 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240600+0000", 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2909999999999996e-06 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240605+0000", 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8799999999999999e-06 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240653+0000", 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8072000000000003e-05 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240733+0000", 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0697000000000003e-05 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.240817+0000", 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3979000000000004e-05 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.241131+0000", 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000313312 2024-06-27T21:18:26.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.241460+0000", 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000329036 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461766+0000", 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22030622999999999 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461813+0000", 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6938999999999999e-05 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461902+0000", 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8901999999999997e-05 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:2785 3.e 3:7e6cc3f7:::benchmark_data_smithi192_74314_object2784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:18:26.754 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.248568+0000", 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.45231403700001, 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.218432768, 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2785 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248568+0000", 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248568+0000", 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248569+0000", 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0130000000000001e-06 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248592+0000", 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2716000000000001e-05 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248603+0000", 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1036000000000001e-05 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248610+0000", 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1740000000000003e-06 2024-06-27T21:18:26.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248653+0000", 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2611999999999998e-05 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248749+0000", 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6628999999999999e-05 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:18:26.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.248864+0000", 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114725 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.249187+0000", 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032285499999999998 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.249627+0000", 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000440226 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.466936+0000", 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21730913099999999 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.466967+0000", 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0252000000000001e-05 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.467001+0000", 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4399e-05 2024-06-27T21:18:26.760 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15132.0:2775 3.e e192/186 ondisk, result = 0)", 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.249478+0000", 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.45140406799999, 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21242808599999999, 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.249478+0000", 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.249478+0000", 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.249479+0000", 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.249481+0000", 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4320000000000002e-06 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461711+0000", 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21223009500000001 2024-06-27T21:18:26.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461718+0000", 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.781e-06 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461757+0000", 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8964999999999997e-05 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461762+0000", 2024-06-27T21:18:26.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9320000000000004e-06 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.461906+0000", 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014433600000000001 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:2792 3.0 3:00224386:::benchmark_data_smithi192_74314_object2791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:18:26.765 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.252624+0000", 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.44825811600001, 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21228609100000001, 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2792 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252624+0000", 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252624+0000", 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252625+0000", 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5600000000000004e-07 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252644+0000", 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8749999999999998e-05 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252657+0000", 2024-06-27T21:18:26.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3286e-05 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252665+0000", 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.52e-06 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252707+0000", 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2583999999999998e-05 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252791+0000", 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3959e-05 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.252874+0000", 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2599e-05 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253233+0000", 2024-06-27T21:18:26.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035919400000000002 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253581+0000", 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000347721 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464846+0000", 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211265447 2024-06-27T21:18:26.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464879+0000", 2024-06-27T21:18:26.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3083999999999997e-05 2024-06-27T21:18:26.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:26.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464910+0000", 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0991000000000002e-05 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:2793 3.0 3:02f22ba0:::benchmark_data_smithi192_74314_object2792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.253674+0000", 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 130.447208558, 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.223241673, 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2793 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.775 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253674+0000", 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253674+0000", 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253675+0000", 2024-06-27T21:18:26.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7599999999999996e-07 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253692+0000", 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7578e-05 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253702+0000", 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253708+0000", 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0789999999999997e-06 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253755+0000", 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7373999999999997e-05 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253823+0000", 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8114000000000005e-05 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253899+0000", 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5733000000000001e-05 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.254234+0000", 2024-06-27T21:18:26.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033521400000000002 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.254552+0000", 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031724800000000001 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.476858+0000", 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22230623399999999 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.476890+0000", 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2005999999999998e-05 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.476915+0000", 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5658999999999999e-05 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.781 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:34092 3.0 3:0d853025:::benchmark_data_smithi192_74314_object34091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.797728+0000", 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.903154395, 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.228574522, 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 34092 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.782 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797728+0000", 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797728+0000", 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797729+0000", 2024-06-27T21:18:26.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3500000000000005e-07 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797746+0000", 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7461000000000001e-05 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797756+0000", 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7119999999999995e-06 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797761+0000", 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2490000000000001e-06 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797812+0000", 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0827e-05 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797896+0000", 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3430999999999995e-05 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797980+0000", 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4709000000000005e-05 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799207+0000", 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012271109999999999 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799469+0000", 2024-06-27T21:18:26.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026209900000000001 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026238+0000", 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.226768789 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026277+0000", 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8489999999999999e-05 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026302+0000", 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5709000000000001e-05 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15132.0:34092 3.0 e199/186 ondisk, result = 0)", 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.799348+0000", 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.90153441699999, 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22695979999999999, 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.788 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799348+0000", 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799348+0000", 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799349+0000", 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2399999999999996e-07 2024-06-27T21:18:26.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799352+0000", 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.225e-06 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026179+0000", 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22682697099999999 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026186+0000", 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.652e-06 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026230+0000", 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4499000000000002e-05 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026235+0000", 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6489999999999999e-06 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026308+0000", 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2879999999999993e-05 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:34099 3.0 3:0e170147:::benchmark_data_smithi192_74314_object34098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.801826+0000", 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.89905623999999, 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22959323800000001, 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.793 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 34099 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801826+0000", 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801826+0000", 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801827+0000", 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9299999999999996e-07 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801850+0000", 2024-06-27T21:18:26.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2745e-05 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801858+0000", 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4849999999999992e-06 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801866+0000", 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0949999999999996e-06 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801910+0000", 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3708999999999998e-05 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801995+0000", 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4843000000000001e-05 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:18:26.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.802076+0000", 2024-06-27T21:18:26.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1186999999999994e-05 2024-06-27T21:18:26.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803095+0000", 2024-06-27T21:18:26.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010192510000000001 2024-06-27T21:18:26.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803471+0000", 2024-06-27T21:18:26.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037532100000000002 2024-06-27T21:18:26.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031348+0000", 2024-06-27T21:18:26.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22787695499999999 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031383+0000", 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5766000000000003e-05 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031419+0000", 2024-06-27T21:18:26.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5988000000000002e-05 2024-06-27T21:18:26.802 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.802 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.802 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.803 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15132.0:34099 3.0 e199/186 ondisk, result = 0)", 2024-06-27T21:18:26.803 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.803252+0000", 2024-06-27T21:18:26.803 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.89762985599999, 2024-06-27T21:18:26.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.228173656, 2024-06-27T21:18:26.804 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.804 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.804 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:18:26.804 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803252+0000", 2024-06-27T21:18:26.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803252+0000", 2024-06-27T21:18:26.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803253+0000", 2024-06-27T21:18:26.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5499999999999998e-07 2024-06-27T21:18:26.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803256+0000", 2024-06-27T21:18:26.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1269999999999999e-06 2024-06-27T21:18:26.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031287+0000", 2024-06-27T21:18:26.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22803076 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031298+0000", 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0888e-05 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031339+0000", 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1446000000000003e-05 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031344+0000", 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8250000000000004e-06 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031426+0000", 2024-06-27T21:18:26.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2055e-05 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:63831 3.0 3:05e624b8:::benchmark_data_smithi192_74314_object63830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:04.451706+0000", 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 82.249176117000005, 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.238552548, 2024-06-27T21:18:26.811 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63831 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451706+0000", 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451706+0000", 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451708+0000", 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.421e-06 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451728+0000", 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0792999999999999e-05 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451741+0000", 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2379e-05 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451748+0000", 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-06-27T21:18:26.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451788+0000", 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0352999999999998e-05 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451881+0000", 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2808999999999999e-05 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:18:26.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451995+0000", 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113669 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.453630+0000", 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001634768 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.454524+0000", 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00089468200000000005 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690192+0000", 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23566730999999999 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690229+0000", 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7013e-05 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690259+0000", 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0175000000000002e-05 2024-06-27T21:18:26.817 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:63865 3.0 3:0a199132:::benchmark_data_smithi192_74314_object63864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:04.485727+0000", 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 82.215155612000004, 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.219408568, 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.818 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63865 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485727+0000", 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485727+0000", 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485728+0000", 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8700000000000004e-07 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485748+0000", 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0051000000000002e-05 2024-06-27T21:18:26.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485758+0000", 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0400999999999999e-05 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485764+0000", 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485809+0000", 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5541999999999999e-05 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485895+0000", 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6080000000000003e-05 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.485992+0000", 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6428000000000006e-05 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.487652+0000", 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016601820000000001 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.704122+0000", 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21647013400000001 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.705034+0000", 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00091235299999999997 2024-06-27T21:18:26.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.705091+0000", 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6286999999999998e-05 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.705135+0000", 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4537999999999999e-05 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:90988 3.e 3:7568d2b6:::benchmark_data_smithi192_74314_object90987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:27.743803+0000", 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 58.957079002999997, 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21457852099999999, 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 90988 2024-06-27T21:18:26.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743803+0000", 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743803+0000", 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743804+0000", 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0200000000000001e-07 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743825+0000", 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0713999999999999e-05 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743835+0000", 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0366e-05 2024-06-27T21:18:26.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743848+0000", 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3105000000000001e-05 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.743895+0000", 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7008999999999997e-05 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.744038+0000", 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014278000000000001 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.744133+0000", 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5326000000000002e-05 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.746459+0000", 2024-06-27T21:18:26.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002326018 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.756046+0000", 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0095863460000000008 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.958298+0000", 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20225185800000001 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.958342+0000", 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4799000000000003e-05 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.958382+0000", 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9498e-05 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:90997 3.e 3:7de0318a:::benchmark_data_smithi192_74314_object90996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:27.752808+0000", 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 58.948074108, 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23434428400000001, 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.832 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 90997 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752808+0000", 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752808+0000", 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752809+0000", 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752827+0000", 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7668000000000001e-05 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752837+0000", 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0035e-05 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752844+0000", 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5480000000000004e-06 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752899+0000", 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5288000000000003e-05 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.753020+0000", 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012061999999999999 2024-06-27T21:18:26.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.753151+0000", 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013093999999999999 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.755646+0000", 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024951349999999999 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.961606+0000", 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20595986099999999 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.987057+0000", 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025450803000000001 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.987111+0000", 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4558999999999997e-05 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.987152+0000", 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1090999999999997e-05 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:123733 3.e 3:7ac25a6b:::benchmark_data_smithi192_74314_object17095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:18:08.845044+0000", 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 17.855837842, 2024-06-27T21:18:26.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22416366900000001, 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123733 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845044+0000", 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845044+0000", 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845046+0000", 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.401e-06 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:18:26.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845051+0000", 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8579999999999999e-06 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845062+0000", 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0817e-05 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845068+0000", 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6579999999999996e-06 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845127+0000", 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8548000000000003e-05 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845256+0000", 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000129092 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.845390+0000", 2024-06-27T21:18:26.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134649 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.848243+0000", 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028529670000000001 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:08.876936+0000", 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028692639999999998 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.068894+0000", 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19195773199999999 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.069162+0000", 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026790700000000001 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.069208+0000", 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6400000000000003e-05 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.848 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:18:26.849 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:18:26.849 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:18:26.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:26.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:26.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:26.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:26 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:26.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1075730355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:18:26.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3860463719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:18:26.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:26 smithi151 ceph-mon[28965]: osdmap e242: 12 total, 12 up, 11 in 2024-06-27T21:18:26.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:27.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:26 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:27.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1075730355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:18:27.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3860463719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:18:27.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:26 smithi190 ceph-mon[32247]: osdmap e242: 12 total, 12 up, 11 in 2024-06-27T21:18:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:27.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:27.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:26 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:27.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1075730355' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:18:27.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3860463719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:18:27.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:26 smithi192 ceph-mon[32956]: osdmap e242: 12 total, 12 up, 11 in 2024-06-27T21:18:27.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:27.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:27.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:27.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:27.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:27.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:27.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:27.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:27.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:27.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:18:27.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:27.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 3 1 2024-06-27T21:18:27.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:27.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:27.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:27.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:27 smithi151 ceph-mon[28965]: pgmap v672: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 892/231327 objects degraded (0.386%); 4998/231327 objects misplaced (2.161%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:18:27.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:27.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:28.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:27 smithi190 ceph-mon[32247]: pgmap v672: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 892/231327 objects degraded (0.386%); 4998/231327 objects misplaced (2.161%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:18:28.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:27 smithi192 ceph-mon[32956]: pgmap v672: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 815 op/s; 892/231327 objects degraded (0.386%); 4998/231327 objects misplaced (2.161%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:18:28.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:28.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:28.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:28.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:28.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:28.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:28.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:28.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:28.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:28.549 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:28.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:28.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:28.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:28.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:28.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:28.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:28.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:29.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:29.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:29.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:29.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:18:29.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:18:29.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:29.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:18:29.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:18:29.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:18:29.274 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:18:29.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:18:29.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:18:29.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:29.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:18:29.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:18:29.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:18:29.362 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:18:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:29.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:29.390 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:18:29.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:18:29.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:18:29.435 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:18:29.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:29.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:18:29.467 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:18:29.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:18:29.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:18:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:29.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:29.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:18:29.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:18:29.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:18:29.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:18:29.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:29.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:29.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:29.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:29.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:29.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:29.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:29.894 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:29 smithi151 ceph-mon[28965]: pgmap v673: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 969 op/s; 1009/226107 objects degraded (0.446%); 4744/226107 objects misplaced (2.098%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:18:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:29 smithi151 ceph-mon[28965]: osdmap e243: 12 total, 12 up, 11 in 2024-06-27T21:18:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2355838783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1009/226107 objects degraded (0.446%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:29.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:30.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:30.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:29 smithi190 ceph-mon[32247]: pgmap v673: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 969 op/s; 1009/226107 objects degraded (0.446%); 4744/226107 objects misplaced (2.098%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:18:30.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:30.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:30.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:29 smithi190 ceph-mon[32247]: osdmap e243: 12 total, 12 up, 11 in 2024-06-27T21:18:30.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2355838783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:30.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1009/226107 objects degraded (0.446%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:29 smithi192 ceph-mon[32956]: pgmap v673: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 969 op/s; 1009/226107 objects degraded (0.446%); 4744/226107 objects misplaced (2.098%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:18:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:29 smithi192 ceph-mon[32956]: osdmap e243: 12 total, 12 up, 11 in 2024-06-27T21:18:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2355838783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1009/226107 objects degraded (0.446%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:30.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:30.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:30.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:30.263 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:30.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:30.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:30.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:30.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:30.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:30.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:30.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:18:30.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:30.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:30.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:30.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:30.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:30.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:30.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:31.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:31.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:31.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:31.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:31.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:31.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:31.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:31.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:31.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:31.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:31.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:31.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:31.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:31.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:31.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:31.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:31.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:32.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:31 smithi190 ceph-mon[32247]: pgmap v675: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 879 op/s; 1009/221631 objects degraded (0.455%); 4744/221631 objects misplaced (2.140%); 1.3 MiB/s, 22 objects/s recovering 2024-06-27T21:18:32.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:32.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:31 smithi192 ceph-mon[32956]: pgmap v675: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 879 op/s; 1009/221631 objects degraded (0.455%); 4744/221631 objects misplaced (2.140%); 1.3 MiB/s, 22 objects/s recovering 2024-06-27T21:18:32.059 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:32.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:32.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:32.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:31 smithi151 ceph-mon[28965]: pgmap v675: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 879 op/s; 1009/221631 objects degraded (0.455%); 4744/221631 objects misplaced (2.140%); 1.3 MiB/s, 22 objects/s recovering 2024-06-27T21:18:32.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:32.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:32.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:32.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:18:32.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:32.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:32.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:32.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:32.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:32.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:32.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:32.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:32.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:32.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:32.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:33.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:33.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:33.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:32 smithi151 ceph-mon[28965]: osdmap e244: 12 total, 12 up, 11 in 2024-06-27T21:18:33.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:33.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:33.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:32 smithi192 ceph-mon[32956]: osdmap e244: 12 total, 12 up, 11 in 2024-06-27T21:18:33.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:33.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:33.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:33.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:33.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:33.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:33.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:32 smithi190 ceph-mon[32247]: osdmap e244: 12 total, 12 up, 11 in 2024-06-27T21:18:33.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:33.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:33.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:33.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:33.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:33.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:33.692 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:33.692 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:33.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:33.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:33.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:33.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:33.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:33 smithi151 ceph-mon[28965]: pgmap v677: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 1009/219321 objects degraded (0.460%); 4744/219321 objects misplaced (2.163%); 1.3 MiB/s, 22 objects/s recovering 2024-06-27T21:18:33.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:33.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:33.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:34.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:33 smithi192 ceph-mon[32956]: pgmap v677: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 1009/219321 objects degraded (0.460%); 4744/219321 objects misplaced (2.163%); 1.3 MiB/s, 22 objects/s recovering 2024-06-27T21:18:34.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:34.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:34.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:34.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:34.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:34.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:33 smithi190 ceph-mon[32247]: pgmap v677: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 1009/219321 objects degraded (0.460%); 4744/219321 objects misplaced (2.163%); 1.3 MiB/s, 22 objects/s recovering 2024-06-27T21:18:34.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:18:34.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:34.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:34.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:34.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:34.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:34.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:34.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:34.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:34.767 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-06-27T21:18:34.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:34.871 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:34.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:34.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:34.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:35.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2234398119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:18:35.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1009/219321 objects degraded (0.460%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:35.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:35.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2234398119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:18:35.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1009/219321 objects degraded (0.460%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:35.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:35.160 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2234398119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:18:35.161 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1009/219321 objects degraded (0.460%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:35.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:35.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:35.342 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:35.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:35.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:35.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:35.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:35.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:35.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:35.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:35.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:35.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:35.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:35.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:36.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:36.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:36.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:35 smithi192 ceph-mon[32956]: pgmap v678: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 729 op/s; 1137/213813 objects degraded (0.532%); 4485/213813 objects misplaced (2.098%); 2.0 MiB/s, 33 objects/s recovering 2024-06-27T21:18:36.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2234398119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:18:36.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:35 smithi192 ceph-mon[32956]: osdmap e245: 12 total, 12 up, 11 in 2024-06-27T21:18:36.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:36.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:36.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:35 smithi151 ceph-mon[28965]: pgmap v678: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 729 op/s; 1137/213813 objects degraded (0.532%); 4485/213813 objects misplaced (2.098%); 2.0 MiB/s, 33 objects/s recovering 2024-06-27T21:18:36.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2234398119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:18:36.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:35 smithi151 ceph-mon[28965]: osdmap e245: 12 total, 12 up, 11 in 2024-06-27T21:18:36.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:35 smithi190 ceph-mon[32247]: pgmap v678: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 729 op/s; 1137/213813 objects degraded (0.532%); 4485/213813 objects misplaced (2.098%); 2.0 MiB/s, 33 objects/s recovering 2024-06-27T21:18:36.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2234398119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:18:36.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:35 smithi190 ceph-mon[32247]: osdmap e245: 12 total, 12 up, 11 in 2024-06-27T21:18:36.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:36.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:36.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:36.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:36.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:36.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:36.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:36.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:36.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:36.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:36.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:36.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:36.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:37.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:37.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:37.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 4 1 2024-06-27T21:18:37.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:37.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:37.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:37.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:37.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:37.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:37.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:37.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:37.630 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:37.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:37.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:37.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:37.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:37.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:37 smithi151 ceph-mon[28965]: pgmap v680: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 892 op/s; 1137/206928 objects degraded (0.549%); 4485/206928 objects misplaced (2.167%); 1.1 MiB/s, 18 objects/s recovering 2024-06-27T21:18:37.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:37.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:37.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:38.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:37 smithi192 ceph-mon[32956]: pgmap v680: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 892 op/s; 1137/206928 objects degraded (0.549%); 4485/206928 objects misplaced (2.167%); 1.1 MiB/s, 18 objects/s recovering 2024-06-27T21:18:38.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:38.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:38.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:38.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:38.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:38.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:37 smithi190 ceph-mon[32247]: pgmap v680: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 892 op/s; 1137/206928 objects degraded (0.549%); 4485/206928 objects misplaced (2.167%); 1.1 MiB/s, 18 objects/s recovering 2024-06-27T21:18:38.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:38.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:38.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:38.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:38.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:38.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:38.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:38.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:38.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:38.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:38.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:38.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:39.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:39.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:18:39.065 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:18:39.090 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:18:39.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:18:39.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:39.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:18:39.115 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:18:39.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:18:39.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:18:39.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:39.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:18:39.167 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:18:39.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:18:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:18:39.219 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:18:39.220 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:18:39.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:39.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:18:39.250 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:18:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:39.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:39.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:18:39.279 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:18:39.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:18:39.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:18:39.344 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:18:39.344 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:18:39.360 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:39.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:39.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:18:39.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:18:39.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:39.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:39.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:39.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:39.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:39.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:39.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:39.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:39.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:40.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:40.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:40.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:39 smithi192 ceph-mon[32956]: pgmap v681: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 1253/201339 objects degraded (0.622%); 4233/201339 objects misplaced (2.102%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:18:40.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3557019316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:40.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1253/201339 objects degraded (0.622%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:40.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1819523852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:40.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:39 smithi151 ceph-mon[28965]: pgmap v681: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 1253/201339 objects degraded (0.622%); 4233/201339 objects misplaced (2.102%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:18:40.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3557019316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:40.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1253/201339 objects degraded (0.622%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:40.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1819523852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:40.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:40.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:40.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:39 smithi190 ceph-mon[32247]: pgmap v681: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 1253/201339 objects degraded (0.622%); 4233/201339 objects misplaced (2.102%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:18:40.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3557019316' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:40.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1253/201339 objects degraded (0.622%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:40.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1819523852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:18:40.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:40.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:40.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:40.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:40.423 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:18:40.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:40.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:40.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:40.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:40.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:40.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:40.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:40.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:40.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:41.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:41.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:40 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:41.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1819523852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:41.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:40 smithi192 ceph-mon[32956]: osdmap e246: 12 total, 12 up, 11 in 2024-06-27T21:18:41.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:41.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:41.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:41.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:40 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:41.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1819523852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:41.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:40 smithi151 ceph-mon[28965]: osdmap e246: 12 total, 12 up, 11 in 2024-06-27T21:18:41.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:40 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:41.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1819523852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:18:41.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:40 smithi190 ceph-mon[32247]: osdmap e246: 12 total, 12 up, 11 in 2024-06-27T21:18:41.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:41.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:41.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:41.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:41.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:41.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:41.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:41.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:41.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:41.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:41.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:41.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:41.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:42.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:41 smithi192 ceph-mon[32956]: pgmap v682: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 934 op/s; 1253/196443 objects degraded (0.638%); 4233/196443 objects misplaced (2.155%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:42.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:42.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:42.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:41 smithi151 ceph-mon[28965]: pgmap v682: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 934 op/s; 1253/196443 objects degraded (0.638%); 4233/196443 objects misplaced (2.155%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:42.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:42.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:42.247 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:18:42.247 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:18:42.247 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:18:42.248 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:18:42.248 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:18:42.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:41 smithi190 ceph-mon[32247]: pgmap v682: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 934 op/s; 1253/196443 objects degraded (0.638%); 4233/196443 objects misplaced (2.155%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:42.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:42.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:42.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:42.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:42.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:42.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:42.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:42.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:42.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:42.841 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:42.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:42.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:42.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:43.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:43.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:43.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:43.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:43.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:43.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:43.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:43.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:43.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:43.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:43.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:43.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:43.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:43.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:43.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:43.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:43.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:18:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:44.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:44.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:43 smithi192 ceph-mon[32956]: pgmap v684: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 1253/194001 objects degraded (0.646%); 4233/194001 objects misplaced (2.182%); 1.1 MiB/s, 16 objects/s recovering 2024-06-27T21:18:44.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:44.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:44.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:43 smithi151 ceph-mon[28965]: pgmap v684: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 1253/194001 objects degraded (0.646%); 4233/194001 objects misplaced (2.182%); 1.1 MiB/s, 16 objects/s recovering 2024-06-27T21:18:44.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:44.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:43 smithi190 ceph-mon[32247]: pgmap v684: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 1253/194001 objects degraded (0.646%); 4233/194001 objects misplaced (2.182%); 1.1 MiB/s, 16 objects/s recovering 2024-06-27T21:18:44.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:44.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:44.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:44.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:44.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:44.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:44.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:44.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:44.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:44.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1253/194001 objects degraded (0.646%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:44.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:44.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:44.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:45.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:45.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:45.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1253/194001 objects degraded (0.646%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:45.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:45.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1253/194001 objects degraded (0.646%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:18:45.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:45.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:45.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:45.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:45.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:18:45.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:45.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:45.642 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:45.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:45.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:45.795 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-06-27T21:18:45.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:45.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:45.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:45.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:45.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:45.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:45 smithi151 ceph-mon[28965]: pgmap v685: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 1424/188403 objects degraded (0.756%); 3945/188403 objects misplaced (2.094%); 1.7 MiB/s, 27 objects/s recovering 2024-06-27T21:18:45.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/146358609' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:18:46.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:46.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:46.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:46.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:46.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:45 smithi190 ceph-mon[32247]: pgmap v685: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 1424/188403 objects degraded (0.756%); 3945/188403 objects misplaced (2.094%); 1.7 MiB/s, 27 objects/s recovering 2024-06-27T21:18:46.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/146358609' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:18:46.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:45 smithi192 ceph-mon[32956]: pgmap v685: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 1424/188403 objects degraded (0.756%); 3945/188403 objects misplaced (2.094%); 1.7 MiB/s, 27 objects/s recovering 2024-06-27T21:18:46.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/146358609' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:18:46.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:18:46.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:46.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:46.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:46.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:46.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:46.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 5 1 2024-06-27T21:18:46.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:46.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:46.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:46.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:46.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:46.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:46.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/146358609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:18:46.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:46 smithi151 ceph-mon[28965]: osdmap e247: 12 total, 12 up, 11 in 2024-06-27T21:18:46.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:46 smithi151 ceph-mon[28965]: pgmap v687: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 1424/181461 objects degraded (0.785%); 3945/181461 objects misplaced (2.174%); 904 KiB/s, 14 objects/s recovering 2024-06-27T21:18:46.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:18:46.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:18:46.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:47.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:47.114 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:47.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:47.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:47.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:47.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/146358609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:18:47.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:46 smithi190 ceph-mon[32247]: osdmap e247: 12 total, 12 up, 11 in 2024-06-27T21:18:47.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:46 smithi190 ceph-mon[32247]: pgmap v687: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 1424/181461 objects degraded (0.785%); 3945/181461 objects misplaced (2.174%); 904 KiB/s, 14 objects/s recovering 2024-06-27T21:18:47.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:18:47.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:18:47.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/146358609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:18:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:46 smithi192 ceph-mon[32956]: osdmap e247: 12 total, 12 up, 11 in 2024-06-27T21:18:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:46 smithi192 ceph-mon[32956]: pgmap v687: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 1424/181461 objects degraded (0.785%); 3945/181461 objects misplaced (2.174%); 904 KiB/s, 14 objects/s recovering 2024-06-27T21:18:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:18:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:18:47.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:47.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:47.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:47.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:47.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:47.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:47.687 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:47.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:47.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:47.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:47.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:47.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:47 smithi151 ceph-mon[28965]: osdmap e248: 12 total, 12 up, 11 in 2024-06-27T21:18:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:47.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:48.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:48.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:48.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:48.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:48.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:47 smithi190 ceph-mon[32247]: osdmap e248: 12 total, 12 up, 11 in 2024-06-27T21:18:48.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:48.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:47 smithi192 ceph-mon[32956]: osdmap e248: 12 total, 12 up, 11 in 2024-06-27T21:18:48.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:48.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:48.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:48.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:48.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:48.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:48.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:48.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:48.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:48.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:48.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:48.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:49.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:49.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:49.143 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:48 smithi151 ceph-mon[28965]: pgmap v689: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 859 op/s; 1549/176343 objects degraded (0.878%); 3691/176343 objects misplaced (2.093%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:49.162 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:49.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:49.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:49.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:48 smithi190 ceph-mon[32247]: pgmap v689: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 859 op/s; 1549/176343 objects degraded (0.878%); 3691/176343 objects misplaced (2.093%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:49.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:48 smithi192 ceph-mon[32956]: pgmap v689: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 859 op/s; 1549/176343 objects degraded (0.878%); 3691/176343 objects misplaced (2.093%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:49.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:49.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:49.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:49.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:49.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:49.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:49.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:49.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:49.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:49.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:49.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:49.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:49.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:50.037 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1549/176343 objects degraded (0.878%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:18:50.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:50.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:50.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:50.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:50.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1549/176343 objects degraded (0.878%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:18:50.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1549/176343 objects degraded (0.878%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:18:50.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:50.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:50.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:50.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:50.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:50.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:50.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:50.754 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:50.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:50.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:50.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:50.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:51.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:51.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:51.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:51.154 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:50 smithi151 ceph-mon[28965]: pgmap v690: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 909 op/s; 1549/172158 objects degraded (0.900%); 3691/172158 objects misplaced (2.144%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:18:51.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:51.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:51.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:50 smithi190 ceph-mon[32247]: pgmap v690: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 909 op/s; 1549/172158 objects degraded (0.900%); 3691/172158 objects misplaced (2.144%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:18:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:51.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:50 smithi192 ceph-mon[32956]: pgmap v690: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 909 op/s; 1549/172158 objects degraded (0.900%); 3691/172158 objects misplaced (2.144%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:18:51.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:51.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:51.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:51.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:51.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:51.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:51.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:51.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:51.834 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:18:51.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:51.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:51.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:51.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.025 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:52.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:52.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:52.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:52.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/42841249' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:18:52.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:52.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/42841249' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:18:52.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:52.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/42841249' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:18:52.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:52.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:52.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:52.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:52.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:52.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:52.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:52.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:52.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:52.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:52.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:52.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:53.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:53.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:53.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:52 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/42841249' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:18:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:52 smithi151 ceph-mon[28965]: osdmap e249: 12 total, 12 up, 11 in 2024-06-27T21:18:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:52 smithi151 ceph-mon[28965]: pgmap v692: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 1549/170016 objects degraded (0.911%); 3691/170016 objects misplaced (2.171%); 1.3 MiB/s, 21 objects/s recovering 2024-06-27T21:18:53.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:53.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:53.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:52 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:53.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/42841249' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:18:53.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:52 smithi190 ceph-mon[32247]: osdmap e249: 12 total, 12 up, 11 in 2024-06-27T21:18:53.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:52 smithi190 ceph-mon[32247]: pgmap v692: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 1549/170016 objects degraded (0.911%); 3691/170016 objects misplaced (2.171%); 1.3 MiB/s, 21 objects/s recovering 2024-06-27T21:18:53.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:53.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:52 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:18:53.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/42841249' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:18:53.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:52 smithi192 ceph-mon[32956]: osdmap e249: 12 total, 12 up, 11 in 2024-06-27T21:18:53.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:52 smithi192 ceph-mon[32956]: pgmap v692: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 1549/170016 objects degraded (0.911%); 3691/170016 objects misplaced (2.171%); 1.3 MiB/s, 21 objects/s recovering 2024-06-27T21:18:53.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:53.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:53.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:53.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:53.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:53.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:53.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:53.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:53.836 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:53.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:53.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:53.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:54.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:54.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:54.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:53 smithi151 ceph-mon[28965]: 3.8 scrub starts 2024-06-27T21:18:54.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1883975339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:54.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:54.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:54.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:53 smithi190 ceph-mon[32247]: 3.8 scrub starts 2024-06-27T21:18:54.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1883975339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:54.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:53 smithi192 ceph-mon[32956]: 3.8 scrub starts 2024-06-27T21:18:54.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1883975339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:18:54.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:54.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:54.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:54.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:18:54.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:18:54.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:54.500 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:18:54.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:18:54.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:18:54.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:18:54.554 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:18:54.554 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:18:54.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:54.581 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:18:54.581 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:18:54.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:54.619 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:18:54.619 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:18:54.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:18:54.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:18:54.681 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:18:54.682 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:18:54.689 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:18:54.690 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:18:54.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:54.710 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:18:54.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:18:54.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:54.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:18:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:18:54.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:54.761 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:18:54.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:18:54.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:18:54.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:18:54.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:54.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:54.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:54 smithi151 ceph-mon[28965]: 3.8 scrub ok 2024-06-27T21:18:54.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:54 smithi151 ceph-mon[28965]: 3.9 scrub starts 2024-06-27T21:18:54.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:54 smithi151 ceph-mon[28965]: pgmap v693: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 1658/165162 objects degraded (1.004%); 3447/165162 objects misplaced (2.087%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:54.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1549/170016 objects degraded (0.911%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:18:54.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:54.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:54.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:55.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:55.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:55.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:54 smithi192 ceph-mon[32956]: 3.8 scrub ok 2024-06-27T21:18:55.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:54 smithi192 ceph-mon[32956]: 3.9 scrub starts 2024-06-27T21:18:55.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:54 smithi192 ceph-mon[32956]: pgmap v693: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 1658/165162 objects degraded (1.004%); 3447/165162 objects misplaced (2.087%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:55.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1549/170016 objects degraded (0.911%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:18:55.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:55.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:55.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:54 smithi190 ceph-mon[32247]: 3.8 scrub ok 2024-06-27T21:18:55.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:54 smithi190 ceph-mon[32247]: 3.9 scrub starts 2024-06-27T21:18:55.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:54 smithi190 ceph-mon[32247]: pgmap v693: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 1658/165162 objects degraded (1.004%); 3447/165162 objects misplaced (2.087%); 2.0 MiB/s, 32 objects/s recovering 2024-06-27T21:18:55.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1549/170016 objects degraded (0.911%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:18:55.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:55.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:55.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:55.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:55.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:18:55.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:18:55.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:55.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:18:55.848 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-06-27T21:18:55.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:55.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:55.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:55.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:55 smithi151 ceph-mon[28965]: 3.9 scrub ok 2024-06-27T21:18:55.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:55 smithi151 ceph-mon[28965]: 3.4 scrub starts 2024-06-27T21:18:55.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1522707185' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:18:55.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:55 smithi151 ceph-mon[28965]: 3.d scrub starts 2024-06-27T21:18:56.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:56.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:56.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:56.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:56.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:56.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:55 smithi190 ceph-mon[32247]: 3.9 scrub ok 2024-06-27T21:18:56.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:55 smithi190 ceph-mon[32247]: 3.4 scrub starts 2024-06-27T21:18:56.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1522707185' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:18:56.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:55 smithi190 ceph-mon[32247]: 3.d scrub starts 2024-06-27T21:18:56.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:56.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:55 smithi192 ceph-mon[32956]: 3.9 scrub ok 2024-06-27T21:18:56.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:55 smithi192 ceph-mon[32956]: 3.4 scrub starts 2024-06-27T21:18:56.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1522707185' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:18:56.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:55 smithi192 ceph-mon[32956]: 3.d scrub starts 2024-06-27T21:18:56.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:56.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:56.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:18:56.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:56.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:56.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:56.680 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:56.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:18:56.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:56.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:56.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:18:56.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:56.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:18:56.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:56 smithi151 ceph-mon[28965]: 3.4 scrub ok 2024-06-27T21:18:56.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1522707185' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:18:56.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:56 smithi151 ceph-mon[28965]: osdmap e250: 12 total, 12 up, 11 in 2024-06-27T21:18:56.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:56 smithi151 ceph-mon[28965]: pgmap v695: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 1658/158280 objects degraded (1.048%); 3447/158280 objects misplaced (2.178%); 1.0 MiB/s, 16 objects/s recovering 2024-06-27T21:18:57.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:57.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:57.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:57.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:57.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 6 1 2024-06-27T21:18:57.234 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:57.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:56 smithi190 ceph-mon[32247]: 3.4 scrub ok 2024-06-27T21:18:57.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1522707185' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:18:57.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:56 smithi190 ceph-mon[32247]: osdmap e250: 12 total, 12 up, 11 in 2024-06-27T21:18:57.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:56 smithi190 ceph-mon[32247]: pgmap v695: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 1658/158280 objects degraded (1.048%); 3447/158280 objects misplaced (2.178%); 1.0 MiB/s, 16 objects/s recovering 2024-06-27T21:18:57.282 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:18:57.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:56 smithi192 ceph-mon[32956]: 3.4 scrub ok 2024-06-27T21:18:57.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1522707185' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:18:57.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:56 smithi192 ceph-mon[32956]: osdmap e250: 12 total, 12 up, 11 in 2024-06-27T21:18:57.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:56 smithi192 ceph-mon[32956]: pgmap v695: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 1658/158280 objects degraded (1.048%); 3447/158280 objects misplaced (2.178%); 1.0 MiB/s, 16 objects/s recovering 2024-06-27T21:18:57.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:57.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:57.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:57.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:57.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:57.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:57.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:57.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:57.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:57.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:57.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:57.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:57 smithi151 ceph-mon[28965]: 3.d scrub ok 2024-06-27T21:18:57.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:57.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:57.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:18:58.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:58.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:58.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:58.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:58.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:57 smithi190 ceph-mon[32247]: 3.d scrub ok 2024-06-27T21:18:58.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:57 smithi192 ceph-mon[32956]: 3.d scrub ok 2024-06-27T21:18:58.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:18:58.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:18:58.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:58.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:58.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:18:58.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:18:58.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:58.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:58.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:58.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:58.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:18:58.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:58.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:18:59.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:59.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:59.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:58 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:18:59.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:58 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:18:59.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:58 smithi151 ceph-mon[28965]: pgmap v696: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 1770/153561 objects degraded (1.153%); 3195/153561 objects misplaced (2.081%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:18:59.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:18:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:18:59.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:18:59.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:58 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:18:59.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:58 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:18:59.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:58 smithi190 ceph-mon[32247]: pgmap v696: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 1770/153561 objects degraded (1.153%); 3195/153561 objects misplaced (2.081%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:18:59.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:58 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:18:59.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:58 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:18:59.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:58 smithi192 ceph-mon[32956]: pgmap v696: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 1770/153561 objects degraded (1.153%); 3195/153561 objects misplaced (2.081%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:18:59.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:18:59.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:18:59.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:18:59.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:18:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:18:59.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:18:59.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:18:59.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:18:59.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:18:59.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:18:59.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:18:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:18:59.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:18:59.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:18:59.985 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:00.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:00.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:00.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:00.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:59 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:19:00.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:59 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:19:00.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:59 smithi151 ceph-mon[28965]: 3.f scrub starts 2024-06-27T21:19:00.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:18:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1770/153561 objects degraded (1.153%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:00.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:00.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:59 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:19:00.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:59 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:19:00.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:59 smithi190 ceph-mon[32247]: 3.f scrub starts 2024-06-27T21:19:00.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:18:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1770/153561 objects degraded (1.153%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:00.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:00.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:59 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:19:00.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:59 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:19:00.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:59 smithi192 ceph-mon[32956]: 3.f scrub starts 2024-06-27T21:19:00.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:18:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1770/153561 objects degraded (1.153%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:00.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:00.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:00.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:00.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:00.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:00.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:00.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:00.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:00.874 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:19:00.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:00.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:00.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:01.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:01.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:01.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:01.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:01.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:00 smithi151 ceph-mon[28965]: 3.0 scrub starts 2024-06-27T21:19:01.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:00 smithi151 ceph-mon[28965]: pgmap v697: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 1770/149451 objects degraded (1.184%); 3195/149451 objects misplaced (2.138%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:19:01.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/346821088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:01.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:00 smithi151 ceph-mon[28965]: 3.f scrub ok 2024-06-27T21:19:01.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:00 smithi190 ceph-mon[32247]: 3.0 scrub starts 2024-06-27T21:19:01.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:00 smithi190 ceph-mon[32247]: pgmap v697: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 1770/149451 objects degraded (1.184%); 3195/149451 objects misplaced (2.138%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:19:01.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/346821088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:01.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:00 smithi190 ceph-mon[32247]: 3.f scrub ok 2024-06-27T21:19:01.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:01.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:00 smithi192 ceph-mon[32956]: 3.0 scrub starts 2024-06-27T21:19:01.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:00 smithi192 ceph-mon[32956]: pgmap v697: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 1770/149451 objects degraded (1.184%); 3195/149451 objects misplaced (2.138%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:19:01.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/346821088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:01.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:00 smithi192 ceph-mon[32956]: 3.f scrub ok 2024-06-27T21:19:01.314 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:01.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:01.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:01.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:01.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:01.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:01.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:01.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:01.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:01.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:01.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:01 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:19:01.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:01 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:19:01.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:01 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:19:01.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/346821088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:01.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:01 smithi151 ceph-mon[28965]: osdmap e251: 12 total, 12 up, 11 in 2024-06-27T21:19:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:01.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:01.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:02.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:02.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:02.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:02.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:02.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:01 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:19:02.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:01 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:19:02.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:01 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:19:02.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/346821088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:02.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:01 smithi190 ceph-mon[32247]: osdmap e251: 12 total, 12 up, 11 in 2024-06-27T21:19:02.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:01 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:19:02.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:01 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:19:02.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:01 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:19:02.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/346821088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:02.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:01 smithi192 ceph-mon[32956]: osdmap e251: 12 total, 12 up, 11 in 2024-06-27T21:19:02.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:02.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:02.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:02.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:02.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:02.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:02.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:02.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:02.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:02.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:02.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:02.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:02.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:02.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:03.036 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:03.067 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:19:03.067 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:19:03.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:02 smithi151 ceph-mon[28965]: 3.0 scrub ok 2024-06-27T21:19:03.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:02 smithi151 ceph-mon[28965]: pgmap v699: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 741 op/s; 1770/147354 objects degraded (1.201%); 3195/147354 objects misplaced (2.168%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:19:03.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:03.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:03.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:03.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:03.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:02 smithi190 ceph-mon[32247]: 3.0 scrub ok 2024-06-27T21:19:03.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:02 smithi190 ceph-mon[32247]: pgmap v699: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 741 op/s; 1770/147354 objects degraded (1.201%); 3195/147354 objects misplaced (2.168%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:19:03.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:02 smithi192 ceph-mon[32956]: 3.0 scrub ok 2024-06-27T21:19:03.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:02 smithi192 ceph-mon[32956]: pgmap v699: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 741 op/s; 1770/147354 objects degraded (1.201%); 3195/147354 objects misplaced (2.168%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:19:03.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:03.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:03.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:03.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:03.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:03.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:03.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:19:03.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:03.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:03.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:03.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:03.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:03.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:04.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:04.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:04.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:04.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:04.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:04.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:04.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:04.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:04.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:04.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:04.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:04.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:04.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:19:04.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:04.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:04.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:04.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:04.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:04 smithi151 ceph-mon[28965]: pgmap v700: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 1871/142185 objects degraded (1.316%); 2953/142185 objects misplaced (2.077%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:19:04.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:04 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:19:04.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:04 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:19:04.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1871/142185 objects degraded (1.316%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:05.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:05.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:05.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:05.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:05.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:05.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:04 smithi190 ceph-mon[32247]: pgmap v700: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 1871/142185 objects degraded (1.316%); 2953/142185 objects misplaced (2.077%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:19:05.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:04 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:19:05.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:04 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:19:05.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1871/142185 objects degraded (1.316%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:05.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:05.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:04 smithi192 ceph-mon[32956]: pgmap v700: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 1871/142185 objects degraded (1.316%); 2953/142185 objects misplaced (2.077%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:19:05.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:04 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:19:05.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:04 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:19:05.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1871/142185 objects degraded (1.316%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:05.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:05.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:05.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:05.521 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.548 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:05.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:05.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:05.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:05.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:05.802 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:05.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:19:05.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:19:05.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:05.920 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:19:05.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:19:05.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:05.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:05.950 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:19:05.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:19:05.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:19:05.980 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:19:06.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:06.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:06.095 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:06.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:06.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:06.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:06.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:05 smithi151 ceph-mon[28965]: osdmap e252: 12 total, 12 up, 11 in 2024-06-27T21:19:06.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4015538587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:06.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:06.214 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:19:06.214 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:19:06.248 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:19:06.248 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:19:06.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:06.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:06.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:05 smithi190 ceph-mon[32247]: osdmap e252: 12 total, 12 up, 11 in 2024-06-27T21:19:06.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4015538587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:06.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:19:06.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:19:06.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:06.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:06.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:05 smithi192 ceph-mon[32956]: osdmap e252: 12 total, 12 up, 11 in 2024-06-27T21:19:06.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4015538587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:06.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:06.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:19:06.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:19:06.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:06.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:19:06.336 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:19:06.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:19:06.362 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:19:06.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:19:06.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:19:06.415 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:19:06.416 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:19:06.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:06.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:06.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:06.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:06.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:06.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:06.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:06.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:06.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:06.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:06.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:06.998 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-06-27T21:19:07.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:07.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:07.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:06 smithi151 ceph-mon[28965]: 3.3 scrub starts 2024-06-27T21:19:07.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:06 smithi151 ceph-mon[28965]: pgmap v702: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.8 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 1871/135609 objects degraded (1.380%); 2953/135609 objects misplaced (2.178%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:19:07.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4139915443' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:19:07.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:06 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:19:07.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:06 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:19:07.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:06 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:19:07.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:07.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:06 smithi190 ceph-mon[32247]: 3.3 scrub starts 2024-06-27T21:19:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:06 smithi190 ceph-mon[32247]: pgmap v702: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.8 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 1871/135609 objects degraded (1.380%); 2953/135609 objects misplaced (2.178%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:19:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4139915443' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:19:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:06 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:19:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:06 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:19:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:06 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:19:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:06 smithi192 ceph-mon[32956]: 3.3 scrub starts 2024-06-27T21:19:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:06 smithi192 ceph-mon[32956]: pgmap v702: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.8 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 1871/135609 objects degraded (1.380%); 2953/135609 objects misplaced (2.178%); 1.1 MiB/s, 17 objects/s recovering 2024-06-27T21:19:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4139915443' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:19:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:06 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:19:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:06 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:19:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:06 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:19:07.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:07.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:07.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:07.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:07.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:07.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:07.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:07.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:07.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:07.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:07.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:07.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:07.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:08.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:08.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:08.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:08.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:08.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:08 smithi190 ceph-mon[32247]: 3.3 scrub ok 2024-06-27T21:19:08.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:08 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:19:08.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:08 smithi190 ceph-mon[32247]: osdmap e253: 12 total, 12 up, 11 in 2024-06-27T21:19:08.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:08 smithi192 ceph-mon[32956]: 3.3 scrub ok 2024-06-27T21:19:08.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:08 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:19:08.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:08 smithi192 ceph-mon[32956]: osdmap e253: 12 total, 12 up, 11 in 2024-06-27T21:19:08.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:08 smithi151 ceph-mon[28965]: 3.3 scrub ok 2024-06-27T21:19:08.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:08 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:19:08.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:08 smithi151 ceph-mon[28965]: osdmap e253: 12 total, 12 up, 11 in 2024-06-27T21:19:08.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:08.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:08.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:08.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:08.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:08.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:08.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:08.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:08.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:08.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:08.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:08.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:09.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:09.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:09.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:09.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:09.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:09 smithi190 ceph-mon[32247]: 3.e deep-scrub starts 2024-06-27T21:19:09.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:09 smithi190 ceph-mon[32247]: pgmap v704: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.7 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 763 op/s; 1955/131223 objects degraded (1.490%); 2705/131223 objects misplaced (2.061%); 2.7 MiB/s, 43 objects/s recovering 2024-06-27T21:19:09.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:09 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:19:09.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:09.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:09 smithi192 ceph-mon[32956]: 3.e deep-scrub starts 2024-06-27T21:19:09.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:09 smithi192 ceph-mon[32956]: pgmap v704: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.7 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 763 op/s; 1955/131223 objects degraded (1.490%); 2705/131223 objects misplaced (2.061%); 2.7 MiB/s, 43 objects/s recovering 2024-06-27T21:19:09.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:09 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:19:09.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:09.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:09.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:09 smithi151 ceph-mon[28965]: 3.e deep-scrub starts 2024-06-27T21:19:09.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:09 smithi151 ceph-mon[28965]: pgmap v704: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.7 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 763 op/s; 1955/131223 objects degraded (1.490%); 2705/131223 objects misplaced (2.061%); 2.7 MiB/s, 43 objects/s recovering 2024-06-27T21:19:09.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:09 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:19:09.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:09.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:09.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:09.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:09.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:09.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:09.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:09.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:09.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:10.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:10.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:10.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:10.161 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:10.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:10 smithi190 ceph-mon[32247]: 3.6 deep-scrub starts 2024-06-27T21:19:10.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:10 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1955/131223 objects degraded (1.490%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:10.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:19:10.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:19:10.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:10 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:19:10.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:10.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:10 smithi151 ceph-mon[28965]: 3.6 deep-scrub starts 2024-06-27T21:19:10.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:10 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1955/131223 objects degraded (1.490%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:10.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:19:10.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:19:10.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:10 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:19:10.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:10.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:10 smithi192 ceph-mon[32956]: 3.6 deep-scrub starts 2024-06-27T21:19:10.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:10 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1955/131223 objects degraded (1.490%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:10.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:19:10.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:19:10.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:10 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:19:10.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:10.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:10.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:10.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:10.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:10.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:10.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:19:10.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:10.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:10.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:10.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:10.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:11.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:11.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:11.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:11.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:11.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:11.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:11.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:11.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 7 1 2024-06-27T21:19:11.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:11.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:11.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:11.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:11.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:11.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:11.698 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:19:11.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:11 smithi151 ceph-mon[28965]: pgmap v705: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 1955/127485 objects degraded (1.534%); 2705/127485 objects misplaced (2.122%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:19:11.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1517015087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:19:11.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:11.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:11.908 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:19:11.908 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:19:11.908 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:19:11.908 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:19:11.908 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:19:11.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:11.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:11.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:12.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:11 smithi190 ceph-mon[32247]: pgmap v705: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 1955/127485 objects degraded (1.534%); 2705/127485 objects misplaced (2.122%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:19:12.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1517015087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:19:12.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:12.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:11 smithi192 ceph-mon[32956]: pgmap v705: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 2.6 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 827 op/s; 1955/127485 objects degraded (1.534%); 2705/127485 objects misplaced (2.122%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:19:12.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1517015087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:19:12.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:12.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:12.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:12.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:12.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:12.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:12.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:12.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:12.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:12.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:12.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:12.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:12.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:12.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:12.862 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:12.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:12.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:12 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:12.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1517015087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:19:12.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:12 smithi151 ceph-mon[28965]: osdmap e254: 12 total, 12 up, 11 in 2024-06-27T21:19:12.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:12.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:13.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:12 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:13.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1517015087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:19:13.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:12 smithi190 ceph-mon[32247]: osdmap e254: 12 total, 12 up, 11 in 2024-06-27T21:19:13.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:12 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:13.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1517015087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:19:13.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:12 smithi192 ceph-mon[32956]: osdmap e254: 12 total, 12 up, 11 in 2024-06-27T21:19:13.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:13.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:13.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:13.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:13.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:13.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:13.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:13.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:13.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:19:13.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:13.634 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:13.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:13.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:13.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:13.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:13.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:13 smithi151 ceph-mon[28965]: pgmap v707: 27 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.6 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 1955/125778 objects degraded (1.554%); 2705/125778 objects misplaced (2.151%); 1.5 MiB/s, 23 objects/s recovering 2024-06-27T21:19:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:13.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:13.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:13.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:13.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:14.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:13 smithi190 ceph-mon[32247]: pgmap v707: 27 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.6 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 1955/125778 objects degraded (1.554%); 2705/125778 objects misplaced (2.151%); 1.5 MiB/s, 23 objects/s recovering 2024-06-27T21:19:14.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:13 smithi192 ceph-mon[32956]: pgmap v707: 27 pgs: 2 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.6 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 1955/125778 objects degraded (1.554%); 2705/125778 objects misplaced (2.151%); 1.5 MiB/s, 23 objects/s recovering 2024-06-27T21:19:14.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:14.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:14.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:14.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:14.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:14.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:14.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:14.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:14.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:14.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:19:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:14.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:14.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:14.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:14.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:14.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:14.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:14.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:14.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1955/125778 objects degraded (1.554%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:14.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:15.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:15.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:15.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1955/125778 objects degraded (1.554%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:15.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:15.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:15.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:15.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1955/125778 objects degraded (1.554%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:15.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:15.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:15.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:15.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:15.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:15.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:15.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:15.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:15.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:15.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:15.691 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:15.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:15.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:15.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:15.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:15.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:15.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:16.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:15 smithi190 ceph-mon[32247]: pgmap v708: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.5 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 547 op/s; 1953/122475 objects degraded (1.595%); 2508/122475 objects misplaced (2.048%); 2.8 MiB/s, 45 objects/s recovering 2024-06-27T21:19:16.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:15 smithi190 ceph-mon[32247]: osdmap e255: 12 total, 12 up, 11 in 2024-06-27T21:19:16.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:16.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:15 smithi192 ceph-mon[32956]: pgmap v708: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.5 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 547 op/s; 1953/122475 objects degraded (1.595%); 2508/122475 objects misplaced (2.048%); 2.8 MiB/s, 45 objects/s recovering 2024-06-27T21:19:16.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:15 smithi192 ceph-mon[32956]: osdmap e255: 12 total, 12 up, 11 in 2024-06-27T21:19:16.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:16.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:16.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:15 smithi151 ceph-mon[28965]: pgmap v708: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.5 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 547 op/s; 1953/122475 objects degraded (1.595%); 2508/122475 objects misplaced (2.048%); 2.8 MiB/s, 45 objects/s recovering 2024-06-27T21:19:16.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:15 smithi151 ceph-mon[28965]: osdmap e255: 12 total, 12 up, 11 in 2024-06-27T21:19:16.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:16.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:16.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:16.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:16.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:16.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:16.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:16.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:16.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:16.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:16.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:16.905 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:16.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:16.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:17.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:17.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:17.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:17.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:17.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:17.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:17.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:17.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:17.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:17.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:17.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:17.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:17.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:17.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:17.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:17.882 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:17 smithi151 ceph-mon[28965]: pgmap v710: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 543 op/s; 1953/118173 objects degraded (1.653%); 2508/118173 objects misplaced (2.122%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:17.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:17.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:17.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:17 smithi151 ceph-mon[28965]: osdmap e256: 12 total, 12 up, 11 in 2024-06-27T21:19:17.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:17.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:17.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:18.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:17 smithi190 ceph-mon[32247]: pgmap v710: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 543 op/s; 1953/118173 objects degraded (1.653%); 2508/118173 objects misplaced (2.122%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:18.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:18.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:18.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:17 smithi190 ceph-mon[32247]: osdmap e256: 12 total, 12 up, 11 in 2024-06-27T21:19:18.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:17 smithi192 ceph-mon[32956]: pgmap v710: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.4 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 543 op/s; 1953/118173 objects degraded (1.653%); 2508/118173 objects misplaced (2.122%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:18.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:18.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:18.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:17 smithi192 ceph-mon[32956]: osdmap e256: 12 total, 12 up, 11 in 2024-06-27T21:19:18.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:18.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:18.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:18.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:18.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:18.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:18.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:18.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:18.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:18.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:18.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:18.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:18.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:18.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:18.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:18.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:18.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:19.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:19.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:19.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:19.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:19.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:19.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:19.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:19.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:19.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:19.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:19.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:19.706 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:19:19.706 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:19:19.721 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-06-27T21:19:19.724 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:19.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:19:19.733 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:19:19.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:19:19.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:19:19.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:19:19.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:19:19.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:19:19.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:19:19.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:19.844 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:19:19.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:19:19.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:19 smithi151 ceph-mon[28965]: pgmap v712: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 1953/114690 objects degraded (1.703%); 2301/114690 objects misplaced (2.006%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:19:19.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4060733993' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:19:19.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:19 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:19:19.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1953/114690 objects degraded (1.703%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:19.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2265490999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:19.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:19.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:19:19.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:19:19.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:19:19.896 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:19:19.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:19:19.928 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:19:19.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:19.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:19:19.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:19:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:19.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:19.983 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:19:19.983 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:19:20.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:19:20.010 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:19:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:19 smithi190 ceph-mon[32247]: pgmap v712: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 1953/114690 objects degraded (1.703%); 2301/114690 objects misplaced (2.006%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:19:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4060733993' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:19:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:19 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:19:20.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1953/114690 objects degraded (1.703%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:20.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2265490999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:19 smithi192 ceph-mon[32956]: pgmap v712: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 1953/114690 objects degraded (1.703%); 2301/114690 objects misplaced (2.006%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:19:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4060733993' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:19:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:19 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:19:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1953/114690 objects degraded (1.703%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:20.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2265490999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:20.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:20.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:20.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:20.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:20.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:20.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:20.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.487 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:20.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:20.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:20.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:20.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:20.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:20.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:20.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:20.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:20.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:20 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:19:20.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:20 smithi151 ceph-mon[28965]: osdmap e257: 12 total, 12 up, 11 in 2024-06-27T21:19:20.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:20.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:20.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:21.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:20 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:19:21.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:20 smithi190 ceph-mon[32247]: osdmap e257: 12 total, 12 up, 11 in 2024-06-27T21:19:21.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:20 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:19:21.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:20 smithi192 ceph-mon[32956]: osdmap e257: 12 total, 12 up, 11 in 2024-06-27T21:19:21.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:21.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:21.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:21.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:21.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:21.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:21.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:21.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:21.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:21.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:21.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:21.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:21.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:21.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:21.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:22.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:21 smithi151 ceph-mon[28965]: pgmap v714: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.3 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 598 op/s; 1953/111705 objects degraded (1.748%); 2301/111705 objects misplaced (2.060%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:19:22.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:21 smithi190 ceph-mon[32247]: pgmap v714: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.3 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 598 op/s; 1953/111705 objects degraded (1.748%); 2301/111705 objects misplaced (2.060%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:19:22.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:22.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:21 smithi192 ceph-mon[32956]: pgmap v714: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.3 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 598 op/s; 1953/111705 objects degraded (1.748%); 2301/111705 objects misplaced (2.060%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:19:22.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:22.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:22.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:22.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:22.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:22.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:22.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:22.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:22.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:22.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:22.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:22.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:22.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:22.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:22.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:23.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:23.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:23.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:23.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:23.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:23.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:23.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 8 1 2024-06-27T21:19:23.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:23.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:19:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:23.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:23.446 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:19:23.446 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:19:23.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:23.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:23.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:23.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:23.735 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:19:23.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:23.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:23.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:23.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:23.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:24.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:23 smithi190 ceph-mon[32247]: pgmap v715: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.2 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 1953/110286 objects degraded (1.771%); 2301/110286 objects misplaced (2.086%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:19:24.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4246118447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:24.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:23 smithi192 ceph-mon[32956]: pgmap v715: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.2 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 1953/110286 objects degraded (1.771%); 2301/110286 objects misplaced (2.086%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:19:24.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4246118447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:24.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:24.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:24.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:24.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:23 smithi151 ceph-mon[28965]: pgmap v715: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.2 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 1953/110286 objects degraded (1.771%); 2301/110286 objects misplaced (2.086%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:19:24.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4246118447' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:24.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:24.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:19:24.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:24.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:24.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:24.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:24.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:24.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:24.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:24.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:24.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:24.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:24.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:24.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4246118447' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: osdmap e258: 12 total, 12 up, 11 in 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:19:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1953/110286 objects degraded (1.771%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:24.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:24.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:25.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4246118447' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: osdmap e258: 12 total, 12 up, 11 in 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:19:25.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1953/110286 objects degraded (1.771%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:25.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:25.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4246118447' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: osdmap e258: 12 total, 12 up, 11 in 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:19:25.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1953/110286 objects degraded (1.771%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:25.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:25.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:25.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:25.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:25.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:25.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:25.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:25.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:25.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:25.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:25.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:25.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:25.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:25.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:25.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:25.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:25.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:25.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:26.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:26.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:26.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:25 smithi192 ceph-mon[32956]: pgmap v717: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.2 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 1953/107118 objects degraded (1.823%); 2064/107118 objects misplaced (1.927%); 3.9 MiB/s, 61 objects/s recovering 2024-06-27T21:19:26.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:26.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:26.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:25 smithi151 ceph-mon[28965]: pgmap v717: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.2 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 1953/107118 objects degraded (1.823%); 2064/107118 objects misplaced (1.927%); 3.9 MiB/s, 61 objects/s recovering 2024-06-27T21:19:26.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:25 smithi190 ceph-mon[32247]: pgmap v717: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.2 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 1953/107118 objects degraded (1.823%); 2064/107118 objects misplaced (1.927%); 3.9 MiB/s, 61 objects/s recovering 2024-06-27T21:19:26.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:26.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:26.386 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:26.389 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:26.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:26.420 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:26.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:26.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:26.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:26.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:26.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:26.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:26.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:26.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:26.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:27.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:27.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:27.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:19:27.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:27.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:27.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:27.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:27.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:27.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:27.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:27.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:27.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:27.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:27.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:27.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:27.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:27.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:27.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:28.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:28.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:27 smithi192 ceph-mon[32956]: pgmap v718: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 517 op/s; 1953/102270 objects degraded (1.910%); 2064/102270 objects misplaced (2.018%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:19:28.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:28.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:28.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:27 smithi151 ceph-mon[28965]: pgmap v718: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 517 op/s; 1953/102270 objects degraded (1.910%); 2064/102270 objects misplaced (2.018%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:19:28.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:28.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:28.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:28.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:28.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:28.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:27 smithi190 ceph-mon[32247]: pgmap v718: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 517 op/s; 1953/102270 objects degraded (1.910%); 2064/102270 objects misplaced (2.018%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:19:28.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:28.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:28.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:28.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:28.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:28.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:28.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:28.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:28.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:28.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:28.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:28.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:28 smithi151 ceph-mon[28965]: osdmap e259: 12 total, 12 up, 11 in 2024-06-27T21:19:28.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:28.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:28.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:29.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:28 smithi192 ceph-mon[32956]: osdmap e259: 12 total, 12 up, 11 in 2024-06-27T21:19:29.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:29.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:29.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:29.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:28 smithi190 ceph-mon[32247]: osdmap e259: 12 total, 12 up, 11 in 2024-06-27T21:19:29.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:29.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:29.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:29.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:29.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:29.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:29.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:29.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:29.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:29 smithi151 ceph-mon[28965]: pgmap v720: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 1953/97458 objects degraded (2.004%); 1881/97458 objects misplaced (1.930%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:29.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:29.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:29.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1953/97458 objects degraded (2.004%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:29.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:29.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:29.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:30.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:29 smithi192 ceph-mon[32956]: pgmap v720: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 1953/97458 objects degraded (2.004%); 1881/97458 objects misplaced (1.930%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:30.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:30.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:30.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1953/97458 objects degraded (2.004%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:30.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:30.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:30.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:30.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:30.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:29 smithi190 ceph-mon[32247]: pgmap v720: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 593 op/s; 1953/97458 objects degraded (2.004%); 1881/97458 objects misplaced (1.930%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:30.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:30.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:30.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1953/97458 objects degraded (2.004%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:30.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:30.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:30.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:30.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:30.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:30.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:30.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:30.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:30.784 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-06-27T21:19:30.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:30.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:30.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:31.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:31.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3368078271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:19:31.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:30 smithi151 ceph-mon[28965]: osdmap e260: 12 total, 12 up, 11 in 2024-06-27T21:19:31.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:31.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3368078271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:19:31.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:30 smithi192 ceph-mon[32956]: osdmap e260: 12 total, 12 up, 11 in 2024-06-27T21:19:31.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:31.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:31.255 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:19:31.255 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:19:31.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3368078271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:19:31.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:30 smithi190 ceph-mon[32247]: osdmap e260: 12 total, 12 up, 11 in 2024-06-27T21:19:31.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:31.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:31.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:31.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:31.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:31.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:31.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:31.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:31.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:31.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:31.900 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:31.907 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:19:31.907 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:19:31.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:31.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:19:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:19:31.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:31.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:31.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:19:31.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:19:31.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:19:31.992 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:19:32.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:31 smithi151 ceph-mon[28965]: pgmap v722: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 768 op/s; 1953/92871 objects degraded (2.103%); 1881/92871 objects misplaced (2.025%); 1.8 MiB/s, 29 objects/s recovering 2024-06-27T21:19:32.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3368078271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:19:32.017 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:31 smithi151 ceph-mon[28965]: osdmap e261: 12 total, 12 up, 11 in 2024-06-27T21:19:32.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:19:32.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:19:32.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:31 smithi192 ceph-mon[32956]: pgmap v722: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 768 op/s; 1953/92871 objects degraded (2.103%); 1881/92871 objects misplaced (2.025%); 1.8 MiB/s, 29 objects/s recovering 2024-06-27T21:19:32.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3368078271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:19:32.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:31 smithi192 ceph-mon[32956]: osdmap e261: 12 total, 12 up, 11 in 2024-06-27T21:19:32.053 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:19:32.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:19:32.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:32.079 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:19:32.080 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:19:32.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:32.110 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:19:32.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:19:32.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:19:32.139 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:19:32.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:19:32.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:19:32.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:32.202 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:19:32.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:19:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:32.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:32.232 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:19:32.232 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:19:32.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:31 smithi190 ceph-mon[32247]: pgmap v722: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 768 op/s; 1953/92871 objects degraded (2.103%); 1881/92871 objects misplaced (2.025%); 1.8 MiB/s, 29 objects/s recovering 2024-06-27T21:19:32.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3368078271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:19:32.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:31 smithi190 ceph-mon[32247]: osdmap e261: 12 total, 12 up, 11 in 2024-06-27T21:19:32.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:32.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:32.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:32.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:32.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:32.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:32.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:32.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:32.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:32.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:32.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:32.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 9 1 2024-06-27T21:19:32.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:32.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:33.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1602866753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:33.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1602866753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:33.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:33.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:33.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:33.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:33.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1602866753' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:33.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:33.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:33.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:33.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:33.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:33.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:33.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:33.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:19:33.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:33.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:33.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:33.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:33.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:33 smithi151 ceph-mon[28965]: pgmap v724: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 1953/90684 objects degraded (2.154%); 1881/90684 objects misplaced (2.074%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:19:33.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:33.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:33.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:33 smithi151 ceph-mon[28965]: osdmap e262: 12 total, 12 up, 11 in 2024-06-27T21:19:33.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:34.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:34.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:33 smithi192 ceph-mon[32956]: pgmap v724: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 1953/90684 objects degraded (2.154%); 1881/90684 objects misplaced (2.074%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:19:34.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:34.041 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:34.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:33 smithi192 ceph-mon[32956]: osdmap e262: 12 total, 12 up, 11 in 2024-06-27T21:19:34.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:34.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:34.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:34.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:33 smithi190 ceph-mon[32247]: pgmap v724: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 1953/90684 objects degraded (2.154%); 1881/90684 objects misplaced (2.074%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:19:34.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:34.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:34.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:33 smithi190 ceph-mon[32247]: osdmap e262: 12 total, 12 up, 11 in 2024-06-27T21:19:34.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:34.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:34.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:34.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:34.531 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:34.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:34.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:34.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:34.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:34.876 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:34.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:34.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:34.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:34.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1953/90684 objects degraded (2.154%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:35.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:35.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:35.161 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1953/90684 objects degraded (2.154%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:35.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:35.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1953/90684 objects degraded (2.154%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:35.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:35.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:35.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:35.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:35.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:35.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:35.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:35.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:35.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:35.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:35.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:35.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:35 smithi151 ceph-mon[28965]: pgmap v726: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.7 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 685 op/s; 1970/85125 objects degraded (2.314%); 1714/85125 objects misplaced (2.014%); 1.6 MiB/s, 24 objects/s recovering 2024-06-27T21:19:35.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:35.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:36.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:35 smithi190 ceph-mon[32247]: pgmap v726: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.7 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 685 op/s; 1970/85125 objects degraded (2.314%); 1714/85125 objects misplaced (2.014%); 1.6 MiB/s, 24 objects/s recovering 2024-06-27T21:19:36.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:36.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:36.168 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:36.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:36.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:36.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:35 smithi192 ceph-mon[32956]: pgmap v726: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.7 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 685 op/s; 1970/85125 objects degraded (2.314%); 1714/85125 objects misplaced (2.014%); 1.6 MiB/s, 24 objects/s recovering 2024-06-27T21:19:36.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:36.355 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:36.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:36.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:36.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:19:36.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:36.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:36.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:36.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:36.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:36.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:36.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:36.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:37.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:37.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:37.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:36 smithi190 ceph-mon[32247]: pgmap v727: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.6 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 792 op/s; 1970/78282 objects degraded (2.517%); 1714/78282 objects misplaced (2.190%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:37.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:37.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:36 smithi151 ceph-mon[28965]: pgmap v727: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.6 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 792 op/s; 1970/78282 objects degraded (2.517%); 1714/78282 objects misplaced (2.190%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:37.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:37.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:37.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:36 smithi192 ceph-mon[32956]: pgmap v727: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.6 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 792 op/s; 1970/78282 objects degraded (2.517%); 1714/78282 objects misplaced (2.190%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:37.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:37.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:37.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:37.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:37.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:37.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:37.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:37.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:37.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:37.863 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:19:37.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:37.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:37.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:38.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:38.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:38.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:38.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2120868348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:19:38.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:38.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2120868348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:19:38.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2120868348' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:19:38.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:38.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:38.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:38.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:38.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:38.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:38.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:38.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:38.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:38.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:38.865 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-06-27T21:19:38.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:38 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:38.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2120868348' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:19:38.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:38 smithi151 ceph-mon[28965]: osdmap e263: 12 total, 12 up, 11 in 2024-06-27T21:19:38.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:38 smithi151 ceph-mon[28965]: pgmap v729: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.5 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 818 op/s; 2045/73161 objects degraded (2.795%); 1466/73161 objects misplaced (2.004%); 2.8 MiB/s, 45 objects/s recovering 2024-06-27T21:19:38.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2783529051' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:19:38.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:38.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:38.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:39.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:39.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:39.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:39.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:39.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:38 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:39.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2120868348' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:19:39.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:38 smithi190 ceph-mon[32247]: osdmap e263: 12 total, 12 up, 11 in 2024-06-27T21:19:39.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:38 smithi190 ceph-mon[32247]: pgmap v729: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.5 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 818 op/s; 2045/73161 objects degraded (2.795%); 1466/73161 objects misplaced (2.004%); 2.8 MiB/s, 45 objects/s recovering 2024-06-27T21:19:39.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2783529051' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:19:39.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:38 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:19:39.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2120868348' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:19:39.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:38 smithi192 ceph-mon[32956]: osdmap e263: 12 total, 12 up, 11 in 2024-06-27T21:19:39.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:38 smithi192 ceph-mon[32956]: pgmap v729: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.5 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 818 op/s; 2045/73161 objects degraded (2.795%); 1466/73161 objects misplaced (2.004%); 2.8 MiB/s, 45 objects/s recovering 2024-06-27T21:19:39.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2783529051' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:19:39.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:39.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:39.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:39.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:39.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:39.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:39.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:39.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:39.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:39.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:39.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:39.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:40.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:40.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:40.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:40.176 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2783529051' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:19:40.176 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:39 smithi151 ceph-mon[28965]: osdmap e264: 12 total, 12 up, 11 in 2024-06-27T21:19:40.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2045/73161 objects degraded (2.795%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:40.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:40.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2783529051' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:19:40.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:39 smithi190 ceph-mon[32247]: osdmap e264: 12 total, 12 up, 11 in 2024-06-27T21:19:40.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2045/73161 objects degraded (2.795%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:40.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:40.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2783529051' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:19:40.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:39 smithi192 ceph-mon[32956]: osdmap e264: 12 total, 12 up, 11 in 2024-06-27T21:19:40.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2045/73161 objects degraded (2.795%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:40.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:40.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:40.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:40.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:40.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:40.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:40.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:40.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:40.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:40.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:40.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:40.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:40.991 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:41.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:41.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 10 1 2024-06-27T21:19:41.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:41.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:41.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:41.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:40 smithi151 ceph-mon[28965]: pgmap v731: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 2045/68679 objects degraded (2.978%); 1466/68679 objects misplaced (2.135%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:41.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:40 smithi190 ceph-mon[32247]: pgmap v731: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 2045/68679 objects degraded (2.978%); 1466/68679 objects misplaced (2.135%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:41.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:41.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:41.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:40 smithi192 ceph-mon[32956]: pgmap v731: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 2045/68679 objects degraded (2.978%); 1466/68679 objects misplaced (2.135%); 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:19:41.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:41.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:41.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:41.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:41.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:41.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:41.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:41.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:41.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:41.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:41.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:41.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:42.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:42.059 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:42.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:42.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:42.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:41 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:19:42.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:41 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:19:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:42.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:19:42.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:41 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:19:42.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:41 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:19:42.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:41 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:19:42.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:41 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:19:42.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:42.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:42.334 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:19:42.334 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:19:42.334 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:19:42.334 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:19:42.334 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:19:42.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:42.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:42.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:42.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:42.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:42.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:42.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:42.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:42.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:42.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:42.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:43.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:43.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:43.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:43.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:42 smithi151 ceph-mon[28965]: pgmap v732: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 2045/66537 objects degraded (3.073%); 1466/66537 objects misplaced (2.203%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:19:43.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:43.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:43.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:43.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:43.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:42 smithi190 ceph-mon[32247]: pgmap v732: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 2045/66537 objects degraded (3.073%); 1466/66537 objects misplaced (2.203%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:19:43.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:43.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:43.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:42 smithi192 ceph-mon[32956]: pgmap v732: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.4 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 2045/66537 objects degraded (3.073%); 1466/66537 objects misplaced (2.203%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:19:43.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:43.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:43.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:43.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:43.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:43.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:43.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:43.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:43.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:43.724 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:43.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:43.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:43.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:19:43.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:43.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:43.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:44.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:44.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:44.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: osdmap e265: 12 total, 12 up, 11 in 2024-06-27T21:19:44.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:44.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:44.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=960500283,config=2353167246,health=20687419,kv=2436679606}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3967100284,logm=620268559}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3837063368}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371099964}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290448443}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2989996166}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299397841}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442493185}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228786262}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1988398456}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162094709}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078881474}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=214256230}) 2024-06-27T21:19:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=363527573}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2555030245}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642410116}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263319353}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094684139}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1364347885}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753127657}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1741382414}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589338430}) 2024-06-27T21:19:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:43 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1160670114}) 2024-06-27T21:19:44.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:44.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:44.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: osdmap e265: 12 total, 12 up, 11 in 2024-06-27T21:19:44.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:44.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:44.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=960500283,config=2353167246,health=20687419,kv=2436679606}) 2024-06-27T21:19:44.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3967100284,logm=620268559}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3837063368}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371099964}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290448443}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2989996166}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299397841}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442493185}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228786262}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1988398456}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162094709}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078881474}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=214256230}) 2024-06-27T21:19:44.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=363527573}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2555030245}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642410116}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263319353}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094684139}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1364347885}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753127657}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1741382414}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589338430}) 2024-06-27T21:19:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:43 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1160670114}) 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: osdmap e265: 12 total, 12 up, 11 in 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=960500283,config=2353167246,health=20687419,kv=2436679606}) 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=3967100284,logm=620268559}) 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3837063368}) 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371099964}) 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290448443}) 2024-06-27T21:19:44.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2989996166}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299397841}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442493185}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=228786262}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1988398456}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2162094709}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4078881474}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=214256230}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=363527573}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2555030245}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642410116}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263319353}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2094684139}) 2024-06-27T21:19:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1364347885}) 2024-06-27T21:19:44.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3753127657}) 2024-06-27T21:19:44.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1741382414}) 2024-06-27T21:19:44.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589338430}) 2024-06-27T21:19:44.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:43 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1160670114}) 2024-06-27T21:19:44.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:44.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:44.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:44.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:44.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:44.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:44.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:44.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:44.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:44.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:44.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:44.899 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:44.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:19:44.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:19:44.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:19:44.935 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:19:44.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:19:44.966 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:19:44.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:44.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:19:44.992 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:19:45.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:45.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:19:45.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181071531}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900236352}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2770152034}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: osdmap e266: 12 total, 12 up, 11 in 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925604649}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448139421}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386674028}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940181037}) 2024-06-27T21:19:45.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=513218317}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=196068837}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=58,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=3} crc {logm=2748752938,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=3646045347,mon_config_key=132289795}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1888713136}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2902841030}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=88,monmap=5,osd_metadata=7} crc {mon_config_key=2301871123,monmap=3596043196,osd_metadata=3269652088}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc {osd_metadata=2322083233,osd_pg_creating=3866290315,osdmap=612377506}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=1303086855}) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: pgmap v735: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 2072/61377 objects degraded (3.376%); 1237/61377 objects misplaced (2.015%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2045/66537 objects degraded (3.073%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: 3.6 scrub starts 2024-06-27T21:19:45.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3073012056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:45.061 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:19:45.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:19:45.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:19:45.088 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:19:45.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:45.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:45.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181071531}) 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900236352}) 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2770152034}) 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: osdmap e266: 12 total, 12 up, 11 in 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925604649}) 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448139421}) 2024-06-27T21:19:45.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386674028}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940181037}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=513218317}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=196068837}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=58,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=3} crc {logm=2748752938,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=3646045347,mon_config_key=132289795}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1888713136}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2902841030}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=88,monmap=5,osd_metadata=7} crc {mon_config_key=2301871123,monmap=3596043196,osd_metadata=3269652088}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc {osd_metadata=2322083233,osd_pg_creating=3866290315,osdmap=612377506}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=1303086855}) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: pgmap v735: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 2072/61377 objects degraded (3.376%); 1237/61377 objects misplaced (2.015%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2045/66537 objects degraded (3.073%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: 3.6 scrub starts 2024-06-27T21:19:45.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3073012056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:45.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:45.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1181071531}) 2024-06-27T21:19:45.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900236352}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2770152034}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: osdmap e266: 12 total, 12 up, 11 in 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925604649}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2448139421}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2386674028}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=940181037}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=513218317}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=196068837}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=58,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=3} crc {logm=2748752938,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=3646045347,mon_config_key=132289795}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1888713136}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2902841030}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=88,monmap=5,osd_metadata=7} crc {mon_config_key=2301871123,monmap=3596043196,osd_metadata=3269652088}) 2024-06-27T21:19:45.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc {osd_metadata=2322083233,osd_pg_creating=3866290315,osdmap=612377506}) 2024-06-27T21:19:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=1303086855}) 2024-06-27T21:19:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: pgmap v735: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 2072/61377 objects degraded (3.376%); 1237/61377 objects misplaced (2.015%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:19:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2045/66537 objects degraded (3.073%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: 3.6 scrub starts 2024-06-27T21:19:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3073012056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:45.316 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:19:45.316 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:19:45.342 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:19:45.342 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:19:45.365 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:19:45.365 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:19:45.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:45.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:45.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:19:45.393 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:19:45.415 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:19:45.415 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:19:45.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:45.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:45.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:45.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:45.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:45.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:45.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:45.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:45.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:46.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:46.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:46.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:46.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:46.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:45 smithi151 ceph-mon[28965]: 3.6 scrub ok 2024-06-27T21:19:46.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:45 smithi151 ceph-mon[28965]: 3.1 scrub starts 2024-06-27T21:19:46.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:45 smithi190 ceph-mon[32247]: 3.6 scrub ok 2024-06-27T21:19:46.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:45 smithi190 ceph-mon[32247]: 3.1 scrub starts 2024-06-27T21:19:46.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:46.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:45 smithi192 ceph-mon[32956]: 3.6 scrub ok 2024-06-27T21:19:46.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:45 smithi192 ceph-mon[32956]: 3.1 scrub starts 2024-06-27T21:19:46.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:46.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:46.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:46.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:46.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:46.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:46.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:46.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:46.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:46.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:46.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:46.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:47.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:47.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:47.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:46 smithi151 ceph-mon[28965]: 3.1 scrub ok 2024-06-27T21:19:47.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:46 smithi151 ceph-mon[28965]: pgmap v736: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.1 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 2072/55380 objects degraded (3.741%); 1237/55380 objects misplaced (2.234%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:19:47.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:46 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:19:47.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:46 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:19:47.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:47.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:47.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:46 smithi190 ceph-mon[32247]: 3.1 scrub ok 2024-06-27T21:19:47.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:46 smithi190 ceph-mon[32247]: pgmap v736: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.1 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 2072/55380 objects degraded (3.741%); 1237/55380 objects misplaced (2.234%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:19:47.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:46 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:19:47.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:46 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:19:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:46 smithi192 ceph-mon[32956]: 3.1 scrub ok 2024-06-27T21:19:47.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:46 smithi192 ceph-mon[32956]: pgmap v736: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.1 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 2072/55380 objects degraded (3.741%); 1237/55380 objects misplaced (2.234%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:19:47.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:46 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:19:47.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:46 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:19:47.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:47.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:47.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:47.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:47.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:47.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:47.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:47.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:47.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:47.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:47.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:47.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:48.003 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:48.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:48.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:48.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:47 smithi151 ceph-mon[28965]: 3.e scrub starts 2024-06-27T21:19:48.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:48.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:47 smithi190 ceph-mon[32247]: 3.e scrub starts 2024-06-27T21:19:48.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:48.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:47 smithi192 ceph-mon[32956]: 3.e scrub starts 2024-06-27T21:19:48.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:48.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:48.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:48.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:48.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:48.646 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:48.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:48.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:48.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:48.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:48.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:48.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:49.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:49.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:49.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:49.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:49.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:48 smithi151 ceph-mon[28965]: 3.e scrub ok 2024-06-27T21:19:49.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:48 smithi151 ceph-mon[28965]: pgmap v737: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.0 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 794 op/s; 2106/49596 objects degraded (4.246%); 988/49596 objects misplaced (1.992%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:49.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:48 smithi190 ceph-mon[32247]: 3.e scrub ok 2024-06-27T21:19:49.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:48 smithi190 ceph-mon[32247]: pgmap v737: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.0 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 794 op/s; 2106/49596 objects degraded (4.246%); 988/49596 objects misplaced (1.992%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:49.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:49.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:48 smithi192 ceph-mon[32956]: 3.e scrub ok 2024-06-27T21:19:49.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:48 smithi192 ceph-mon[32956]: pgmap v737: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.0 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 794 op/s; 2106/49596 objects degraded (4.246%); 988/49596 objects misplaced (1.992%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:49.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:49.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:49.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:49.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:49.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:19:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:49.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:49.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:49.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:49.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:49.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:49.920 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-06-27T21:19:49.920 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:19:49.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:49.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:49.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:50.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:50.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:50.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:50.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1279733846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:50.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:50.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:50.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2106/49596 objects degraded (4.246%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:50.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2463086630' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:19:50.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:50.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1279733846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:50.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:50.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:50.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2106/49596 objects degraded (4.246%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:50.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2463086630' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:19:50.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1279733846' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:19:50.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:50.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:50.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2106/49596 objects degraded (4.246%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:50.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2463086630' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:19:50.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:50.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:50.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:50.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:50.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:50.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:50.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:50.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:50.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:50.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:50.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:50.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:50.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 11 1 2024-06-27T21:19:50.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:50.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:50.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:51.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:51.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:51.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:50 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:19:51.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1279733846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:51.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2463086630' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:19:51.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:50 smithi151 ceph-mon[28965]: osdmap e267: 12 total, 12 up, 11 in 2024-06-27T21:19:51.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:50 smithi151 ceph-mon[28965]: pgmap v738: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 947 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 878 op/s; 2106/45456 objects degraded (4.633%); 988/45456 objects misplaced (2.174%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:51.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:51.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:51.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:50 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:19:51.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1279733846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:51.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2463086630' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:19:51.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:50 smithi190 ceph-mon[32247]: osdmap e267: 12 total, 12 up, 11 in 2024-06-27T21:19:51.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:50 smithi190 ceph-mon[32247]: pgmap v738: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 947 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 878 op/s; 2106/45456 objects degraded (4.633%); 988/45456 objects misplaced (2.174%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:51.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:50 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:19:51.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1279733846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:19:51.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2463086630' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:19:51.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:50 smithi192 ceph-mon[32956]: osdmap e267: 12 total, 12 up, 11 in 2024-06-27T21:19:51.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:50 smithi192 ceph-mon[32956]: pgmap v738: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 947 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 878 op/s; 2106/45456 objects degraded (4.633%); 988/45456 objects misplaced (2.174%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:19:51.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:51.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:51.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:51.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:51.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:51.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:51.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:51.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:51.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:51.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:51.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:51.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:51 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:19:51.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:51 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:19:52.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:52.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:52.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:52.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:52.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:51 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:19:52.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:51 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:19:52.271 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:52.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:52.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:51 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:19:52.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:51 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:19:52.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:52.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:52.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:52.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:52.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:52.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:52.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:52.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:52.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:52.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:52.904 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:52.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:19:52.917 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:19:52.917 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:19:52.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:52.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:19:53.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:53.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:52 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:53.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:52 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:53.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:52 smithi151 ceph-mon[28965]: osdmap e268: 12 total, 12 up, 11 in 2024-06-27T21:19:53.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:52 smithi151 ceph-mon[28965]: pgmap v741: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 903 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 2106/43311 objects degraded (4.863%); 988/43311 objects misplaced (2.281%); 1.7 MiB/s, 26 objects/s recovering 2024-06-27T21:19:53.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:53.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:53.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:53.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:52 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:53.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:52 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:53.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:52 smithi190 ceph-mon[32247]: osdmap e268: 12 total, 12 up, 11 in 2024-06-27T21:19:53.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:52 smithi190 ceph-mon[32247]: pgmap v741: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 903 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 2106/43311 objects degraded (4.863%); 988/43311 objects misplaced (2.281%); 1.7 MiB/s, 26 objects/s recovering 2024-06-27T21:19:53.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:52 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:53.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:52 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:53.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:52 smithi192 ceph-mon[32956]: osdmap e268: 12 total, 12 up, 11 in 2024-06-27T21:19:53.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:52 smithi192 ceph-mon[32956]: pgmap v741: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 903 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 2106/43311 objects degraded (4.863%); 988/43311 objects misplaced (2.281%); 1.7 MiB/s, 26 objects/s recovering 2024-06-27T21:19:53.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:53.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:53.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:53.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:53.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:53.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:53.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:53.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:53.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:19:53.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:19:53.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:53.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:53.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:53.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:53.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:54.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:54.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:54.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:54.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:54.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:54.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:54.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:54.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:54.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:54.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:54.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:54.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:54.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:54.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:54.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:54.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:54.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:55.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:55.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:55.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:54 smithi151 ceph-mon[28965]: pgmap v742: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 800 MiB data, 3.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 2123/38376 objects degraded (5.532%); 758/38376 objects misplaced (1.975%); 3.3 MiB/s, 53 objects/s recovering 2024-06-27T21:19:55.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:54 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:19:55.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:54 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:19:55.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2123/38376 objects degraded (5.532%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:55.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:55.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:55.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:54 smithi190 ceph-mon[32247]: pgmap v742: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 800 MiB data, 3.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 2123/38376 objects degraded (5.532%); 758/38376 objects misplaced (1.975%); 3.3 MiB/s, 53 objects/s recovering 2024-06-27T21:19:55.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:54 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:19:55.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:54 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:19:55.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2123/38376 objects degraded (5.532%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:55.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:54 smithi192 ceph-mon[32956]: pgmap v742: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 800 MiB data, 3.9 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 2123/38376 objects degraded (5.532%); 758/38376 objects misplaced (1.975%); 3.3 MiB/s, 53 objects/s recovering 2024-06-27T21:19:55.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:54 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:19:55.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:54 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:19:55.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2123/38376 objects degraded (5.532%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:19:55.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:55.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:55.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:19:55.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:19:55.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:55.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:55.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:19:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:55.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:19:55.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:55.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:55.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:55.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:56.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:56.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:56.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:56.177 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:56.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:56.257 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:55 smithi151 ceph-mon[28965]: osdmap e269: 12 total, 12 up, 11 in 2024-06-27T21:19:56.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:55 smithi190 ceph-mon[32247]: osdmap e269: 12 total, 12 up, 11 in 2024-06-27T21:19:56.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:56.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:55 smithi192 ceph-mon[32956]: osdmap e269: 12 total, 12 up, 11 in 2024-06-27T21:19:56.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:56.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:56.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:56.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:56.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:56.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:56.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:56.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:56.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:56.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:56.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:57.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:57.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:57.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:56 smithi151 ceph-mon[28965]: pgmap v744: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 664 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 2123/31848 objects degraded (6.666%); 758/31848 objects misplaced (2.380%); 2.2 MiB/s, 35 objects/s recovering 2024-06-27T21:19:57.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:57.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:57.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:56 smithi190 ceph-mon[32247]: pgmap v744: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 664 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 2123/31848 objects degraded (6.666%); 758/31848 objects misplaced (2.380%); 2.2 MiB/s, 35 objects/s recovering 2024-06-27T21:19:57.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:56 smithi192 ceph-mon[32956]: pgmap v744: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 664 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 2123/31848 objects degraded (6.666%); 758/31848 objects misplaced (2.380%); 2.2 MiB/s, 35 objects/s recovering 2024-06-27T21:19:57.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:57.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:57.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:19:57.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:19:57.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:19:57.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:19:57.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:19:57.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:19:57.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:19:57.372 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:19:57.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:57.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:57.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:57.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:57.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:57.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:57.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:19:57.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:19:57.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:57.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:57.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:58.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:58.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:58.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/492693112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:58.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:19:58.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:19:58.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/492693112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:58.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/492693112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:19:58.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:58.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:58.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:58.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:58.518 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:19:58.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:19:58.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:19:58.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:19:58.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:19:58.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:58.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:58.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:19:58.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:58.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:19:59.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:58 smithi151 ceph-mon[28965]: pgmap v745: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 556 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 2161/26646 objects degraded (8.110%); 483/26646 objects misplaced (1.813%); 3.5 MiB/s, 56 objects/s recovering 2024-06-27T21:19:59.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:19:59.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:19:59.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:59.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:59.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:58 smithi190 ceph-mon[32247]: pgmap v745: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 556 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 2161/26646 objects degraded (8.110%); 483/26646 objects misplaced (1.813%); 3.5 MiB/s, 56 objects/s recovering 2024-06-27T21:19:59.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:58 smithi192 ceph-mon[32956]: pgmap v745: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 556 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 2161/26646 objects degraded (8.110%); 483/26646 objects misplaced (1.813%); 3.5 MiB/s, 56 objects/s recovering 2024-06-27T21:19:59.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:59.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:59.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:19:59.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:19:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:19:59.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:19:59.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:19:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:19:59.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:59.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:59.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:19:59.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:19:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:19:59.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:19:59.963 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-06-27T21:19:59.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:00.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:00.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:00.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1733283404' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:20:00.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:19:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2161/26646 objects degraded (8.110%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:00.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:00.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:20:00.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:00.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1733283404' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:20:00.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:19:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2161/26646 objects degraded (8.110%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:00.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:00.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1733283404' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:20:00.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:19:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2161/26646 objects degraded (8.110%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:00.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:00.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:00.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:00.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:00.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:00.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:00.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:00.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:00.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:00.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:00.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:01.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:01.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:01.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:01.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:01.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:00 smithi151 ceph-mon[28965]: pgmap v746: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 464 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 2161/22251 objects degraded (9.712%); 483/22251 objects misplaced (2.171%); 3.5 MiB/s, 56 objects/s recovering 2024-06-27T21:20:01.194 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1733283404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:20:01.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:00 smithi151 ceph-mon[28965]: osdmap e270: 12 total, 12 up, 11 in 2024-06-27T21:20:01.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:00 smithi151 ceph-mon[28965]: Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:20:01.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:00 smithi151 ceph-mon[28965]: [WRN] PG_DEGRADED: Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:20:01.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:00 smithi151 ceph-mon[28965]: pg 3.5 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,3] 2024-06-27T21:20:01.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:00 smithi190 ceph-mon[32247]: pgmap v746: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 464 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 2161/22251 objects degraded (9.712%); 483/22251 objects misplaced (2.171%); 3.5 MiB/s, 56 objects/s recovering 2024-06-27T21:20:01.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1733283404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:20:01.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:00 smithi190 ceph-mon[32247]: osdmap e270: 12 total, 12 up, 11 in 2024-06-27T21:20:01.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:00 smithi190 ceph-mon[32247]: Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:20:01.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:00 smithi190 ceph-mon[32247]: [WRN] PG_DEGRADED: Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:20:01.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:00 smithi190 ceph-mon[32247]: pg 3.5 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,3] 2024-06-27T21:20:01.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:01.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:00 smithi192 ceph-mon[32956]: pgmap v746: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 464 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 877 op/s; 2161/22251 objects degraded (9.712%); 483/22251 objects misplaced (2.171%); 3.5 MiB/s, 56 objects/s recovering 2024-06-27T21:20:01.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1733283404' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:20:01.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:00 smithi192 ceph-mon[32956]: osdmap e270: 12 total, 12 up, 11 in 2024-06-27T21:20:01.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:00 smithi192 ceph-mon[32956]: Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:20:01.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:00 smithi192 ceph-mon[32956]: [WRN] PG_DEGRADED: Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:20:01.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:00 smithi192 ceph-mon[32956]: pg 3.5 is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [5,3] 2024-06-27T21:20:01.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:01.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:01.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:01.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:01.569 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:01.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:01.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:01.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:01.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:01.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:01.945 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-27T21:20:01.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:20:01.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:01.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:01.979 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:20:02.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:02.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:02.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:02.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3421802029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:02.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:01 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:02.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:02.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3421802029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:02.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:01 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:02.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3421802029' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:02.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:01 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:02.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:02.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:02.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:02.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:02.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:02.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:02.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:02.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:02.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:02.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:02.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:02.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:02.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:03.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:02 smithi151 ceph-mon[28965]: pgmap v748: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 417 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2161/20004 objects degraded (10.803%); 483/20004 objects misplaced (2.415%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:20:03.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:02 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:03.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:02 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:20:03.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:02 smithi151 ceph-mon[28965]: osdmap e271: 12 total, 12 up, 11 in 2024-06-27T21:20:03.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:03.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:03.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:03.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:03.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:02 smithi190 ceph-mon[32247]: pgmap v748: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 417 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2161/20004 objects degraded (10.803%); 483/20004 objects misplaced (2.415%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:20:03.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:02 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:03.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:02 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:20:03.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:02 smithi190 ceph-mon[32247]: osdmap e271: 12 total, 12 up, 11 in 2024-06-27T21:20:03.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:02 smithi192 ceph-mon[32956]: pgmap v748: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 417 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2161/20004 objects degraded (10.803%); 483/20004 objects misplaced (2.415%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:20:03.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:02 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:03.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:02 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:20:03.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:02 smithi192 ceph-mon[32956]: osdmap e271: 12 total, 12 up, 11 in 2024-06-27T21:20:03.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:03.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:03.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:03.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:03.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:03.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.15132.0:210390 3.5 e271/227 3:ae30801e:::benchmark_data_smithi192_74314_object103752:head v 271'13426, mlcod=224'6617)", 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:03.648974+0000", 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0097292409999999996, 2024-06-27T21:20:03.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.009756766, 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.648974+0000", 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.648974+0000", 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.648975+0000", 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7800000000000002e-07 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.648980+0000", 2024-06-27T21:20:03.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8069999999999999e-06 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.656115+0000", 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0071349719999999998 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.656122+0000", 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.029e-06 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.656172+0000", 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.079e-05 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.656191+0000", 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8270999999999998e-05 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:03.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:210404 3.2 3:4e71821b:::benchmark_data_smithi192_74314_object103766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:03.658186+0000", 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00051725600000000001, 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060446399999999998, 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:03.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 210404 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658186+0000", 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:03.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658186+0000", 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658187+0000", 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658191+0000", 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9439999999999998e-06 2024-06-27T21:20:03.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658199+0000", 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0749999999999998e-06 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658204+0000", 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4209999999999997e-06 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:03.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658246+0000", 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1899e-05 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658376+0000", 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012991500000000001 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.658502+0000", 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126344 2024-06-27T21:20:03.665 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:210384 3.2 3:41718bb5:::benchmark_data_smithi192_74314_object103746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:03.636584+0000", 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.022119883999999999, 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022258955, 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:03.666 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 210384 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636584+0000", 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636584+0000", 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:03.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636586+0000", 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9e-06 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636592+0000", 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6590000000000001e-06 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636608+0000", 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5342e-05 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636616+0000", 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1829999999999994e-06 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636669+0000", 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2981e-05 2024-06-27T21:20:03.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636775+0000", 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106087 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.636853+0000", 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8374999999999994e-05 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:03.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.641414+0000", 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045610690000000001 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:03.646144+0000", 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047298059999999996 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:20:03.671 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-06-27T21:20:03.672 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:20:03.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:03.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:03.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:03.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:03.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:04.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:04.038 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:04.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:04.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:04.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:04.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:04.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:04.291 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:04.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:04.419 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:04.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:04.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:04.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:20:04.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:04.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:04.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:04.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:04.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:04.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:05.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:05.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:05.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:20:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:04 smithi151 ceph-mon[28965]: pgmap v750: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 312 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 702 op/s; 2172/14976 objects degraded (14.503%); 240/14976 objects misplaced (1.603%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:20:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2172/14976 objects degraded (14.503%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:05.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:04 smithi151 ceph-mon[28965]: osdmap e272: 12 total, 12 up, 11 in 2024-06-27T21:20:05.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:05.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:05.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:04 smithi190 ceph-mon[32247]: pgmap v750: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 312 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 702 op/s; 2172/14976 objects degraded (14.503%); 240/14976 objects misplaced (1.603%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:20:05.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2172/14976 objects degraded (14.503%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:05.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:04 smithi190 ceph-mon[32247]: osdmap e272: 12 total, 12 up, 11 in 2024-06-27T21:20:05.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:04 smithi192 ceph-mon[32956]: pgmap v750: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 312 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 702 op/s; 2172/14976 objects degraded (14.503%); 240/14976 objects misplaced (1.603%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:20:05.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2172/14976 objects degraded (14.503%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:05.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:04 smithi192 ceph-mon[32956]: osdmap e272: 12 total, 12 up, 11 in 2024-06-27T21:20:05.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:05.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:05.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:05.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:05.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:05.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:05.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:05.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:05.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:05.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:05.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:05.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:05.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:06.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:06.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:06.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:06.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:06.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:06.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:06.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:06.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:06.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:06.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:06.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:06.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:06.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:06.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:06.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:06.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:07.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:07.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:07.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:07.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:06 smithi151 ceph-mon[28965]: pgmap v752: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 167 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 791 op/s; 2172/8007 objects degraded (27.126%); 240/8007 objects misplaced (2.997%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:20:07.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:07.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:06 smithi190 ceph-mon[32247]: pgmap v752: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 167 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 791 op/s; 2172/8007 objects degraded (27.126%); 240/8007 objects misplaced (2.997%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:20:07.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:07.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:06 smithi192 ceph-mon[32956]: pgmap v752: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 167 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 791 op/s; 2172/8007 objects degraded (27.126%); 240/8007 objects misplaced (2.997%); 2.4 MiB/s, 38 objects/s recovering 2024-06-27T21:20:07.356 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:20:07.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:07.378 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:20:07.378 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:20:07.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:07.399 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:20:07.400 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:20:07.421 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:20:07.421 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:20:07.444 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:20:07.444 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:20:07.465 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:20:07.466 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:20:07.487 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:20:07.487 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:20:07.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:07.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:20:07.509 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:20:07.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.516 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:07.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:20:07.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:20:07.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:20:07.551 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:20:07.572 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:20:07.572 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:20:07.594 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:20:07.594 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:20:07.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:07.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:07.705 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 106635 objects 2024-06-27T21:20:07.705 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :140.099 2024-06-27T21:20:07.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:07.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:07.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:07.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:07.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:07.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:08.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:08.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:08.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:08.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:08.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:08.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/641616825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:08.291 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/641616825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:08.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/641616825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:08.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:08.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:08.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:08.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:08.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:08.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:08.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:08.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:08.800 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:08.800 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:08.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:08.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:08.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:08.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:09.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:09.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:09.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:09.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:09.238 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:08 smithi151 ceph-mon[28965]: pgmap v753: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 64 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 803 op/s; 2084/3048 objects degraded (68.373%); 3.6 MiB/s, 70 objects/s recovering 2024-06-27T21:20:09.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:09.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:08 smithi190 ceph-mon[32247]: pgmap v753: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 64 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 803 op/s; 2084/3048 objects degraded (68.373%); 3.6 MiB/s, 70 objects/s recovering 2024-06-27T21:20:09.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:08 smithi192 ceph-mon[32956]: pgmap v753: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 64 MiB data, 1.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 803 op/s; 2084/3048 objects degraded (68.373%); 3.6 MiB/s, 70 objects/s recovering 2024-06-27T21:20:09.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:09.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:09.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:09.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:09.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:09.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:09.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:09.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:09.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:09.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:09.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:09.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:10.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:10.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:10.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:10.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:10.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:10.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2084/3048 objects degraded (68.373%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:10.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:09 smithi190 ceph-mon[32247]: osdmap e273: 12 total, 12 up, 11 in 2024-06-27T21:20:10.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2084/3048 objects degraded (68.373%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:10.292 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:09 smithi192 ceph-mon[32956]: osdmap e273: 12 total, 12 up, 11 in 2024-06-27T21:20:10.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2084/3048 objects degraded (68.373%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:10.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:09 smithi151 ceph-mon[28965]: osdmap e273: 12 total, 12 up, 11 in 2024-06-27T21:20:10.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:20:10.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:10.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:10.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:10.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:10.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:10.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:10.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:10.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:10.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:10.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:10.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:10.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:11.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:11.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:11.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:11.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:11.166 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:20:11.166 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:20:11.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:11.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:11.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:11.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:11.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:11.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:11.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:11.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:11.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:11 smithi151 ceph-mon[28965]: pgmap v755: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 14 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 811 op/s; 2084/627 objects degraded (332.376%); 3.6 MiB/s, 70 objects/s recovering 2024-06-27T21:20:11.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:11.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:11.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:11 smithi190 ceph-mon[32247]: pgmap v755: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 14 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 811 op/s; 2084/627 objects degraded (332.376%); 3.6 MiB/s, 70 objects/s recovering 2024-06-27T21:20:11.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:11 smithi192 ceph-mon[32956]: pgmap v755: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 14 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 811 op/s; 2084/627 objects degraded (332.376%); 3.6 MiB/s, 70 objects/s recovering 2024-06-27T21:20:11.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:11.886 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:11.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:11.917 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-06-27T21:20:09.933737+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":14156320,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213065,"num_write_kb":6824641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4720,"num_bytes_recovered":301339168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161699,"ondisk_log_size":161699,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1886616,"kb_used_data":104048,"kb_used_omap":17,"kb_used_meta":1782446,"kb_avail":1029310056,"statfs":{"total":1055945392128,"available":1054013497344,"internally_reserved":0,"allocated":106545152,"data_stored":103071714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1825225589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-423231512,"num_objects":-6459,"num_object_clones":0,"num_object_copies":-19377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":-483,"num_objects_unfound":0,"num_objects_dirty":-6459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6459,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":560,"num_bytes_recovered":30277632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.956958"},"pg_stats":[{"pgid":"3.f","version":"272'13214","reported_seq":13507,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.021990+0000","last_change":"2024-06-27T21:19:00.478810+0000","last_active":"2024-06-27T21:20:07.021990+0000","last_peered":"2024-06-27T21:20:07.021990+0000","last_clean":"2024-06-27T21:20:07.021990+0000","last_became_active":"2024-06-27T21:16:02.437136+0000","last_became_peered":"2024-06-27T21:16:02.437136+0000","last_unstale":"2024-06-27T21:20:07.021990+0000","last_undegraded":"2024-06-27T21:20:07.021990+0000","last_fullsized":"2024-06-27T21:20:07.021990+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10257","last_scrub_stamp":"2024-06-27T21:19:00.478347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:00.478347+0000","objects_scrubbed":2984,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:14.380809+0000","scrub_duration":1157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13214,"num_write_kb":422848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"272'15119","reported_seq":15461,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.081148+0000","last_change":"2024-06-27T21:19:47.365005+0000","last_active":"2024-06-27T21:20:07.081148+0000","last_peered":"2024-06-27T21:20:07.081148+0000","last_clean":"2024-06-27T21:20:07.081148+0000","last_became_active":"2024-06-27T21:16:02.436824+0000","last_became_peered":"2024-06-27T21:16:02.436824+0000","last_unstale":"2024-06-27T21:20:07.081148+0000","last_undegraded":"2024-06-27T21:20:07.081148+0000","last_fullsized":"2024-06-27T21:20:07.081148+0000","mapping_epoch":186,"log_start":"213'5101","ondisk_log_start":"213'5101","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'14175","last_scrub_stamp":"2024-06-27T21:19:47.364792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:47.364792+0000","objects_scrubbed":958,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:03.339363+0000","scrub_duration":459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13556,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"272'13534","reported_seq":13834,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.098351+0000","last_change":"2024-06-27T21:18:56.791587+0000","last_active":"2024-06-27T21:20:07.098351+0000","last_peered":"2024-06-27T21:20:07.098351+0000","last_clean":"2024-06-27T21:20:07.098351+0000","last_became_active":"2024-06-27T21:16:02.856526+0000","last_became_peered":"2024-06-27T21:16:02.856526+0000","last_unstale":"2024-06-27T21:20:07.098351+0000","last_undegraded":"2024-06-27T21:20:07.098351+0000","last_fullsized":"2024-06-27T21:20:07.098351+0000","mapping_epoch":186,"log_start":"205'3500","ondisk_log_start":"205'3500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10243","last_scrub_stamp":"2024-06-27T21:18:56.790801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:56.790801+0000","objects_scrubbed":3328,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:12.174408+0000","scrub_duration":1523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13534,"num_write_kb":433025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"272'13224","reported_seq":13477,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.100972+0000","last_change":"2024-06-27T21:17:39.000875+0000","last_active":"2024-06-27T21:20:07.100972+0000","last_peered":"2024-06-27T21:20:07.100972+0000","last_clean":"2024-06-27T21:20:07.100972+0000","last_became_active":"2024-06-27T21:16:02.439289+0000","last_became_peered":"2024-06-27T21:16:02.439289+0000","last_unstale":"2024-06-27T21:20:07.100972+0000","last_undegraded":"2024-06-27T21:20:07.100972+0000","last_fullsized":"2024-06-27T21:20:07.100972+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6283","last_scrub_stamp":"2024-06-27T21:17:39.000676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:39.000676+0000","objects_scrubbed":6223,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"272'13264","reported_seq":13561,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.082342+0000","last_change":"2024-06-27T21:18:54.798170+0000","last_active":"2024-06-27T21:20:07.082342+0000","last_peered":"2024-06-27T21:20:07.082342+0000","last_clean":"2024-06-27T21:20:07.082342+0000","last_became_active":"2024-06-27T21:16:02.435740+0000","last_became_peered":"2024-06-27T21:16:02.435740+0000","last_unstale":"2024-06-27T21:20:07.082342+0000","last_undegraded":"2024-06-27T21:20:07.082342+0000","last_fullsized":"2024-06-27T21:20:07.082342+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10086","last_scrub_stamp":"2024-06-27T21:18:54.797841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:54.797841+0000","objects_scrubbed":3199,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:22.934044+0000","scrub_duration":906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13264,"num_write_kb":424448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"272'13256","reported_seq":13511,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.076511+0000","last_change":"2024-06-27T21:17:36.697283+0000","last_active":"2024-06-27T21:20:07.076511+0000","last_peered":"2024-06-27T21:20:07.076511+0000","last_clean":"2024-06-27T21:20:07.076511+0000","last_became_active":"2024-06-27T21:16:02.856475+0000","last_became_peered":"2024-06-27T21:16:02.856475+0000","last_unstale":"2024-06-27T21:20:07.076511+0000","last_undegraded":"2024-06-27T21:20:07.076511+0000","last_fullsized":"2024-06-27T21:20:07.076511+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'6177","last_scrub_stamp":"2024-06-27T21:17:36.697124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:36.697124+0000","objects_scrubbed":6104,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13256,"num_write_kb":424192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":548,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.459728+0000","last_change":"2024-06-27T21:19:46.475053+0000","last_active":"2024-06-27T21:20:04.459728+0000","last_peered":"2024-06-27T21:20:04.459728+0000","last_clean":"2024-06-27T21:20:04.459728+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:20:04.459728+0000","last_undegraded":"2024-06-27T21:20:04.459728+0000","last_fullsized":"2024-06-27T21:20:04.459728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:46.474944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:46.474944+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:48.986062+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"272'13457","reported_seq":13768,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:05.742873+0000","last_change":"2024-06-27T21:19:45.738578+0000","last_active":"2024-06-27T21:20:05.742873+0000","last_peered":"2024-06-27T21:20:05.742873+0000","last_clean":"2024-06-27T21:20:05.742873+0000","last_became_active":"2024-06-27T21:16:03.380506+0000","last_became_peered":"2024-06-27T21:16:03.380506+0000","last_unstale":"2024-06-27T21:20:05.742873+0000","last_undegraded":"2024-06-27T21:20:05.742873+0000","last_fullsized":"2024-06-27T21:20:05.742873+0000","mapping_epoch":186,"log_start":"204'3400","ondisk_log_start":"204'3400","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'12465","last_scrub_stamp":"2024-06-27T21:19:45.738454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:45.738454+0000","objects_scrubbed":1059,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:56.490662+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13457,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":133,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.461782+0000","last_change":"2024-06-27T21:18:11.238129+0000","last_active":"2024-06-27T21:20:04.461782+0000","last_peered":"2024-06-27T21:20:04.461782+0000","last_clean":"2024-06-27T21:20:04.461782+0000","last_became_active":"2024-06-27T21:17:58.594952+0000","last_became_peered":"2024-06-27T21:17:58.594952+0000","last_unstale":"2024-06-27T21:20:04.461782+0000","last_undegraded":"2024-06-27T21:20:04.461782+0000","last_fullsized":"2024-06-27T21:20:04.461782+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:18:11.238027+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:18:11.238027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"272'13374","reported_seq":13668,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:06.999608+0000","last_change":"2024-06-27T21:19:01.088777+0000","last_active":"2024-06-27T21:20:06.999608+0000","last_peered":"2024-06-27T21:20:06.999608+0000","last_clean":"2024-06-27T21:20:06.999608+0000","last_became_active":"2024-06-27T21:16:02.856410+0000","last_became_peered":"2024-06-27T21:16:02.856410+0000","last_unstale":"2024-06-27T21:20:06.999608+0000","last_undegraded":"2024-06-27T21:20:06.999608+0000","last_fullsized":"2024-06-27T21:20:06.999608+0000","mapping_epoch":186,"log_start":"204'3300","ondisk_log_start":"204'3300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'10375","last_scrub_stamp":"2024-06-27T21:19:01.088693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:01.088693+0000","objects_scrubbed":3039,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:28.983057+0000","scrub_duration":1340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":552,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.459518+0000","last_change":"2024-06-27T21:19:04.347536+0000","last_active":"2024-06-27T21:20:04.459518+0000","last_peered":"2024-06-27T21:20:04.459518+0000","last_clean":"2024-06-27T21:20:04.459518+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:20:04.459518+0000","last_undegraded":"2024-06-27T21:20:04.459518+0000","last_fullsized":"2024-06-27T21:20:04.459518+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:04.347424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:04.347424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:04.901896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"272'12952","reported_seq":13240,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.102523+0000","last_change":"2024-06-27T21:19:06.671521+0000","last_active":"2024-06-27T21:20:07.102523+0000","last_peered":"2024-06-27T21:20:07.102523+0000","last_clean":"2024-06-27T21:20:07.102523+0000","last_became_active":"2024-06-27T21:16:02.437264+0000","last_became_peered":"2024-06-27T21:16:02.437264+0000","last_unstale":"2024-06-27T21:20:07.102523+0000","last_undegraded":"2024-06-27T21:20:07.102523+0000","last_fullsized":"2024-06-27T21:20:07.102523+0000","mapping_epoch":186,"log_start":"203'2900","ondisk_log_start":"203'2900","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'10309","last_scrub_stamp":"2024-06-27T21:19:06.671332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:06.671332+0000","objects_scrubbed":2661,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:19.196852+0000","scrub_duration":1167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12952,"num_write_kb":414464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":259,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.465669+0000","last_change":"2024-06-27T21:18:57.759488+0000","last_active":"2024-06-27T21:20:04.465669+0000","last_peered":"2024-06-27T21:20:04.465669+0000","last_clean":"2024-06-27T21:20:04.465669+0000","last_became_active":"2024-06-27T21:14:55.718653+0000","last_became_peered":"2024-06-27T21:14:55.718653+0000","last_unstale":"2024-06-27T21:20:04.465669+0000","last_undegraded":"2024-06-27T21:20:04.465669+0000","last_fullsized":"2024-06-27T21:20:04.465669+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:18:57.759379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:18:57.759379+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"272'13618","reported_seq":23313,"reported_epoch":272,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:20:07.648484+0000","last_change":"2024-06-27T21:17:48.680969+0000","last_active":"2024-06-27T21:20:07.648484+0000","last_peered":"2024-06-27T21:20:07.648484+0000","last_clean":"2024-06-27T21:17:44.498417+0000","last_became_active":"2024-06-27T21:17:46.964668+0000","last_became_peered":"2024-06-27T21:17:46.964668+0000","last_unstale":"2024-06-27T21:20:07.648484+0000","last_undegraded":"2024-06-27T21:17:48.677261+0000","last_fullsized":"2024-06-27T21:17:46.505450+0000","mapping_epoch":227,"log_start":"205'3600","ondisk_log_start":"205'3600","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'6285","last_scrub_stamp":"2024-06-27T21:17:35.788328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:35.788328+0000","objects_scrubbed":6220,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13618,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4719,"num_bytes_recovered":301137920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"3,5","objects":-2084}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":269,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.461743+0000","last_change":"2024-06-27T21:16:46.599756+0000","last_active":"2024-06-27T21:20:04.461743+0000","last_peered":"2024-06-27T21:20:04.461743+0000","last_clean":"2024-06-27T21:20:04.461743+0000","last_became_active":"2024-06-27T21:14:55.861874+0000","last_became_peered":"2024-06-27T21:14:55.861874+0000","last_unstale":"2024-06-27T21:20:04.461743+0000","last_undegraded":"2024-06-27T21:20:04.461743+0000","last_fullsized":"2024-06-27T21:20:04.461743+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:46.599672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:46.599672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"272'13548","reported_seq":13863,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.091920+0000","last_change":"2024-06-27T21:18:55.817183+0000","last_active":"2024-06-27T21:20:07.091920+0000","last_peered":"2024-06-27T21:20:07.091920+0000","last_clean":"2024-06-27T21:20:07.091920+0000","last_became_active":"2024-06-27T21:17:58.595091+0000","last_became_peered":"2024-06-27T21:17:58.595091+0000","last_unstale":"2024-06-27T21:20:07.091920+0000","last_undegraded":"2024-06-27T21:20:07.091920+0000","last_fullsized":"2024-06-27T21:20:07.091920+0000","mapping_epoch":230,"log_start":"204'3500","ondisk_log_start":"204'3500","created":186,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10303","last_scrub_stamp":"2024-06-27T21:18:55.817070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:55.817070+0000","objects_scrubbed":3265,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:13.237624+0000","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13548,"num_write_kb":433536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":259,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.462636+0000","last_change":"2024-06-27T21:19:00.785414+0000","last_active":"2024-06-27T21:20:04.462636+0000","last_peered":"2024-06-27T21:20:04.462636+0000","last_clean":"2024-06-27T21:20:04.462636+0000","last_became_active":"2024-06-27T21:14:55.711705+0000","last_became_peered":"2024-06-27T21:14:55.711705+0000","last_unstale":"2024-06-27T21:20:04.462636+0000","last_undegraded":"2024-06-27T21:20:04.462636+0000","last_fullsized":"2024-06-27T21:20:04.462636+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:00.785339+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:19:00.785339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"272'13246","reported_seq":13501,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.059339+0000","last_change":"2024-06-27T21:17:37.829478+0000","last_active":"2024-06-27T21:20:07.059339+0000","last_peered":"2024-06-27T21:20:07.059339+0000","last_clean":"2024-06-27T21:20:07.059339+0000","last_became_active":"2024-06-27T21:16:02.436035+0000","last_became_peered":"2024-06-27T21:16:02.436035+0000","last_unstale":"2024-06-27T21:20:07.059339+0000","last_undegraded":"2024-06-27T21:20:07.059339+0000","last_fullsized":"2024-06-27T21:20:07.059339+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6242","last_scrub_stamp":"2024-06-27T21:17:37.829344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:37.829344+0000","objects_scrubbed":6174,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13246,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":265,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.462632+0000","last_change":"2024-06-27T21:19:40.674095+0000","last_active":"2024-06-27T21:20:04.462632+0000","last_peered":"2024-06-27T21:20:04.462632+0000","last_clean":"2024-06-27T21:20:04.462632+0000","last_became_active":"2024-06-27T21:17:45.838234+0000","last_became_peered":"2024-06-27T21:17:45.838234+0000","last_unstale":"2024-06-27T21:20:04.462632+0000","last_undegraded":"2024-06-27T21:20:04.462632+0000","last_fullsized":"2024-06-27T21:20:04.462632+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:40.673987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:40.673987+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:06.405572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.6","version":"272'14474","reported_seq":14805,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:05.712326+0000","last_change":"2024-06-27T21:19:44.837277+0000","last_active":"2024-06-27T21:20:05.712326+0000","last_peered":"2024-06-27T21:20:05.712326+0000","last_clean":"2024-06-27T21:20:05.712326+0000","last_became_active":"2024-06-27T21:16:02.436785+0000","last_became_peered":"2024-06-27T21:16:02.436785+0000","last_unstale":"2024-06-27T21:20:05.712326+0000","last_undegraded":"2024-06-27T21:20:05.712326+0000","last_fullsized":"2024-06-27T21:20:05.712326+0000","mapping_epoch":186,"log_start":"212'4401","ondisk_log_start":"212'4401","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'13439","last_scrub_stamp":"2024-06-27T21:19:44.836934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:44.836934+0000","objects_scrubbed":1107,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:50.976863+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13113,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":356,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.456011+0000","last_change":"2024-06-27T21:19:50.905384+0000","last_active":"2024-06-27T21:20:04.456011+0000","last_peered":"2024-06-27T21:20:04.456011+0000","last_clean":"2024-06-27T21:20:04.456011+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:20:04.456011+0000","last_undegraded":"2024-06-27T21:20:04.456011+0000","last_fullsized":"2024-06-27T21:20:04.456011+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:50.905308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:50.905308+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:11.504423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"272'13186","reported_seq":13460,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.091297+0000","last_change":"2024-06-27T21:18:53.835918+0000","last_active":"2024-06-27T21:20:07.091297+0000","last_peered":"2024-06-27T21:20:07.091297+0000","last_clean":"2024-06-27T21:20:07.091297+0000","last_became_active":"2024-06-27T21:16:02.436507+0000","last_became_peered":"2024-06-27T21:16:02.436507+0000","last_unstale":"2024-06-27T21:20:07.091297+0000","last_undegraded":"2024-06-27T21:20:07.091297+0000","last_fullsized":"2024-06-27T21:20:07.091297+0000","mapping_epoch":186,"log_start":"203'3100","ondisk_log_start":"203'3100","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'9881","last_scrub_stamp":"2024-06-27T21:18:53.835815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:53.835815+0000","objects_scrubbed":3330,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:10.580591+0000","scrub_duration":878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":260,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.475375+0000","last_change":"2024-06-27T21:18:58.933419+0000","last_active":"2024-06-27T21:20:04.475375+0000","last_peered":"2024-06-27T21:20:04.475375+0000","last_clean":"2024-06-27T21:20:04.475375+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:20:04.475375+0000","last_undegraded":"2024-06-27T21:20:04.475375+0000","last_fullsized":"2024-06-27T21:20:04.475375+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:18:58.933335+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:18:58.933335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:16.443386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.b","version":"272'13302","reported_seq":13532,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:07.086985+0000","last_change":"2024-06-27T21:17:04.020522+0000","last_active":"2024-06-27T21:20:07.086985+0000","last_peered":"2024-06-27T21:20:07.086985+0000","last_clean":"2024-06-27T21:20:07.086985+0000","last_became_active":"2024-06-27T21:16:02.435710+0000","last_became_peered":"2024-06-27T21:16:02.435710+0000","last_unstale":"2024-06-27T21:20:07.086985+0000","last_undegraded":"2024-06-27T21:20:07.086985+0000","last_fullsized":"2024-06-27T21:20:07.086985+0000","mapping_epoch":186,"log_start":"204'3300","ondisk_log_start":"204'3300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'3907","last_scrub_stamp":"2024-06-27T21:17:04.020273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:04.020273+0000","objects_scrubbed":3842,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":425664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":558,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.459542+0000","last_change":"2024-06-27T21:19:06.368723+0000","last_active":"2024-06-27T21:20:04.459542+0000","last_peered":"2024-06-27T21:20:04.459542+0000","last_clean":"2024-06-27T21:20:04.459542+0000","last_became_active":"2024-06-27T21:17:45.505725+0000","last_became_peered":"2024-06-27T21:17:45.505725+0000","last_unstale":"2024-06-27T21:20:04.459542+0000","last_undegraded":"2024-06-27T21:20:04.459542+0000","last_fullsized":"2024-06-27T21:20:04.459542+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:06.368611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:06.368611+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:30.059095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"272'13221","reported_seq":13474,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:05.275224+0000","last_change":"2024-06-27T21:17:36.221000+0000","last_active":"2024-06-27T21:20:05.275224+0000","last_peered":"2024-06-27T21:20:05.275224+0000","last_clean":"2024-06-27T21:20:05.275224+0000","last_became_active":"2024-06-27T21:16:02.436617+0000","last_became_peered":"2024-06-27T21:16:02.436617+0000","last_unstale":"2024-06-27T21:20:05.275224+0000","last_undegraded":"2024-06-27T21:20:05.275224+0000","last_fullsized":"2024-06-27T21:20:05.275224+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'6177","last_scrub_stamp":"2024-06-27T21:17:36.220814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:36.220814+0000","objects_scrubbed":6143,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13221,"num_write_kb":425856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":268,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-06-27T21:20:04.462106+0000","last_change":"2024-06-27T21:17:45.505887+0000","last_active":"2024-06-27T21:20:04.462106+0000","last_peered":"2024-06-27T21:20:04.462106+0000","last_clean":"2024-06-27T21:20:04.462106+0000","last_became_active":"2024-06-27T21:17:45.502472+0000","last_became_peered":"2024-06-27T21:17:45.502472+0000","last_unstale":"2024-06-27T21:20:04.462106+0000","last_undegraded":"2024-06-27T21:20:04.462106+0000","last_fullsized":"2024-06-27T21:20:04.462106+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:17:36.398858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:17:36.398858+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":13565952,"num_objects":207,"num_object_clones":0,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213065,"num_write_kb":6824641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4719,"num_bytes_recovered":301137920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":514138112,"data_stored":514129968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252994,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225272,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93519880,"statfs":{"total":95995035648,"available":95764357120,"internally_reserved":0,"allocated":1163264,"data_stored":849206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514631,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80568,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93664584,"statfs":{"total":95995035648,"available":95912534016,"internally_reserved":0,"allocated":835584,"data_stored":521526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743562,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":167616,"kb_used_data":23988,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93577536,"statfs":{"total":95995035648,"available":95823396864,"internally_reserved":0,"allocated":24563712,"data_stored":24245582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972495,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184388,"kb_used_data":4664,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93560764,"statfs":{"total":95995035648,"available":95806222336,"internally_reserved":0,"allocated":4775936,"data_stored":4458326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158648,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93586504,"statfs":{"total":95995035648,"available":95832580096,"internally_reserved":0,"allocated":1163264,"data_stored":849206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528472,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":253692,"kb_used_data":33712,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93491460,"statfs":{"total":95995035648,"available":95735255040,"internally_reserved":0,"allocated":34521088,"data_stored":34207030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225245638},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757406,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112708,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632444,"statfs":{"total":95995035648,"available":95879622656,"internally_reserved":0,"allocated":1044480,"data_stored":722798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986339,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190264,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554888,"statfs":{"total":95995035648,"available":95800205312,"internally_reserved":0,"allocated":835584,"data_stored":521526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182567,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181120,"kb_used_data":9720,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93564032,"statfs":{"total":95995035648,"available":95809568768,"internally_reserved":0,"allocated":9953280,"data_stored":9635670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411500,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126392,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618760,"statfs":{"total":95995035648,"available":95865610240,"internally_reserved":0,"allocated":835584,"data_stored":521526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205948,"kb_used_data":26224,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93539204,"statfs":{"total":95995035648,"available":95784144896,"internally_reserved":0,"allocated":26853376,"data_stored":26539318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21561344,"data_stored":21561344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":445841408,"data_stored":445841408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28770304,"data_stored":28770304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":17960960,"data_stored":17956888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:20:11.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:20:11.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:12.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:12.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:12.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:12.161 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:12.161 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:12.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:12.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:12.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:12.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:12.483 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-06-27T21:20:12.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-06-27T21:20:12.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:12.507 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:12.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:12.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:12.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:12.753 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:12 smithi151 ceph-mon[28965]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:12.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:12.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:12 smithi190 ceph-mon[32247]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:12.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:20:12.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:12 smithi192 ceph-mon[32956]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:12.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:12.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:12.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:12.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:13.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:13.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:13.119 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:13.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:13.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:13.263 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:13.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:13.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:13.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:13.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:13.609 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:13.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:13.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:13 smithi151 ceph-mon[28965]: pgmap v756: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 2084/6 objects degraded (34733.333%); 1.8 MiB/s, 40 objects/s recovering 2024-06-27T21:20:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:13.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:13.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:13.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:13 smithi190 ceph-mon[32247]: pgmap v756: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 2084/6 objects degraded (34733.333%); 1.8 MiB/s, 40 objects/s recovering 2024-06-27T21:20:13.762 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:13.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:13 smithi192 ceph-mon[32956]: pgmap v756: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 623 op/s; 2084/6 objects degraded (34733.333%); 1.8 MiB/s, 40 objects/s recovering 2024-06-27T21:20:13.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:13.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:13.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:13.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:14.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:14.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:14.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:14.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:14.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:14.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:14.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:14.455 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:20:14.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:14.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:14.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:14.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:14.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2471764660' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:14.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2084/6 objects degraded (34733.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:14.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:14 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:14.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2471764660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:14.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:14 smithi151 ceph-mon[28965]: osdmap e274: 12 total, 12 up, 11 in 2024-06-27T21:20:14.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:14.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:14.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2471764660' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:14.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2084/6 objects degraded (34733.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:14.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:14 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:14.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2471764660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:14.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:14 smithi190 ceph-mon[32247]: osdmap e274: 12 total, 12 up, 11 in 2024-06-27T21:20:14.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2471764660' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:14.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2084/6 objects degraded (34733.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:14.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:14 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:14.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2471764660' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:14.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:14 smithi192 ceph-mon[32956]: osdmap e274: 12 total, 12 up, 11 in 2024-06-27T21:20:14.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:14.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:14.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:14.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:14.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:15.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:15.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:15.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:15.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:15.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:15.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:15.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:15.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:15.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:15.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:15.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:15.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:15 smithi151 ceph-mon[28965]: pgmap v757: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 1240/6 objects degraded (20666.667%); 1.5 MiB/s, 123 objects/s recovering 2024-06-27T21:20:15.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:15.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:15.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:15.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:15.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:15 smithi190 ceph-mon[32247]: pgmap v757: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 1240/6 objects degraded (20666.667%); 1.5 MiB/s, 123 objects/s recovering 2024-06-27T21:20:15.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:15.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:15.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:15 smithi192 ceph-mon[32956]: pgmap v757: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 1240/6 objects degraded (20666.667%); 1.5 MiB/s, 123 objects/s recovering 2024-06-27T21:20:15.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:15.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:15.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:15.886 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:15.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:15.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:16.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:16.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:16.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:16.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:16.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:16.320 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:16.321 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:16.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:16.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:16.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:16.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:16.571 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:16 smithi151 ceph-mon[28965]: osdmap e275: 12 total, 12 up, 11 in 2024-06-27T21:20:16.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:16.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:16.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:16.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:16 smithi190 ceph-mon[32247]: osdmap e275: 12 total, 12 up, 11 in 2024-06-27T21:20:16.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:16.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:16 smithi192 ceph-mon[32956]: osdmap e275: 12 total, 12 up, 11 in 2024-06-27T21:20:16.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:16.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:16.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:16.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:20:16.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:20:16.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:17.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:20:17.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:20:17.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:17.031 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:20:17.031 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:20:17.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:20:17.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:20:17.073 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:20:17.074 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:20:17.096 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:20:17.096 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:20:17.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:20:17.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:20:17.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:17.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:20:17.139 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:20:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:17.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:17.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:20:17.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:20:17.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:20:17.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:20:17.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:20:17.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:20:17.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:20:17.228 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:20:17.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:17.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:17.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:17.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:17.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:17.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:17.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:17.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:17.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:17.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:17 smithi151 ceph-mon[28965]: pgmap v760: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 156 op/s; 1240/6 objects degraded (20666.667%); 0 B/s, 130 objects/s recovering 2024-06-27T21:20:17.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:17 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1719930851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:17.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:17.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:17 smithi190 ceph-mon[32247]: pgmap v760: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 156 op/s; 1240/6 objects degraded (20666.667%); 0 B/s, 130 objects/s recovering 2024-06-27T21:20:17.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:17 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1719930851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:17.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:17.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:17 smithi192 ceph-mon[32956]: pgmap v760: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 156 op/s; 1240/6 objects degraded (20666.667%); 0 B/s, 130 objects/s recovering 2024-06-27T21:20:17.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:17 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1719930851' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:17.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:17.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:17.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:18.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:18.022 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:18.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:18.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:18.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:18.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:18.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:18.376 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:20:18.376 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:20:18.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:18.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:18.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:18.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:18.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:18.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:18 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:18.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:18 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:18.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:18 smithi151 ceph-mon[28965]: osdmap e276: 12 total, 12 up, 11 in 2024-06-27T21:20:18.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:18.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:18 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:18.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:18 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:18.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:18 smithi190 ceph-mon[32247]: osdmap e276: 12 total, 12 up, 11 in 2024-06-27T21:20:18.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:18.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:18 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:18.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:18 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:18.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:18 smithi192 ceph-mon[32956]: osdmap e276: 12 total, 12 up, 11 in 2024-06-27T21:20:18.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:18.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:18.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:19.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:19.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:19.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:19.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:19.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:19.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:19.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:19.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:19.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:20:19.487 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-06-27T21:20:17.935371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213272,"num_write_kb":6824641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6319,"num_bytes_recovered":301339168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161606,"ondisk_log_size":161606,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1807292,"kb_used_data":10800,"kb_used_omap":17,"kb_used_meta":1796334,"kb_avail":1029389380,"statfs":{"total":1055945392128,"available":1054094725120,"internally_reserved":0,"allocated":11059200,"data_stored":7494688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1839446901},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1599,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001210"},"pg_stats":[{"pgid":"3.f","version":"272'13214","reported_seq":13511,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557173+0000","last_change":"2024-06-27T21:19:00.478810+0000","last_active":"2024-06-27T21:20:14.557173+0000","last_peered":"2024-06-27T21:20:14.557173+0000","last_clean":"2024-06-27T21:20:14.557173+0000","last_became_active":"2024-06-27T21:16:02.437136+0000","last_became_peered":"2024-06-27T21:16:02.437136+0000","last_unstale":"2024-06-27T21:20:14.557173+0000","last_undegraded":"2024-06-27T21:20:14.557173+0000","last_fullsized":"2024-06-27T21:20:14.557173+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10257","last_scrub_stamp":"2024-06-27T21:19:00.478347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:00.478347+0000","objects_scrubbed":2984,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13214,"num_write_kb":422848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"272'15119","reported_seq":15465,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.459688+0000","last_change":"2024-06-27T21:19:47.365005+0000","last_active":"2024-06-27T21:20:14.459688+0000","last_peered":"2024-06-27T21:20:14.459688+0000","last_clean":"2024-06-27T21:20:14.459688+0000","last_became_active":"2024-06-27T21:16:02.436824+0000","last_became_peered":"2024-06-27T21:16:02.436824+0000","last_unstale":"2024-06-27T21:20:14.459688+0000","last_undegraded":"2024-06-27T21:20:14.459688+0000","last_fullsized":"2024-06-27T21:20:14.459688+0000","mapping_epoch":186,"log_start":"213'5101","ondisk_log_start":"213'5101","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'14175","last_scrub_stamp":"2024-06-27T21:19:47.364792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:47.364792+0000","objects_scrubbed":958,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:03.339363+0000","scrub_duration":459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13556,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"272'13534","reported_seq":13838,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557068+0000","last_change":"2024-06-27T21:18:56.791587+0000","last_active":"2024-06-27T21:20:14.557068+0000","last_peered":"2024-06-27T21:20:14.557068+0000","last_clean":"2024-06-27T21:20:14.557068+0000","last_became_active":"2024-06-27T21:16:02.856526+0000","last_became_peered":"2024-06-27T21:16:02.856526+0000","last_unstale":"2024-06-27T21:20:14.557068+0000","last_undegraded":"2024-06-27T21:20:14.557068+0000","last_fullsized":"2024-06-27T21:20:14.557068+0000","mapping_epoch":186,"log_start":"205'3500","ondisk_log_start":"205'3500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10243","last_scrub_stamp":"2024-06-27T21:18:56.790801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:56.790801+0000","objects_scrubbed":3328,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13534,"num_write_kb":433025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"272'13224","reported_seq":13485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-06-27T21:20:17.538958+0000","last_change":"2024-06-27T21:17:39.000875+0000","last_active":"2024-06-27T21:20:17.538958+0000","last_peered":"2024-06-27T21:20:17.538958+0000","last_clean":"2024-06-27T21:20:17.538958+0000","last_became_active":"2024-06-27T21:16:02.439289+0000","last_became_peered":"2024-06-27T21:16:02.439289+0000","last_unstale":"2024-06-27T21:20:17.538958+0000","last_undegraded":"2024-06-27T21:20:17.538958+0000","last_fullsized":"2024-06-27T21:20:17.538958+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6283","last_scrub_stamp":"2024-06-27T21:17:39.000676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:39.000676+0000","objects_scrubbed":6223,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1574,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"272'13264","reported_seq":13567,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-27T21:20:15.495994+0000","last_change":"2024-06-27T21:18:54.798170+0000","last_active":"2024-06-27T21:20:15.495994+0000","last_peered":"2024-06-27T21:20:15.495994+0000","last_clean":"2024-06-27T21:20:15.495994+0000","last_became_active":"2024-06-27T21:16:02.435740+0000","last_became_peered":"2024-06-27T21:16:02.435740+0000","last_unstale":"2024-06-27T21:20:15.495994+0000","last_undegraded":"2024-06-27T21:20:15.495994+0000","last_fullsized":"2024-06-27T21:20:15.495994+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10086","last_scrub_stamp":"2024-06-27T21:18:54.797841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:54.797841+0000","objects_scrubbed":3199,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:22.934044+0000","scrub_duration":906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13264,"num_write_kb":424448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"272'13256","reported_seq":13518,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-27T21:20:15.634596+0000","last_change":"2024-06-27T21:17:36.697283+0000","last_active":"2024-06-27T21:20:15.634596+0000","last_peered":"2024-06-27T21:20:15.634596+0000","last_clean":"2024-06-27T21:20:15.634596+0000","last_became_active":"2024-06-27T21:16:02.856475+0000","last_became_peered":"2024-06-27T21:16:02.856475+0000","last_unstale":"2024-06-27T21:20:15.634596+0000","last_undegraded":"2024-06-27T21:20:15.634596+0000","last_fullsized":"2024-06-27T21:20:15.634596+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'6177","last_scrub_stamp":"2024-06-27T21:17:36.697124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:36.697124+0000","objects_scrubbed":6104,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13256,"num_write_kb":424192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":552,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.556936+0000","last_change":"2024-06-27T21:19:46.475053+0000","last_active":"2024-06-27T21:20:14.556936+0000","last_peered":"2024-06-27T21:20:14.556936+0000","last_clean":"2024-06-27T21:20:14.556936+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:20:14.556936+0000","last_undegraded":"2024-06-27T21:20:14.556936+0000","last_fullsized":"2024-06-27T21:20:14.556936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:46.474944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:46.474944+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:48.986062+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"272'13518","reported_seq":13836,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-27T21:20:15.634524+0000","last_change":"2024-06-27T21:19:45.738578+0000","last_active":"2024-06-27T21:20:15.634524+0000","last_peered":"2024-06-27T21:20:15.634524+0000","last_clean":"2024-06-27T21:20:15.634524+0000","last_became_active":"2024-06-27T21:16:03.380506+0000","last_became_peered":"2024-06-27T21:16:03.380506+0000","last_unstale":"2024-06-27T21:20:15.634524+0000","last_undegraded":"2024-06-27T21:20:15.634524+0000","last_fullsized":"2024-06-27T21:20:15.634524+0000","mapping_epoch":186,"log_start":"204'3500","ondisk_log_start":"204'3500","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'12465","last_scrub_stamp":"2024-06-27T21:19:45.738454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:45.738454+0000","objects_scrubbed":1059,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:56.490662+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":137,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557169+0000","last_change":"2024-06-27T21:18:11.238129+0000","last_active":"2024-06-27T21:20:14.557169+0000","last_peered":"2024-06-27T21:20:14.557169+0000","last_clean":"2024-06-27T21:20:14.557169+0000","last_became_active":"2024-06-27T21:17:58.594952+0000","last_became_peered":"2024-06-27T21:17:58.594952+0000","last_unstale":"2024-06-27T21:20:14.557169+0000","last_undegraded":"2024-06-27T21:20:14.557169+0000","last_fullsized":"2024-06-27T21:20:14.557169+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:18:11.238027+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:18:11.238027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"272'13374","reported_seq":13672,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.459487+0000","last_change":"2024-06-27T21:19:01.088777+0000","last_active":"2024-06-27T21:20:14.459487+0000","last_peered":"2024-06-27T21:20:14.459487+0000","last_clean":"2024-06-27T21:20:14.459487+0000","last_became_active":"2024-06-27T21:16:02.856410+0000","last_became_peered":"2024-06-27T21:16:02.856410+0000","last_unstale":"2024-06-27T21:20:14.459487+0000","last_undegraded":"2024-06-27T21:20:14.459487+0000","last_fullsized":"2024-06-27T21:20:14.459487+0000","mapping_epoch":186,"log_start":"204'3300","ondisk_log_start":"204'3300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'10375","last_scrub_stamp":"2024-06-27T21:19:01.088693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:01.088693+0000","objects_scrubbed":3039,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:28.983057+0000","scrub_duration":1340,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":556,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557305+0000","last_change":"2024-06-27T21:19:04.347536+0000","last_active":"2024-06-27T21:20:14.557305+0000","last_peered":"2024-06-27T21:20:14.557305+0000","last_clean":"2024-06-27T21:20:14.557305+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:20:14.557305+0000","last_undegraded":"2024-06-27T21:20:14.557305+0000","last_fullsized":"2024-06-27T21:20:14.557305+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:04.347424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:04.347424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"272'12952","reported_seq":13249,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-06-27T21:20:17.642416+0000","last_change":"2024-06-27T21:19:06.671521+0000","last_active":"2024-06-27T21:20:17.642416+0000","last_peered":"2024-06-27T21:20:17.642416+0000","last_clean":"2024-06-27T21:20:17.642416+0000","last_became_active":"2024-06-27T21:16:02.437264+0000","last_became_peered":"2024-06-27T21:16:02.437264+0000","last_unstale":"2024-06-27T21:20:17.642416+0000","last_undegraded":"2024-06-27T21:20:17.642416+0000","last_fullsized":"2024-06-27T21:20:17.642416+0000","mapping_epoch":186,"log_start":"203'2900","ondisk_log_start":"203'2900","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'10309","last_scrub_stamp":"2024-06-27T21:19:06.671332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:06.671332+0000","objects_scrubbed":2661,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:19.196852+0000","scrub_duration":1167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12952,"num_write_kb":414464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":263,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.556948+0000","last_change":"2024-06-27T21:18:57.759488+0000","last_active":"2024-06-27T21:20:14.556948+0000","last_peered":"2024-06-27T21:20:14.556948+0000","last_clean":"2024-06-27T21:20:14.556948+0000","last_became_active":"2024-06-27T21:14:55.718653+0000","last_became_peered":"2024-06-27T21:14:55.718653+0000","last_unstale":"2024-06-27T21:20:14.556948+0000","last_undegraded":"2024-06-27T21:20:14.556948+0000","last_fullsized":"2024-06-27T21:20:14.556948+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:18:57.759379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:18:57.759379+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:14.844302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"272'13618","reported_seq":26520,"reported_epoch":276,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:20:17.651595+0000","last_change":"2024-06-27T21:17:48.680969+0000","last_active":"2024-06-27T21:20:17.651595+0000","last_peered":"2024-06-27T21:20:17.651595+0000","last_clean":"2024-06-27T21:17:44.498417+0000","last_became_active":"2024-06-27T21:17:46.964668+0000","last_became_peered":"2024-06-27T21:17:46.964668+0000","last_unstale":"2024-06-27T21:20:17.651595+0000","last_undegraded":"2024-06-27T21:17:48.677261+0000","last_fullsized":"2024-06-27T21:17:46.505450+0000","mapping_epoch":227,"log_start":"205'3600","ondisk_log_start":"205'3600","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'6285","last_scrub_stamp":"2024-06-27T21:17:35.788328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:35.788328+0000","objects_scrubbed":6220,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13618,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6318,"num_bytes_recovered":301137920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"3,5","objects":-485}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":273,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557362+0000","last_change":"2024-06-27T21:16:46.599756+0000","last_active":"2024-06-27T21:20:14.557362+0000","last_peered":"2024-06-27T21:20:14.557362+0000","last_clean":"2024-06-27T21:20:14.557362+0000","last_became_active":"2024-06-27T21:14:55.861874+0000","last_became_peered":"2024-06-27T21:14:55.861874+0000","last_unstale":"2024-06-27T21:20:14.557362+0000","last_undegraded":"2024-06-27T21:20:14.557362+0000","last_fullsized":"2024-06-27T21:20:14.557362+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:16:46.599672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:16:46.599672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"272'13548","reported_seq":13867,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.459575+0000","last_change":"2024-06-27T21:18:55.817183+0000","last_active":"2024-06-27T21:20:14.459575+0000","last_peered":"2024-06-27T21:20:14.459575+0000","last_clean":"2024-06-27T21:20:14.459575+0000","last_became_active":"2024-06-27T21:17:58.595091+0000","last_became_peered":"2024-06-27T21:17:58.595091+0000","last_unstale":"2024-06-27T21:20:14.459575+0000","last_undegraded":"2024-06-27T21:20:14.459575+0000","last_fullsized":"2024-06-27T21:20:14.459575+0000","mapping_epoch":230,"log_start":"204'3500","ondisk_log_start":"204'3500","created":186,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10303","last_scrub_stamp":"2024-06-27T21:18:55.817070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:55.817070+0000","objects_scrubbed":3265,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":956,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13548,"num_write_kb":433536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":263,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.556941+0000","last_change":"2024-06-27T21:19:00.785414+0000","last_active":"2024-06-27T21:20:14.556941+0000","last_peered":"2024-06-27T21:20:14.556941+0000","last_clean":"2024-06-27T21:20:14.556941+0000","last_became_active":"2024-06-27T21:14:55.711705+0000","last_became_peered":"2024-06-27T21:14:55.711705+0000","last_unstale":"2024-06-27T21:20:14.556941+0000","last_undegraded":"2024-06-27T21:20:14.556941+0000","last_fullsized":"2024-06-27T21:20:14.556941+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:00.785339+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:19:00.785339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:14.844355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"272'13246","reported_seq":13507,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-27T21:20:15.634750+0000","last_change":"2024-06-27T21:17:37.829478+0000","last_active":"2024-06-27T21:20:15.634750+0000","last_peered":"2024-06-27T21:20:15.634750+0000","last_clean":"2024-06-27T21:20:15.634750+0000","last_became_active":"2024-06-27T21:16:02.436035+0000","last_became_peered":"2024-06-27T21:16:02.436035+0000","last_unstale":"2024-06-27T21:20:15.634750+0000","last_undegraded":"2024-06-27T21:20:15.634750+0000","last_fullsized":"2024-06-27T21:20:15.634750+0000","mapping_epoch":186,"log_start":"203'3200","ondisk_log_start":"203'3200","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6242","last_scrub_stamp":"2024-06-27T21:17:37.829344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:37.829344+0000","objects_scrubbed":6174,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13246,"num_write_kb":423872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":269,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.556940+0000","last_change":"2024-06-27T21:19:40.674095+0000","last_active":"2024-06-27T21:20:14.556940+0000","last_peered":"2024-06-27T21:20:14.556940+0000","last_clean":"2024-06-27T21:20:14.556940+0000","last_became_active":"2024-06-27T21:17:45.838234+0000","last_became_peered":"2024-06-27T21:17:45.838234+0000","last_unstale":"2024-06-27T21:20:14.556940+0000","last_undegraded":"2024-06-27T21:20:14.556940+0000","last_fullsized":"2024-06-27T21:20:14.556940+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:40.673987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:40.673987+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:06.405572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.6","version":"272'14533","reported_seq":14871,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-27T21:20:15.634663+0000","last_change":"2024-06-27T21:19:44.837277+0000","last_active":"2024-06-27T21:20:15.634663+0000","last_peered":"2024-06-27T21:20:15.634663+0000","last_clean":"2024-06-27T21:20:15.634663+0000","last_became_active":"2024-06-27T21:16:02.436785+0000","last_became_peered":"2024-06-27T21:16:02.436785+0000","last_unstale":"2024-06-27T21:20:15.634663+0000","last_undegraded":"2024-06-27T21:20:15.634663+0000","last_fullsized":"2024-06-27T21:20:15.634663+0000","mapping_epoch":186,"log_start":"212'4501","ondisk_log_start":"212'4501","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"266'13439","last_scrub_stamp":"2024-06-27T21:19:44.836934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:19:44.836934+0000","objects_scrubbed":1107,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:50.976863+0000","scrub_duration":366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13172,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":360,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.459650+0000","last_change":"2024-06-27T21:19:50.905384+0000","last_active":"2024-06-27T21:20:14.459650+0000","last_peered":"2024-06-27T21:20:14.459650+0000","last_clean":"2024-06-27T21:20:14.459650+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:20:14.459650+0000","last_undegraded":"2024-06-27T21:20:14.459650+0000","last_fullsized":"2024-06-27T21:20:14.459650+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:50.905308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:50.905308+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:11.504423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"272'13186","reported_seq":13464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.459476+0000","last_change":"2024-06-27T21:18:53.835918+0000","last_active":"2024-06-27T21:20:14.459476+0000","last_peered":"2024-06-27T21:20:14.459476+0000","last_clean":"2024-06-27T21:20:14.459476+0000","last_became_active":"2024-06-27T21:16:02.436507+0000","last_became_peered":"2024-06-27T21:16:02.436507+0000","last_unstale":"2024-06-27T21:20:14.459476+0000","last_undegraded":"2024-06-27T21:20:14.459476+0000","last_fullsized":"2024-06-27T21:20:14.459476+0000","mapping_epoch":186,"log_start":"203'3100","ondisk_log_start":"203'3100","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'9881","last_scrub_stamp":"2024-06-27T21:18:53.835815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:18:53.835815+0000","objects_scrubbed":3330,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:16.799526+0000","scrub_duration":878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":264,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.556691+0000","last_change":"2024-06-27T21:18:58.933419+0000","last_active":"2024-06-27T21:20:14.556691+0000","last_peered":"2024-06-27T21:20:14.556691+0000","last_clean":"2024-06-27T21:20:14.556691+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:20:14.556691+0000","last_undegraded":"2024-06-27T21:20:14.556691+0000","last_fullsized":"2024-06-27T21:20:14.556691+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:18:58.933335+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:18:58.933335+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:16.443386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.b","version":"272'13302","reported_seq":13538,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-06-27T21:20:15.634363+0000","last_change":"2024-06-27T21:17:04.020522+0000","last_active":"2024-06-27T21:20:15.634363+0000","last_peered":"2024-06-27T21:20:15.634363+0000","last_clean":"2024-06-27T21:20:15.634363+0000","last_became_active":"2024-06-27T21:16:02.435710+0000","last_became_peered":"2024-06-27T21:16:02.435710+0000","last_unstale":"2024-06-27T21:20:15.634363+0000","last_undegraded":"2024-06-27T21:20:15.634363+0000","last_fullsized":"2024-06-27T21:20:15.634363+0000","mapping_epoch":186,"log_start":"204'3300","ondisk_log_start":"204'3300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'3907","last_scrub_stamp":"2024-06-27T21:17:04.020273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:04.020273+0000","objects_scrubbed":3842,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":425664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":562,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557067+0000","last_change":"2024-06-27T21:19:06.368723+0000","last_active":"2024-06-27T21:20:14.557067+0000","last_peered":"2024-06-27T21:20:14.557067+0000","last_clean":"2024-06-27T21:20:14.557067+0000","last_became_active":"2024-06-27T21:17:45.505725+0000","last_became_peered":"2024-06-27T21:17:45.505725+0000","last_unstale":"2024-06-27T21:20:14.557067+0000","last_undegraded":"2024-06-27T21:20:14.557067+0000","last_fullsized":"2024-06-27T21:20:14.557067+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:06.368611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:06.368611+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:30.059095+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"272'13308","reported_seq":13565,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-06-27T21:20:14.557178+0000","last_change":"2024-06-27T21:17:36.221000+0000","last_active":"2024-06-27T21:20:14.557178+0000","last_peered":"2024-06-27T21:20:14.557178+0000","last_clean":"2024-06-27T21:20:14.557178+0000","last_became_active":"2024-06-27T21:16:02.436617+0000","last_became_peered":"2024-06-27T21:16:02.436617+0000","last_unstale":"2024-06-27T21:20:14.557178+0000","last_undegraded":"2024-06-27T21:20:14.557178+0000","last_fullsized":"2024-06-27T21:20:14.557178+0000","mapping_epoch":186,"log_start":"203'3300","ondisk_log_start":"203'3300","created":186,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'6177","last_scrub_stamp":"2024-06-27T21:17:36.220814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:16:01.415884+0000","last_clean_scrub_stamp":"2024-06-27T21:17:36.220814+0000","objects_scrubbed":6143,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13308,"num_write_kb":425856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":276,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-06-27T21:20:17.538933+0000","last_change":"2024-06-27T21:17:45.505887+0000","last_active":"2024-06-27T21:20:17.538933+0000","last_peered":"2024-06-27T21:20:17.538933+0000","last_clean":"2024-06-27T21:20:17.538933+0000","last_became_active":"2024-06-27T21:17:45.502472+0000","last_became_peered":"2024-06-27T21:17:45.502472+0000","last_unstale":"2024-06-27T21:20:17.538933+0000","last_undegraded":"2024-06-27T21:20:17.538933+0000","last_fullsized":"2024-06-27T21:20:17.538933+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:17:36.398858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:17:36.398858+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213272,"num_write_kb":6824641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6318,"num_bytes_recovered":301137920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445841408,"data_stored":445841408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160594,"ondisk_log_size":160594,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252995,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225288,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93519864,"statfs":{"total":95995035648,"available":95764340736,"internally_reserved":0,"allocated":1175552,"data_stored":856975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514632,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80584,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93664568,"statfs":{"total":95995035648,"available":95912517632,"internally_reserved":0,"allocated":847872,"data_stored":529295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743564,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144468,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600684,"statfs":{"total":95995035648,"available":95847100416,"internally_reserved":0,"allocated":860160,"data_stored":537064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972497,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181100,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564052,"statfs":{"total":95995035648,"available":95809589248,"internally_reserved":0,"allocated":1404928,"data_stored":1073811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158664,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93586488,"statfs":{"total":95995035648,"available":95832563712,"internally_reserved":0,"allocated":1175552,"data_stored":856975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528474,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216660,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93528492,"statfs":{"total":95995035648,"available":95773175808,"internally_reserved":0,"allocated":860160,"data_stored":537064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757408,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112744,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632408,"statfs":{"total":95995035648,"available":95879585792,"internally_reserved":0,"allocated":1077248,"data_stored":746131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986341,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190308,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554844,"statfs":{"total":95995035648,"available":95800160256,"internally_reserved":0,"allocated":872448,"data_stored":544883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182569,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190504,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554648,"statfs":{"total":95995035648,"available":95799959552,"internally_reserved":0,"allocated":1077248,"data_stored":746131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":193984967},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411501,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126408,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618744,"statfs":{"total":95995035648,"available":95865593856,"internally_reserved":0,"allocated":847872,"data_stored":529295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180564,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564588,"statfs":{"total":95995035648,"available":95810138112,"internally_reserved":0,"allocated":860160,"data_stored":537064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":445841408,"data_stored":445841408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:20:19.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph status --format=json 2024-06-27T21:20:19.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:19.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:19.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:19.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:19.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:19.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:19 smithi190 ceph-mon[32247]: pgmap v762: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 266 objects/s recovering 2024-06-27T21:20:19.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:19 smithi190 ceph-mon[32247]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:19.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 485/6 objects degraded (8083.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:19.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:19 smithi190 ceph-mon[32247]: osdmap e277: 12 total, 12 up, 11 in 2024-06-27T21:20:19.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:19.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:19 smithi192 ceph-mon[32956]: pgmap v762: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 266 objects/s recovering 2024-06-27T21:20:19.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:19 smithi192 ceph-mon[32956]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:19.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 485/6 objects degraded (8083.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:19.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:19 smithi192 ceph-mon[32956]: osdmap e277: 12 total, 12 up, 11 in 2024-06-27T21:20:19.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:19 smithi151 ceph-mon[28965]: pgmap v762: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 266 objects/s recovering 2024-06-27T21:20:19.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:19 smithi151 ceph-mon[28965]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:19.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 485/6 objects degraded (8083.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:20:19.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:19 smithi151 ceph-mon[28965]: osdmap e277: 12 total, 12 up, 11 in 2024-06-27T21:20:19.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:19.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:19.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:20.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:20.022 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:20.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:20.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:20.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:20.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:20.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:20.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:20.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:20.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:20.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:20:20.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:20.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:20.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:20.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:20.830 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:121198 2.d e140/104 2:bd9f4ce1:::benchmark_data_smithi192_59818_object29123:head v 140'7584, mlcod=140'7581)", 2024-06-27T21:20:20.841 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:45.243183+0000", 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 395.59917667299999, 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21566449900000001, 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.243183+0000", 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.243183+0000", 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.243185+0000", 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.17e-06 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.243192+0000", 2024-06-27T21:20:20.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1400000000000002e-06 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.456235+0000", 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.213042908 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.456241+0000", 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1199999999999999e-06 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.456295+0000", 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4546999999999998e-05 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.456312+0000", 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6642000000000001e-05 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.458816+0000", 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002504334 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:45.458848+0000", 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1637999999999998e-05 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.846 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:121455 2.9 e141/104 2:95514c70:::benchmark_data_smithi192_59818_object29380:head v 141'7617, mlcod=141'7615)", 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:46.142437+0000", 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 394.69992308500002, 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34835318999999998, 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.142437+0000", 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.142437+0000", 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.142438+0000", 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4399999999999999e-07 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.142443+0000", 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1089999999999997e-06 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.487841+0000", 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34539838499999997 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.487847+0000", 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3940000000000004e-06 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.487897+0000", 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0018000000000001e-05 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.487915+0000", 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8379000000000001e-05 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.490757+0000", 2024-06-27T21:20:20.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028418100000000002 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.490790+0000", 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3351e-05 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:121520 2.6 e142/104 2:6c84e51e:::benchmark_data_smithi192_59818_object29445:head v 142'7746, mlcod=141'7744)", 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:46.647529+0000", 2024-06-27T21:20:20.851 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 394.19483091500001, 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21449154400000001, 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.647529+0000", 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.647529+0000", 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.647530+0000", 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9999999999999996e-07 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.647536+0000", 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7899999999999996e-06 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.859192+0000", 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21165578299999999 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.859197+0000", 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9529999999999998e-06 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.859242+0000", 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4966999999999998e-05 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.859255+0000", 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3903000000000001e-05 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.861992+0000", 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027365699999999998 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:46.862021+0000", 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8778000000000001e-05 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:34105 3.7 3:ea959f77:::benchmark_data_smithi192_74314_object34104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:40.813118+0000", 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 220.02924239699999, 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216328097, 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.857 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 34105 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813118+0000", 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813118+0000", 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813119+0000", 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2079999999999999e-06 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813166+0000", 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7472000000000002e-05 2024-06-27T21:20:20.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813178+0000", 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1172000000000001e-05 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813185+0000", 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2899999999999997e-06 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813238+0000", 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3229999999999997e-05 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813326+0000", 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7414e-05 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.813415+0000", 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9734000000000005e-05 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.814443+0000", 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010279060000000001 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.028872+0000", 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214428433 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.029351+0000", 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047974500000000001 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.029402+0000", 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.041e-05 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.029446+0000", 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4082999999999998e-05 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.15132.0:34105 3.7 e199/186 ondisk, result = 0)", 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:40.814633+0000", 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 220.027727019, 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21424296500000001, 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.814633+0000", 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.814633+0000", 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.814634+0000", 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9800000000000004e-07 2024-06-27T21:20:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:40.814636+0000", 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.802e-06 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.028820+0000", 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214183387 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.028827+0000", 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8260000000000004e-06 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.028863+0000", 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6772e-05 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.028868+0000", 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5510000000000001e-06 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:41.028876+0000", 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1289999999999996e-06 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:41292 3.b 3:d0ac0269:::benchmark_data_smithi192_74314_object41291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:46.476413+0000", 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 214.36594706700001, 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.234755995, 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41292 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.476413+0000", 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.476413+0000", 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.476414+0000", 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.35e-06 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.476462+0000", 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7879e-05 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.476474+0000", 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2164000000000001e-05 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.476482+0000", 2024-06-27T21:20:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2459999999999997e-06 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478460+0000", 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019787590000000001 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478687+0000", 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000226169 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478835+0000", 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014836299999999999 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480524+0000", 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016890329999999999 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480644+0000", 2024-06-27T21:20:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120066 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.711042+0000", 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.230398092 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.711098+0000", 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6258000000000002e-05 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.711169+0000", 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0616000000000003e-05 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:41325 3.7 3:e87d7191:::benchmark_data_smithi192_74314_object41324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:46.497061+0000", 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 214.34529889000001, 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21533022299999999, 2024-06-27T21:20:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.876 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.876 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.876 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41325 2024-06-27T21:20:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497061+0000", 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497061+0000", 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497062+0000", 2024-06-27T21:20:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.11e-06 2024-06-27T21:20:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497120+0000", 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7791999999999998e-05 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497139+0000", 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8814e-05 2024-06-27T21:20:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497149+0000", 2024-06-27T21:20:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8199999999999992e-06 2024-06-27T21:20:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497218+0000", 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8942999999999994e-05 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497330+0000", 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011238 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497422+0000", 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1689999999999998e-05 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.498711+0000", 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001288909 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.498720+0000", 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7110000000000007e-06 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712275+0000", 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21355497700000001 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712325+0000", 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9363999999999997e-05 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712391+0000", 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6712999999999996e-05 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.15132.0:41324 3.c e202/186 3:35d56568:::benchmark_data_smithi192_74314_object41323:head v 202'2630, mlcod=202'2627)", 2024-06-27T21:20:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:46.497304+0000", 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 214.345056602, 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216467196, 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497304+0000", 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497304+0000", 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497304+0000", 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7899999999999998e-07 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.497362+0000", 2024-06-27T21:20:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8130000000000001e-05 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712170+0000", 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214807359 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712174+0000", 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0210000000000001e-06 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712216+0000", 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2432999999999998e-05 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.712238+0000", 2024-06-27T21:20:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1506999999999999e-05 2024-06-27T21:20:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.713735+0000", 2024-06-27T21:20:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001497434 2024-06-27T21:20:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.713771+0000", 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5633000000000003e-05 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:42491 3.7 3:ed28703d:::benchmark_data_smithi192_74314_object42490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:47.294504+0000", 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 213.54785658099999, 2024-06-27T21:20:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21460622100000001, 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 42491 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294504+0000", 2024-06-27T21:20:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294504+0000", 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294505+0000", 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8299999999999995e-07 2024-06-27T21:20:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294565+0000", 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0668000000000003e-05 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294578+0000", 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2785e-05 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294586+0000", 2024-06-27T21:20:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6240000000000002e-06 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294696+0000", 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109948 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294823+0000", 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012738100000000001 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.294947+0000", 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012417299999999999 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.296771+0000", 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018241360000000001 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.299236+0000", 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024649730000000001 2024-06-27T21:20:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.508990+0000", 2024-06-27T21:20:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20975349500000001 2024-06-27T21:20:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.509057+0000", 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7317000000000006e-05 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:47.509110+0000", 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2738000000000002e-05 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:63858 3.2 3:48978ee4:::benchmark_data_smithi192_74314_object63857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:04.476943+0000", 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 196.365417412, 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216185562, 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63858 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.476943+0000", 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.476943+0000", 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.476945+0000", 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8220000000000001e-06 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.476992+0000", 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7540000000000002e-05 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477013+0000", 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1177000000000001e-05 2024-06-27T21:20:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477024+0000", 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0644e-05 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477093+0000", 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9524999999999996e-05 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477236+0000", 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014218 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477383+0000", 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014755000000000001 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.479123+0000", 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017397090000000001 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.693027+0000", 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21390368100000001 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.693062+0000", 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5382000000000001e-05 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.693098+0000", 2024-06-27T21:20:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5784000000000002e-05 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.693128+0000", 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0568000000000003e-05 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:63863 3.b 3:d8eed84c:::benchmark_data_smithi192_74314_object63862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:04.482759+0000", 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 196.35960154200001, 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22064873199999999, 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63863 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482759+0000", 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482759+0000", 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482763+0000", 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9790000000000004e-06 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482799+0000", 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6900999999999998e-05 2024-06-27T21:20:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482814+0000", 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4603999999999999e-05 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482821+0000", 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5560000000000004e-06 2024-06-27T21:20:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482869+0000", 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8816999999999997e-05 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.482977+0000", 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107533 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:20:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.483078+0000", 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010101499999999999 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.484945+0000", 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018671149999999999 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.485667+0000", 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00072143999999999995 2024-06-27T21:20:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703336+0000", 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21766940300000001 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703374+0000", 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8528000000000001e-05 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703407+0000", 2024-06-27T21:20:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2840999999999999e-05 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.15132.0:63863 3.b e209/186 ondisk, result = 0)", 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:04.485252+0000", 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 196.35710795599999, 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21816058299999999, 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:20:20.921 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.485252+0000", 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.485252+0000", 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.485253+0000", 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7999999999999993e-07 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.485258+0000", 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8300000000000003e-06 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703282+0000", 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21802437899999999 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703287+0000", 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8770000000000001e-06 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703327+0000", 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0157999999999997e-05 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703332+0000", 2024-06-27T21:20:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8729999999999998e-06 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.703413+0000", 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0486000000000002e-05 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:90963 3.b 3:d314aad7:::benchmark_data_smithi192_74314_object90962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:27.723975+0000", 2024-06-27T21:20:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 173.11838505399999, 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26558297400000003, 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90963 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.723975+0000", 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.723975+0000", 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.723976+0000", 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4200000000000004e-07 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.724047+0000", 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0893e-05 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.724060+0000", 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3172e-05 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.724067+0000", 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1949999999999997e-06 2024-06-27T21:20:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.724112+0000", 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5192999999999999e-05 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.724202+0000", 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9198999999999998e-05 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.724315+0000", 2024-06-27T21:20:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113355 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.726952+0000", 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002636514 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.735682+0000", 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087308060000000007 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.989450+0000", 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25376716500000002 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.989509+0000", 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9318999999999999e-05 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.989558+0000", 2024-06-27T21:20:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9221000000000002e-05 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.15132.0:90963 3.b e218/186 ondisk, result = 0)", 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:27.727131+0000", 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 173.11522910900001, 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26243591599999999, 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.727131+0000", 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.727131+0000", 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.727132+0000", 2024-06-27T21:20:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9400000000000005e-07 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.727139+0000", 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9349999999999997e-06 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.984973+0000", 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25783470400000003 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.984983+0000", 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7550000000000006e-06 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.989436+0000", 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044529280000000001 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.989444+0000", 2024-06-27T21:20:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7829999999999999e-06 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.989567+0000", 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012321700000000001 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.15132.0:91006 3.c e218/186 3:3f210be7:::benchmark_data_smithi192_74314_object91005:head v 218'5652, mlcod=218'5649)", 2024-06-27T21:20:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:27.763410+0000", 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 173.07895028999999, 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21955356200000001, 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.763410+0000", 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.763410+0000", 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.763411+0000", 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7199999999999998e-07 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.763458+0000", 2024-06-27T21:20:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7364000000000003e-05 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.977582+0000", 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21412394500000001 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.977598+0000", 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6552999999999999e-05 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.977665+0000", 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6326000000000002e-05 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.977680+0000", 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4714e-05 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.982268+0000", 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045880360000000002 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.982963+0000", 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00069585199999999995 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:91262 3.2 3:444f31c2:::benchmark_data_smithi192_74314_object91261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:28.195794+0000", 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 172.64656588299999, 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.229012454, 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 91262 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195794+0000", 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195794+0000", 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195795+0000", 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.184e-06 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195856+0000", 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0522000000000001e-05 2024-06-27T21:20:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195873+0000", 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.751e-05 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195887+0000", 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3363000000000001e-05 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.195968+0000", 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1229000000000004e-05 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.196117+0000", 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014847600000000001 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.196255+0000", 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013871099999999999 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.199256+0000", 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030009360000000001 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.199270+0000", 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4021e-05 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.424749+0000", 2024-06-27T21:20:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22547869800000001 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.424784+0000", 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4669000000000003e-05 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:28.424807+0000", 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3135000000000001e-05 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:156135 3.b 3:d5104d80:::benchmark_data_smithi192_74314_object49497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:18:48.180279+0000", 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 92.662080908999997, 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23998867600000001, 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156135 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180279+0000", 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180279+0000", 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180280+0000", 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9599999999999999e-07 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180284+0000", 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8840000000000004e-06 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180296+0000", 2024-06-27T21:20:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.189e-05 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180304+0000", 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.49e-06 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180356+0000", 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1527000000000003e-05 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180468+0000", 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112515 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.180568+0000", 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100233 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.184419+0000", 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003850203 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.185219+0000", 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079988599999999998 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.420144+0000", 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23492502200000001 2024-06-27T21:20:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.420206+0000", 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2266000000000003e-05 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.420268+0000", 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2063999999999994e-05 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:156150 3.7 3:e60eb879:::benchmark_data_smithi192_74314_object49512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:18:48.199761+0000", 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 92.642599391999994, 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21741544500000001, 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156150 2024-06-27T21:20:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199761+0000", 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199761+0000", 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199762+0000", 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.71e-07 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199766+0000", 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2729999999999996e-06 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199776+0000", 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0134e-05 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199782+0000", 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6760000000000004e-06 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199822+0000", 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0175000000000001e-05 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.199905+0000", 2024-06-27T21:20:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3225000000000004e-05 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.200010+0000", 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104924 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.203310+0000", 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033001079999999999 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.204024+0000", 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071364599999999996 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.417059+0000", 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21303561200000001 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.417109+0000", 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0071999999999998e-05 2024-06-27T21:20:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:18:48.417176+0000", 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6729000000000004e-05 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:172376 3.b 3:d4d8ae51:::benchmark_data_smithi192_74314_object65738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:10.777080+0000", 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 70.065279902, 2024-06-27T21:20:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22013977600000001, 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172376 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.777080+0000", 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.777080+0000", 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.777082+0000", 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.471e-06 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.777088+0000", 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4320000000000004e-06 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.777105+0000", 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7207e-05 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.777113+0000", 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4939999999999997e-06 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.779112+0000", 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001998845 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.779225+0000", 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112972 2024-06-27T21:20:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.779336+0000", 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111433 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.784387+0000", 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005050489 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.785963+0000", 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001575981 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.997068+0000", 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21110522000000001 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.997170+0000", 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010211 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:10.997220+0000", 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0121999999999999e-05 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:207898 3.b 3:dccc71c3:::benchmark_data_smithi192_74314_object101260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:00.499379+0000", 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.342981317, 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21627613400000001, 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207898 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:20:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499379+0000", 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499379+0000", 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499379+0000", 2024-06-27T21:20:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.75e-07 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499383+0000", 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.331e-06 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499392+0000", 2024-06-27T21:20:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8300000000000002e-06 2024-06-27T21:20:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499398+0000", 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.348e-06 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499444+0000", 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6346e-05 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499528+0000", 2024-06-27T21:20:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3658000000000004e-05 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.499601+0000", 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2630000000000001e-05 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.504185+0000", 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045847079999999998 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.504399+0000", 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021364600000000001 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.715547+0000", 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.211147683 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.715597+0000", 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0813000000000003e-05 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:20:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:00.715655+0000", 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7466000000000002e-05 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:20:20.977 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:20:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:20.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:20.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:21.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:21.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:21.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:21.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:21.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:21.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:21.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:21.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:21.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:21.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:21.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:21.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:21 smithi192 ceph-mon[32956]: pgmap v764: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 137 objects/s recovering 2024-06-27T21:20:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:21 smithi192 ceph-mon[32956]: osdmap e278: 12 total, 12 up, 11 in 2024-06-27T21:20:21.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:21.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:21 smithi151 ceph-mon[28965]: pgmap v764: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 137 objects/s recovering 2024-06-27T21:20:21.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:21 smithi151 ceph-mon[28965]: osdmap e278: 12 total, 12 up, 11 in 2024-06-27T21:20:21.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:21.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:20:21.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:21 smithi190 ceph-mon[32247]: pgmap v764: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 137 objects/s recovering 2024-06-27T21:20:21.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:21 smithi190 ceph-mon[32247]: osdmap e278: 12 total, 12 up, 11 in 2024-06-27T21:20:21.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:21.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:21.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:22.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:22.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:22.141 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:22.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:22.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:22.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:22.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:22.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:22.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:22.508 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:22.545 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:22.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:22.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:22.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:22.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:22.773 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:22 smithi151 ceph-mon[28965]: osdmap e279: 12 total, 12 up, 11 in 2024-06-27T21:20:22.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:22 smithi192 ceph-mon[32956]: osdmap e279: 12 total, 12 up, 11 in 2024-06-27T21:20:22.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:22.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:22.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:22.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:22.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:23.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:22 smithi190 ceph-mon[32247]: osdmap e279: 12 total, 12 up, 11 in 2024-06-27T21:20:23.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:23.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:23.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:23.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:23.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:23.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:23.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:23.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:23.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:23.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:23.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:23 smithi190 ceph-mon[32247]: pgmap v767: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 170 objects/s recovering 2024-06-27T21:20:23.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:23.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:23.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:23 smithi192 ceph-mon[32956]: pgmap v767: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 170 objects/s recovering 2024-06-27T21:20:23.839 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:23.839 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:20:23.839 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:20:23.839 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:23.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:23.878 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:23.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:23.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:23 smithi151 ceph-mon[28965]: pgmap v767: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 485/6 objects degraded (8083.333%); 0 B/s, 170 objects/s recovering 2024-06-27T21:20:23.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:24.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:24.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:24.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:24.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:24.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:24.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:24.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:24.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:24.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:24.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:20:24.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:24.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:24.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:24.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:24.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:24.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:24.795 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:24.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:24.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:24.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:24.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:24.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:24.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:25.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:25.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:25.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:25.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:25.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:25.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:25.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:25.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:25.538 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_1' removed 2024-06-27T21:20:25.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:25.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:25.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:25.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: pgmap v768: 27 pgs: 27 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s, 80 objects/s recovering 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 485/6 objects degraded (8083.333%), 1 pg degraded, 1 pg undersized) 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/771411115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:25.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:25.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:25.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:25.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:25.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: pgmap v768: 27 pgs: 27 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s, 80 objects/s recovering 2024-06-27T21:20:25.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 485/6 objects degraded (8083.333%), 1 pg degraded, 1 pg undersized) 2024-06-27T21:20:25.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/771411115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:25.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: pgmap v768: 27 pgs: 27 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s, 80 objects/s recovering 2024-06-27T21:20:25.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 485/6 objects degraded (8083.333%), 1 pg degraded, 1 pg undersized) 2024-06-27T21:20:25.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/771411115' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:20:25.966 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:25.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:25.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:25.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:26.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:26.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:26.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:26.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:26.303 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:20:26.304 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:20:26.304 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-06-27T21:20:26.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-06-27T21:20:26.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:26.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:26.457 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:26.457 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:20:26.457 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:20:26.457 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:20:26.457 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:26.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:26.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:26.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:26.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:26.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:26.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/771411115' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:20:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:26 smithi192 ceph-mon[32956]: osdmap e280: 12 total, 12 up, 11 in 2024-06-27T21:20:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:26.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:26.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/771411115' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:20:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:26 smithi151 ceph-mon[28965]: osdmap e280: 12 total, 12 up, 11 in 2024-06-27T21:20:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:26.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:26.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:26.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:26.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/771411115' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:20:26.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:26 smithi190 ceph-mon[32247]: osdmap e280: 12 total, 12 up, 11 in 2024-06-27T21:20:26.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:26.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:26.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:26.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:27.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:20:27.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:27.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:27.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:27.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:27.282 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:27.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:27.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:27.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:27.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:27.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:27.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:27.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:27 smithi190 ceph-mon[32247]: pgmap v770: 11 pgs: 11 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:27.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:27.757 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:27 smithi151 ceph-mon[28965]: pgmap v770: 11 pgs: 11 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:27.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:27.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:27.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:27.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:27.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:28.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:28.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:27 smithi192 ceph-mon[32956]: pgmap v770: 11 pgs: 11 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:28.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:28.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:28.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:28.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:28.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:28.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:28.463 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:28.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:28.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:28.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:28.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:28.646 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:20:28.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:28.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:28.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2474014113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:20:28.716 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:20:28.716 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":944,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":281,"num_osds":12,"num_up_osds":12,"osd_up_since":1719522577,"num_in_osds":11,"osd_in_since":1719523064,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11}],"num_pgs":11,"num_pools":1,"num_objects":2,"data_bytes":590368,"bytes_used":1842872320,"bytes_avail":1054102519808,"bytes_total":1055945392128},"fsmap":{"epoch":1,"btime":"2024-06-27T21:02:54:280563+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":22,"modified":"2024-06-27T21:19:47.927770+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-06-27T21:20:28.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:28.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:28.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:28.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:28.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:28.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:28.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:28.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2474014113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:15439 2.5 e110/104 2:aca3d0f0:::benchmark_data_smithi192_59818_object15438:head v 110'900, mlcod=110'898)", 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:55.565577+0000", 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 513.32004594099999, 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18853719999999999, 2024-06-27T21:20:28.922 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565577+0000", 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565577+0000", 2024-06-27T21:20:28.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565578+0000", 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.071e-06 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.565648+0000", 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9678000000000006e-05 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753307+0000", 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18765964600000001 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753322+0000", 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4707e-05 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753441+0000", 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011891 2024-06-27T21:20:28.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.753458+0000", 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7104000000000001e-05 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.754083+0000", 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00062465600000000002 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.754114+0000", 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1427999999999998e-05 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24508.0:15465 2.f e110/104 2:f871ccb0:::benchmark_data_smithi192_59818_object15464:head v 110'946, mlcod=110'944)", 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 513.30876695100005, 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.211785213, 2024-06-27T21:20:28.927 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:20:28.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576856+0000", 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4700000000000002e-07 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.576894+0000", 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7351999999999997e-05 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787747+0000", 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.210853076 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787752+0000", 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3489999999999999e-06 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787815+0000", 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2385000000000007e-05 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.787848+0000", 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3575999999999998e-05 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.788593+0000", 2024-06-27T21:20:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074521400000000001 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.788641+0000", 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7814e-05 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(2.8 158/154 [PushOp(2:1ef68a07:::benchmark_data_smithi192_59818_object66542:head, version: 125'4118, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1ef68a07:::benchmark_data_smithi192_59818_object66542:head@125'4118, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:20:28.932 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:14:41.642894+0000", 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 347.24272902899997, 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.233094159, 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642894+0000", 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642894+0000", 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642895+0000", 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0979999999999999e-06 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.642940+0000", 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4817000000000001e-05 2024-06-27T21:20:28.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.873921+0000", 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.230981196 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.873936+0000", 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5299999999999999e-05 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.874003+0000", 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7003999999999997e-05 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.874011+0000", 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5979999999999999e-06 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:41.875988+0000", 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019771459999999999 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.936 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:2786 3.b e192/186 3:d5b8f08f:::benchmark_data_smithi192_74314_object2785:head v 192'170, mlcod=192'167)", 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:16.249262+0000", 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 252.636360696, 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.212590011, 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249262+0000", 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249262+0000", 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249263+0000", 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.249315+0000", 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2621000000000003e-05 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461452+0000", 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.212136667 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461457+0000", 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5020000000000003e-06 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461502+0000", 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5300999999999999e-05 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461515+0000", 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2909e-05 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461821+0000", 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030602 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:16.461852+0000", 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1319000000000003e-05 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.941 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:6040 3.8 e192/186 3:18c08d67:::benchmark_data_smithi192_74314_object6039:head v 192'392, mlcod=192'389)", 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 250.700502868, 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201679577, 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185120+0000", 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9399999999999995e-07 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185217+0000", 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6491999999999997e-05 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386204+0000", 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20098743599999999 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386212+0000", 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5730000000000002e-06 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386285+0000", 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3151999999999994e-05 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386313+0000", 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8017999999999998e-05 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386769+0000", 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045611700000000002 2024-06-27T21:20:28.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386800+0000", 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0295e-05 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:6041 3.8 e192/186 3:1fe618f9:::benchmark_data_smithi192_74314_object6040:head v 192'393, mlcod=192'389)", 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:18.185644+0000", 2024-06-27T21:20:28.946 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 250.69997839199999, 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201844833, 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185644+0000", 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185644+0000", 2024-06-27T21:20:28.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185645+0000", 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1899999999999998e-07 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.185700+0000", 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5392000000000001e-05 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386986+0000", 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201285557 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.386990+0000", 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2830000000000003e-06 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387035+0000", 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4728000000000003e-05 2024-06-27T21:20:28.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387056+0000", 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0859000000000001e-05 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387454+0000", 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039844799999999998 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:18.387489+0000", 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5147000000000001e-05 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:11628 3.b e194/186 3:d81385b5:::benchmark_data_smithi192_74314_object11627:head v 194'698, mlcod=194'695)", 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 246.86079505399999, 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18984046600000001, 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.951 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024828+0000", 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4799999999999998e-07 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.024890+0000", 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1645999999999999e-05 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214049+0000", 2024-06-27T21:20:28.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.1891593 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214053+0000", 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.738e-06 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214095+0000", 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2193000000000001e-05 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214117+0000", 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1214999999999999e-05 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214634+0000", 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051760599999999999 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:22.214668+0000", 2024-06-27T21:20:28.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4119999999999999e-05 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:41330 3.3 3:c3cab74f:::benchmark_data_smithi192_74314_object41329:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:46.500500+0000", 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 222.38512256499999, 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20752989099999999, 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:28.956 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 41330 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500500+0000", 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500500+0000", 2024-06-27T21:20:28.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500502+0000", 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.206e-06 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500561+0000", 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.948e-05 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500572+0000", 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0818999999999999e-05 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500577+0000", 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3120000000000001e-06 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500626+0000", 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8449000000000003e-05 2024-06-27T21:20:28.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500732+0000", 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106128 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.500895+0000", 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000163282 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.502196+0000", 2024-06-27T21:20:28.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001300867 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.502914+0000", 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071834000000000004 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707895+0000", 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20498121599999999 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707998+0000", 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010217099999999999 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.708030+0000", 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2620999999999998e-05 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.962 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.15132.0:41330 3.3 e202/186 ondisk, result = 0)", 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:46.514144+0000", 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 222.37147892300001, 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19389235699999999, 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514144+0000", 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514144+0000", 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514145+0000", 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3499999999999996e-07 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.514148+0000", 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.19e-06 2024-06-27T21:20:28.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707715+0000", 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19356746599999999 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707729+0000", 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.384e-05 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707868+0000", 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013926300000000001 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.707886+0000", 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7753000000000001e-05 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.708036+0000", 2024-06-27T21:20:28.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015020999999999999 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:63857 3.3 3:ccd25bbb:::benchmark_data_smithi192_74314_object63856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:04.475553+0000", 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 204.41007009099999, 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21602158199999999, 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.970 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 63857 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475553+0000", 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475553+0000", 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475554+0000", 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4950000000000001e-06 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475608+0000", 2024-06-27T21:20:28.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3335999999999999e-05 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475626+0000", 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8119000000000001e-05 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475633+0000", 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0690000000000004e-06 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475693+0000", 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0194e-05 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.475847+0000", 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015441599999999999 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.476076+0000", 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022837399999999999 2024-06-27T21:20:28.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.478815+0000", 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027391020000000002 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.478856+0000", 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1196000000000003e-05 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691507+0000", 2024-06-27T21:20:28.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21265120200000001 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691545+0000", 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7818000000000003e-05 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691574+0000", 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9261e-05 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.976 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:63864 3.3 3:cdb1142a:::benchmark_data_smithi192_74314_object63863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:04.485645+0000", 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 204.39997765499999, 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20605995899999999, 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:28.977 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 63864 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485645+0000", 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485645+0000", 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485646+0000", 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1880000000000001e-06 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485700+0000", 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3418999999999998e-05 2024-06-27T21:20:28.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485712+0000", 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1904e-05 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485718+0000", 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0560000000000001e-06 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485756+0000", 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8611e-05 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485858+0000", 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101432 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.485953+0000", 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5674999999999995e-05 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.487617+0000", 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001663159 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.488072+0000", 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045525600000000002 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691621+0000", 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20354951299999999 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691670+0000", 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8455999999999998e-05 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.691705+0000", 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5290000000000003e-05 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:91008 3.a e218/186 3:579cf028:::benchmark_data_smithi192_74314_object91007:head v 218'5688, mlcod=218'5683)", 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.768617+0000", 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 181.117005783, 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20529583300000001, 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768617+0000", 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768617+0000", 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768618+0000", 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.71e-07 2024-06-27T21:20:28.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768644+0000", 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6098e-05 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.971789+0000", 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20314495799999999 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.971793+0000", 2024-06-27T21:20:28.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3320000000000002e-06 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.971851+0000", 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7293000000000002e-05 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.971866+0000", 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5543e-05 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.973870+0000", 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002003566 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.973913+0000", 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3171999999999999e-05 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:124242 3.3 3:c49022aa:::benchmark_data_smithi192_74314_object17604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:09.692163+0000", 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 139.193460278, 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18940742899999999, 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 124242 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.989 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692163+0000", 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692163+0000", 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692164+0000", 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1990000000000001e-06 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692169+0000", 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5339999999999999e-06 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692182+0000", 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2415999999999999e-05 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692188+0000", 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3119999999999999e-06 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692233+0000", 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4557e-05 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692312+0000", 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9734999999999995e-05 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.692381+0000", 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8601000000000003e-05 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.695921+0000", 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035399390000000002 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.696534+0000", 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00061349500000000003 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.881405+0000", 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18487020600000001 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.881480+0000", 2024-06-27T21:20:28.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5074999999999995e-05 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.881570+0000", 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0359999999999995e-05 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:137029 3.b e241/186 3:d038d6f8:::benchmark_data_smithi192_74314_object30391:head v 241'8519, mlcod=241'8515)", 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:24.872485+0000", 2024-06-27T21:20:28.995 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 124.013137812, 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19146871600000001, 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:24.872485+0000", 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:28.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:24.872485+0000", 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:24.872486+0000", 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5499999999999996e-07 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:24.872492+0000", 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2779999999999999e-06 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:25.060491+0000", 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18799860299999999 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:25.060497+0000", 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5749999999999997e-06 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:25.060548+0000", 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0382e-05 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:25.060568+0000", 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0242999999999999e-05 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:25.063915+0000", 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033473769999999999 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:25.063954+0000", 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8303000000000002e-05 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:28.999 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:153549 3.e e246/186 3:7baf14dc:::benchmark_data_smithi192_74314_object46911:head v 246'9766, mlcod=246'9763)", 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:44.778184+0000", 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 104.10743926400001, 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22610499000000001, 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.778184+0000", 2024-06-27T21:20:29.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.778184+0000", 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.778185+0000", 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5199999999999995e-07 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:29.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.778191+0000", 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8929999999999999e-06 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.999169+0000", 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.220977112 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.999178+0000", 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6579999999999997e-06 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.999241+0000", 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2898000000000007e-05 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:44.999269+0000", 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7852e-05 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:45.004250+0000", 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049814150000000003 2024-06-27T21:20:29.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:45.004289+0000", 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8210000000000002e-05 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:156135 3.b e248/186 3:d5104d80:::benchmark_data_smithi192_74314_object49497:head v 248'9747, mlcod=248'9744)", 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:48.183495+0000", 2024-06-27T21:20:29.004 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 100.702128145, 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23217312700000001, 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.183495+0000", 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:29.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.183495+0000", 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.183496+0000", 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.540000000000001e-07 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.183502+0000", 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6409999999999996e-06 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.408451+0000", 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.224949019 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.408457+0000", 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0340000000000002e-06 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.408520+0000", 2024-06-27T21:20:29.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2354999999999995e-05 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.408543+0000", 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3243000000000001e-05 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.415636+0000", 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007092976 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:29.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.415668+0000", 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1905e-05 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:170449 3.3 3:cbfdbad3:::benchmark_data_smithi192_74314_object63811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:19:07.233339+0000", 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 81.652283994000001, 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.211246138, 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:29.009 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 170449 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.233339+0000", 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:29.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.233339+0000", 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.233340+0000", 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.494e-06 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.233349+0000", 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.115e-06 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.233366+0000", 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6140000000000001e-05 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.233373+0000", 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2450000000000001e-06 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235270+0000", 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018968069999999999 2024-06-27T21:20:29.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235403+0000", 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013337200000000001 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235535+0000", 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013178199999999999 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.245088+0000", 2024-06-27T21:20:29.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0095530730000000005 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.245095+0000", 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4499999999999998e-06 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.444478+0000", 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199383059 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.444539+0000", 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0983999999999998e-05 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.444585+0000", 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5617e-05 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.015 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:170451 3.3 3:c6047184:::benchmark_data_smithi192_74314_object63813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:19:07.235881+0000", 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 81.649742313000004, 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22428039899999999, 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 170451 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235881+0000", 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235881+0000", 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235882+0000", 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0899999999999999e-06 2024-06-27T21:20:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235887+0000", 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1479999999999997e-06 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235899+0000", 2024-06-27T21:20:29.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2058e-05 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.235905+0000", 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5209999999999999e-06 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240463+0000", 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045574739999999997 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240605+0000", 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014234999999999999 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240748+0000", 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014291400000000001 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.245246+0000", 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004498129 2024-06-27T21:20:29.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.245410+0000", 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016392499999999999 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.460049+0000", 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.214639253 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:29.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.460111+0000", 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1789000000000001e-05 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.460161+0000", 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9747999999999999e-05 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:170462 3.3 3:c24eed55:::benchmark_data_smithi192_74314_object63824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-06-27T21:20:29.022 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:19:07.251755+0000", 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 81.633868093999993, 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.217943365, 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 170462 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251755+0000", 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251755+0000", 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251757+0000", 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7579999999999999e-06 2024-06-27T21:20:29.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251762+0000", 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6160000000000001e-06 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251774+0000", 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2241e-05 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251782+0000", 2024-06-27T21:20:29.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3599999999999998e-06 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251823+0000", 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1653999999999998e-05 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251910+0000", 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6404999999999997e-05 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.251979+0000", 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9484000000000001e-05 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.256096+0000", 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041164490000000003 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.256281+0000", 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000185668 2024-06-27T21:20:29.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.469590+0000", 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.213308356 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.469648+0000", 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7911000000000002e-05 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:29.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.469698+0000", 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0463000000000001e-05 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:172376 3.b e253/186 3:d4d8ae51:::benchmark_data_smithi192_74314_object65738:head v 253'10722, mlcod=253'10718)", 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:19:10.779476+0000", 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 78.106146621999997, 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21313219999999999, 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:29.029 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.779476+0000", 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.779476+0000", 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.779477+0000", 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1999999999999998e-07 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.779482+0000", 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8679999999999998e-06 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.987814+0000", 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.208331766 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.987823+0000", 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1810000000000007e-06 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.987884+0000", 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1498e-05 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.987934+0000", 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9994999999999998e-05 2024-06-27T21:20:29.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.992550+0000", 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046155939999999998 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:10.992608+0000", 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8378000000000002e-05 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:29.033 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:29.034 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:29.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:29.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:29.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:29.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2474014113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:20:29.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:29.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:29.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:29.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:29.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:29.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:29.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:29.474 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-27T21:20:29.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:20:29.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:29.548 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:29.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:29.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:29.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:29 smithi151 ceph-mon[28965]: pgmap v771: 11 pgs: 11 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:29.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:29 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:29.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2474014113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:20:29.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:29 smithi151 ceph-mon[28965]: osdmap e281: 12 total, 12 up, 11 in 2024-06-27T21:20:29.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3614461188' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-27T21:20:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:29 smithi190 ceph-mon[32247]: pgmap v771: 11 pgs: 11 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:29 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2474014113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:20:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:29 smithi190 ceph-mon[32247]: osdmap e281: 12 total, 12 up, 11 in 2024-06-27T21:20:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3614461188' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-27T21:20:29.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:29.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:29.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:20:29.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:29.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:29.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:30.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:29 smithi192 ceph-mon[32956]: pgmap v771: 11 pgs: 11 active+clean; 577 KiB data, 1.7 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:29 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2474014113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:20:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:29 smithi192 ceph-mon[32956]: osdmap e281: 12 total, 12 up, 11 in 2024-06-27T21:20:30.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3614461188' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-06-27T21:20:30.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:30.119 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:30.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:30.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:30.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:30.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:30.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:30.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:30.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:30.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:30.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:30.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:30.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:30.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:30.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:20:30.854 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:20:30.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:30 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:20:30.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:30 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:20:30.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:30 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:20:30.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:30 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:20:30.880 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:20:30.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:20:30.901 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:20:30.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:20:30.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:30.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:30 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:20:30.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:30 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:20:30.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:30 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:20:30.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:30 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:20:30.922 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:20:30.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:20:30.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:30.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:30.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:20:30.945 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:20:30.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:20:30.967 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:20:30.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:20:30.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:20:31.012 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:20:31.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:20:31.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:31.035 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:20:31.035 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:20:31.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:30 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:20:31.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:30 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:20:31.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:30 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:20:31.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:30 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:20:31.057 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:20:31.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:20:31.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:31.080 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:20:31.081 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:20:31.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:20:31.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:20:31.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:31.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:31.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:31.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:31.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:31.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:31.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:31.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:31.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:31.666 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:31.667 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:31.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:31 smithi190 ceph-mon[32247]: pgmap v773: 11 pgs: 11 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1167767214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:31 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:20:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:31 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:20:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:31 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:20:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:31 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:20:31.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:31.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:31.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:31.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:31.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:31.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:31 smithi151 ceph-mon[28965]: pgmap v773: 11 pgs: 11 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:31.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1167767214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:31.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:31 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:20:31.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:31 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:20:31.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:31 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:20:31.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:31 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:20:32.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:32.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:31 smithi192 ceph-mon[32956]: pgmap v773: 11 pgs: 11 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1167767214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:31 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:31 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:31 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:20:32.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:31 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:20:32.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:32.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:32.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:32.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:32.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:32.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:32.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:32.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:32.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:32.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:20:32.659 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_2' created 2024-06-27T21:20:32.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:32.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:32.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:32.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:32.874 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:32.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:32.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:32.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:32.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:32 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:20:32.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:32 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:20:32.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1950139222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-27T21:20:32.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:32 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:20:32.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:32 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:20:32.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1950139222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-27T21:20:33.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:33.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:32 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:20:33.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:32 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:20:33.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1950139222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-06-27T21:20:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:33.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:33.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:33.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:33.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:33.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:33.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-06-27T21:20:33.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:20:33.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:33.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:33.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:33.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:33.661 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:33 smithi190 ceph-mon[32247]: pgmap v774: 11 pgs: 11 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:33.661 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:33 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:20:33.661 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:33 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:20:33.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1950139222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-27T21:20:33.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:33 smithi190 ceph-mon[32247]: osdmap e282: 12 total, 12 up, 11 in 2024-06-27T21:20:33.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:33.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:33.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:33.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:33.828 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:33.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:33.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:33 smithi151 ceph-mon[28965]: pgmap v774: 11 pgs: 11 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:33.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:33 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:20:33.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:33 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:20:33.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1950139222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-27T21:20:33.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:33 smithi151 ceph-mon[28965]: osdmap e282: 12 total, 12 up, 11 in 2024-06-27T21:20:33.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:33.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:34.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:33 smithi192 ceph-mon[32956]: pgmap v774: 11 pgs: 11 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:34.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:33 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:20:34.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:33 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:20:34.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1950139222' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-06-27T21:20:34.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:33 smithi192 ceph-mon[32956]: osdmap e282: 12 total, 12 up, 11 in 2024-06-27T21:20:34.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:34.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:34.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:34.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:34.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:34.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:34.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:20:34.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:34.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:34.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:34.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:34.604 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:34.604 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:34.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:34.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:34.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:34.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:34.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:34.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:34.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:34.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: osdmap e283: 12 total, 12 up, 11 in 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:34.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:34.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:20:34.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:34 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:20:34.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: osdmap e283: 12 total, 12 up, 11 in 2024-06-27T21:20:34.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:20:34.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:34 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:20:35.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: osdmap e283: 12 total, 12 up, 11 in 2024-06-27T21:20:35.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:35.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:35.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:35.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:35.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:35.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:35.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:35.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:20:35.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:34 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:20:35.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:35.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:35.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:35.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:35.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:20:35.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:35.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:35.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:35.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:35.474 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:35.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:35.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:35.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:35.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:35 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:20:35.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:35 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:20:35.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:35 smithi151 ceph-mon[28965]: osdmap e284: 12 total, 12 up, 11 in 2024-06-27T21:20:35.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:35.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:35.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:35.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:35 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:20:35.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:35 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:20:35.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:35 smithi190 ceph-mon[32247]: osdmap e284: 12 total, 12 up, 11 in 2024-06-27T21:20:35.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:35.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:35.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:36.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:35 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:20:36.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:35 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:20:36.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:35 smithi192 ceph-mon[32956]: osdmap e284: 12 total, 12 up, 11 in 2024-06-27T21:20:36.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:36.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:36.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:36.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:36.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:36.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:36.417 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:36.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:36.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:36.449 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:36.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:36.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:36.619 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:20:36.620 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:20:36.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:36.677 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:20:36.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:36.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:36.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:36.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:36.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:36.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1342982095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:36.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:36.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:36.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:36.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1342982095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:36.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:36.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:36.996 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:36.996 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:37.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:37.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:37.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1342982095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:37.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:37.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:37.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:37.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:37.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:37.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:37.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:37.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:37.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:37.534 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":779,"stamp":"2024-06-27T21:20:35.939219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1854224,"kb_used_data":11912,"kb_used_omap":17,"kb_used_meta":1842158,"kb_avail":1029342448,"statfs":{"total":1055945392128,"available":1054046666752,"internally_reserved":0,"allocated":12197888,"data_stored":8185163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1886370677},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001020"},"pg_stats":[{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":282,"state":"creating+peering","last_fresh":"2024-06-27T21:20:32.668884+0000","last_change":"2024-06-27T21:20:32.668884+0000","last_active":"2024-06-27T21:20:32.657175+0000","last_peered":"2024-06-27T21:20:32.657175+0000","last_clean":"2024-06-27T21:20:32.657175+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-27T21:20:32.668884+0000","last_undegraded":"2024-06-27T21:20:32.668884+0000","last_fullsized":"2024-06-27T21:20:32.668884+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-06-27T21:20:35.729810+0000","last_change":"2024-06-27T21:20:33.674258+0000","last_active":"2024-06-27T21:20:35.729810+0000","last_peered":"2024-06-27T21:20:35.729810+0000","last_clean":"2024-06-27T21:20:35.729810+0000","last_became_active":"2024-06-27T21:20:33.673842+0000","last_became_peered":"2024-06-27T21:20:33.673842+0000","last_unstale":"2024-06-27T21:20:35.729810+0000","last_undegraded":"2024-06-27T21:20:35.729810+0000","last_fullsized":"2024-06-27T21:20:35.729810+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:47.214701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.960570+0000","last_change":"2024-06-27T21:20:33.960114+0000","last_active":"2024-06-27T21:20:33.960570+0000","last_peered":"2024-06-27T21:20:33.960570+0000","last_clean":"2024-06-27T21:20:33.960570+0000","last_became_active":"2024-06-27T21:20:33.959756+0000","last_became_peered":"2024-06-27T21:20:33.959756+0000","last_unstale":"2024-06-27T21:20:33.960570+0000","last_undegraded":"2024-06-27T21:20:33.960570+0000","last_fullsized":"2024-06-27T21:20:33.960570+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:38.161350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.673692+0000","last_change":"2024-06-27T21:20:33.673013+0000","last_active":"2024-06-27T21:20:33.673692+0000","last_peered":"2024-06-27T21:20:33.673692+0000","last_clean":"2024-06-27T21:20:33.673692+0000","last_became_active":"2024-06-27T21:20:33.672659+0000","last_became_peered":"2024-06-27T21:20:33.672659+0000","last_unstale":"2024-06-27T21:20:33.673692+0000","last_undegraded":"2024-06-27T21:20:33.673692+0000","last_fullsized":"2024-06-27T21:20:33.673692+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:53.883713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.962244+0000","last_change":"2024-06-27T21:20:33.961841+0000","last_active":"2024-06-27T21:20:33.962244+0000","last_peered":"2024-06-27T21:20:33.962244+0000","last_clean":"2024-06-27T21:20:33.962244+0000","last_became_active":"2024-06-27T21:20:33.961523+0000","last_became_peered":"2024-06-27T21:20:33.961523+0000","last_unstale":"2024-06-27T21:20:33.962244+0000","last_undegraded":"2024-06-27T21:20:33.962244+0000","last_fullsized":"2024-06-27T21:20:33.962244+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:37.868065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.961916+0000","last_change":"2024-06-27T21:20:33.961410+0000","last_active":"2024-06-27T21:20:33.961916+0000","last_peered":"2024-06-27T21:20:33.961916+0000","last_clean":"2024-06-27T21:20:33.961916+0000","last_became_active":"2024-06-27T21:20:33.961114+0000","last_became_peered":"2024-06-27T21:20:33.961114+0000","last_unstale":"2024-06-27T21:20:33.961916+0000","last_undegraded":"2024-06-27T21:20:33.961916+0000","last_fullsized":"2024-06-27T21:20:33.961916+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:44.089251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":570,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.959706+0000","last_change":"2024-06-27T21:19:46.475053+0000","last_active":"2024-06-27T21:20:33.959706+0000","last_peered":"2024-06-27T21:20:33.959706+0000","last_clean":"2024-06-27T21:20:33.959706+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:20:33.959706+0000","last_undegraded":"2024-06-27T21:20:33.959706+0000","last_fullsized":"2024-06-27T21:20:33.959706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:46.474944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:46.474944+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:48.986062+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.961742+0000","last_change":"2024-06-27T21:20:33.961290+0000","last_active":"2024-06-27T21:20:33.961742+0000","last_peered":"2024-06-27T21:20:33.961742+0000","last_clean":"2024-06-27T21:20:33.961742+0000","last_became_active":"2024-06-27T21:20:33.960775+0000","last_became_peered":"2024-06-27T21:20:33.960775+0000","last_unstale":"2024-06-27T21:20:33.961742+0000","last_undegraded":"2024-06-27T21:20:33.961742+0000","last_fullsized":"2024-06-27T21:20:33.961742+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:46.979078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":163,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:34.359794+0000","last_change":"2024-06-27T21:20:34.359816+0000","last_active":"2024-06-27T21:20:34.359794+0000","last_peered":"2024-06-27T21:20:34.359794+0000","last_clean":"2024-06-27T21:20:34.359794+0000","last_became_active":"2024-06-27T21:17:58.594952+0000","last_became_peered":"2024-06-27T21:17:58.594952+0000","last_unstale":"2024-06-27T21:20:34.359794+0000","last_undegraded":"2024-06-27T21:20:34.359794+0000","last_fullsized":"2024-06-27T21:20:34.359794+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:34.359736+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:20:34.359736+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:54.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":4,"reported_epoch":283,"state":"creating+activating","last_fresh":"2024-06-27T21:20:33.673528+0000","last_change":"2024-06-27T21:20:33.671429+0000","last_active":"2024-06-27T21:20:32.657175+0000","last_peered":"2024-06-27T21:20:32.657175+0000","last_clean":"2024-06-27T21:20:32.657175+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-27T21:20:33.673528+0000","last_undegraded":"2024-06-27T21:20:33.673528+0000","last_fullsized":"2024-06-27T21:20:33.673528+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":["1","0","10"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":582,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.959672+0000","last_change":"2024-06-27T21:20:30.322903+0000","last_active":"2024-06-27T21:20:33.959672+0000","last_peered":"2024-06-27T21:20:33.959672+0000","last_clean":"2024-06-27T21:20:33.959672+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:20:33.959672+0000","last_undegraded":"2024-06-27T21:20:33.959672+0000","last_fullsized":"2024-06-27T21:20:33.959672+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:30.322810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:30.322810+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:56.787406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-27T21:20:32.657175+0000","last_change":"2024-06-27T21:20:32.657175+0000","last_active":"2024-06-27T21:20:32.657175+0000","last_peered":"2024-06-27T21:20:32.657175+0000","last_clean":"2024-06-27T21:20:32.657175+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-27T21:20:32.657175+0000","last_undegraded":"2024-06-27T21:20:32.657175+0000","last_fullsized":"2024-06-27T21:20:32.657175+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":289,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.958923+0000","last_change":"2024-06-27T21:20:30.922701+0000","last_active":"2024-06-27T21:20:33.958923+0000","last_peered":"2024-06-27T21:20:33.958923+0000","last_clean":"2024-06-27T21:20:33.958923+0000","last_became_active":"2024-06-27T21:14:55.718653+0000","last_became_peered":"2024-06-27T21:14:55.718653+0000","last_unstale":"2024-06-27T21:20:33.958923+0000","last_undegraded":"2024-06-27T21:20:33.958923+0000","last_fullsized":"2024-06-27T21:20:33.958923+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:30.922606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:30.922606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:32.414507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.677162+0000","last_change":"2024-06-27T21:20:33.676832+0000","last_active":"2024-06-27T21:20:33.677162+0000","last_peered":"2024-06-27T21:20:33.677162+0000","last_clean":"2024-06-27T21:20:33.677162+0000","last_became_active":"2024-06-27T21:20:33.676501+0000","last_became_peered":"2024-06-27T21:20:33.676501+0000","last_unstale":"2024-06-27T21:20:33.677162+0000","last_undegraded":"2024-06-27T21:20:33.677162+0000","last_fullsized":"2024-06-27T21:20:33.677162+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:46.240537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":299,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.669427+0000","last_change":"2024-06-27T21:20:32.377288+0000","last_active":"2024-06-27T21:20:33.669427+0000","last_peered":"2024-06-27T21:20:33.669427+0000","last_clean":"2024-06-27T21:20:33.669427+0000","last_became_active":"2024-06-27T21:14:55.861874+0000","last_became_peered":"2024-06-27T21:14:55.861874+0000","last_unstale":"2024-06-27T21:20:33.669427+0000","last_undegraded":"2024-06-27T21:20:33.669427+0000","last_fullsized":"2024-06-27T21:20:33.669427+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:32.377138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.377138+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:46.615022+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.962030+0000","last_change":"2024-06-27T21:20:33.961473+0000","last_active":"2024-06-27T21:20:33.962030+0000","last_peered":"2024-06-27T21:20:33.962030+0000","last_clean":"2024-06-27T21:20:33.962030+0000","last_became_active":"2024-06-27T21:20:33.961025+0000","last_became_peered":"2024-06-27T21:20:33.961025+0000","last_unstale":"2024-06-27T21:20:33.962030+0000","last_undegraded":"2024-06-27T21:20:33.962030+0000","last_fullsized":"2024-06-27T21:20:33.962030+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:00.916695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":289,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.959984+0000","last_change":"2024-06-27T21:20:33.960049+0000","last_active":"2024-06-27T21:20:33.959984+0000","last_peered":"2024-06-27T21:20:33.959984+0000","last_clean":"2024-06-27T21:20:33.959984+0000","last_became_active":"2024-06-27T21:14:55.711705+0000","last_became_peered":"2024-06-27T21:14:55.711705+0000","last_unstale":"2024-06-27T21:20:33.959984+0000","last_undegraded":"2024-06-27T21:20:33.959984+0000","last_fullsized":"2024-06-27T21:20:33.959984+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:33.959884+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:20:33.959884+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:35.765314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-06-27T21:20:32.657175+0000","last_change":"2024-06-27T21:20:32.657175+0000","last_active":"2024-06-27T21:20:32.657175+0000","last_peered":"2024-06-27T21:20:32.657175+0000","last_clean":"2024-06-27T21:20:32.657175+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-27T21:20:32.657175+0000","last_undegraded":"2024-06-27T21:20:32.657175+0000","last_fullsized":"2024-06-27T21:20:32.657175+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":287,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.958301+0000","last_change":"2024-06-27T21:19:40.674095+0000","last_active":"2024-06-27T21:20:33.958301+0000","last_peered":"2024-06-27T21:20:33.958301+0000","last_clean":"2024-06-27T21:20:33.958301+0000","last_became_active":"2024-06-27T21:17:45.838234+0000","last_became_peered":"2024-06-27T21:17:45.838234+0000","last_unstale":"2024-06-27T21:20:33.958301+0000","last_undegraded":"2024-06-27T21:20:33.958301+0000","last_fullsized":"2024-06-27T21:20:33.958301+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:40.673987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:40.673987+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:06.405572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.677273+0000","last_change":"2024-06-27T21:20:33.676743+0000","last_active":"2024-06-27T21:20:33.677273+0000","last_peered":"2024-06-27T21:20:33.677273+0000","last_clean":"2024-06-27T21:20:33.677273+0000","last_became_active":"2024-06-27T21:20:33.676398+0000","last_became_peered":"2024-06-27T21:20:33.676398+0000","last_unstale":"2024-06-27T21:20:33.677273+0000","last_undegraded":"2024-06-27T21:20:33.677273+0000","last_fullsized":"2024-06-27T21:20:33.677273+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:58.784233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":378,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.669383+0000","last_change":"2024-06-27T21:19:50.905384+0000","last_active":"2024-06-27T21:20:33.669383+0000","last_peered":"2024-06-27T21:20:33.669383+0000","last_clean":"2024-06-27T21:20:33.669383+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:20:33.669383+0000","last_undegraded":"2024-06-27T21:20:33.669383+0000","last_fullsized":"2024-06-27T21:20:33.669383+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:50.905308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:50.905308+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:11.504423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.675034+0000","last_change":"2024-06-27T21:20:33.674679+0000","last_active":"2024-06-27T21:20:33.675034+0000","last_peered":"2024-06-27T21:20:33.675034+0000","last_clean":"2024-06-27T21:20:33.675034+0000","last_became_active":"2024-06-27T21:20:33.674388+0000","last_became_peered":"2024-06-27T21:20:33.674388+0000","last_unstale":"2024-06-27T21:20:33.675034+0000","last_undegraded":"2024-06-27T21:20:33.675034+0000","last_fullsized":"2024-06-27T21:20:33.675034+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:47.542741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":290,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.673161+0000","last_change":"2024-06-27T21:20:31.763820+0000","last_active":"2024-06-27T21:20:33.673161+0000","last_peered":"2024-06-27T21:20:33.673161+0000","last_clean":"2024-06-27T21:20:33.673161+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:20:33.673161+0000","last_undegraded":"2024-06-27T21:20:33.673161+0000","last_fullsized":"2024-06-27T21:20:33.673161+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.763737+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.763737+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:53.584279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":282,"state":"creating+peering","last_fresh":"2024-06-27T21:20:32.668895+0000","last_change":"2024-06-27T21:20:32.668895+0000","last_active":"2024-06-27T21:20:32.657175+0000","last_peered":"2024-06-27T21:20:32.657175+0000","last_clean":"2024-06-27T21:20:32.657175+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-27T21:20:32.668895+0000","last_undegraded":"2024-06-27T21:20:32.668895+0000","last_fullsized":"2024-06-27T21:20:32.668895+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":588,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-06-27T21:20:33.959581+0000","last_change":"2024-06-27T21:20:31.336781+0000","last_active":"2024-06-27T21:20:33.959581+0000","last_peered":"2024-06-27T21:20:33.959581+0000","last_clean":"2024-06-27T21:20:33.959581+0000","last_became_active":"2024-06-27T21:17:45.505725+0000","last_became_peered":"2024-06-27T21:17:45.505725+0000","last_unstale":"2024-06-27T21:20:33.959581+0000","last_undegraded":"2024-06-27T21:20:33.959581+0000","last_fullsized":"2024-06-27T21:20:33.959581+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.336705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.336705+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:54.989693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":282,"state":"creating+peering","last_fresh":"2024-06-27T21:20:32.668939+0000","last_change":"2024-06-27T21:20:32.668939+0000","last_active":"2024-06-27T21:20:32.657175+0000","last_peered":"2024-06-27T21:20:32.657175+0000","last_clean":"2024-06-27T21:20:32.657175+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-06-27T21:20:32.668939+0000","last_undegraded":"2024-06-27T21:20:32.668939+0000","last_fullsized":"2024-06-27T21:20:32.668939+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":294,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-06-27T21:20:29.675016+0000","last_change":"2024-06-27T21:20:29.675042+0000","last_active":"2024-06-27T21:20:29.675016+0000","last_peered":"2024-06-27T21:20:29.675016+0000","last_clean":"2024-06-27T21:20:29.675016+0000","last_became_active":"2024-06-27T21:17:45.502472+0000","last_became_peered":"2024-06-27T21:17:45.502472+0000","last_unstale":"2024-06-27T21:20:29.675016+0000","last_undegraded":"2024-06-27T21:20:29.675016+0000","last_fullsized":"2024-06-27T21:20:29.675016+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:29.674960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:29.674960+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:42.541237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":42,"acting":42,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252999,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225416,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93519736,"statfs":{"total":95995035648,"available":95764209664,"internally_reserved":0,"allocated":1302528,"data_stored":934723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514636,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80696,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93664456,"statfs":{"total":95995035648,"available":95912402944,"internally_reserved":0,"allocated":962560,"data_stored":599260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743568,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144580,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600572,"statfs":{"total":95995035648,"available":95846985728,"internally_reserved":0,"allocated":974848,"data_stored":607043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972500,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181172,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93563980,"statfs":{"total":95995035648,"available":95809515520,"internally_reserved":0,"allocated":1482752,"data_stored":1120089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158788,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93586364,"statfs":{"total":95995035648,"available":95832436736,"internally_reserved":0,"allocated":1302528,"data_stored":934723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528478,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208452,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93536700,"statfs":{"total":95995035648,"available":95781580800,"internally_reserved":0,"allocated":974848,"data_stored":607043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130868,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614284,"statfs":{"total":95995035648,"available":95861026816,"internally_reserved":0,"allocated":1155072,"data_stored":792409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986344,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190380,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554772,"statfs":{"total":95995035648,"available":95800086528,"internally_reserved":0,"allocated":950272,"data_stored":591161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190580,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554572,"statfs":{"total":95995035648,"available":95799881728,"internally_reserved":0,"allocated":1155072,"data_stored":792409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":193984967},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411505,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162616,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582536,"statfs":{"total":95995035648,"available":95828516864,"internally_reserved":0,"allocated":962560,"data_stored":599260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738555,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180676,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564476,"statfs":{"total":95995035648,"available":95810023424,"internally_reserved":0,"allocated":974848,"data_stored":607043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:20:37.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:37.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:37.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:20:37.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:37.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:37.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:37.835 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:37.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:37.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:37.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: pgmap v779: 27 pgs: 1 creating+activating, 3 creating+peering, 2 unknown, 21 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1342982095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:37.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: osdmap e285: 12 total, 12 up, 11 in 2024-06-27T21:20:37.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:37.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:37.958 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: pgmap v779: 27 pgs: 1 creating+activating, 3 creating+peering, 2 unknown, 21 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1342982095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: osdmap e285: 12 total, 12 up, 11 in 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:37.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:38.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: pgmap v779: 27 pgs: 1 creating+activating, 3 creating+peering, 2 unknown, 21 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1342982095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: osdmap e285: 12 total, 12 up, 11 in 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:38.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:38.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:38.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:38.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:38.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:38.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:38.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:38.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:38.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:38.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:38.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:38.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:38.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:38.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:38.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:38.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:38.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:38 smithi151 ceph-mon[28965]: osdmap e286: 12 total, 12 up, 11 in 2024-06-27T21:20:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:38.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:38.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:39.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:38 smithi190 ceph-mon[32247]: osdmap e286: 12 total, 12 up, 11 in 2024-06-27T21:20:39.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:38 smithi192 ceph-mon[32956]: osdmap e286: 12 total, 12 up, 11 in 2024-06-27T21:20:39.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:39.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:39.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:39.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:39.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:39.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:39.383 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:39.383 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:39.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:39.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:39.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:39.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:39.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:39.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:39.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:39.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:39.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:39.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:39.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:39.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:39 smithi151 ceph-mon[28965]: pgmap v782: 27 pgs: 1 creating+activating, 26 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:39.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:39 smithi151 ceph-mon[28965]: osdmap e287: 12 total, 12 up, 11 in 2024-06-27T21:20:39.961 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:39 smithi190 ceph-mon[32247]: pgmap v782: 27 pgs: 1 creating+activating, 26 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:39.961 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:39 smithi190 ceph-mon[32247]: osdmap e287: 12 total, 12 up, 11 in 2024-06-27T21:20:39.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:20:40.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:40.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:39 smithi192 ceph-mon[32956]: pgmap v782: 27 pgs: 1 creating+activating, 26 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:40.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:39 smithi192 ceph-mon[32956]: osdmap e287: 12 total, 12 up, 11 in 2024-06-27T21:20:40.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:40.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:40.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:40.239 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:40.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:40.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:40.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:40.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:40.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:20:40.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:40.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:40.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:40.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:40.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:40.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:40.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:20:40.927 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:20:40.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:20:40.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:20:40.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:40.961 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:40.971 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:20:40.972 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:20:40.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:40.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:40.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:40.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:40.995 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:20:40.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:20:41.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:41.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:41.018 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:20:41.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:20:41.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:20:41.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:20:41.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:20:41.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:41.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:41.061 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:20:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:20:41.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:41.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:20:41.084 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:20:41.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:41.108 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:20:41.108 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:20:41.130 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:20:41.130 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:20:41.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:20:41.152 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:20:41.174 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:20:41.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:20:41.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:41.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:41.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:41.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:41.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:41.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:41.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:41.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:41.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:41.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:41.703 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-06-27T21:20:41.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:41.788 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:41.788 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:20:41.788 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:20:41.789 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:20:41.789 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:41.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:41.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:41.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:41 smithi151 ceph-mon[28965]: pgmap v784: 27 pgs: 27 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:41.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4095100842' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:20:41.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2914173040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:41.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:41.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:41.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:42.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:41 smithi190 ceph-mon[32247]: pgmap v784: 27 pgs: 27 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:42.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4095100842' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:20:42.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2914173040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:42.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:41 smithi192 ceph-mon[32956]: pgmap v784: 27 pgs: 27 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:42.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4095100842' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:20:42.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2914173040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:42.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:42.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:42.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:42.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:42.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:42.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:42.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:20:42.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:42.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:42.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:42.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:42.661 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:42.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:42.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:42.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:42.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:42.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4095100842' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:20:42.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: osdmap e288: 12 total, 12 up, 11 in 2024-06-27T21:20:42.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:42.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:42.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4095100842' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:20:42.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: osdmap e288: 12 total, 12 up, 11 in 2024-06-27T21:20:42.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:42.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:42.959 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:42.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:42.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:42.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:42.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:42.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:42.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:43.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4095100842' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:20:43.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: osdmap e288: 12 total, 12 up, 11 in 2024-06-27T21:20:43.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:20:43.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:20:43.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:20:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:20:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:20:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:20:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:20:43.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:43.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:43.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:43.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:43.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:43.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:43.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:43.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:43.473 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:43.531 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-06-27T21:20:43.534 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:20:43.535 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:20:43.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:43.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:43.597 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:20:43.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:20:43.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_88961 2024-06-27T21:20:43.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:20:43.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:20:43.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:43.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:43.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:43.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:43.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:20:43.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:43.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:43.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:43.974 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:43 smithi151 ceph-mon[28965]: pgmap v786: 27 pgs: 27 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:43.974 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:43 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:20:43.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:43 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:20:44.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:43 smithi190 ceph-mon[32247]: pgmap v786: 27 pgs: 27 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:44.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:43 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:20:44.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:43 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:20:44.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:43 smithi192 ceph-mon[32956]: pgmap v786: 27 pgs: 27 active+clean; 577 KiB data, 1.8 GiB used, 982 GiB / 983 GiB avail 2024-06-27T21:20:44.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:43 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:20:44.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:43 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:20:44.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:44.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:44.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:44.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:44.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:44.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:44.344 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:44.344 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:20:44.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:20:44.344 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:15444 2.7 2:e9beafb2:::benchmark_data_smithi192_59818_object15443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:11:55.566933+0000", 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 528.77867332000005, 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22234326200000001, 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.345 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 15444 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566933+0000", 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566933+0000", 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566935+0000", 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4589999999999999e-06 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566983+0000", 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8597999999999998e-05 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566994+0000", 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0373e-05 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.566999+0000", 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4389999999999999e-06 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567045+0000", 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5927000000000003e-05 2024-06-27T21:20:44.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567159+0000", 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113916 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567276+0000", 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011696699999999999 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.567884+0000", 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000608005 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.568622+0000", 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073796400000000002 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.789206+0000", 2024-06-27T21:20:44.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22058386599999999 2024-06-27T21:20:44.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.789239+0000", 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3337000000000003e-05 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:11:55.789277+0000", 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7410999999999998e-05 2024-06-27T21:20:44.352 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:43157 2.7 2:e7d51da8:::benchmark_data_smithi192_59818_object43156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:30.778192+0000", 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 493.56741498600002, 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22017993299999999, 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.353 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.354 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.354 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 43157 2024-06-27T21:20:44.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.354 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778192+0000", 2024-06-27T21:20:44.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778192+0000", 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778193+0000", 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4139999999999999e-06 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778253+0000", 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9471e-05 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778264+0000", 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0968000000000001e-05 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778269+0000", 2024-06-27T21:20:44.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4609999999999999e-06 2024-06-27T21:20:44.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778321+0000", 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2071999999999999e-05 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778442+0000", 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012131200000000001 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.778586+0000", 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014394200000000001 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779754+0000", 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011675749999999999 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780861+0000", 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001107219 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.998270+0000", 2024-06-27T21:20:44.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21740916099999999 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.998326+0000", 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5892e-05 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.998372+0000", 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5445999999999998e-05 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:43158 2.7 2:ee159f29:::benchmark_data_smithi192_59818_object43157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 493.56571477900002, 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22168315799999999, 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.362 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.363 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.363 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.363 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.363 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 43158 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779892+0000", 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.44e-07 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779952+0000", 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9070000000000002e-05 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779960+0000", 2024-06-27T21:20:44.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4570000000000005e-06 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779964+0000", 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0770000000000001e-06 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.779995+0000", 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1319000000000003e-05 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780079+0000", 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3338999999999996e-05 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780162+0000", 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3011999999999997e-05 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.780961+0000", 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079940899999999995 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:30.781132+0000", 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017053899999999999 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:31.001476+0000", 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22034437300000001 2024-06-27T21:20:44.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:31.001530+0000", 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3934000000000002e-05 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:31.001575+0000", 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5084999999999999e-05 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:48511 2.7 2:e603054b:::benchmark_data_smithi192_59818_object48510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.481199+0000", 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 489.86440763299998, 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.234965234, 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.370 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 48511 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481199+0000", 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481199+0000", 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481201+0000", 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2840000000000001e-06 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481250+0000", 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8384999999999998e-05 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481279+0000", 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9283000000000001e-05 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481295+0000", 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5472e-05 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481396+0000", 2024-06-27T21:20:44.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101228 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481513+0000", 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117568 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.481600+0000", 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6884000000000004e-05 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.482792+0000", 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011914639999999999 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483325+0000", 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053293100000000005 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716091+0000", 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23276621 2024-06-27T21:20:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716124+0000", 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2898999999999998e-05 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716164+0000", 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0626e-05 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.376 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:48511 2.7 e123/104 ondisk, result = 0)", 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:12:34.483126+0000", 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 489.86248074899999, 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23304767200000001, 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483126+0000", 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483126+0000", 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483127+0000", 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.483130+0000", 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0520000000000002e-06 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.714854+0000", 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23172449000000001 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.714858+0000", 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0090000000000001e-06 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716081+0000", 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001222929 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716086+0000", 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1329999999999998e-06 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:12:34.716174+0000", 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7328000000000001e-05 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:79085 2.7 2:e5effe75:::benchmark_data_smithi192_59818_object79084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:01.904400+0000", 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 462.44120647599999, 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21667033599999999, 2024-06-27T21:20:44.381 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 79085 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904400+0000", 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904400+0000", 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904402+0000", 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.545e-06 2024-06-27T21:20:44.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904448+0000", 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5906999999999999e-05 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904458+0000", 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0796e-05 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904464+0000", 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2140000000000004e-06 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904520+0000", 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6617999999999999e-05 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904643+0000", 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012265799999999999 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.904785+0000", 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014182600000000001 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.906755+0000", 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019700809999999998 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:01.907117+0000", 2024-06-27T21:20:44.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000361689 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.120970+0000", 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21385347399999999 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.121026+0000", 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6329999999999998e-05 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:02.121071+0000", 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4197999999999999e-05 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121181 2.7 2:edd475b0:::benchmark_data_smithi192_59818_object29106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.209941+0000", 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 419.13566588600003, 2024-06-27T21:20:44.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.248862888, 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121181 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209941+0000", 2024-06-27T21:20:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209941+0000", 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209943+0000", 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.747e-06 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209949+0000", 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3570000000000003e-06 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209966+0000", 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6786000000000002e-05 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.209976+0000", 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7969999999999995e-06 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.210053+0000", 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7533999999999993e-05 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.210179+0000", 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125926 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.210304+0000", 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012508699999999999 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.213723+0000", 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003419395 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.227334+0000", 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013610224000000001 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.458736+0000", 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.231402099 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.458776+0000", 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0691000000000001e-05 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.458804+0000", 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7245000000000001e-05 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121194 2.7 2:ede15273:::benchmark_data_smithi192_59818_object29119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.235274+0000", 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 419.11033304, 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.233618453, 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.395 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121194 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235274+0000", 2024-06-27T21:20:44.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235274+0000", 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235275+0000", 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5400000000000001e-06 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235298+0000", 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2521e-05 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235310+0000", 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2680000000000001e-05 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235316+0000", 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999998e-06 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235363+0000", 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6100999999999998e-05 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235443+0000", 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0623000000000004e-05 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.235508+0000", 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5108000000000001e-05 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.238187+0000", 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026788820000000001 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.238196+0000", 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.251e-06 2024-06-27T21:20:44.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.468818+0000", 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.230621254 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.468863+0000", 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5043999999999998e-05 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.468892+0000", 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9408999999999999e-05 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121370 2.7 2:e447b26c:::benchmark_data_smithi192_59818_object29295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:45.665171+0000", 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 418.680435676, 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31690380899999998, 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.402 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121370 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665171+0000", 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665171+0000", 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665172+0000", 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.302e-06 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665177+0000", 2024-06-27T21:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9309999999999999e-06 2024-06-27T21:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665189+0000", 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1612e-05 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665195+0000", 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.755e-06 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665252+0000", 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7067000000000001e-05 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665383+0000", 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000131765 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.665504+0000", 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120136 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.668112+0000", 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002607945 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.668584+0000", 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047197599999999998 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.982000+0000", 2024-06-27T21:20:44.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31341653200000003 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.982040+0000", 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0159e-05 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:45.982075+0000", 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4629000000000003e-05 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.416 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(2.0 e176,142 2:096fcc4d:::benchmark_data_smithi192_59818_object22725:head,139'7076)", 2024-06-27T21:20:44.416 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:15:39.096407+0000", 2024-06-27T21:20:44.416 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 305.24919989900002, 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62392472799999998, 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096407+0000", 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096407+0000", 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096408+0000", 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0500000000000003e-07 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.096411+0000", 2024-06-27T21:20:44.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8990000000000003e-06 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720154+0000", 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62374265200000001 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720176+0000", 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2195999999999999e-05 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720266+0000", 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0172000000000003e-05 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720278+0000", 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1715e-05 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:15:39.720332+0000", 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3389e-05 2024-06-27T21:20:44.420 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:34085 3.c e199/186 3:31ab7127:::benchmark_data_smithi192_74314_object34084:head v 199'2184, mlcod=199'2181)", 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:40.797794+0000", 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 243.54781267300001, 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23472686700000001, 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797794+0000", 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797794+0000", 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797795+0000", 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4e-07 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.797819+0000", 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4051999999999999e-05 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030409+0000", 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23259012300000001 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030414+0000", 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7160000000000002e-06 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030486+0000", 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2717000000000002e-05 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.030507+0000", 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0649000000000002e-05 2024-06-27T21:20:44.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032473+0000", 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019658929999999998 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032521+0000", 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7877000000000003e-05 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:63865 3.0 e209/186 3:0a199132:::benchmark_data_smithi192_74314_object63864:head v 209'4000, mlcod=209'3996)", 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:04.486048+0000", 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 219.85955841200001, 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21823163800000001, 2024-06-27T21:20:44.431 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486048+0000", 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486048+0000", 2024-06-27T21:20:44.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486049+0000", 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.486103+0000", 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4200999999999998e-05 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702091+0000", 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21598763300000001 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702097+0000", 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1580000000000002e-06 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702148+0000", 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.083e-05 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.702168+0000", 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9839000000000001e-05 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.704149+0000", 2024-06-27T21:20:44.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001981029 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:04.704280+0000", 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013115800000000001 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:74026 3.a 3:5e5f5656:::benchmark_data_smithi192_74314_object74025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:12.910360+0000", 2024-06-27T21:20:44.437 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 211.435246925, 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22282300999999999, 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 74026 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910360+0000", 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910360+0000", 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910361+0000", 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3149999999999999e-06 2024-06-27T21:20:44.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910406+0000", 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4926999999999999e-05 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910418+0000", 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1685000000000001e-05 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910425+0000", 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.306e-06 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910473+0000", 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8204999999999997e-05 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910574+0000", 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100437 2024-06-27T21:20:44.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.910663+0000", 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8849000000000003e-05 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.912821+0000", 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002158872 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:12.946702+0000", 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.033880217999999997 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.133064+0000", 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18636245500000001 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.133130+0000", 2024-06-27T21:20:44.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5468000000000004e-05 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:13.133183+0000", 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3273000000000003e-05 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.445 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91001 3.a 3:5cd187b6:::benchmark_data_smithi192_74314_object91000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.755993+0000", 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 196.58961356500001, 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21809930599999999, 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91001 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.446 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.755993+0000", 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.755993+0000", 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.755994+0000", 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.299e-06 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756041+0000", 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6492999999999997e-05 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756054+0000", 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2577e-05 2024-06-27T21:20:44.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756061+0000", 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3479999999999998e-06 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756120+0000", 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8684999999999998e-05 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756259+0000", 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000139376 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.756404+0000", 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014487199999999999 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.758788+0000", 2024-06-27T21:20:44.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023840179999999999 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.758856+0000", 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8298000000000002e-05 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974025+0000", 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.215169042 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974065+0000", 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0036000000000001e-05 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.974092+0000", 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7262000000000001e-05 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.452 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91005 3.a 3:5b53ffe7:::benchmark_data_smithi192_74314_object91004:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.762103+0000", 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 196.583503976, 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23945181400000001, 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:44.453 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91005 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762103+0000", 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762103+0000", 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762104+0000", 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.172e-06 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762160+0000", 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5951e-05 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762173+0000", 2024-06-27T21:20:44.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3087e-05 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762179+0000", 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7629999999999997e-06 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762221+0000", 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2635000000000001e-05 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762319+0000", 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7162000000000002e-05 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762406+0000", 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7294999999999997e-05 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.765175+0000", 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027689920000000001 2024-06-27T21:20:44.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.765304+0000", 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128787 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001466+0000", 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23616261299999999 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001507+0000", 2024-06-27T21:20:44.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.121e-05 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001555+0000", 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7147000000000001e-05 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91008 3.a 3:579cf028:::benchmark_data_smithi192_74314_object91007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.768226+0000", 2024-06-27T21:20:44.459 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 196.57738084799999, 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.236103593, 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91008 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768226+0000", 2024-06-27T21:20:44.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768226+0000", 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768227+0000", 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1850000000000001e-06 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768287+0000", 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0040000000000001e-05 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768299+0000", 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1762e-05 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768305+0000", 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6969999999999998e-06 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768342+0000", 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7653e-05 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768434+0000", 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1506000000000001e-05 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768532+0000", 2024-06-27T21:20:44.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7875999999999996e-05 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.770803+0000", 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022713009999999999 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.770820+0000", 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6983999999999999e-05 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.004218+0000", 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23339822399999999 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.004283+0000", 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4589000000000001e-05 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.004329+0000", 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6776000000000001e-05 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.15132.0:91005 3.a e218/186 ondisk, result = 0)", 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.768794+0000", 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 196.576812518, 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.232767417, 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768794+0000", 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768794+0000", 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768795+0000", 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7400000000000002e-07 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.768798+0000", 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9699999999999999e-06 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.998115+0000", 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22931665300000001 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.998119+0000", 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7369999999999997e-06 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001457+0000", 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033374720000000002 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001463+0000", 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7339999999999996e-06 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:28.001562+0000", 2024-06-27T21:20:44.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9077e-05 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:156122 3.a 3:5b74c1b2:::benchmark_data_smithi192_74314_object49484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:48.166023+0000", 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 116.17958401200001, 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27862849299999998, 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.470 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 156122 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.166023+0000", 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.166023+0000", 2024-06-27T21:20:44.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.166024+0000", 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.587e-06 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.166032+0000", 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9759999999999995e-06 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.166048+0000", 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5274999999999999e-05 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.166058+0000", 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0419e-05 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.167100+0000", 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010420270000000001 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.167206+0000", 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010594600000000001 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.167318+0000", 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011230000000000001 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.173424+0000", 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006105727 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.179349+0000", 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059246189999999999 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.444499+0000", 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26515030699999997 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.444575+0000", 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6477999999999995e-05 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.444651+0000", 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5832000000000002e-05 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.476 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:170454 3.a 3:5579447d:::benchmark_data_smithi192_74314_object63816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:19:07.240592+0000", 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 97.105014650000001, 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22079354000000001, 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 170454 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240592+0000", 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240592+0000", 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240594+0000", 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.561e-06 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240600+0000", 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1659999999999999e-06 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240614+0000", 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4443999999999999e-05 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240623+0000", 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1069999999999994e-06 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240686+0000", 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2401000000000001e-05 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240811+0000", 2024-06-27T21:20:44.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012489200000000001 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.240930+0000", 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119472 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.245842+0000", 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049120409999999998 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.248554+0000", 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027122119999999999 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.461279+0000", 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.212724521 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.461342+0000", 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3019000000000001e-05 2024-06-27T21:20:44.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:19:07.461386+0000", 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3704e-05 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(3.5 e276,227 3:a36527c4:::benchmark_data_smithi192_74314_object57984:head,250'10498)", 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:17.842224+0000", 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.503382357, 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62875785100000003, 2024-06-27T21:20:44.483 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.842224+0000", 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.842224+0000", 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:20:44.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.842225+0000", 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4199999999999996e-07 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.842229+0000", 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7610000000000001e-06 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:18.470854+0000", 2024-06-27T21:20:44.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.628625455 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:18.470862+0000", 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0569999999999993e-06 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:18.470917+0000", 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5095e-05 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:18.470925+0000", 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4630000000000004e-06 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:18.470982+0000", 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7478000000000001e-05 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:20:44.487 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:44.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:44.494 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:20:44.494 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:20:44.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:44.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 1184 1168 72.9963 73 0.0287432 0.0133597 2024-06-27T21:20:44.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:44.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:44.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:44.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:44.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:44.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:44.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:44.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:45.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:45.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:45.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:45.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:20:45.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:45.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:45.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:45.406 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:45.506 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:45.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:45.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:45.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 2792 2776 86.7421 100.5 0.00121865 0.0112465 2024-06-27T21:20:45.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:45.657 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:45.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:45 smithi151 ceph-mon[28965]: pgmap v787: 27 pgs: 27 active+clean; 2.0 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 236 KiB/s wr, 3 op/s 2024-06-27T21:20:45.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:45.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:45.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:45 smithi151 ceph-mon[28965]: osdmap e289: 12 total, 12 up, 11 in 2024-06-27T21:20:45.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:45 smithi151 ceph-mon[28965]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:45.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:45.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:45 smithi190 ceph-mon[32247]: pgmap v787: 27 pgs: 27 active+clean; 2.0 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 236 KiB/s wr, 3 op/s 2024-06-27T21:20:45.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:45.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:45.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:45 smithi190 ceph-mon[32247]: osdmap e289: 12 total, 12 up, 11 in 2024-06-27T21:20:45.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:45 smithi190 ceph-mon[32247]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:45.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:45.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:45 smithi192 ceph-mon[32956]: pgmap v787: 27 pgs: 27 active+clean; 2.0 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 236 KiB/s wr, 3 op/s 2024-06-27T21:20:45.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:45.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:45.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:45 smithi192 ceph-mon[32956]: osdmap e289: 12 total, 12 up, 11 in 2024-06-27T21:20:45.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:45 smithi192 ceph-mon[32956]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:20:45.819 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":788,"stamp":"2024-06-27T21:20:44.465870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2097696,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1035,"ondisk_log_size":1035,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1854872,"kb_used_data":12536,"kb_used_omap":17,"kb_used_meta":1842158,"kb_avail":1029341800,"statfs":{"total":1055945392128,"available":1054046003200,"internally_reserved":0,"allocated":12836864,"data_stored":8594847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17547,"internal_metadata":1886370677},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.526271"},"pg_stats":[{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:41.751888+0000","last_change":"2024-06-27T21:20:33.960530+0000","last_active":"2024-06-27T21:20:41.751888+0000","last_peered":"2024-06-27T21:20:41.751888+0000","last_clean":"2024-06-27T21:20:41.751888+0000","last_became_active":"2024-06-27T21:20:33.960150+0000","last_became_peered":"2024-06-27T21:20:33.960150+0000","last_unstale":"2024-06-27T21:20:41.751888+0000","last_undegraded":"2024-06-27T21:20:41.751888+0000","last_fullsized":"2024-06-27T21:20:41.751888+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:51.498368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:39.840297+0000","last_change":"2024-06-27T21:20:33.674258+0000","last_active":"2024-06-27T21:20:39.840297+0000","last_peered":"2024-06-27T21:20:39.840297+0000","last_clean":"2024-06-27T21:20:39.840297+0000","last_became_active":"2024-06-27T21:20:33.673842+0000","last_became_peered":"2024-06-27T21:20:33.673842+0000","last_unstale":"2024-06-27T21:20:39.840297+0000","last_undegraded":"2024-06-27T21:20:39.840297+0000","last_fullsized":"2024-06-27T21:20:39.840297+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:47.214701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032578+0000","last_change":"2024-06-27T21:20:33.960114+0000","last_active":"2024-06-27T21:20:40.032578+0000","last_peered":"2024-06-27T21:20:40.032578+0000","last_clean":"2024-06-27T21:20:40.032578+0000","last_became_active":"2024-06-27T21:20:33.959756+0000","last_became_peered":"2024-06-27T21:20:33.959756+0000","last_unstale":"2024-06-27T21:20:40.032578+0000","last_undegraded":"2024-06-27T21:20:40.032578+0000","last_fullsized":"2024-06-27T21:20:40.032578+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:33.826085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:39.465519+0000","last_change":"2024-06-27T21:20:33.673013+0000","last_active":"2024-06-27T21:20:39.465519+0000","last_peered":"2024-06-27T21:20:39.465519+0000","last_clean":"2024-06-27T21:20:39.465519+0000","last_became_active":"2024-06-27T21:20:33.672659+0000","last_became_peered":"2024-06-27T21:20:33.672659+0000","last_unstale":"2024-06-27T21:20:39.465519+0000","last_undegraded":"2024-06-27T21:20:39.465519+0000","last_fullsized":"2024-06-27T21:20:39.465519+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:43.076526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:39.465699+0000","last_change":"2024-06-27T21:20:33.961841+0000","last_active":"2024-06-27T21:20:39.465699+0000","last_peered":"2024-06-27T21:20:39.465699+0000","last_clean":"2024-06-27T21:20:39.465699+0000","last_became_active":"2024-06-27T21:20:33.961523+0000","last_became_peered":"2024-06-27T21:20:33.961523+0000","last_unstale":"2024-06-27T21:20:39.465699+0000","last_undegraded":"2024-06-27T21:20:39.465699+0000","last_fullsized":"2024-06-27T21:20:39.465699+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:56.701093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032684+0000","last_change":"2024-06-27T21:20:33.961410+0000","last_active":"2024-06-27T21:20:40.032684+0000","last_peered":"2024-06-27T21:20:40.032684+0000","last_clean":"2024-06-27T21:20:40.032684+0000","last_became_active":"2024-06-27T21:20:33.961114+0000","last_became_peered":"2024-06-27T21:20:33.961114+0000","last_unstale":"2024-06-27T21:20:40.032684+0000","last_undegraded":"2024-06-27T21:20:40.032684+0000","last_fullsized":"2024-06-27T21:20:40.032684+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:36.597989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":576,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-27T21:20:37.831714+0000","last_change":"2024-06-27T21:19:46.475053+0000","last_active":"2024-06-27T21:20:37.831714+0000","last_peered":"2024-06-27T21:20:37.831714+0000","last_clean":"2024-06-27T21:20:37.831714+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:20:37.831714+0000","last_undegraded":"2024-06-27T21:20:37.831714+0000","last_fullsized":"2024-06-27T21:20:37.831714+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:46.474944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:46.474944+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:20:48.986062+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032725+0000","last_change":"2024-06-27T21:20:33.961290+0000","last_active":"2024-06-27T21:20:40.032725+0000","last_peered":"2024-06-27T21:20:40.032725+0000","last_clean":"2024-06-27T21:20:40.032725+0000","last_became_active":"2024-06-27T21:20:33.960775+0000","last_became_peered":"2024-06-27T21:20:33.960775+0000","last_unstale":"2024-06-27T21:20:40.032725+0000","last_undegraded":"2024-06-27T21:20:40.032725+0000","last_fullsized":"2024-06-27T21:20:40.032725+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:41.016669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,7],"acting":[9,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":171,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032671+0000","last_change":"2024-06-27T21:20:34.359816+0000","last_active":"2024-06-27T21:20:40.032671+0000","last_peered":"2024-06-27T21:20:40.032671+0000","last_clean":"2024-06-27T21:20:40.032671+0000","last_became_active":"2024-06-27T21:17:58.594952+0000","last_became_peered":"2024-06-27T21:17:58.594952+0000","last_unstale":"2024-06-27T21:20:40.032671+0000","last_undegraded":"2024-06-27T21:20:40.032671+0000","last_fullsized":"2024-06-27T21:20:40.032671+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:34.359736+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:20:34.359736+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:54.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"288'10","reported_seq":32,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:43.654660+0000","last_change":"2024-06-27T21:20:33.960470+0000","last_active":"2024-06-27T21:20:43.654660+0000","last_peered":"2024-06-27T21:20:43.654660+0000","last_clean":"2024-06-27T21:20:43.654660+0000","last_became_active":"2024-06-27T21:20:33.960064+0000","last_became_peered":"2024-06-27T21:20:33.960064+0000","last_unstale":"2024-06-27T21:20:43.654660+0000","last_undegraded":"2024-06-27T21:20:43.654660+0000","last_fullsized":"2024-06-27T21:20:43.654660+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:46.429646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":588,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-27T21:20:37.831546+0000","last_change":"2024-06-27T21:20:30.322903+0000","last_active":"2024-06-27T21:20:37.831546+0000","last_peered":"2024-06-27T21:20:37.831546+0000","last_clean":"2024-06-27T21:20:37.831546+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:20:37.831546+0000","last_undegraded":"2024-06-27T21:20:37.831546+0000","last_fullsized":"2024-06-27T21:20:37.831546+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:30.322810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:30.322810+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:56.787406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:42.233465+0000","last_change":"2024-06-27T21:20:33.960230+0000","last_active":"2024-06-27T21:20:42.233465+0000","last_peered":"2024-06-27T21:20:42.233465+0000","last_clean":"2024-06-27T21:20:42.233465+0000","last_became_active":"2024-06-27T21:20:33.959886+0000","last_became_peered":"2024-06-27T21:20:33.959886+0000","last_unstale":"2024-06-27T21:20:42.233465+0000","last_undegraded":"2024-06-27T21:20:42.233465+0000","last_fullsized":"2024-06-27T21:20:42.233465+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:43.206873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,10],"acting":[3,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":295,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-27T21:20:37.831456+0000","last_change":"2024-06-27T21:20:30.922701+0000","last_active":"2024-06-27T21:20:37.831456+0000","last_peered":"2024-06-27T21:20:37.831456+0000","last_clean":"2024-06-27T21:20:37.831456+0000","last_became_active":"2024-06-27T21:14:55.718653+0000","last_became_peered":"2024-06-27T21:14:55.718653+0000","last_unstale":"2024-06-27T21:20:37.831456+0000","last_undegraded":"2024-06-27T21:20:37.831456+0000","last_fullsized":"2024-06-27T21:20:37.831456+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:30.922606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:30.922606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:32.414507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032706+0000","last_change":"2024-06-27T21:20:33.676832+0000","last_active":"2024-06-27T21:20:40.032706+0000","last_peered":"2024-06-27T21:20:40.032706+0000","last_clean":"2024-06-27T21:20:40.032706+0000","last_became_active":"2024-06-27T21:20:33.676501+0000","last_became_peered":"2024-06-27T21:20:33.676501+0000","last_unstale":"2024-06-27T21:20:40.032706+0000","last_undegraded":"2024-06-27T21:20:40.032706+0000","last_fullsized":"2024-06-27T21:20:40.032706+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:49.176697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":307,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032467+0000","last_change":"2024-06-27T21:20:32.377288+0000","last_active":"2024-06-27T21:20:40.032467+0000","last_peered":"2024-06-27T21:20:40.032467+0000","last_clean":"2024-06-27T21:20:40.032467+0000","last_became_active":"2024-06-27T21:14:55.861874+0000","last_became_peered":"2024-06-27T21:14:55.861874+0000","last_unstale":"2024-06-27T21:20:40.032467+0000","last_undegraded":"2024-06-27T21:20:40.032467+0000","last_fullsized":"2024-06-27T21:20:40.032467+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:32.377138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.377138+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:46.615022+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:39.465570+0000","last_change":"2024-06-27T21:20:33.961473+0000","last_active":"2024-06-27T21:20:39.465570+0000","last_peered":"2024-06-27T21:20:39.465570+0000","last_clean":"2024-06-27T21:20:39.465570+0000","last_became_active":"2024-06-27T21:20:33.961025+0000","last_became_peered":"2024-06-27T21:20:33.961025+0000","last_unstale":"2024-06-27T21:20:39.465570+0000","last_undegraded":"2024-06-27T21:20:39.465570+0000","last_fullsized":"2024-06-27T21:20:39.465570+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:42.777470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":295,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-27T21:20:37.831531+0000","last_change":"2024-06-27T21:20:33.960049+0000","last_active":"2024-06-27T21:20:37.831531+0000","last_peered":"2024-06-27T21:20:37.831531+0000","last_clean":"2024-06-27T21:20:37.831531+0000","last_became_active":"2024-06-27T21:14:55.711705+0000","last_became_peered":"2024-06-27T21:14:55.711705+0000","last_unstale":"2024-06-27T21:20:37.831531+0000","last_undegraded":"2024-06-27T21:20:37.831531+0000","last_fullsized":"2024-06-27T21:20:37.831531+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:33.959884+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:20:33.959884+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:35.765314+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:42.233387+0000","last_change":"2024-06-27T21:20:33.961484+0000","last_active":"2024-06-27T21:20:42.233387+0000","last_peered":"2024-06-27T21:20:42.233387+0000","last_clean":"2024-06-27T21:20:42.233387+0000","last_became_active":"2024-06-27T21:20:33.961176+0000","last_became_peered":"2024-06-27T21:20:33.961176+0000","last_unstale":"2024-06-27T21:20:42.233387+0000","last_undegraded":"2024-06-27T21:20:42.233387+0000","last_fullsized":"2024-06-27T21:20:42.233387+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:48.516434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":293,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-27T21:20:37.831521+0000","last_change":"2024-06-27T21:19:40.674095+0000","last_active":"2024-06-27T21:20:37.831521+0000","last_peered":"2024-06-27T21:20:37.831521+0000","last_clean":"2024-06-27T21:20:37.831521+0000","last_became_active":"2024-06-27T21:17:45.838234+0000","last_became_peered":"2024-06-27T21:17:45.838234+0000","last_unstale":"2024-06-27T21:20:37.831521+0000","last_undegraded":"2024-06-27T21:20:37.831521+0000","last_fullsized":"2024-06-27T21:20:37.831521+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:40.673987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:40.673987+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:06.405572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:39.465608+0000","last_change":"2024-06-27T21:20:33.676743+0000","last_active":"2024-06-27T21:20:39.465608+0000","last_peered":"2024-06-27T21:20:39.465608+0000","last_clean":"2024-06-27T21:20:39.465608+0000","last_became_active":"2024-06-27T21:20:33.676398+0000","last_became_peered":"2024-06-27T21:20:33.676398+0000","last_unstale":"2024-06-27T21:20:39.465608+0000","last_undegraded":"2024-06-27T21:20:39.465608+0000","last_fullsized":"2024-06-27T21:20:39.465608+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:38.769312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":386,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:39.465629+0000","last_change":"2024-06-27T21:19:50.905384+0000","last_active":"2024-06-27T21:20:39.465629+0000","last_peered":"2024-06-27T21:20:39.465629+0000","last_clean":"2024-06-27T21:20:39.465629+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:20:39.465629+0000","last_undegraded":"2024-06-27T21:20:39.465629+0000","last_fullsized":"2024-06-27T21:20:39.465629+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:19:50.905308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:19:50.905308+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:11.504423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"288'13","reported_seq":35,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:43.650639+0000","last_change":"2024-06-27T21:20:33.674679+0000","last_active":"2024-06-27T21:20:43.650639+0000","last_peered":"2024-06-27T21:20:43.650639+0000","last_clean":"2024-06-27T21:20:43.650639+0000","last_became_active":"2024-06-27T21:20:33.674388+0000","last_became_peered":"2024-06-27T21:20:33.674388+0000","last_unstale":"2024-06-27T21:20:43.650639+0000","last_undegraded":"2024-06-27T21:20:43.650639+0000","last_fullsized":"2024-06-27T21:20:43.650639+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:58.600518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":851968,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":298,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-06-27T21:20:40.032649+0000","last_change":"2024-06-27T21:20:31.763820+0000","last_active":"2024-06-27T21:20:40.032649+0000","last_peered":"2024-06-27T21:20:40.032649+0000","last_clean":"2024-06-27T21:20:40.032649+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:20:40.032649+0000","last_undegraded":"2024-06-27T21:20:40.032649+0000","last_fullsized":"2024-06-27T21:20:40.032649+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.763737+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.763737+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:53.584279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":22,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:41.751891+0000","last_change":"2024-06-27T21:20:33.674183+0000","last_active":"2024-06-27T21:20:41.751891+0000","last_peered":"2024-06-27T21:20:41.751891+0000","last_clean":"2024-06-27T21:20:41.751891+0000","last_became_active":"2024-06-27T21:20:33.673827+0000","last_became_peered":"2024-06-27T21:20:33.673827+0000","last_unstale":"2024-06-27T21:20:41.751891+0000","last_undegraded":"2024-06-27T21:20:41.751891+0000","last_fullsized":"2024-06-27T21:20:41.751891+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:35.536302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":594,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-06-27T21:20:37.831678+0000","last_change":"2024-06-27T21:20:31.336781+0000","last_active":"2024-06-27T21:20:37.831678+0000","last_peered":"2024-06-27T21:20:37.831678+0000","last_clean":"2024-06-27T21:20:37.831678+0000","last_became_active":"2024-06-27T21:17:45.505725+0000","last_became_peered":"2024-06-27T21:17:45.505725+0000","last_unstale":"2024-06-27T21:20:37.831678+0000","last_undegraded":"2024-06-27T21:20:37.831678+0000","last_fullsized":"2024-06-27T21:20:37.831678+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.336705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.336705+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:54.989693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:41.751880+0000","last_change":"2024-06-27T21:20:33.674639+0000","last_active":"2024-06-27T21:20:41.751880+0000","last_peered":"2024-06-27T21:20:41.751880+0000","last_clean":"2024-06-27T21:20:41.751880+0000","last_became_active":"2024-06-27T21:20:33.674351+0000","last_became_peered":"2024-06-27T21:20:33.674351+0000","last_unstale":"2024-06-27T21:20:41.751880+0000","last_undegraded":"2024-06-27T21:20:41.751880+0000","last_fullsized":"2024-06-27T21:20:41.751880+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:01.896721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,1],"acting":[4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":308,"reported_epoch":288,"state":"active+clean","last_fresh":"2024-06-27T21:20:42.233185+0000","last_change":"2024-06-27T21:20:29.675042+0000","last_active":"2024-06-27T21:20:42.233185+0000","last_peered":"2024-06-27T21:20:42.233185+0000","last_clean":"2024-06-27T21:20:42.233185+0000","last_became_active":"2024-06-27T21:17:45.502472+0000","last_became_peered":"2024-06-27T21:17:45.502472+0000","last_unstale":"2024-06-27T21:20:42.233185+0000","last_undegraded":"2024-06-27T21:20:42.233185+0000","last_fullsized":"2024-06-27T21:20:42.233185+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:29.674960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:29.674960+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:42.541237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7208960,"data_stored":7208960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253000,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225452,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93519700,"statfs":{"total":95995035648,"available":95764172800,"internally_reserved":0,"allocated":1339392,"data_stored":958331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514637,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80752,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93664400,"statfs":{"total":95995035648,"available":95912345600,"internally_reserved":0,"allocated":1011712,"data_stored":630651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743569,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144620,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600532,"statfs":{"total":95995035648,"available":95846944768,"internally_reserved":0,"allocated":1011712,"data_stored":630651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972502,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181248,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93563904,"statfs":{"total":95995035648,"available":95809437696,"internally_reserved":0,"allocated":1556480,"data_stored":1167356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158828,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93586324,"statfs":{"total":95995035648,"available":95832395776,"internally_reserved":0,"allocated":1339392,"data_stored":958331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528479,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208492,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93536660,"statfs":{"total":95995035648,"available":95781539840,"internally_reserved":0,"allocated":1011712,"data_stored":630651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757413,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130952,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614200,"statfs":{"total":95995035648,"available":95860940800,"internally_reserved":0,"allocated":1241088,"data_stored":847756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986346,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190464,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554688,"statfs":{"total":95995035648,"available":95800000512,"internally_reserved":0,"allocated":1036288,"data_stored":646508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182574,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190652,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554500,"statfs":{"total":95995035648,"available":95799808000,"internally_reserved":0,"allocated":1228800,"data_stored":839676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":193984967},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411507,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162688,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582464,"statfs":{"total":95995035648,"available":95828443136,"internally_reserved":0,"allocated":1036288,"data_stored":646508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180724,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564428,"statfs":{"total":95995035648,"available":95809974272,"internally_reserved":0,"allocated":1024000,"data_stored":638428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7208960,"data_stored":7208960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:20:45.821 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-27T21:20:45.821 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:20:45.821 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:20:45.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 10 2024-06-27T21:20:45.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:45.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:45.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:46.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:46.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:46.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:46.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:46.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:46.381 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:46.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:46.414 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:46.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:46.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:46.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:46.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 16 4318 4302 89.6173 95.375 0.00115205 0.0110687 2024-06-27T21:20:46.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:46.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:46.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:46.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:46.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:46.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:46.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:46.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:47.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:47.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:47.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:47.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:47.237 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:47.237 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:20:47.237 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:20:47.237 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:47.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:47.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:47.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:47.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:47.473 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:20:47.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:47 smithi190 ceph-mon[32247]: pgmap v789: 27 pgs: 27 active+clean; 65 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 10 MiB/s wr, 159 op/s 2024-06-27T21:20:47.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/631855622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:47.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:47.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:47.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 5667 5651 88.2895 84.3125 0.00144189 0.011291 2024-06-27T21:20:47.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:47.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:47.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:47.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:47 smithi192 ceph-mon[32956]: pgmap v789: 27 pgs: 27 active+clean; 65 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 10 MiB/s wr, 159 op/s 2024-06-27T21:20:47.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/631855622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:47.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:47.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:47 smithi151 ceph-mon[28965]: pgmap v789: 27 pgs: 27 active+clean; 65 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 10 MiB/s wr, 159 op/s 2024-06-27T21:20:47.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/631855622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:20:47.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:47.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:20:47.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:47.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:48.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:48.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:48.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:48.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:48.248 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:48.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:48.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:48.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:48.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:48.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:48.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:48.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 7274 7258 90.7176 100.438 0.00553473 0.0109803 2024-06-27T21:20:48.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:48 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:48.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/631855622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:20:48.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:48 smithi190 ceph-mon[32247]: osdmap e290: 12 total, 12 up, 11 in 2024-06-27T21:20:48.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:48.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:48 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:48.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/631855622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:20:48.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:48 smithi151 ceph-mon[28965]: osdmap e290: 12 total, 12 up, 11 in 2024-06-27T21:20:48.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:48.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:48 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:48.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/631855622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:20:48.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:48 smithi192 ceph-mon[32956]: osdmap e290: 12 total, 12 up, 11 in 2024-06-27T21:20:48.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:48.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:48.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:48.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:48.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:49.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:49.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:49.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:49.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:49.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:49.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:49.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:49.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:49.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:49.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:49.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 8375 8359 87.0658 68.8125 0.00190311 0.011473 2024-06-27T21:20:49.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:49.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:49 smithi151 ceph-mon[28965]: pgmap v791: 27 pgs: 27 active+clean; 183 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 29 MiB/s wr, 468 op/s 2024-06-27T21:20:49.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:49 smithi151 ceph-mon[28965]: osdmap e291: 12 total, 12 up, 11 in 2024-06-27T21:20:49.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:49.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:49 smithi190 ceph-mon[32247]: pgmap v791: 27 pgs: 27 active+clean; 183 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 29 MiB/s wr, 468 op/s 2024-06-27T21:20:49.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:49 smithi190 ceph-mon[32247]: osdmap e291: 12 total, 12 up, 11 in 2024-06-27T21:20:49.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:49 smithi192 ceph-mon[32956]: pgmap v791: 27 pgs: 27 active+clean; 183 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 29 MiB/s wr, 468 op/s 2024-06-27T21:20:49.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:49 smithi192 ceph-mon[32956]: osdmap e291: 12 total, 12 up, 11 in 2024-06-27T21:20:49.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:49.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:49.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:49.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:49.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:50.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:50.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:50.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:50.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:50.299 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:20:50.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:20:50.300 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:20:50.300 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:20:50.300 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:20:50.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:50.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:50.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:50.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:50 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:20:50.155+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e291 definitely_dead 0 2024-06-27T21:20:50.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:50.483 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.10. 2024-06-27T21:20:50.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:50.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:50.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 9524 9508 84.8854 71.8125 0.00138366 0.0116194 2024-06-27T21:20:50.607 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2176985568' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-06-27T21:20:50.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:50 smithi190 ceph-mon[32247]: Client client.admin marked osd.10 out, while it was still marked up 2024-06-27T21:20:50.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:50.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:50.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2176985568' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-06-27T21:20:50.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:50 smithi192 ceph-mon[32956]: Client client.admin marked osd.10 out, while it was still marked up 2024-06-27T21:20:50.795 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2176985568' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-06-27T21:20:50.795 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:50 smithi151 ceph-mon[28965]: Client client.admin marked osd.10 out, while it was still marked up 2024-06-27T21:20:50.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:50.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:50.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:50.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:20:50.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:50.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:51.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:20:51.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:51.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:51.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:51.186 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:20:51.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:51.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:51.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:51.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:51.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:51.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:51.564 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:51.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:20:51.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:20:51.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:51.595 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:20:51.595 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:20:51.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:51 smithi151 ceph-mon[28965]: pgmap v793: 27 pgs: 27 active+clean; 350 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 58 MiB/s wr, 928 op/s 2024-06-27T21:20:51.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2176985568' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-06-27T21:20:51.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:51 smithi151 ceph-mon[28965]: osdmap e292: 12 total, 12 up, 10 in 2024-06-27T21:20:51.598 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 9541 9525 74.4073 1.0625 0.0019212 0.0119493 2024-06-27T21:20:51.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:51 smithi190 ceph-mon[32247]: pgmap v793: 27 pgs: 27 active+clean; 350 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 58 MiB/s wr, 928 op/s 2024-06-27T21:20:51.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2176985568' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-06-27T21:20:51.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:51 smithi190 ceph-mon[32247]: osdmap e292: 12 total, 12 up, 10 in 2024-06-27T21:20:51.616 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:20:51.616 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:20:51.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:20:51.637 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:20:51.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:20:51.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:20:51.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:51.680 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:20:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:20:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:51.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:51.702 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:20:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:20:51.724 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:20:51.724 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:20:51.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:20:51.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:20:51.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:20:51.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:20:51.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:20:51.790 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:20:51.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:51 smithi192 ceph-mon[32956]: pgmap v793: 27 pgs: 27 active+clean; 350 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 58 MiB/s wr, 928 op/s 2024-06-27T21:20:51.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2176985568' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-06-27T21:20:51.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:51 smithi192 ceph-mon[32956]: osdmap e292: 12 total, 12 up, 10 in 2024-06-27T21:20:51.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:51.811 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:20:51.812 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:20:51.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:51.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:51.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:51.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:52.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:52.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:52.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:52.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:52.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:52.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:52.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:52.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:52.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:52.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:52.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 9729 9713 67.4453 11.75 0.00287823 0.014815 2024-06-27T21:20:52.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:52.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:52.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:52 smithi190 ceph-mon[32247]: osdmap e293: 12 total, 12 up, 10 in 2024-06-27T21:20:52.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/930272133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:52.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:52 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:20:52.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:52 smithi190 ceph-mon[32247]: osdmap e294: 12 total, 12 up, 10 in 2024-06-27T21:20:52.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:52 smithi192 ceph-mon[32956]: osdmap e293: 12 total, 12 up, 10 in 2024-06-27T21:20:52.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/930272133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:52.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:52 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:20:52.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:52 smithi192 ceph-mon[32956]: osdmap e294: 12 total, 12 up, 10 in 2024-06-27T21:20:52.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:20:52.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:20:52.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:52.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:52 smithi151 ceph-mon[28965]: osdmap e293: 12 total, 12 up, 10 in 2024-06-27T21:20:52.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/930272133' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:20:52.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:52 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:20:52.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:52 smithi151 ceph-mon[28965]: osdmap e294: 12 total, 12 up, 10 in 2024-06-27T21:20:52.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:52.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:53.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:53.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:53.095 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:53.096 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:53.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:53.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:53.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:53.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:53.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:53.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:53.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:53 smithi151 ceph-mon[28965]: pgmap v796: 27 pgs: 27 active+clean; 487 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-27T21:20:53.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:53 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:20:53.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:53 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:20:53.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:53 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:20:53.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:53 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:20:53.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:53.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 11300 11284 70.5187 98.1875 0.028214 0.0141697 2024-06-27T21:20:53.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:53.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:53.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:53.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:53 smithi190 ceph-mon[32247]: pgmap v796: 27 pgs: 27 active+clean; 487 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-27T21:20:53.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:53 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:20:53.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:53 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:20:53.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:53 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:20:53.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:53 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:20:53.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:53 smithi192 ceph-mon[32956]: pgmap v796: 27 pgs: 27 active+clean; 487 MiB data, 2.9 GiB used, 891 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-27T21:20:53.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:53 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:20:53.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:53 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:20:53.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:53 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:20:53.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:53 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:20:53.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:53.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:53.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:53.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:53.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:54.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:20:54.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:54.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:54.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:54.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:54.302 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:20:54.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:54.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:54.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:54.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:54.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:54.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 12493 12477 70.8858 74.5625 0.0117485 0.0140955 2024-06-27T21:20:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:54.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:54.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:54.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:54.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:54.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:54.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:54.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:55.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:55.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:55.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:55.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:55.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:55.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:55.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:55.500 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:20:55.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:55.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:55.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 14120 14104 73.4519 101.688 0.0204241 0.0135833 2024-06-27T21:20:55.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.659 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:55.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:55.748 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:55 smithi192 ceph-mon[32956]: pgmap v798: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 600 MiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s; 581/28785 objects degraded (2.018%); 1131/28785 objects misplaced (3.929%); 1015 KiB/s, 15 objects/s recovering 2024-06-27T21:20:55.749 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:55 smithi192 ceph-mon[32956]: osdmap e295: 12 total, 12 up, 10 in 2024-06-27T21:20:55.749 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:55 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 581/28785 objects degraded (2.018%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:20:55.749 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4262454668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:55.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:55 smithi190 ceph-mon[32247]: pgmap v798: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 600 MiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s; 581/28785 objects degraded (2.018%); 1131/28785 objects misplaced (3.929%); 1015 KiB/s, 15 objects/s recovering 2024-06-27T21:20:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:55 smithi190 ceph-mon[32247]: osdmap e295: 12 total, 12 up, 10 in 2024-06-27T21:20:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:55 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 581/28785 objects degraded (2.018%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:20:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4262454668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:55.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:55.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:55 smithi151 ceph-mon[28965]: pgmap v798: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 600 MiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s; 581/28785 objects degraded (2.018%); 1131/28785 objects misplaced (3.929%); 1015 KiB/s, 15 objects/s recovering 2024-06-27T21:20:55.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:55 smithi151 ceph-mon[28965]: osdmap e295: 12 total, 12 up, 10 in 2024-06-27T21:20:55.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:55 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 581/28785 objects degraded (2.018%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:20:55.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4262454668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:20:55.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:55.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:55.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:56.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:56.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:56.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [2, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:20:56.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:20:56.104 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-27T21:20:56.104 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.323815 2024-06-27T21:20:56.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 4 0.3238153682786499 2024-06-27T21:20:56.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:56.183 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:56.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:56.343 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:56.343 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:56.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:20:56.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:56.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:56.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:20:56.561 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:56.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:20:56.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:56 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:56.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4262454668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:56.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:56 smithi192 ceph-mon[32956]: osdmap e296: 12 total, 12 up, 10 in 2024-06-27T21:20:56.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 15894 15878 76.3297 110.875 0.0816815 0.0130263 2024-06-27T21:20:56.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:56.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:56 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:56.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4262454668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:56.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:56 smithi151 ceph-mon[28965]: osdmap e296: 12 total, 12 up, 10 in 2024-06-27T21:20:56.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:56.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:56 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:20:56.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4262454668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:20:56.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:56 smithi190 ceph-mon[32247]: osdmap e296: 12 total, 12 up, 10 in 2024-06-27T21:20:56.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:56.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:56.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:56.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:56.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:57.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:20:57.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:57.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:57.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:57.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:57.264 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:20:57.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:57.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:57.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:57.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:57.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:57.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:57.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 16183 16167 72.1677 18.0625 0.00406105 0.013851 2024-06-27T21:20:57.602 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:57 smithi192 ceph-mon[32956]: pgmap v801: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 22 active+clean; 748 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 59 MiB/s wr, 939 op/s; 581/35895 objects degraded (1.619%); 1447/35895 objects misplaced (4.031%); 13 MiB/s, 211 objects/s recovering 2024-06-27T21:20:57.603 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:57 smithi192 ceph-mon[32956]: osdmap e297: 12 total, 12 up, 10 in 2024-06-27T21:20:57.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:57.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:57 smithi151 ceph-mon[28965]: pgmap v801: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 22 active+clean; 748 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 59 MiB/s wr, 939 op/s; 581/35895 objects degraded (1.619%); 1447/35895 objects misplaced (4.031%); 13 MiB/s, 211 objects/s recovering 2024-06-27T21:20:57.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:57 smithi151 ceph-mon[28965]: osdmap e297: 12 total, 12 up, 10 in 2024-06-27T21:20:57.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:57.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:57.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:20:57.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:20:57.875 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:20:57.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:57.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:57.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:58.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:57 smithi190 ceph-mon[32247]: pgmap v801: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 22 active+clean; 748 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 59 MiB/s wr, 939 op/s; 581/35895 objects degraded (1.619%); 1447/35895 objects misplaced (4.031%); 13 MiB/s, 211 objects/s recovering 2024-06-27T21:20:58.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:57 smithi190 ceph-mon[32247]: osdmap e297: 12 total, 12 up, 10 in 2024-06-27T21:20:58.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:20:58.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:20:58.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:58.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:58.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:58.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:58.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:58.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:58.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:20:58.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:20:58.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 17650 17634 73.4685 91.6875 0.00656256 0.0136044 2024-06-27T21:20:58.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:20:58.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:58 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:20:58.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:58 smithi192 ceph-mon[32956]: osdmap e298: 12 total, 12 up, 10 in 2024-06-27T21:20:58.682 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:58 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:20:58.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:58.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:58.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:58.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:58.920 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:20:58.920 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:20:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:58 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:20:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:58 smithi151 ceph-mon[28965]: osdmap e298: 12 total, 12 up, 10 in 2024-06-27T21:20:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:58 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:20:58.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:20:58.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:58.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:20:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:58 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:20:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:58 smithi190 ceph-mon[32247]: osdmap e298: 12 total, 12 up, 10 in 2024-06-27T21:20:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:58 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:20:59.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:20:59.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:20:59.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:20:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:59.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:20:59.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:20:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:59.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:20:59.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:59.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:59.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 19042 19026 74.3136 87 0.0063467 0.0134026 2024-06-27T21:20:59.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:20:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:20:59.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:20:59.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:20:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:20:59.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:20:59.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:59 smithi192 ceph-mon[32956]: pgmap v804: 27 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 24 active+clean; 870 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s; 580/41739 objects degraded (1.390%); 316/41739 objects misplaced (0.757%); 31 MiB/s, 500 objects/s recovering 2024-06-27T21:20:59.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:59 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:20:59.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:20:59 smithi192 ceph-mon[32956]: osdmap e299: 12 total, 12 up, 10 in 2024-06-27T21:20:59.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:20:59.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:59 smithi151 ceph-mon[28965]: pgmap v804: 27 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 24 active+clean; 870 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s; 580/41739 objects degraded (1.390%); 316/41739 objects misplaced (0.757%); 31 MiB/s, 500 objects/s recovering 2024-06-27T21:20:59.874 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:59 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:20:59.874 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:20:59 smithi151 ceph-mon[28965]: osdmap e299: 12 total, 12 up, 10 in 2024-06-27T21:20:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:20:59.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:20:59.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:20:59.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:20:59.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:00.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:59 smithi190 ceph-mon[32247]: pgmap v804: 27 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 24 active+clean; 870 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s; 580/41739 objects degraded (1.390%); 316/41739 objects misplaced (0.757%); 31 MiB/s, 500 objects/s recovering 2024-06-27T21:21:00.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:59 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:00.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:20:59 smithi190 ceph-mon[32247]: osdmap e299: 12 total, 12 up, 10 in 2024-06-27T21:21:00.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:00.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:00.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:00.215 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:00.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:00.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:00.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:00.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:00.465 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:00.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:00.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:00.599 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 19354 19338 71.0892 19.5 0.0116129 0.014057 2024-06-27T21:21:00.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:00.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:00.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:00.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:00.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:00.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:00.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:01.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:01.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:01.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:01.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:01.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:01.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:01.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:01.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:01.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:01.531 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.4 primary-affinity to 0.323815 (8212212) 2024-06-27T21:21:01.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:01 smithi192 ceph-mon[32956]: pgmap v806: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1024 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 66 MiB/s wr, 1.06k op/s; 505/49110 objects misplaced (1.028%); 21 MiB/s, 343 objects/s recovering 2024-06-27T21:21:01.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:01 smithi192 ceph-mon[32956]: osdmap e300: 12 total, 12 up, 10 in 2024-06-27T21:21:01.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:01 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 580/41739 objects degraded (1.390%), 1 pg degraded) 2024-06-27T21:21:01.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4155707158' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.3238153682786499}]: dispatch 2024-06-27T21:21:01.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:01.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:01 smithi151 ceph-mon[28965]: pgmap v806: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1024 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 66 MiB/s wr, 1.06k op/s; 505/49110 objects misplaced (1.028%); 21 MiB/s, 343 objects/s recovering 2024-06-27T21:21:01.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:01 smithi151 ceph-mon[28965]: osdmap e300: 12 total, 12 up, 10 in 2024-06-27T21:21:01.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:01 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 580/41739 objects degraded (1.390%), 1 pg degraded) 2024-06-27T21:21:01.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4155707158' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.3238153682786499}]: dispatch 2024-06-27T21:21:01.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 16 20982 20966 72.792 101.75 0.0142789 0.0136849 2024-06-27T21:21:01.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:01.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:01.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:01.708 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:21:01.708 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:21:01.730 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:21:01.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:21:01.751 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:21:01.751 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:21:01.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:01 smithi190 ceph-mon[32247]: pgmap v806: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1024 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 66 MiB/s wr, 1.06k op/s; 505/49110 objects misplaced (1.028%); 21 MiB/s, 343 objects/s recovering 2024-06-27T21:21:01.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:01 smithi190 ceph-mon[32247]: osdmap e300: 12 total, 12 up, 10 in 2024-06-27T21:21:01.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:01 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 580/41739 objects degraded (1.390%), 1 pg degraded) 2024-06-27T21:21:01.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4155707158' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.3238153682786499}]: dispatch 2024-06-27T21:21:01.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:01.772 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:21:01.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:21:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:01.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:01.794 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:21:01.794 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:21:01.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:21:01.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:21:01.838 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:21:01.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:21:01.860 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:21:01.860 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:21:01.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:21:01.881 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:21:01.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:01.903 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:21:01.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:21:01.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:01.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:01.925 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:21:01.925 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:21:01.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:21:01.947 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:21:02.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:02.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:02.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:02.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:02.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:02.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:02.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:02.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:02.488 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:02.488 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:21:02.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:21:02.488 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:21:02.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.488 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:915 2.b 2:d2e05b7a:::benchmark_data_smithi192_59818_object914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:46.543532+0000", 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 555.92232554600002, 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.210178481, 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 915 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543532+0000", 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543532+0000", 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543533+0000", 2024-06-27T21:21:02.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2399999999999996e-07 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543550+0000", 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6626999999999999e-05 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543562+0000", 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2191000000000001e-05 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543570+0000", 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7179999999999996e-06 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543621+0000", 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1082000000000003e-05 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543740+0000", 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119135 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.543840+0000", 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9821999999999994e-05 2024-06-27T21:21:02.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.544356+0000", 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051664300000000001 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.544567+0000", 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021065999999999999 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753592+0000", 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20902500399999999 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753653+0000", 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0615000000000002e-05 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:46.753711+0000", 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8060000000000003e-05 2024-06-27T21:21:02.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7681 2.6 e108/104 2:6be1fbc1:::benchmark_data_smithi192_59818_object7680:head v 108'492, mlcod=108'488)", 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.217816+0000", 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 552.24804128799997, 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22328922400000001, 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.496 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217816+0000", 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217816+0000", 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217817+0000", 2024-06-27T21:21:02.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3699999999999999e-07 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.217853+0000", 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5932000000000001e-05 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440547+0000", 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.222693483 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440552+0000", 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.507e-06 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440602+0000", 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9755000000000001e-05 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.440624+0000", 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2209999999999999e-05 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.441073+0000", 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044889600000000001 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.441106+0000", 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2503999999999999e-05 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:7692 2.b 2:de479498:::benchmark_data_smithi192_59818_object7691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.225536+0000", 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 552.24032155400005, 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21296707500000001, 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7692 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225536+0000", 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225536+0000", 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225537+0000", 2024-06-27T21:21:02.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7499999999999999e-07 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225554+0000", 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7331000000000001e-05 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225565+0000", 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0642000000000001e-05 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225572+0000", 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9970000000000001e-06 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225613+0000", 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0567e-05 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225689+0000", 2024-06-27T21:21:02.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6981e-05 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.225772+0000", 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2256999999999995e-05 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226219+0000", 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044744499999999998 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226663+0000", 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044416399999999998 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438441+0000", 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211778035 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438477+0000", 2024-06-27T21:21:02.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5577999999999997e-05 2024-06-27T21:21:02.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438503+0000", 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6203e-05 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:7692 2.b e108/104 ondisk, result = 0)", 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.226423+0000", 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 552.23943430500003, 2024-06-27T21:21:02.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21208665900000001, 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226423+0000", 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226423+0000", 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226424+0000", 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.226427+0000", 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.225e-06 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438395+0000", 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21196757899999999 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438399+0000", 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7319999999999998e-06 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438433+0000", 2024-06-27T21:21:02.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3351e-05 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438437+0000", 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.438510+0000", 2024-06-27T21:21:02.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2614999999999995e-05 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:15465 2.f 2:f871ccb0:::benchmark_data_smithi192_59818_object15464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:55.576351+0000", 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 546.88950632900003, 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21259978600000001, 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 15465 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576351+0000", 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576351+0000", 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576353+0000", 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5510000000000001e-06 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576377+0000", 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4511e-05 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576396+0000", 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8802999999999999e-05 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576405+0000", 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3569999999999998e-06 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576470+0000", 2024-06-27T21:21:02.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5220000000000002e-05 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576582+0000", 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112493 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.576717+0000", 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134583 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.577463+0000", 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000745998 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.577877+0000", 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041452699999999998 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.788844+0000", 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21096653700000001 2024-06-27T21:21:02.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.788893+0000", 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9224999999999997e-05 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:55.788951+0000", 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7980999999999999e-05 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:79089 2.6 e128/104 2:61ffccd4:::benchmark_data_smithi192_59818_object79088:head v 128'5050, mlcod=128'5047)", 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:01.911357+0000", 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 480.55450061900001, 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.219698061, 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911357+0000", 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911357+0000", 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911358+0000", 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.75e-07 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:01.911409+0000", 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999999e-05 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128590+0000", 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21718090700000001 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128597+0000", 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6340000000000003e-06 2024-06-27T21:21:02.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128659+0000", 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1767999999999995e-05 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.128678+0000", 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8901000000000001e-05 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.131007+0000", 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023295070000000002 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:02.131055+0000", 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7969000000000001e-05 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:107417 2.b 2:d5dd94fd:::benchmark_data_smithi192_59818_object15342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-06-27T21:21:02.524 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:31.694933+0000", 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 450.77092426399997, 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22405386099999999, 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 107417 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694933+0000", 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694933+0000", 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694934+0000", 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8899999999999998e-07 2024-06-27T21:21:02.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694938+0000", 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7019999999999999e-06 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694946+0000", 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4679999999999992e-06 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.694951+0000", 2024-06-27T21:21:02.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6319999999999999e-06 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.698026+0000", 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030754419999999998 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.698115+0000", 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8578999999999993e-05 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.698192+0000", 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7249000000000005e-05 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.701121+0000", 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002928833 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.701939+0000", 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00081798400000000001 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.918912+0000", 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.216973104 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.918948+0000", 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6171999999999999e-05 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:31.918987+0000", 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8906999999999999e-05 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:121356 2.6 e141/104 2:672e567d:::benchmark_data_smithi192_59818_object29281:head v 141'7736, mlcod=140'7733)", 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.82456882899999, 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.249253161, 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.531 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641289+0000", 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999996e-07 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.641294+0000", 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2629999999999997e-06 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888046+0000", 2024-06-27T21:21:02.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24675217899999999 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888051+0000", 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8280000000000002e-06 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888089+0000", 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8661000000000001e-05 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.888103+0000", 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3981e-05 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.890517+0000", 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024139869999999998 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.890542+0000", 2024-06-27T21:21:02.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4658e-05 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121372 2.5 2:a1740290:::benchmark_data_smithi192_59818_object29297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.672471+0000", 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.79338702400003, 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20977246999999999, 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.536 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121372 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672471+0000", 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672471+0000", 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672472+0000", 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5000000000000001e-07 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672476+0000", 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3959999999999999e-06 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672487+0000", 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0558e-05 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672492+0000", 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4609999999999999e-06 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672537+0000", 2024-06-27T21:21:02.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5380999999999998e-05 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672617+0000", 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9810999999999999e-05 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.672697+0000", 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9793e-05 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.675988+0000", 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003291165 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.685499+0000", 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0095106500000000007 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882089+0000", 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19659014699999999 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882183+0000", 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3621000000000004e-05 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.882243+0000", 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0537e-05 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:121411 2.6 e141/104 2:678a3e62:::benchmark_data_smithi192_59818_object29336:head v 141'7739, mlcod=141'7736)", 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:46.011735+0000", 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.45412237800002, 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47895283999999999, 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011735+0000", 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011735+0000", 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011736+0000", 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6099999999999999e-07 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.011742+0000", 2024-06-27T21:21:02.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8109999999999999e-06 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.487949+0000", 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47620712700000001 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.487954+0000", 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9590000000000003e-06 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.488011+0000", 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6526000000000001e-05 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.488035+0000", 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4139999999999999e-05 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.490659+0000", 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026242729999999999 2024-06-27T21:21:02.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:46.490688+0000", 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9142999999999999e-05 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:159644 2.b 2:de2dfeb6:::benchmark_data_smithi192_59818_object67569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:43.373805+0000", 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 379.09205260200002, 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23554080699999999, 2024-06-27T21:21:02.547 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 159644 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373805+0000", 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373805+0000", 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373806+0000", 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7800000000000001e-07 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373809+0000", 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2480000000000001e-06 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373817+0000", 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7309999999999993e-06 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373826+0000", 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3819999999999993e-06 2024-06-27T21:21:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.373874+0000", 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7593999999999999e-05 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.374002+0000", 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012876900000000001 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.374122+0000", 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011917399999999999 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.378543+0000", 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044213569999999999 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.399632+0000", 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021089100999999999 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609246+0000", 2024-06-27T21:21:02.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20961380600000001 2024-06-27T21:21:02.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609298+0000", 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2250999999999998e-05 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609346+0000", 2024-06-27T21:21:02.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7716000000000002e-05 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:159644 2.b e158/104 ondisk, result = 0)", 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:43.378757+0000", 2024-06-27T21:21:02.556 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 379.08710044899999, 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23059312800000001, 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.378757+0000", 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.378757+0000", 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.378758+0000", 2024-06-27T21:21:02.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8500000000000001e-07 2024-06-27T21:21:02.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.378761+0000", 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9919999999999999e-06 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609178+0000", 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23041729699999999 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609185+0000", 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6549999999999998e-06 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609233+0000", 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609240+0000", 2024-06-27T21:21:02.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.764e-06 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:43.609350+0000", 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000110294 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:176910 2.f 2:f5665235:::benchmark_data_smithi192_59818_object84835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:15:08.321552+0000", 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 354.14430559499999, 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22149084999999999, 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.566 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 176910 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321552+0000", 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321552+0000", 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321554+0000", 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.669e-06 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321561+0000", 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3019999999999997e-06 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321577+0000", 2024-06-27T21:21:02.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6160000000000001e-05 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321586+0000", 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8769999999999999e-06 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321649+0000", 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3173e-05 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321754+0000", 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104895 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.321830+0000", 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5780999999999998e-05 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.325485+0000", 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036552529999999998 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.325673+0000", 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000187849 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.542942+0000", 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21726931699999999 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543012+0000", 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9832000000000005e-05 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543043+0000", 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0741999999999998e-05 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:176911 2.5 2:a1118f5a:::benchmark_data_smithi192_59818_object84836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:15:08.322161+0000", 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 354.14369652800002, 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22089508299999999, 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 176911 2024-06-27T21:21:02.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.576 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322161+0000", 2024-06-27T21:21:02.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322161+0000", 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322162+0000", 2024-06-27T21:21:02.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7199999999999997e-07 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322167+0000", 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0420000000000002e-06 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322180+0000", 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2928999999999999e-05 2024-06-27T21:21:02.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322187+0000", 2024-06-27T21:21:02.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3849999999999996e-06 2024-06-27T21:21:02.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322240+0000", 2024-06-27T21:21:02.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2403e-05 2024-06-27T21:21:02.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322313+0000", 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3300999999999995e-05 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.322377+0000", 2024-06-27T21:21:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3903000000000001e-05 2024-06-27T21:21:02.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.325802+0000", 2024-06-27T21:21:02.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034252890000000002 2024-06-27T21:21:02.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.327696+0000", 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018937800000000001 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.542914+0000", 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21521791200000001 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543003+0000", 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8805000000000001e-05 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543056+0000", 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3362000000000002e-05 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:176917 2.f 2:f636705f:::benchmark_data_smithi192_59818_object84842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:15:08.329021+0000", 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 354.13683637999998, 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21412619299999999, 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 176917 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329021+0000", 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329021+0000", 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329023+0000", 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9429999999999999e-06 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329039+0000", 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5829e-05 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329051+0000", 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2258e-05 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329059+0000", 2024-06-27T21:21:02.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3549999999999999e-06 2024-06-27T21:21:02.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329111+0000", 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2147000000000001e-05 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329199+0000", 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8415e-05 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.329266+0000", 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7267000000000005e-05 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.333128+0000", 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038619190000000001 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.333438+0000", 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030979500000000002 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543074+0000", 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.209635812 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543111+0000", 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7290999999999999e-05 2024-06-27T21:21:02.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:08.543147+0000", 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6161999999999997e-05 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:2751 3.9 3:93a6ec87:::benchmark_data_smithi192_74314_object2750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.227056+0000", 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.23880197199998, 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23014690099999999, 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.595 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2751 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227056+0000", 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227056+0000", 2024-06-27T21:21:02.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227057+0000", 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.654e-06 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227079+0000", 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1290000000000001e-05 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227093+0000", 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4178999999999999e-05 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227099+0000", 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6889999999999998e-06 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227154+0000", 2024-06-27T21:21:02.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4311000000000002e-05 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227236+0000", 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2307999999999999e-05 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227346+0000", 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011014900000000001 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227801+0000", 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045487700000000002 2024-06-27T21:21:02.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.231003+0000", 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003201509 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457084+0000", 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.226081009 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457133+0000", 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9775999999999999e-05 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457203+0000", 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.915e-05 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15132.0:2751 3.9 e192/186 ondisk, result = 0)", 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.227922+0000", 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.23793609400002, 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22929560500000001, 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227922+0000", 2024-06-27T21:21:02.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227922+0000", 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227922+0000", 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0999999999999999e-07 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.227925+0000", 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0070000000000002e-06 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457020+0000", 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22909479699999999 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457024+0000", 2024-06-27T21:21:02.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3549999999999998e-06 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457071+0000", 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6468999999999998e-05 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457078+0000", 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2860000000000002e-06 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457217+0000", 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013918099999999999 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:2751 3.9 3:93a6ec87:::benchmark_data_smithi192_74314_object2750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.233061+0000", 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 286.23279646100002, 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22408557400000001, 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.609 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for ondisk", 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2751 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233061+0000", 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233061+0000", 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233062+0000", 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3900000000000004e-07 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233079+0000", 2024-06-27T21:21:02.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7476999999999999e-05 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233087+0000", 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4039999999999998e-06 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233093+0000", 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8429999999999998e-06 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233159+0000", 2024-06-27T21:21:02.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:02.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6253000000000004e-05 2024-06-27T21:21:02.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for ondisk", 2024-06-27T21:21:02.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.233196+0000", 2024-06-27T21:21:02.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7401999999999999e-05 2024-06-27T21:21:02.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.457147+0000", 2024-06-27T21:21:02.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.223950556 2024-06-27T21:21:02.617 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.617 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.617 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.617 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:123893 3.9 3:9178fb98:::benchmark_data_smithi192_74314_object17255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-06-27T21:21:02.618 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:18:09.135523+0000", 2024-06-27T21:21:02.618 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 173.33033468400001, 2024-06-27T21:21:02.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.235212846, 2024-06-27T21:21:02.618 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.618 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.619 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:02.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:02.619 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123893 2024-06-27T21:21:02.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.620 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135523+0000", 2024-06-27T21:21:02.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135523+0000", 2024-06-27T21:21:02.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135525+0000", 2024-06-27T21:21:02.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5719999999999999e-06 2024-06-27T21:21:02.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135529+0000", 2024-06-27T21:21:02.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.899e-06 2024-06-27T21:21:02.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135543+0000", 2024-06-27T21:21:02.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3728e-05 2024-06-27T21:21:02.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135550+0000", 2024-06-27T21:21:02.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9110000000000004e-06 2024-06-27T21:21:02.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135595+0000", 2024-06-27T21:21:02.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4870999999999998e-05 2024-06-27T21:21:02.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135706+0000", 2024-06-27T21:21:02.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011105899999999999 2024-06-27T21:21:02.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.135816+0000", 2024-06-27T21:21:02.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010951999999999999 2024-06-27T21:21:02.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.138504+0000", 2024-06-27T21:21:02.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002688232 2024-06-27T21:21:02.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.140163+0000", 2024-06-27T21:21:02.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00165912 2024-06-27T21:21:02.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.370674+0000", 2024-06-27T21:21:02.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23051102600000001 2024-06-27T21:21:02.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.370708+0000", 2024-06-27T21:21:02.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3819000000000003e-05 2024-06-27T21:21:02.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.370736+0000", 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8089000000000002e-05 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.634 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:123913 3.9 3:9fff0412:::benchmark_data_smithi192_74314_object17275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-06-27T21:21:02.634 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:18:09.161762+0000", 2024-06-27T21:21:02.634 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 173.304095902, 2024-06-27T21:21:02.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22383867499999999, 2024-06-27T21:21:02.634 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123913 2024-06-27T21:21:02.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161762+0000", 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161762+0000", 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161764+0000", 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.232e-06 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161771+0000", 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3220000000000004e-06 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161788+0000", 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6387000000000001e-05 2024-06-27T21:21:02.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161796+0000", 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1210000000000007e-06 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161857+0000", 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1289000000000002e-05 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.161952+0000", 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4438999999999995e-05 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.162042+0000", 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.009e-05 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.164825+0000", 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027833369999999999 2024-06-27T21:21:02.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.165083+0000", 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00025791600000000002 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.385540+0000", 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22045681 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:02.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.385572+0000", 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2750000000000003e-05 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.385600+0000", 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7982000000000001e-05 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:02.641 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:02.642 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 21189 21173 69.6417 12.9375 0.00288886 0.0143517 2024-06-27T21:21:02.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:02.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:02.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:02.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4155707158' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.3238153682786499}]': finished 2024-06-27T21:21:02.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: osdmap e301: 12 total, 12 up, 10 in 2024-06-27T21:21:02.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:02.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:02.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4247119605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:02.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4155707158' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.3238153682786499}]': finished 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: osdmap e301: 12 total, 12 up, 10 in 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:02.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4247119605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:02.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:02.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:02.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:02.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:03.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:03.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:03.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4155707158' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.3238153682786499}]': finished 2024-06-27T21:21:03.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: osdmap e301: 12 total, 12 up, 10 in 2024-06-27T21:21:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4247119605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:03.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:03.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:03.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:03.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:03.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:03.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:03.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:03.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:21:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:03.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:03.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:03.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:03.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:21:03.603404+0000 min lat: 0.00102932 max lat: 2.15488 avg lat: 0.0143156 2024-06-27T21:21:03.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:21:03.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 22356 22340 69.8061 72.9375 0.0161925 0.0143156 2024-06-27T21:21:03.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:03.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:03.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:03 smithi151 ceph-mon[28965]: pgmap v809: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.1 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s; 189/54192 objects misplaced (0.349%); 26 MiB/s, 415 objects/s recovering 2024-06-27T21:21:03.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:03 smithi151 ceph-mon[28965]: osdmap e302: 12 total, 12 up, 10 in 2024-06-27T21:21:03.724 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:03.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:03.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:03 smithi192 ceph-mon[32956]: pgmap v809: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.1 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s; 189/54192 objects misplaced (0.349%); 26 MiB/s, 415 objects/s recovering 2024-06-27T21:21:03.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:03 smithi192 ceph-mon[32956]: osdmap e302: 12 total, 12 up, 10 in 2024-06-27T21:21:03.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:03.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:03.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:03.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:03.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:04.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:03 smithi190 ceph-mon[32247]: pgmap v809: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.1 GiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s; 189/54192 objects misplaced (0.349%); 26 MiB/s, 415 objects/s recovering 2024-06-27T21:21:04.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:03 smithi190 ceph-mon[32247]: osdmap e302: 12 total, 12 up, 10 in 2024-06-27T21:21:04.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:04.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:04.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:04.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:04.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:04.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:04.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:04.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:04.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:04.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:04.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 15 23551 23536 70.0412 74.75 0.00393323 0.0142718 2024-06-27T21:21:04.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:04.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:04.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:04.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:04.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:04.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:04 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:21:04.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:04 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:04.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:04 smithi192 ceph-mon[32956]: osdmap e303: 12 total, 12 up, 10 in 2024-06-27T21:21:04.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:04.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:04.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:04.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:04.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:04.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:04.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:04 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:21:04.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:04 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:04.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:04 smithi151 ceph-mon[28965]: osdmap e303: 12 total, 12 up, 10 in 2024-06-27T21:21:05.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:05.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:05.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:04 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:21:05.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:04 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:05.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:04 smithi190 ceph-mon[32247]: osdmap e303: 12 total, 12 up, 10 in 2024-06-27T21:21:05.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:05.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:05.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:05.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:05.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:05.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:05.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:05.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:24981 4.2 4:420dd53a:::benchmark_data_smithi192_88961_object24980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:05.464739+0000", 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0014928090000000001, 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015141239999999999, 2024-06-27T21:21:05.464 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 24981 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464739+0000", 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:05.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464739+0000", 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464740+0000", 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.029e-06 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464762+0000", 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1256000000000001e-05 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464775+0000", 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2938999999999999e-05 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464780+0000", 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4530000000000001e-06 2024-06-27T21:21:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464816+0000", 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5741000000000002e-05 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464892+0000", 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5899e-05 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,11", 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.464973+0000", 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0983999999999996e-05 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.465607+0000", 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063431900000000005 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:05.465991+0000", 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038449000000000003 2024-06-27T21:21:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:21:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-06-27T21:21:05.470 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:05.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:05.550 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:21:05.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:05.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 25101 25085 71.2576 96.8125 0.0467452 0.0140042 2024-06-27T21:21:05.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:05.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:05.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:05 smithi192 ceph-mon[32956]: pgmap v812: 27 pgs: 27 active+clean; 1.2 GiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 50 MiB/s wr, 802 op/s; 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:21:05.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:05 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:05.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4238588378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:05.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:05 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:05.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:05.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:05.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:05 smithi190 ceph-mon[32247]: pgmap v812: 27 pgs: 27 active+clean; 1.2 GiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 50 MiB/s wr, 802 op/s; 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:21:05.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:05 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:05.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4238588378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:05.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:05 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:05.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:05.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:05 smithi151 ceph-mon[28965]: pgmap v812: 27 pgs: 27 active+clean; 1.2 GiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 50 MiB/s wr, 802 op/s; 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:21:05.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:05 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:05.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4238588378' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:05.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:05 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:05.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:05.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:06.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:06.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:06.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:21:06.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:06.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:06.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:06.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:06.405 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:06.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:06.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:06.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:06.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:06.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 26289 26273 71.3875 74.25 0.0165987 0.013991 2024-06-27T21:21:06.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:06.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:06 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:06.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:06 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:21:06.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:06 smithi151 ceph-mon[28965]: osdmap e304: 12 total, 12 up, 10 in 2024-06-27T21:21:06.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:06.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:06 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:06.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:06 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:21:06.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:06 smithi192 ceph-mon[32956]: osdmap e304: 12 total, 12 up, 10 in 2024-06-27T21:21:06.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:06.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:06.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:06.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:06.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:07.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:06 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:07.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:06 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:21:07.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:06 smithi190 ceph-mon[32247]: osdmap e304: 12 total, 12 up, 10 in 2024-06-27T21:21:07.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:07.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:07.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:07.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:07.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:07.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:07.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:07.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:07.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:07.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 16 27737 27721 72.1835 90.5 0.00242875 0.0138477 2024-06-27T21:21:07.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:07.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:07.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:07.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:07 smithi190 ceph-mon[32247]: pgmap v814: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s; 2.7 MiB/s, 42 objects/s recovering 2024-06-27T21:21:07.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:07 smithi192 ceph-mon[32956]: pgmap v814: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s; 2.7 MiB/s, 42 objects/s recovering 2024-06-27T21:21:07.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:07.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:07.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:07 smithi151 ceph-mon[28965]: pgmap v814: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.4 GiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s; 2.7 MiB/s, 42 objects/s recovering 2024-06-27T21:21:07.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:07.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:07.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:08.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:08.113 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:08.196 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:08.196 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:21:08.196 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:21:08.196 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:21:08.196 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:08.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:08.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:08.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:08.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:08.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:21:08.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:08.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:08.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:21:08.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 29387 29371 73.4209 103.125 0.00759714 0.0136107 2024-06-27T21:21:08.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:08.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:08.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:08.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:08.770 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:08.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:08.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:08.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:08.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:21:08.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:09.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:09.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:09.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:09.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:09.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:09.287 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:09.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:09.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:09.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:09.600 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 31372 31356 75.3682 124.062 0.00524495 0.0132628 2024-06-27T21:21:09.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:09.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:09.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:09 smithi192 ceph-mon[32956]: pgmap v815: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:21:09.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:09 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:09.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:09 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:21:09.678 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:09 smithi192 ceph-mon[32956]: osdmap e305: 12 total, 12 up, 10 in 2024-06-27T21:21:09.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:09.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:09.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:09.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:09.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:09.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:09 smithi151 ceph-mon[28965]: pgmap v815: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:21:09.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:09 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:09.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:09 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:21:09.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:09 smithi151 ceph-mon[28965]: osdmap e305: 12 total, 12 up, 10 in 2024-06-27T21:21:10.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:09 smithi190 ceph-mon[32247]: pgmap v815: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.5 GiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:21:10.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:09 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:10.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:09 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:21:10.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:09 smithi190 ceph-mon[32247]: osdmap e305: 12 total, 12 up, 10 in 2024-06-27T21:21:10.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:10.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:10.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:10.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:10.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:10.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:10.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:10.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:10.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:10.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:10.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 16 32847 32831 75.9907 92.1875 0.0219273 0.0131361 2024-06-27T21:21:10.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:10.670 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:10 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:21:10.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:10 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:21:10.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:10 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:10.671 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:10 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:21:10.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:10.721 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:10.752 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [2, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:21:10.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:21:10.753 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-06-27T21:21:10.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 10 2024-06-27T21:21:10.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:10.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:10.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:10.832 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:10 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:21:10.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:10 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:21:10.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:10 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:10.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:10 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:21:10.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:10.923 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:10.923 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:7672 2.d 2:bd9a34d0:::benchmark_data_smithi192_59818_object7671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.208270+0000", 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 560.71616186699998, 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.227281121, 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7672 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208270+0000", 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208270+0000", 2024-06-27T21:21:10.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208271+0000", 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2729999999999999e-06 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208291+0000", 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9899e-05 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208304+0000", 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.256e-05 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208311+0000", 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.294e-06 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208351+0000", 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9774000000000002e-05 2024-06-27T21:21:10.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208433+0000", 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2402000000000002e-05 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.208517+0000", 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3277999999999998e-05 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209005+0000", 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000487997 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209476+0000", 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047170399999999998 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435492+0000", 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22601585399999999 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435524+0000", 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2202000000000001e-05 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435551+0000", 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6883999999999999e-05 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:7672 2.d e108/104 ondisk, result = 0)", 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.209290+0000", 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 560.71514167999999, 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.226265196, 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209290+0000", 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209290+0000", 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209291+0000", 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8599999999999998e-07 2024-06-27T21:21:10.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.209294+0000", 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9979999999999999e-06 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435435+0000", 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22614099600000001 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435439+0000", 2024-06-27T21:21:10.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5120000000000002e-06 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435479+0000", 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9264999999999998e-05 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435488+0000", 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3279999999999995e-06 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.435556+0000", 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7510999999999998e-05 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.24508.0:7689 2.0 e108/104 2:05a01aa7:::benchmark_data_smithi192_59818_object7688:head v 108'502, mlcod=108'499)", 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.224260+0000", 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 560.70017208399997, 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22816329499999999, 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224260+0000", 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224260+0000", 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224261+0000", 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6000000000000003e-07 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.224357+0000", 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6608999999999996e-05 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451882+0000", 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.227524636 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451887+0000", 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.724e-06 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451931+0000", 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4607000000000002e-05 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.451945+0000", 2024-06-27T21:21:10.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3582e-05 2024-06-27T21:21:10.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.452386+0000", 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044162299999999998 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.452423+0000", 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6854000000000003e-05 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:43154 2.d 2:b53185c5:::benchmark_data_smithi192_59818_object43153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.776149+0000", 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 520.14828345399997, 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24192956500000001, 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 43154 2024-06-27T21:21:10.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776149+0000", 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776149+0000", 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776150+0000", 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.052e-06 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776171+0000", 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1206e-05 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776184+0000", 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2835e-05 2024-06-27T21:21:10.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776191+0000", 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1600000000000001e-06 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776226+0000", 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5216000000000003e-05 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776296+0000", 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0240000000000005e-05 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.776376+0000", 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9312000000000002e-05 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777045+0000", 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066960100000000003 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777729+0000", 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00068351800000000004 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017986+0000", 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24025681700000001 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.018034+0000", 2024-06-27T21:21:10.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8449999999999999e-05 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.018078+0000", 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4158e-05 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.947 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:43154 2.d e123/104 ondisk, result = 0)", 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 520.14683700099999, 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24048932100000001, 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777595+0000", 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:30.777598+0000", 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3640000000000002e-06 2024-06-27T21:21:10.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017922+0000", 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24032410600000001 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017928+0000", 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8950000000000003e-06 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017969+0000", 2024-06-27T21:21:10.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1403999999999999e-05 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.017980+0000", 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0722e-05 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:31.018084+0000", 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104425 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.951 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:50215 2.d 2:b84cda7e:::benchmark_data_smithi192_59818_object50214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:12:35.962578+0000", 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 514.96185408899998, 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21343746199999999, 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50215 2024-06-27T21:21:10.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962578+0000", 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962578+0000", 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962580+0000", 2024-06-27T21:21:10.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6220000000000001e-06 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962600+0000", 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9893000000000001e-05 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962611+0000", 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1416999999999999e-05 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962616+0000", 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2290000000000002e-06 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962666+0000", 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0210000000000002e-05 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962761+0000", 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4616000000000003e-05 2024-06-27T21:21:10.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.962851+0000", 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9772999999999996e-05 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.964264+0000", 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001413414 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:35.964542+0000", 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027784499999999998 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.175922+0000", 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21138020299999999 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.175987+0000", 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4610000000000007e-05 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:12:36.176015+0000", 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8629999999999999e-05 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121198 2.d 2:bd9f4ce1:::benchmark_data_smithi192_59818_object29123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:45.242800+0000", 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 445.681632536, 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21629783799999999, 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.958 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121198 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242800+0000", 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242800+0000", 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242801+0000", 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.068e-06 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242805+0000", 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2930000000000002e-06 2024-06-27T21:21:10.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242817+0000", 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1787000000000001e-05 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242822+0000", 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.614e-06 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.242875+0000", 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2468e-05 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.243001+0000", 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012661199999999999 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.243118+0000", 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011650999999999999 2024-06-27T21:21:10.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.245693+0000", 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025746229999999998 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.245906+0000", 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000213122 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.459027+0000", 2024-06-27T21:21:10.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21312176299999999 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.459061+0000", 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3868999999999997e-05 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:45.459097+0000", 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6109000000000003e-05 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:161706 2.d 2:bf6d504a:::benchmark_data_smithi192_59818_object69631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:46.391941+0000", 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.53249117199999, 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.219282276, 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161706 2024-06-27T21:21:10.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391941+0000", 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391941+0000", 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391943+0000", 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.531e-06 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391950+0000", 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7460000000000003e-06 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.391969+0000", 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8564999999999999e-05 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.392007+0000", 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8473999999999998e-05 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.392986+0000", 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00097909499999999992 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.393087+0000", 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010039000000000001 2024-06-27T21:21:10.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.393186+0000", 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9653000000000003e-05 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.399148+0000", 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059618889999999997 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.405716+0000", 2024-06-27T21:21:10.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0065678999999999998 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.611040+0000", 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20532411 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.611160+0000", 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120061 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.611223+0000", 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2861999999999995e-05 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:161711 2.d 2:bda238be:::benchmark_data_smithi192_59818_object69636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:14:46.397659+0000", 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 384.52677315699998, 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23285805100000001, 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.971 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161711 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397659+0000", 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397659+0000", 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397660+0000", 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5719999999999999e-06 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397667+0000", 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1240000000000002e-06 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397680+0000", 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3346999999999999e-05 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.397686+0000", 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0469999999999999e-06 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.399444+0000", 2024-06-27T21:21:10.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001757488 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.399717+0000", 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027336100000000001 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.400227+0000", 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051034599999999996 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.405741+0000", 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055133339999999999 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.406373+0000", 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063264199999999999 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.630405+0000", 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22403214199999999 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.630472+0000", 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6199999999999996e-05 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:14:46.630517+0000", 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5448000000000003e-05 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15132.0:2793 3.0 e192/186 3:02f22ba0:::benchmark_data_smithi192_74314_object2792:head v 192'178, mlcod=192'174)", 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:16.253973+0000", 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 294.67045880900002, 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21077112000000001, 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253973+0000", 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253973+0000", 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.253979+0000", 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3890000000000004e-06 2024-06-27T21:21:10.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.254000+0000", 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1155999999999999e-05 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464256+0000", 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.210255739 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464262+0000", 2024-06-27T21:21:10.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8809999999999998e-06 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464314+0000", 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1462999999999998e-05 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464335+0000", 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1256000000000001e-05 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464700+0000", 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036450399999999998 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:16.464744+0000", 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4731999999999998e-05 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15132.0:155734 3.0 e248/186 3:0cb6e170:::benchmark_data_smithi192_74314_object49096:head v 248'9715, mlcod=248'9713)", 2024-06-27T21:21:10.982 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:18:47.541803+0000", 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 143.382629237, 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21411283, 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.541803+0000", 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.541803+0000", 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.541804+0000", 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2099999999999996e-07 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.541809+0000", 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.31e-06 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.752269+0000", 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21046035199999999 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.752274+0000", 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9579999999999998e-06 2024-06-27T21:21:10.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.752342+0000", 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8195999999999996e-05 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.752364+0000", 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1902000000000001e-05 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.755878+0000", 2024-06-27T21:21:10.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035135800000000001 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:47.755916+0000", 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7811000000000001e-05 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:653 4.6 4:697d5697:::benchmark_data_smithi192_88961_object652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.796944+0000", 2024-06-27T21:21:10.989 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.127488375999999, 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25617681399999997, 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 653 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796944+0000", 2024-06-27T21:21:10.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796944+0000", 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796945+0000", 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1500000000000003e-07 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796964+0000", 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8958e-05 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796974+0000", 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0054999999999999e-05 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796986+0000", 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2773e-05 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797028+0000", 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1832000000000002e-05 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797115+0000", 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6712999999999995e-05 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797195+0000", 2024-06-27T21:21:10.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0402999999999996e-05 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797550+0000", 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035431999999999998 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.805651+0000", 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081016370000000001 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.053058+0000", 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.247406986 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.053093+0000", 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4881e-05 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.053121+0000", 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7341000000000001e-05 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.995 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:672 4.6 4:6f4ffc5a:::benchmark_data_smithi192_88961_object671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.804539+0000", 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.119893327, 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25584938200000001, 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:10.996 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 672 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804539+0000", 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804539+0000", 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804540+0000", 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4419999999999999e-06 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804564+0000", 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3887999999999999e-05 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804577+0000", 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2992e-05 2024-06-27T21:21:10.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804583+0000", 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0290000000000002e-06 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804639+0000", 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6381000000000001e-05 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804724+0000", 2024-06-27T21:21:10.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4730999999999999e-05 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804805+0000", 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1168999999999995e-05 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.805244+0000", 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00043857899999999998 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.829263+0000", 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024019511 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.060311+0000", 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23104725100000001 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.060359+0000", 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7987e-05 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.060388+0000", 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9422000000000001e-05 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:699 4.6 4:6c5f4d04:::benchmark_data_smithi192_88961_object698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.828311+0000", 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.096121263000001, 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23322895599999999, 2024-06-27T21:21:11.002 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 699 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828311+0000", 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828311+0000", 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828312+0000", 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.226e-06 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828332+0000", 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0293e-05 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828345+0000", 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2317000000000001e-05 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828350+0000", 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3530000000000002e-06 2024-06-27T21:21:11.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828401+0000", 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0878999999999999e-05 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828488+0000", 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7100000000000003e-05 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828572+0000", 2024-06-27T21:21:11.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4259999999999996e-05 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.828971+0000", 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039863399999999998 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.948510+0000", 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.119539164 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.061470+0000", 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.112960407 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.061500+0000", 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9594000000000002e-05 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.061540+0000", 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9728999999999999e-05 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.008 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:7293 4.6 4:618f99dc:::benchmark_data_smithi192_88961_object7292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:48.611719+0000", 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.312712748999999, 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20982527600000001, 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.009 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7293 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611719+0000", 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611719+0000", 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611720+0000", 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1500000000000005e-07 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611739+0000", 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8760999999999999e-05 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611745+0000", 2024-06-27T21:21:11.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.46e-06 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611749+0000", 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8999999999999999e-06 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611786+0000", 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7503000000000003e-05 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611859+0000", 2024-06-27T21:21:11.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2162000000000005e-05 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.611937+0000", 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8549999999999998e-05 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.612367+0000", 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000430307 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.612895+0000", 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052724100000000002 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.821446+0000", 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.208551454 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.821504+0000", 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7717999999999999e-05 2024-06-27T21:21:11.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.821545+0000", 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0704999999999997e-05 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:7562 4.6 4:6d071455:::benchmark_data_smithi192_88961_object7561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:48.874423+0000", 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.050009273000001, 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.210606338, 2024-06-27T21:21:11.015 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7562 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.874423+0000", 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.874423+0000", 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.874424+0000", 2024-06-27T21:21:11.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.009e-06 2024-06-27T21:21:11.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.874446+0000", 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2507e-05 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.874459+0000", 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2903e-05 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.874467+0000", 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5059999999999998e-06 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.875201+0000", 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00073470999999999996 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.875296+0000", 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4436000000000002e-05 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.875417+0000", 2024-06-27T21:21:11.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120919 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.876003+0000", 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00058646399999999997 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.876417+0000", 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041383300000000001 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:49.084948+0000", 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20853084 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:49.084995+0000", 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7472000000000002e-05 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:49.085029+0000", 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3738999999999997e-05 2024-06-27T21:21:11.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:23377 4.6 4:6629100e:::benchmark_data_smithi192_88961_object23376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:04.370270+0000", 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.5541623539999998, 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211717135, 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23377 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370270+0000", 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370270+0000", 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370270+0000", 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3200000000000004e-07 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370289+0000", 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8828999999999999e-05 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370299+0000", 2024-06-27T21:21:11.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9729999999999992e-06 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370304+0000", 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9919999999999998e-06 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370356+0000", 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2108000000000003e-05 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370450+0000", 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4003000000000001e-05 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370557+0000", 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106949 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371873+0000", 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001315582 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.381699+0000", 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0098264200000000006 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.581919+0000", 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20021974300000001 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.581953+0000", 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4311999999999999e-05 2024-06-27T21:21:11.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.581987+0000", 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3491999999999997e-05 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:23379 4.6 4:6df5925b:::benchmark_data_smithi192_88961_object23378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:04.370843+0000", 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.5535889630000002, 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.211939291, 2024-06-27T21:21:11.029 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23379 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370843+0000", 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370843+0000", 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370844+0000", 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9500000000000001e-07 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370872+0000", 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8405e-05 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370885+0000", 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2945000000000001e-05 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.370894+0000", 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5750000000000001e-06 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371199+0000", 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030476 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371297+0000", 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8004000000000006e-05 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371392+0000", 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5728000000000002e-05 2024-06-27T21:21:11.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.372573+0000", 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001180841 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.391349+0000", 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018775909 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.582729+0000", 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19138029400000001 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.582761+0000", 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1900000000000003e-05 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.582782+0000", 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1035000000000001e-05 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.035 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:23380 4.6 4:6dcfcda4:::benchmark_data_smithi192_88961_object23379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:04.371539+0000", 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.5528929339999999, 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21134383700000001, 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.036 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23380 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371539+0000", 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371539+0000", 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371540+0000", 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2999999999999999e-07 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371570+0000", 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9898000000000001e-05 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371583+0000", 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2659e-05 2024-06-27T21:21:11.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371591+0000", 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2530000000000004e-06 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.371896+0000", 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030538199999999998 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.372022+0000", 2024-06-27T21:21:11.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000125599 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.372159+0000", 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013669600000000001 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.372763+0000", 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060458300000000001 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.391445+0000", 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018681996999999999 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.582817+0000", 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19137222300000001 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.582857+0000", 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9251000000000001e-05 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.582883+0000", 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6366000000000001e-05 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:23389 4.6 4:6f9ef948:::benchmark_data_smithi192_88961_object23388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:04.375595+0000", 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.5488374760000001, 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20815449699999999, 2024-06-27T21:21:11.042 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23389 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375595+0000", 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375595+0000", 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375595+0000", 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9199999999999999e-07 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375614+0000", 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8253999999999998e-05 2024-06-27T21:21:11.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375624+0000", 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.049e-05 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375630+0000", 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3759999999999999e-06 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:11.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375682+0000", 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2277000000000001e-05 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375836+0000", 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000154291 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.375936+0000", 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9518999999999994e-05 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.376607+0000", 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000671424 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.392645+0000", 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016037880000000001 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:11.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.583692+0000", 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19104676600000001 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.583726+0000", 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4035999999999997e-05 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:04.583749+0000", 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3292e-05 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.048 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:11.049 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:11.049 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:11.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:11.050 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:10 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:21:11.050 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:10 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:21:11.050 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:10 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:11.050 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:10 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:21:11.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:11.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:11.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:11.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:11.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:11.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:11.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:11.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:11.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 34209 34193 76.3166 85.125 0.00892889 0.0130951 2024-06-27T21:21:11.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:11.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:11.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:11 smithi151 ceph-mon[28965]: pgmap v817: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.7 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s; 1.8 MiB/s, 29 objects/s recovering 2024-06-27T21:21:11.769 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:11 smithi192 ceph-mon[32956]: pgmap v817: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.7 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s; 1.8 MiB/s, 29 objects/s recovering 2024-06-27T21:21:11.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:11.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:21:11.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:11.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:11.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:11.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:12.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:11 smithi190 ceph-mon[32247]: pgmap v817: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 1.7 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 84 MiB/s wr, 1.34k op/s; 1.8 MiB/s, 29 objects/s recovering 2024-06-27T21:21:12.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:12.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:12.131 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:12.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:12.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:12.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:12.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:12.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:12.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:12.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:12.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:12.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 35676 35660 76.8464 91.6875 0.0128905 0.0130066 2024-06-27T21:21:12.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:12.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:12.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:12 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:21:12.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:12 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:21:12.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:12.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:12.935 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:12.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:12.958 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:21:12.958 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:21:12.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:12.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:21:12.989 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:21:13.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:12 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:21:13.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:12 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:21:13.016 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:21:13.016 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:21:13.039 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:21:13.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:21:13.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:12 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:21:13.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:12 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:21:13.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:21:13.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:21:13.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:13.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:21:13.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:21:13.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:13.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:21:13.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:21:13.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:21:13.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:21:13.157 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:21:13.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:21:13.186 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:21:13.186 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:21:13.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:13.207 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:21:13.207 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:21:13.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:13.231 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:21:13.232 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:21:13.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:13.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:13.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:13.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:13.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:13.584 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:21:13.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 36938 36922 76.9138 78.875 0.0174408 0.0129954 2024-06-27T21:21:13.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:13 smithi192 ceph-mon[32956]: pgmap v818: 27 pgs: 27 active+clean; 1.9 GiB data, 7.4 GiB used, 887 GiB / 894 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-06-27T21:21:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3620455134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:21:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3107464121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:13 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:21:13.657 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:13 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:21:13.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:13 smithi151 ceph-mon[28965]: pgmap v818: 27 pgs: 27 active+clean; 1.9 GiB data, 7.4 GiB used, 887 GiB / 894 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-06-27T21:21:13.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3620455134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:21:13.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3107464121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:13.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:13 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:21:13.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:13 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:21:13.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:13.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:13.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:13.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:13.931 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:21:13.931 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:21:13.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:13.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:13.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:14.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:13 smithi190 ceph-mon[32247]: pgmap v818: 27 pgs: 27 active+clean; 1.9 GiB data, 7.4 GiB used, 887 GiB / 894 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-06-27T21:21:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3620455134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:21:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3107464121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:13 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:21:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:13 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:21:14.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:14.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:14.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:14.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:14.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:14.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:14.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:14.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:14.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:14.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 38438 38422 77.4567 93.75 0.0253218 0.0129019 2024-06-27T21:21:14.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:21:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:14.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:14 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:21:14.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:14 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:21:14.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3620455134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:21:14.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:14 smithi151 ceph-mon[28965]: osdmap e306: 12 total, 12 up, 10 in 2024-06-27T21:21:14.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:14 smithi151 ceph-mon[28965]: osdmap e307: 12 total, 12 up, 10 in 2024-06-27T21:21:14.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:14.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:14.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:14.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:14.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:14.903 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:14.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:15.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:15.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:14 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:21:15.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:14 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:21:15.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3620455134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:21:15.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:14 smithi190 ceph-mon[32247]: osdmap e306: 12 total, 12 up, 10 in 2024-06-27T21:21:15.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:14 smithi190 ceph-mon[32247]: osdmap e307: 12 total, 12 up, 10 in 2024-06-27T21:21:15.029 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:14 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:21:15.029 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:14 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:21:15.029 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3620455134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:21:15.030 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:14 smithi192 ceph-mon[32956]: osdmap e306: 12 total, 12 up, 10 in 2024-06-27T21:21:15.030 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:14 smithi192 ceph-mon[32956]: osdmap e307: 12 total, 12 up, 10 in 2024-06-27T21:21:15.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:15.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:15.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:15.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:15.257 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:15.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:15.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:15.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:15.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:15.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 16 39676 39660 77.4539 77.375 0.00198189 0.0128775 2024-06-27T21:21:15.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:15.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:15.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:15.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:15.849 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:15 smithi192 ceph-mon[32956]: pgmap v820: 27 pgs: 27 active+clean; 2.0 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:21:15.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:15.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:15.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:15.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:15 smithi151 ceph-mon[28965]: pgmap v820: 27 pgs: 27 active+clean; 2.0 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:21:15.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:16.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:15 smithi190 ceph-mon[32247]: pgmap v820: 27 pgs: 27 active+clean; 2.0 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:21:16.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:16.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:16.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:16.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:16.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:16.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:16.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:16.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:16.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:16.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 41192 41176 77.9778 94.75 0.0143704 0.0128183 2024-06-27T21:21:16.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:16.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:16.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:16.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:16.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:21:16.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:16.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:16.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:17.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:17.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:17.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:17.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:17.226 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:17.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:17.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:17.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:17 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:21:17.007+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e307 definitely_dead 0 2024-06-27T21:21:17.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:17.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:17.498 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:21:17.498 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:21:17.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:17.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:17.594 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.10. 2024-06-27T21:21:17.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 42741 42725 78.5315 96.8125 0.0109399 0.0127259 2024-06-27T21:21:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:17 smithi151 ceph-mon[28965]: pgmap v822: 27 pgs: 27 active+clean; 2.3 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-06-27T21:21:17.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:17 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/698620327' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-06-27T21:21:17.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:21:17.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:17.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:17.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:17 smithi190 ceph-mon[32247]: pgmap v822: 27 pgs: 27 active+clean; 2.3 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-06-27T21:21:17.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:17 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/698620327' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-06-27T21:21:17.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:17 smithi192 ceph-mon[32956]: pgmap v822: 27 pgs: 27 active+clean; 2.3 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s 2024-06-27T21:21:17.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:17 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/698620327' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-06-27T21:21:17.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:17.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:17.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:17.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:17.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:18.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:18.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:18.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:21:18.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:18.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:18.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:18.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:18.328 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:18.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:18.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:18.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:18.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:18.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 16 42776 42760 76.3503 2.1875 0.147968 0.012795 2024-06-27T21:21:18.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:18.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:18.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/698620327' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-06-27T21:21:18.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:18 smithi151 ceph-mon[28965]: osdmap e308: 12 total, 12 up, 11 in 2024-06-27T21:21:18.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:18 smithi151 ceph-mon[28965]: osdmap e309: 12 total, 12 up, 11 in 2024-06-27T21:21:18.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/698620327' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-06-27T21:21:18.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:18 smithi192 ceph-mon[32956]: osdmap e308: 12 total, 12 up, 11 in 2024-06-27T21:21:18.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:18 smithi192 ceph-mon[32956]: osdmap e309: 12 total, 12 up, 11 in 2024-06-27T21:21:18.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:18.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:18.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:18.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:19.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/698620327' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-06-27T21:21:19.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:18 smithi190 ceph-mon[32247]: osdmap e308: 12 total, 12 up, 11 in 2024-06-27T21:21:19.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:18 smithi190 ceph-mon[32247]: osdmap e309: 12 total, 12 up, 11 in 2024-06-27T21:21:19.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:19.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:19.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:19.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:19.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:19.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:19.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:19.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:19.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:19.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:19.577 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-06-27T21:21:19.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 42776 42760 74.2294 0 - 0.012795 2024-06-27T21:21:19.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:19.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: pgmap v824: 27 pgs: 27 active+clean; 2.4 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:21:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: 1.5 deep-scrub starts 2024-06-27T21:21:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: 1.5 deep-scrub ok 2024-06-27T21:21:19.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3918026812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:21:19.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:19.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3918026812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:21:19.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:19 smithi151 ceph-mon[28965]: osdmap e310: 12 total, 12 up, 11 in 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: pgmap v824: 27 pgs: 27 active+clean; 2.4 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: 1.5 deep-scrub starts 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: 1.5 deep-scrub ok 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3918026812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3918026812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:21:19.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:19 smithi192 ceph-mon[32956]: osdmap e310: 12 total, 12 up, 11 in 2024-06-27T21:21:19.736 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:21:19.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:19.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:19.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:19.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:19.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: pgmap v824: 27 pgs: 27 active+clean; 2.4 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:21:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: 1.5 deep-scrub starts 2024-06-27T21:21:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: 1.5 deep-scrub ok 2024-06-27T21:21:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3918026812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:21:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3918026812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:21:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:19 smithi190 ceph-mon[32247]: osdmap e310: 12 total, 12 up, 11 in 2024-06-27T21:21:20.032 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:21:20.033 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:21:20.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:20.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:20.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:20.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:20.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:20.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:20.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:20.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:20.601 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 44002 43986 74.294 38.3125 0.00735426 0.0134544 2024-06-27T21:21:20.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:20.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:21:20.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:20.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:20.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:20.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:20.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:20.911 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:20 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:20.947 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:20.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:21.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:20 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:21.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:21.038 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:21.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:20 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:21.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:21.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:21.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:21.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:21.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:21.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:21.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:21.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:21.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 45243 45227 74.3798 77.5625 0.00859862 0.0134387 2024-06-27T21:21:21.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.664 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:21.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:21.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:21 smithi192 ceph-mon[32956]: pgmap v827: 27 pgs: 2 remapped+peering, 25 active+clean; 2.5 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-06-27T21:21:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:21 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:21.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:21.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:21.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:21.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:21.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:21 smithi151 ceph-mon[28965]: pgmap v827: 27 pgs: 2 remapped+peering, 25 active+clean; 2.5 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-06-27T21:21:21.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:21 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:22.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:21 smithi190 ceph-mon[32247]: pgmap v827: 27 pgs: 2 remapped+peering, 25 active+clean; 2.5 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-06-27T21:21:22.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:21 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:22.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:22.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:22.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:22.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:22.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:22.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:22.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:22.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:22.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:22.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:22.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 46633 46617 74.7 86.875 0.0304994 0.0133774 2024-06-27T21:21:22.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:22.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:22.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:22 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 2746/129444 objects degraded (2.121%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:22.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:22.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:22.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:21:22.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:22.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:22.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:23.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:22 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 2746/129444 objects degraded (2.121%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:23.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:22 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 2746/129444 objects degraded (2.121%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:23.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:23.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:23.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:23.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:23.267 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:23.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:23.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:23.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:23.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:23.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:23.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:21:23.605150+0000 min lat: 0.00102932 max lat: 2.15488 avg lat: 0.0133741 2024-06-27T21:21:23.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:21:23.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 47831 47815 74.7043 74.875 0.00907495 0.0133741 2024-06-27T21:21:23.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:23.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:23.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:23.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:21:23.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:21:23.804 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:21:23.804 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:21:23.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:23.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:21:23.837 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:23.868 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:21:23.868 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:21:23.901 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:21:23.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:21:23.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:21:23.934 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:21:23.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:23 smithi151 ceph-mon[28965]: pgmap v828: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 23 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 52 MiB/s wr, 832 op/s; 2746/129444 objects degraded (2.121%); 2557/129444 objects misplaced (1.975%); 576 KiB/s, 8 objects/s recovering 2024-06-27T21:21:23.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1060060501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:23.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:23.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:21:23.969 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:21:23.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:23.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:23.999 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:21:24.000 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:21:24.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:23 smithi190 ceph-mon[32247]: pgmap v828: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 23 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 52 MiB/s wr, 832 op/s; 2746/129444 objects degraded (2.121%); 2557/129444 objects misplaced (1.975%); 576 KiB/s, 8 objects/s recovering 2024-06-27T21:21:24.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1060060501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:24.029 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:23 smithi192 ceph-mon[32956]: pgmap v828: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 23 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 52 MiB/s wr, 832 op/s; 2746/129444 objects degraded (2.121%); 2557/129444 objects misplaced (1.975%); 576 KiB/s, 8 objects/s recovering 2024-06-27T21:21:24.029 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1060060501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:24.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:21:24.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:21:24.065 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:21:24.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:21:24.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:24.098 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:21:24.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:21:24.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:24.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:21:24.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:21:24.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:24.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:24.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:24.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:24.458 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:24.458 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:48610 4.1 4:83a8df1c:::benchmark_data_smithi192_88961_object48609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:24.442686+0000", 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017405411999999999, 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017449914, 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:24.459 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 48610 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442686+0000", 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442686+0000", 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:24.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442688+0000", 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1059999999999998e-06 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442744+0000", 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6604000000000002e-05 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442759+0000", 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5211000000000001e-05 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442768+0000", 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.789e-06 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442830+0000", 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1394000000000002e-05 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.442985+0000", 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000155604 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,6", 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.443131+0000", 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000145348 2024-06-27T21:21:24.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.445323+0000", 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021924190000000001 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.445494+0000", 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017071299999999999 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:24.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:48632 4.2 e310/282 4:48bd876c:::benchmark_data_smithi192_88961_object48631:head v 310'3075, mlcod=310'3071)", 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:24.457878+0000", 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0022132969999999999, 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023342559999999998, 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.457878+0000", 2024-06-27T21:21:24.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.457878+0000", 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.457879+0000", 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1500000000000003e-07 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.457906+0000", 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7351999999999998e-05 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.457924+0000", 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7621999999999998e-05 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.457931+0000", 2024-06-27T21:21:24.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0469999999999996e-06 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.458159+0000", 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000228101 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.458177+0000", 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8216999999999999e-05 2024-06-27T21:21:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:24.460213+0000", 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020360069999999998 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-06-27T21:21:24.469 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:24.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:24.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:24.577 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:21:24.577 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:21:24.578 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-06-27T21:21:24.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 2 2024-06-27T21:21:24.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 15 48819 48804 74.3897 61.8125 0.00514537 0.0134361 2024-06-27T21:21:24.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:24.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:24.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:24.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:24.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:24.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:24.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:25.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:25.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:25.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:25.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:25.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:21:25.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:25.282 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:25.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:25.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:25.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:25.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:25.547 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:25.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:25 smithi151 ceph-mon[28965]: pgmap v829: 27 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 53 MiB/s wr, 844 op/s; 2746/134514 objects degraded (2.041%); 10158/134514 objects misplaced (7.552%); 4.6 MiB/s, 74 objects/s recovering 2024-06-27T21:21:25.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:25 smithi151 ceph-mon[28965]: osdmap e311: 12 total, 12 up, 11 in 2024-06-27T21:21:25.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 49877 49861 74.1913 66.0625 0.0299819 0.0134703 2024-06-27T21:21:25.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:25.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:25.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:25 smithi190 ceph-mon[32247]: pgmap v829: 27 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 53 MiB/s wr, 844 op/s; 2746/134514 objects degraded (2.041%); 10158/134514 objects misplaced (7.552%); 4.6 MiB/s, 74 objects/s recovering 2024-06-27T21:21:25.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:25 smithi190 ceph-mon[32247]: osdmap e311: 12 total, 12 up, 11 in 2024-06-27T21:21:25.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:25.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:25 smithi192 ceph-mon[32956]: pgmap v829: 27 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 53 MiB/s wr, 844 op/s; 2746/134514 objects degraded (2.041%); 10158/134514 objects misplaced (7.552%); 4.6 MiB/s, 74 objects/s recovering 2024-06-27T21:21:25.791 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:25 smithi192 ceph-mon[32956]: osdmap e311: 12 total, 12 up, 11 in 2024-06-27T21:21:25.810 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:25.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:25.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:25.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:25.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:26.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:26.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:26.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:26.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:26.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:26.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:26.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:26.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:26.486 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:21:26.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:26.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 51006 50990 74.1068 70.5625 0.0072246 0.0134887 2024-06-27T21:21:26.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:26.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:26.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:26.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:26.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:26.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/174472712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:21:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:26.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:26.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/174472712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:21:26.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:26.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:26.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:26.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:26.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/174472712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:21:26.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:27.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:27.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:27.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:27.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:27.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:27.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:21:27.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:27.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:27.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:27.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:27.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:27 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:27.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:27 smithi192 ceph-mon[32956]: pgmap v831: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 2746/145635 objects degraded (1.886%); 9732/145635 objects misplaced (6.682%); 7.6 MiB/s, 122 objects/s recovering 2024-06-27T21:21:27.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:27 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:27.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/174472712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:21:27.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:27 smithi192 ceph-mon[32956]: osdmap e312: 12 total, 12 up, 11 in 2024-06-27T21:21:27.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 52136 52120 74.0275 70.625 0.00835276 0.0135015 2024-06-27T21:21:27.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:27.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:27.701 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:27.701 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:21:27.701 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:21:27.701 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:21:27.701 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:27.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:27.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:27 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:27.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:27 smithi190 ceph-mon[32247]: pgmap v831: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 2746/145635 objects degraded (1.886%); 9732/145635 objects misplaced (6.682%); 7.6 MiB/s, 122 objects/s recovering 2024-06-27T21:21:27.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:27 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:27.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/174472712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:21:27.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:27 smithi190 ceph-mon[32247]: osdmap e312: 12 total, 12 up, 11 in 2024-06-27T21:21:27.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:27.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:27.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:27.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:27.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:27 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:27.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:27 smithi151 ceph-mon[28965]: pgmap v831: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 2746/145635 objects degraded (1.886%); 9732/145635 objects misplaced (6.682%); 7.6 MiB/s, 122 objects/s recovering 2024-06-27T21:21:27.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:27 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:27.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/174472712' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:21:27.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:27 smithi151 ceph-mon[28965]: osdmap e312: 12 total, 12 up, 11 in 2024-06-27T21:21:28.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:28.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:28.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:28.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:28.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:28.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:28.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:28.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:28.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:28.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 53245 53229 73.9226 69.3125 0.0125099 0.0135227 2024-06-27T21:21:28.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:28.669 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:28.681 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:28 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:28.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:21:28.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:28 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:28.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:28.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:28.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:28 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:28.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:28.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:29.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:29.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:29.086 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:21:29.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:29.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:29.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:29.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:29.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:29.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:29.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:29.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:29.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:29.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:29.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 54349 54333 73.8155 69 0.0792599 0.0135344 2024-06-27T21:21:29.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:29.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:29 smithi190 ceph-mon[32247]: pgmap v833: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 2746/148725 objects degraded (1.846%); 9368/148725 objects misplaced (6.299%); 9.9 MiB/s, 157 objects/s recovering 2024-06-27T21:21:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2746/148725 objects degraded (1.846%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:29.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:29 smithi190 ceph-mon[32247]: osdmap e313: 12 total, 12 up, 11 in 2024-06-27T21:21:29.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:29 smithi192 ceph-mon[32956]: pgmap v833: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 2746/148725 objects degraded (1.846%); 9368/148725 objects misplaced (6.299%); 9.9 MiB/s, 157 objects/s recovering 2024-06-27T21:21:29.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2746/148725 objects degraded (1.846%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:29.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:29 smithi192 ceph-mon[32956]: osdmap e313: 12 total, 12 up, 11 in 2024-06-27T21:21:29.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:29.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:29.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:29 smithi151 ceph-mon[28965]: pgmap v833: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 2746/148725 objects degraded (1.846%); 9368/148725 objects misplaced (6.299%); 9.9 MiB/s, 157 objects/s recovering 2024-06-27T21:21:29.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2746/148725 objects degraded (1.846%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:29.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:29 smithi151 ceph-mon[28965]: osdmap e313: 12 total, 12 up, 11 in 2024-06-27T21:21:29.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:29.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:30.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:30.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:30.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:30.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:30.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:30.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:30.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:30.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:30.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:30.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 55071 55055 73.205 45.125 0.0136429 0.0136545 2024-06-27T21:21:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:30.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:30.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:30.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:30.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:7679 2.c 2:39129954:::benchmark_data_smithi192_59818_object7678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:11:50.214661+0000", 2024-06-27T21:21:30.881 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 580.66705892499999, 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.247452693, 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7679 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214661+0000", 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214661+0000", 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214662+0000", 2024-06-27T21:21:30.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214680+0000", 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7581999999999999e-05 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214689+0000", 2024-06-27T21:21:30.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0489999999999993e-06 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214693+0000", 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7520000000000004e-06 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214737+0000", 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3136000000000002e-05 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214846+0000", 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109242 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.214987+0000", 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014072600000000001 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.215443+0000", 2024-06-27T21:21:30.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000456274 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.215988+0000", 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054537599999999998 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.462014+0000", 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24602538800000001 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.462071+0000", 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7448000000000003e-05 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:11:50.462114+0000", 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2984e-05 2024-06-27T21:21:30.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121553 2.c 2:388c4988:::benchmark_data_smithi192_59818_object29478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:47.089910+0000", 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 463.79181023299998, 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61414529399999995, 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.889 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121553 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089910+0000", 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089910+0000", 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089911+0000", 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1349999999999999e-06 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089916+0000", 2024-06-27T21:21:30.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9429999999999999e-06 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089929+0000", 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2551000000000001e-05 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089935+0000", 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6279999999999999e-06 2024-06-27T21:21:30.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.089990+0000", 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4555000000000002e-05 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.090074+0000", 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4549999999999995e-05 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.090141+0000", 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6808000000000001e-05 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.092673+0000", 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025313480000000001 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093694+0000", 2024-06-27T21:21:30.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010215420000000001 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703914+0000", 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61022028100000003 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703985+0000", 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0164000000000001e-05 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.704055+0000", 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0789000000000003e-05 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.24508.0:121553 2.c e142/104 ondisk, result = 0)", 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:47.093550+0000", 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 463.788170417, 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61051329200000004, 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.896 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:30.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093550+0000", 2024-06-27T21:21:30.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:30.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093550+0000", 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093552+0000", 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9599999999999999e-06 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.093560+0000", 2024-06-27T21:21:30.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3170000000000002e-06 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703821+0000", 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61026075000000002 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703831+0000", 2024-06-27T21:21:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6739999999999992e-06 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703895+0000", 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4135999999999997e-05 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.703907+0000", 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2373e-05 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.704063+0000", 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015608199999999999 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.24508.0:121562 2.c 2:356cce12:::benchmark_data_smithi192_59818_object29487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:13:47.763119+0000", 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 463.11860114299998, 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30556668300000001, 2024-06-27T21:21:30.902 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.24508", 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 121562 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763119+0000", 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763119+0000", 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763121+0000", 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.917e-06 2024-06-27T21:21:30.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763130+0000", 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1160000000000005e-06 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763147+0000", 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7024000000000002e-05 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763159+0000", 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1724e-05 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763237+0000", 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8146999999999996e-05 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763334+0000", 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6927000000000002e-05 2024-06-27T21:21:30.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.763408+0000", 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4337000000000003e-05 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.766248+0000", 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002839622 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:47.766887+0000", 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063907000000000005 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:48.068540+0000", 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30165340899999998 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:48.068637+0000", 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6249999999999995e-05 2024-06-27T21:21:30.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:13:48.068686+0000", 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.914e-05 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(2.0 e177,142 2:07b9da72:::benchmark_data_smithi192_59818_object24729:head,139'7201)", 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:15:40.453073+0000", 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 350.42864678299998, 2024-06-27T21:21:30.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98256926600000005, 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453073+0000", 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453073+0000", 2024-06-27T21:21:30.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453074+0000", 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7800000000000001e-07 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:40.453080+0000", 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:41.435537+0000", 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.98245643100000002 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:41.435549+0000", 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2714000000000001e-05 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:15:41.435643+0000", 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3451999999999999e-05 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.912 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:34092 3.0 3:0d853025:::benchmark_data_smithi192_74314_object34091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.797728+0000", 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 290.083992352, 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.228574522, 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:30.913 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 34092 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797728+0000", 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797728+0000", 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797729+0000", 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3500000000000005e-07 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797746+0000", 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7461000000000001e-05 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797756+0000", 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7119999999999995e-06 2024-06-27T21:21:30.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797761+0000", 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2490000000000001e-06 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797812+0000", 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0827e-05 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797896+0000", 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3430999999999995e-05 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.797980+0000", 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4709000000000005e-05 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799207+0000", 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012271109999999999 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799469+0000", 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026209900000000001 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026238+0000", 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.226768789 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026277+0000", 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8489999999999999e-05 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026302+0000", 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5709000000000001e-05 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.919 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15132.0:34092 3.0 e199/186 ondisk, result = 0)", 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.799348+0000", 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 290.08237237399999, 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22695979999999999, 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799348+0000", 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799348+0000", 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799349+0000", 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2399999999999996e-07 2024-06-27T21:21:30.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.799352+0000", 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.225e-06 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026179+0000", 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22682697099999999 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026186+0000", 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.652e-06 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026230+0000", 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4499000000000002e-05 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026235+0000", 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6489999999999999e-06 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.026308+0000", 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2879999999999993e-05 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:34099 3.0 3:0e170147:::benchmark_data_smithi192_74314_object34098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.801826+0000", 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 290.07989419699999, 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22959323800000001, 2024-06-27T21:21:30.924 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 34099 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801826+0000", 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801826+0000", 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801827+0000", 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9299999999999996e-07 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801850+0000", 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2745e-05 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801858+0000", 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4849999999999992e-06 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801866+0000", 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0949999999999996e-06 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801910+0000", 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3708999999999998e-05 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.801995+0000", 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4843000000000001e-05 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.802076+0000", 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1186999999999994e-05 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803095+0000", 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010192510000000001 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803471+0000", 2024-06-27T21:21:30.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037532100000000002 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031348+0000", 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22787695499999999 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031383+0000", 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5766000000000003e-05 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031419+0000", 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5988000000000002e-05 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.15132.0:34099 3.0 e199/186 ondisk, result = 0)", 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:40.803252+0000", 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 290.07846781299997, 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.228173656, 2024-06-27T21:21:30.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803252+0000", 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803252+0000", 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803253+0000", 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5499999999999998e-07 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:40.803256+0000", 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1269999999999999e-06 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031287+0000", 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22803076 2024-06-27T21:21:30.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031298+0000", 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0888e-05 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031339+0000", 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1446000000000003e-05 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031344+0000", 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8250000000000004e-06 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:41.031426+0000", 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2055e-05 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:63831 3.0 3:05e624b8:::benchmark_data_smithi192_74314_object63830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:04.451706+0000", 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 266.43001407399998, 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.238552548, 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63831 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451706+0000", 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451706+0000", 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451708+0000", 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.421e-06 2024-06-27T21:21:30.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451728+0000", 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0792999999999999e-05 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451741+0000", 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2379e-05 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451748+0000", 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451788+0000", 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0352999999999998e-05 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451881+0000", 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2808999999999999e-05 2024-06-27T21:21:30.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.451995+0000", 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113669 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.453630+0000", 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001634768 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.454524+0000", 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00089468200000000005 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690192+0000", 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23566730999999999 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690229+0000", 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7013e-05 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:04.690259+0000", 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0175000000000002e-05 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:90997 3.e 3:7de0318a:::benchmark_data_smithi192_74314_object90996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:17:27.752808+0000", 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 243.12891206500001, 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23434428400000001, 2024-06-27T21:21:30.942 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 90997 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752808+0000", 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752808+0000", 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752809+0000", 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752827+0000", 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7668000000000001e-05 2024-06-27T21:21:30.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752837+0000", 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0035e-05 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752844+0000", 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5480000000000004e-06 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.752899+0000", 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5288000000000003e-05 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.753020+0000", 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012061999999999999 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.753151+0000", 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013093999999999999 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.755646+0000", 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024951349999999999 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.961606+0000", 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.20595986099999999 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.987057+0000", 2024-06-27T21:21:30.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025450803000000001 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.987111+0000", 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4558999999999997e-05 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:17:27.987152+0000", 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1090999999999997e-05 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:175433 3.e 3:7e1bf03b:::benchmark_data_smithi192_74314_object68795:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:19:16.763555+0000", 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 134.11816566499999, 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23101063299999999, 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 175433 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.763555+0000", 2024-06-27T21:21:30.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.763555+0000", 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.763555+0000", 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9499999999999998e-07 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.763556+0000", 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.579e-06 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.763561+0000", 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7550000000000002e-06 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.763564+0000", 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0690000000000002e-06 2024-06-27T21:21:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.835397+0000", 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.071832992999999998 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.835502+0000", 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104578 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,4", 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.835623+0000", 2024-06-27T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012123 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.842503+0000", 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0068794650000000004 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.960950+0000", 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.118447128 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.994478+0000", 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033528611999999999 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.994517+0000", 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8829999999999999e-05 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:19:16.994565+0000", 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8099000000000001e-05 2024-06-27T21:21:30.955 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:213264 3.4 3:20d1c801:::benchmark_data_smithi192_74314_object106626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:07.087204+0000", 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 83.794515802999996, 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.380334114, 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:21:30.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 213264 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087204+0000", 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087204+0000", 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087206+0000", 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1829999999999999e-06 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087210+0000", 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3950000000000003e-06 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087221+0000", 2024-06-27T21:21:30.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0711e-05 2024-06-27T21:21:30.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:30.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:30.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087226+0000", 2024-06-27T21:21:30.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2989999999999996e-06 2024-06-27T21:21:30.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087272+0000", 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5618000000000003e-05 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087358+0000", 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6531000000000002e-05 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-06-27T21:21:30.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087434+0000", 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5946999999999996e-05 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.091650+0000", 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042159090000000003 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.102657+0000", 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011007082 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.467432+0000", 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36477513700000003 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.467494+0000", 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1838e-05 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.467539+0000", 2024-06-27T21:21:30.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4464e-05 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:693 4.1 4:851bfe3a:::benchmark_data_smithi192_88961_object692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.821276+0000", 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.060444154000002, 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23055603899999999, 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 693 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821276+0000", 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821276+0000", 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821277+0000", 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3500000000000005e-07 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821297+0000", 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0421e-05 2024-06-27T21:21:30.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821306+0000", 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.918e-06 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821314+0000", 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1519999999999992e-06 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821375+0000", 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0597000000000003e-05 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821460+0000", 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4739000000000003e-05 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,6", 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821570+0000", 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109994 2024-06-27T21:21:30.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.821901+0000", 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033129200000000001 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.937496+0000", 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11559525399999999 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.051764+0000", 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.11426755600000001 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.051801+0000", 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7336999999999998e-05 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.051832+0000", 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0944000000000001e-05 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.970 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:7285 4.9 4:9ee302e1:::benchmark_data_smithi192_88961_object7284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:48.607517+0000", 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 42.274203040000003, 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22868163899999999, 2024-06-27T21:21:30.971 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7285 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607517+0000", 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607517+0000", 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607518+0000", 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7499999999999998e-07 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607556+0000", 2024-06-27T21:21:30.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7984000000000001e-05 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607566+0000", 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6989999999999998e-06 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607572+0000", 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0939999999999996e-06 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607624+0000", 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2122e-05 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607743+0000", 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011906299999999999 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.607850+0000", 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107329 2024-06-27T21:21:30.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.608322+0000", 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047108600000000001 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.609765+0000", 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014430809999999999 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.836104+0000", 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22633894399999999 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.836158+0000", 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4305000000000003e-05 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:48.836199+0000", 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0957000000000001e-05 2024-06-27T21:21:30.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:9486 4.3 4:c476da72:::benchmark_data_smithi192_88961_object9485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.492698623000003, 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.31799833300000002, 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 9486 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4700000000000001e-07 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389039+0000", 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6892999999999999e-05 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389049+0000", 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9620000000000005e-06 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389055+0000", 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3770000000000004e-06 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389102+0000", 2024-06-27T21:21:30.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7261e-05 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389216+0000", 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114268 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389358+0000", 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014186 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389912+0000", 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00055357200000000001 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.396996+0000", 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070839789999999998 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.706949+0000", 2024-06-27T21:21:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30995374999999997 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.706995+0000", 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6125000000000003e-05 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.707020+0000", 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4539e-05 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:9521 4.3 4:c314fc9a:::benchmark_data_smithi192_88961_object9520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:50.565666+0000", 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.316054282000003, 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53533422500000005, 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 9521 2024-06-27T21:21:30.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565666+0000", 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565666+0000", 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565667+0000", 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1800000000000005e-07 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565706+0000", 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9322000000000001e-05 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565715+0000", 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3780000000000007e-06 2024-06-27T21:21:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565720+0000", 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.656e-06 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565770+0000", 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9424e-05 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565887+0000", 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011731900000000001 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.566033+0000", 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014573599999999999 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.566532+0000", 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049953499999999995 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.657244+0000", 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.090711438000000005 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:51.100924+0000", 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.443680713 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:51.100959+0000", 2024-06-27T21:21:30.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4941000000000003e-05 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:51.101000+0000", 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0945000000000002e-05 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:20969 4.9 e301/282 ondisk, result = 0)", 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:21:30.991 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.338477187999999, 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42756724600000001, 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5199999999999998e-07 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543245+0000", 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.815e-06 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970713+0000", 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42746736499999999 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970738+0000", 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5360000000000001e-05 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970792+0000", 2024-06-27T21:21:30.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4327999999999999e-05 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970800+0000", 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4120000000000004e-06 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970810+0000", 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0614e-05 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:30.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:20969 4.9 e301/282 ondisk, result = 0)", 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.261732168000002, 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.378104671, 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:30.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9400000000000001e-07 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619990+0000", 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.827e-06 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997896+0000", 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.377906097 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:30.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997906+0000", 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5459999999999997e-06 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997967+0000", 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0730999999999998e-05 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997980+0000", 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3223e-05 2024-06-27T21:21:30.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.998093+0000", 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112853 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:20970 4.3 e301/282 ondisk, result = 0)", 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:01.850376+0000", 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.031344333, 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.340704061, 2024-06-27T21:21:31.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850376+0000", 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850376+0000", 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:21:31.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850378+0000", 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.796e-06 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850386+0000", 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6370000000000005e-06 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:21:31.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190672+0000", 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34028516800000003 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190689+0000", 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7036000000000001e-05 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190815+0000", 2024-06-27T21:21:31.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012613199999999999 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190833+0000", 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7813e-05 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.191080+0000", 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024747899999999999 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:31.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:31.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:21:31.005 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:21:31.005 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:21:31.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:31.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:31.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:31.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:31.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:31.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:31.454 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:31.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:31.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:31.602 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 56191 56175 73.1381 70 0.0293728 0.0136654 2024-06-27T21:21:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:31.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:31.736 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:31.751 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:31 smithi192 ceph-mon[32956]: pgmap v835: 27 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 2746/155841 objects degraded (1.762%); 9032/155841 objects misplaced (5.796%); 12 MiB/s, 187 objects/s recovering 2024-06-27T21:21:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:31 smithi190 ceph-mon[32247]: pgmap v835: 27 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 2746/155841 objects degraded (1.762%); 9032/155841 objects misplaced (5.796%); 12 MiB/s, 187 objects/s recovering 2024-06-27T21:21:31.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:31 smithi151 ceph-mon[28965]: pgmap v835: 27 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 2746/155841 objects degraded (1.762%); 9032/155841 objects misplaced (5.796%); 12 MiB/s, 187 objects/s recovering 2024-06-27T21:21:31.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:21:31.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:31.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:31.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:31.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:31.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:32.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:32.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:32.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:32.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:32.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:32.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:32.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:32.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:32.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 16 57287 57271 73.0433 68.5 0.0140097 0.0136766 2024-06-27T21:21:32.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:32.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:32.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:32.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:32 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:32.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:32.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:32 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:32.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:32.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:32.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:32.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:32 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:33.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:33.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:33.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:33.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:33.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:33.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:33.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:33.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:33.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:33 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:21:33.192+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e313 definitely_dead 0 2024-06-27T21:21:33.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:33.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:33.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 58326 58310 72.881 64.9375 0.0426527 0.013705 2024-06-27T21:21:33.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:33.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:33.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:33.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:21:33.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:33 smithi192 ceph-mon[32956]: pgmap v836: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s; 2746/161931 objects degraded (1.696%); 8576/161931 objects misplaced (5.296%); 13 MiB/s, 211 objects/s recovering 2024-06-27T21:21:33.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:33 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:33.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1840981467' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-27T21:21:33.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:33.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:33 smithi151 ceph-mon[28965]: pgmap v836: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s; 2746/161931 objects degraded (1.696%); 8576/161931 objects misplaced (5.296%); 13 MiB/s, 211 objects/s recovering 2024-06-27T21:21:33.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:33 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:33.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1840981467' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-27T21:21:33.832 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.2. 2024-06-27T21:21:33.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:33.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:33.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:34.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:33 smithi190 ceph-mon[32247]: pgmap v836: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s; 2746/161931 objects degraded (1.696%); 8576/161931 objects misplaced (5.296%); 13 MiB/s, 211 objects/s recovering 2024-06-27T21:21:34.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:33 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:34.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1840981467' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-06-27T21:21:34.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:34.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:34.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:34.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:34.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:34.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:34.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:34.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:34.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:34.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:34.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 58376 58360 71.5133 3.125 0.00273401 0.0137648 2024-06-27T21:21:34.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:34.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:34.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:34.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1840981467' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-27T21:21:34.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:34 smithi192 ceph-mon[32956]: osdmap e314: 12 total, 12 up, 12 in 2024-06-27T21:21:34.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2746/161931 objects degraded (1.696%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:34.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:34 smithi192 ceph-mon[32956]: osdmap e315: 12 total, 12 up, 12 in 2024-06-27T21:21:34.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:34.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:34.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:34.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1840981467' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-27T21:21:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:34 smithi151 ceph-mon[28965]: osdmap e314: 12 total, 12 up, 12 in 2024-06-27T21:21:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2746/161931 objects degraded (1.696%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:34.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:34 smithi151 ceph-mon[28965]: osdmap e315: 12 total, 12 up, 12 in 2024-06-27T21:21:35.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1840981467' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-06-27T21:21:35.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:34 smithi190 ceph-mon[32247]: osdmap e314: 12 total, 12 up, 12 in 2024-06-27T21:21:35.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2746/161931 objects degraded (1.696%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:35.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:34 smithi190 ceph-mon[32247]: osdmap e315: 12 total, 12 up, 12 in 2024-06-27T21:21:35.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:35.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:35.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:35.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:35.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:35.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:35.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:35.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:35.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:35.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:35.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 58532 58516 70.3255 9.75 0.00283183 0.013886 2024-06-27T21:21:35.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:35.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:35.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:35.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:35 smithi192 ceph-mon[32956]: pgmap v838: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 972 op/s; 2746/167403 objects degraded (1.640%); 7906/167403 objects misplaced (4.723%); 15 MiB/s, 244 objects/s recovering 2024-06-27T21:21:35.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:35 smithi192 ceph-mon[32956]: osdmap e316: 12 total, 12 up, 12 in 2024-06-27T21:21:35.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:35.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:35.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:35 smithi190 ceph-mon[32247]: pgmap v838: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 972 op/s; 2746/167403 objects degraded (1.640%); 7906/167403 objects misplaced (4.723%); 15 MiB/s, 244 objects/s recovering 2024-06-27T21:21:35.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:35 smithi190 ceph-mon[32247]: osdmap e316: 12 total, 12 up, 12 in 2024-06-27T21:21:35.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:35.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:35.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:35 smithi151 ceph-mon[28965]: pgmap v838: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 972 op/s; 2746/167403 objects degraded (1.640%); 7906/167403 objects misplaced (4.723%); 15 MiB/s, 244 objects/s recovering 2024-06-27T21:21:35.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:35 smithi151 ceph-mon[28965]: osdmap e316: 12 total, 12 up, 12 in 2024-06-27T21:21:36.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:36.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:36.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:36.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:36.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:36.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:36.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:36.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:36.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:36.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:36.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 59321 59305 69.929 49.3125 0.00413709 0.0142935 2024-06-27T21:21:36.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:36.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:36.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:36.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:36.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:36.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:36.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:37.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:37.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:37.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:37.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:37.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:37.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:37.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:37.448 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:37.468 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-06-27T21:21:37.468 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:37.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:37.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:37.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 15 60282 60267 69.7474 60.125 0.00322966 0.0143318 2024-06-27T21:21:37.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:37.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:37 smithi151 ceph-mon[28965]: pgmap v841: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 3.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 2746/174423 objects degraded (1.574%); 7365/174423 objects misplaced (4.222%); 17 MiB/s, 277 objects/s recovering 2024-06-27T21:21:37.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:37.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:37.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:37 smithi190 ceph-mon[32247]: pgmap v841: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 3.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 2746/174423 objects degraded (1.574%); 7365/174423 objects misplaced (4.222%); 17 MiB/s, 277 objects/s recovering 2024-06-27T21:21:37.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:37 smithi192 ceph-mon[32956]: pgmap v841: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 3.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 2746/174423 objects degraded (1.574%); 7365/174423 objects misplaced (4.222%); 17 MiB/s, 277 objects/s recovering 2024-06-27T21:21:37.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:37.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:37.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:37.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:38.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:38.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:38.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:38.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:38.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:38.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:38.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:38.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:38.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:38.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:38.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 61328 61312 69.6666 65.3125 0.00732931 0.0143477 2024-06-27T21:21:38.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:38.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:38.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:38 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:38.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:38 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:38.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:38 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:38.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:38.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:38.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:38 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:38.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:38 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:38.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:38 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:38.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:38.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:38.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:39.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:38 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:39.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:38 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:39.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:38 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:39.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:39.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:39.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:39.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:39.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:39.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:39.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:39.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:39.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:39.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:39.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 62396 62380 69.6145 66.75 0.0390352 0.0143582 2024-06-27T21:21:39.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:39.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:39.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:39 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:39.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:39 smithi192 ceph-mon[32956]: pgmap v842: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s; 2746/176106 objects degraded (1.559%); 9026/176106 objects misplaced (5.125%); 15 MiB/s, 246 objects/s recovering 2024-06-27T21:21:39.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:39 smithi192 ceph-mon[32956]: osdmap e317: 12 total, 12 up, 12 in 2024-06-27T21:21:39.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:39.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:39.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2746/176106 objects degraded (1.559%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:39.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:39.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:39.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:39.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:39 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:39.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:39 smithi151 ceph-mon[28965]: pgmap v842: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s; 2746/176106 objects degraded (1.559%); 9026/176106 objects misplaced (5.125%); 15 MiB/s, 246 objects/s recovering 2024-06-27T21:21:39.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:39 smithi151 ceph-mon[28965]: osdmap e317: 12 total, 12 up, 12 in 2024-06-27T21:21:39.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:39.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:39.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2746/176106 objects degraded (1.559%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:39.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:39.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:40.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:39 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:39 smithi190 ceph-mon[32247]: pgmap v842: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s; 2746/176106 objects degraded (1.559%); 9026/176106 objects misplaced (5.125%); 15 MiB/s, 246 objects/s recovering 2024-06-27T21:21:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:39 smithi190 ceph-mon[32247]: osdmap e317: 12 total, 12 up, 12 in 2024-06-27T21:21:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2746/176106 objects degraded (1.559%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:40.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:40.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:40.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:40.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:40.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:40.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:40.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:40.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:40.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:40.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:40.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:40.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 63447 63431 69.5455 65.6875 0.0105787 0.0143695 2024-06-27T21:21:40.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:40.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:40.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:40.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:40.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:40 smithi151 ceph-mon[28965]: osdmap e318: 12 total, 12 up, 12 in 2024-06-27T21:21:40.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:40.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:40.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:41.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:40 smithi190 ceph-mon[32247]: osdmap e318: 12 total, 12 up, 12 in 2024-06-27T21:21:41.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:40 smithi192 ceph-mon[32956]: osdmap e318: 12 total, 12 up, 12 in 2024-06-27T21:21:41.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:41.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:41.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:41.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:41.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:41.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:41.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:41.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:41.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:41.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 64318 64302 69.2849 54.4375 0.00403799 0.0144245 2024-06-27T21:21:41.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:41.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:41.754 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:41.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:41.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:41.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:41.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:41 smithi151 ceph-mon[28965]: pgmap v845: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 21 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 2746/180873 objects degraded (1.518%); 8281/180873 objects misplaced (4.578%); 18 MiB/s, 283 objects/s recovering 2024-06-27T21:21:41.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:42.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:41 smithi190 ceph-mon[32247]: pgmap v845: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 21 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 2746/180873 objects degraded (1.518%); 8281/180873 objects misplaced (4.578%); 18 MiB/s, 283 objects/s recovering 2024-06-27T21:21:42.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:42.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:41 smithi192 ceph-mon[32956]: pgmap v845: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 21 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 2746/180873 objects degraded (1.518%); 8281/180873 objects misplaced (4.578%); 18 MiB/s, 283 objects/s recovering 2024-06-27T21:21:42.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:42.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:42.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:42.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:42.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:42.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:42.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:21:42.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:21:42.470 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-06-27T21:21:42.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:21:42.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:42.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:42.603 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 65242 65226 69.0893 57.75 0.107731 0.0144666 2024-06-27T21:21:42.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:42.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:42.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:42.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:42.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:42.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:42 smithi151 ceph-mon[28965]: osdmap e319: 12 total, 12 up, 12 in 2024-06-27T21:21:42.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:21:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:42.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:43.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:43.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:43.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:42 smithi190 ceph-mon[32247]: osdmap e319: 12 total, 12 up, 12 in 2024-06-27T21:21:43.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:21:43.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:42 smithi192 ceph-mon[32956]: osdmap e319: 12 total, 12 up, 12 in 2024-06-27T21:21:43.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:21:43.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:43.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:43.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:43.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:43.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:43.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:43.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:43.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:43.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:43.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:43.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:21:43.606794+0000 min lat: 0.00102932 max lat: 2.15488 avg lat: 0.0145504 2024-06-27T21:21:43.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:21:43.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 65973 65957 68.6992 45.6875 0.0449544 0.0145504 2024-06-27T21:21:43.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:43.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:43.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:43.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:43.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:43.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:43 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:43.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:43 smithi151 ceph-mon[28965]: pgmap v847: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 649 op/s; 2746/186120 objects degraded (1.475%); 10976/186120 objects misplaced (5.897%); 20 MiB/s, 326 objects/s recovering 2024-06-27T21:21:43.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:21:43.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:21:43.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:43.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:44.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:43 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:44.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:43 smithi190 ceph-mon[32247]: pgmap v847: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 649 op/s; 2746/186120 objects degraded (1.475%); 10976/186120 objects misplaced (5.897%); 20 MiB/s, 326 objects/s recovering 2024-06-27T21:21:44.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:21:44.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:21:44.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:21:44.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:43 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:44.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:43 smithi192 ceph-mon[32956]: pgmap v847: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 649 op/s; 2746/186120 objects degraded (1.475%); 10976/186120 objects misplaced (5.897%); 20 MiB/s, 326 objects/s recovering 2024-06-27T21:21:44.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:21:44.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:21:44.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:21:44.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:44.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:44.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:44.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:44.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:44.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:44.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:44.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:44.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:44.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 66822 66806 68.4428 53.0625 0.00390929 0.0145897 2024-06-27T21:21:44.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:44.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:44.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:44.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:44.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:44.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:44 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:44.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2746/186120 objects degraded (1.475%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:44.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:44 smithi151 ceph-mon[28965]: osdmap e320: 12 total, 12 up, 12 in 2024-06-27T21:21:44.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:21:44.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:21:44.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:44.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:44.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:45.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:44 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:45.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2746/186120 objects degraded (1.475%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:45.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:44 smithi190 ceph-mon[32247]: osdmap e320: 12 total, 12 up, 12 in 2024-06-27T21:21:45.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:21:45.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:21:45.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:44 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:45.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2746/186120 objects degraded (1.475%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:21:45.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:44 smithi192 ceph-mon[32956]: osdmap e320: 12 total, 12 up, 12 in 2024-06-27T21:21:45.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:21:45.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:21:45.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:45.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:45.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:45.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:45.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:45.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:45.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:45.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:45.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:45.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 66981 66965 67.4992 9.9375 0.00606829 0.0148099 2024-06-27T21:21:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:45.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:45.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:45.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:45.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:45.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:45.900 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:45 smithi151 ceph-mon[28965]: pgmap v848: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 864 op/s; 2746/191673 objects degraded (1.433%); 10124/191673 objects misplaced (5.282%); 26 MiB/s, 423 objects/s recovering 2024-06-27T21:21:45.901 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:45 smithi151 ceph-mon[28965]: osdmap e321: 12 total, 12 up, 12 in 2024-06-27T21:21:45.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:45.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:45.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:46.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:45 smithi190 ceph-mon[32247]: pgmap v848: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 864 op/s; 2746/191673 objects degraded (1.433%); 10124/191673 objects misplaced (5.282%); 26 MiB/s, 423 objects/s recovering 2024-06-27T21:21:46.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:45 smithi190 ceph-mon[32247]: osdmap e321: 12 total, 12 up, 12 in 2024-06-27T21:21:46.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:45 smithi192 ceph-mon[32956]: pgmap v848: 27 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 864 op/s; 2746/191673 objects degraded (1.433%); 10124/191673 objects misplaced (5.282%); 26 MiB/s, 423 objects/s recovering 2024-06-27T21:21:46.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:45 smithi192 ceph-mon[32956]: osdmap e321: 12 total, 12 up, 12 in 2024-06-27T21:21:46.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:46.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:46.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:46.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:46.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:46.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:46.453 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:21:46.454 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:21:46.454 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:21:46.454 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:21:46.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:46.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:46.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 67931 67915 67.3702 59.375 0.0173682 0.014838 2024-06-27T21:21:46.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:46.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:46.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:46.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:46.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:46 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2746/191673 objects degraded (1.433%), 1 pg degraded) 2024-06-27T21:21:46.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:46.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:46.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:47.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:47.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:46 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2746/191673 objects degraded (1.433%), 1 pg degraded) 2024-06-27T21:21:47.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:47.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:46 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2746/191673 objects degraded (1.433%), 1 pg degraded) 2024-06-27T21:21:47.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:47.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:47.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:47.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:47.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:47.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:47.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:21:47.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:47.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:47.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 68853 68837 67.2178 57.625 0.00844335 0.0148706 2024-06-27T21:21:47.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:47.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:47.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:47 smithi151 ceph-mon[28965]: pgmap v851: 27 pgs: 1 peering, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 11969/199044 objects misplaced (6.013%); 28 MiB/s, 449 objects/s recovering 2024-06-27T21:21:47.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:47 smithi190 ceph-mon[32247]: pgmap v851: 27 pgs: 1 peering, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 11969/199044 objects misplaced (6.013%); 28 MiB/s, 449 objects/s recovering 2024-06-27T21:21:47.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:47.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:47.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:47.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:47.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:48.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:48.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:47 smithi192 ceph-mon[32956]: pgmap v851: 27 pgs: 1 peering, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 11969/199044 objects misplaced (6.013%); 28 MiB/s, 449 objects/s recovering 2024-06-27T21:21:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:48.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:48.163 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:48.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:48.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:48.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:21:48.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:21:48.263 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:21:48.264 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:21:48.298 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:21:48.298 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:21:48.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:48.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:21:48.335 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:21:48.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:48.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:21:48.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:21:48.397 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:21:48.397 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:21:48.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:21:48.432 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:21:48.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:48.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:21:48.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:21:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:48.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:48.509 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:21:48.509 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:21:48.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:21:48.545 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:21:48.580 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:21:48.580 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:21:48.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:48.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 69670 69654 66.9692 51.0625 0.00501047 0.0149266 2024-06-27T21:21:48.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:21:48.614 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:21:48.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:48.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/723183172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:48.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:48.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:48.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:48.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:48.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:48.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:49.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/723183172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:49.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:49.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/723183172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:21:49.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:49.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:49.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:49.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:49.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:49.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:49.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:49.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:49.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 70822 70806 67.0453 72 0.0130261 0.0149101 2024-06-27T21:21:49.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:49.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:49.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:49.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:49.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:49.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:49.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:49.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:49 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:49.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:49 smithi151 ceph-mon[28965]: pgmap v852: 27 pgs: 1 peering, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 775 op/s; 11668/200997 objects misplaced (5.805%); 20 MiB/s, 321 objects/s recovering 2024-06-27T21:21:50.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:49 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:50.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:49 smithi190 ceph-mon[32247]: pgmap v852: 27 pgs: 1 peering, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 775 op/s; 11668/200997 objects misplaced (5.805%); 20 MiB/s, 321 objects/s recovering 2024-06-27T21:21:50.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:50.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:49 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:50.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:49 smithi192 ceph-mon[32956]: pgmap v852: 27 pgs: 1 peering, 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 21 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 775 op/s; 11668/200997 objects misplaced (5.805%); 20 MiB/s, 321 objects/s recovering 2024-06-27T21:21:50.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:50.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:50.185 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:50.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:50.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:50.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:50.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:50.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:50.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:50.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:50.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 16 71510 71494 66.6864 43 0.0195299 0.0149905 2024-06-27T21:21:50.655 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:21:50.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:50.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:50.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:50.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:50.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:50.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:50 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:50.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3487683693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:50.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:50.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:51.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:51.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:50 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:51.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3487683693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:51.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:51.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:50 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:51.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3487683693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:51.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:21:51.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:51.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:51.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:51.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:51.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:51.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:51.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:51.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:51.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 15 72476 72461 66.5944 60.4375 0.0105582 0.0150094 2024-06-27T21:21:51.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:51.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: pgmap v853: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 1 peering, 4 active+recovering+undersized+remapped, 19 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 833 op/s; 10719/206130 objects misplaced (5.200%); 23 MiB/s, 375 objects/s recovering 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: osdmap e322: 12 total, 12 up, 12 in 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:21:51.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:51 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:21:51.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:51.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:51.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:51.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:51.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:52.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: pgmap v853: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 1 peering, 4 active+recovering+undersized+remapped, 19 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 833 op/s; 10719/206130 objects misplaced (5.200%); 23 MiB/s, 375 objects/s recovering 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: osdmap e322: 12 total, 12 up, 12 in 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:21:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:51 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:21:52.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:52.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: pgmap v853: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 1 peering, 4 active+recovering+undersized+remapped, 19 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 833 op/s; 10719/206130 objects misplaced (5.200%); 23 MiB/s, 375 objects/s recovering 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: osdmap e322: 12 total, 12 up, 12 in 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:21:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:51 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:21:52.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:52.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:52.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:52.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:21:52.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:52.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:52.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:52.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:52.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:52.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:52.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 73427 73411 66.4897 59.375 0.0378171 0.0150338 2024-06-27T21:21:52.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:52.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:52.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:52.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:52.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:52.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:52 smithi151 ceph-mon[28965]: 4.8 scrub starts 2024-06-27T21:21:52.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:52.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:53.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:52 smithi190 ceph-mon[32247]: 4.8 scrub starts 2024-06-27T21:21:53.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:52 smithi192 ceph-mon[32956]: 4.8 scrub starts 2024-06-27T21:21:53.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:53.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:53.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:53.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:53.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:53.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:53.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:53.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:53.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:53.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:53.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:53.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 73659 73643 65.747 14.5 0.0138655 0.015079 2024-06-27T21:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:21:53.612 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:21:53.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:53.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:53.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:53.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:53.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:53 smithi151 ceph-mon[28965]: pgmap v855: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 20 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s; 10110/211185 objects misplaced (4.787%); 23 MiB/s, 369 objects/s recovering 2024-06-27T21:21:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:53 smithi151 ceph-mon[28965]: 4.8 scrub ok 2024-06-27T21:21:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:53 smithi151 ceph-mon[28965]: osdmap e323: 12 total, 12 up, 12 in 2024-06-27T21:21:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:53 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:21:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:53 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:21:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:53.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:53.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:53 smithi190 ceph-mon[32247]: pgmap v855: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 20 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s; 10110/211185 objects misplaced (4.787%); 23 MiB/s, 369 objects/s recovering 2024-06-27T21:21:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:53 smithi190 ceph-mon[32247]: 4.8 scrub ok 2024-06-27T21:21:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:53 smithi190 ceph-mon[32247]: osdmap e323: 12 total, 12 up, 12 in 2024-06-27T21:21:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:53 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:21:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:53 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:21:54.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:53 smithi192 ceph-mon[32956]: pgmap v855: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 20 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s; 10110/211185 objects misplaced (4.787%); 23 MiB/s, 369 objects/s recovering 2024-06-27T21:21:54.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:53 smithi192 ceph-mon[32956]: 4.8 scrub ok 2024-06-27T21:21:54.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:53 smithi192 ceph-mon[32956]: osdmap e323: 12 total, 12 up, 12 in 2024-06-27T21:21:54.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:53 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:21:54.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:53 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:21:54.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:54.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:54.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:54.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:54.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:54.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:21:54.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:54.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:54.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:54.491 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":856,"stamp":"2024-06-27T21:21:52.666708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4668391968,"num_objects":71227,"num_object_clones":0,"num_object_copies":213681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9888,"num_objects_unfound":0,"num_objects_dirty":71227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71225,"num_write_kb":4558400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12029,"num_bytes_recovered":788603968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72237,"ondisk_log_size":72237,"up":81,"acting":77,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16825720,"kb_used_data":14261912,"kb_used_omap":18,"kb_used_meta":2563309,"kb_avail":1108116104,"statfs":{"total":1151940427776,"available":1134710890496,"internally_reserved":0,"allocated":14604197888,"data_stored":14601021230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":2624828730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,1,1,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2081,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":325440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2081,"num_bytes_recovered":136380416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.178279"},"pg_stats":[{"pgid":"4.e","version":"321'4345","reported_seq":7562,"reported_epoch":321,"state":"active+recovering+remapped","last_fresh":"2024-06-27T21:21:48.693560+0000","last_change":"2024-06-27T21:21:33.522416+0000","last_active":"2024-06-27T21:21:48.693560+0000","last_peered":"2024-06-27T21:21:48.693560+0000","last_clean":"2024-06-27T21:21:17.589750+0000","last_became_active":"2024-06-27T21:21:19.720628+0000","last_became_peered":"2024-06-27T21:21:19.720628+0000","last_unstale":"2024-06-27T21:21:48.693560+0000","last_undegraded":"2024-06-27T21:21:48.693560+0000","last_fullsized":"2024-06-27T21:21:48.693560+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284753920,"num_objects":4345,"num_object_clones":0,"num_object_copies":13035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1681,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":278080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1555,"num_bytes_recovered":101908480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":2664}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"322'4548","reported_seq":4638,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-06-27T21:21:50.773383+0000","last_change":"2024-06-27T21:20:33.674258+0000","last_active":"2024-06-27T21:21:50.773383+0000","last_peered":"2024-06-27T21:21:50.773383+0000","last_clean":"2024-06-27T21:21:50.773383+0000","last_became_active":"2024-06-27T21:20:33.673842+0000","last_became_peered":"2024-06-27T21:20:33.673842+0000","last_unstale":"2024-06-27T21:21:50.773383+0000","last_undegraded":"2024-06-27T21:21:50.773383+0000","last_fullsized":"2024-06-27T21:21:50.773383+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"321'4520","reported_seq":6840,"reported_epoch":321,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:21:50.381117+0000","last_change":"2024-06-27T21:21:44.050294+0000","last_active":"2024-06-27T21:21:50.381117+0000","last_peered":"2024-06-27T21:21:50.381117+0000","last_clean":"2024-06-27T21:21:17.595411+0000","last_became_active":"2024-06-27T21:21:19.721053+0000","last_became_peered":"2024-06-27T21:21:19.721053+0000","last_unstale":"2024-06-27T21:21:50.381117+0000","last_undegraded":"2024-06-27T21:21:50.381117+0000","last_fullsized":"2024-06-27T21:21:19.602313+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4520,"log_dups_size":0,"ondisk_log_size":4520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2233,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":289280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1109,"num_bytes_recovered":72679424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6","objects":2287},{"shards":"1,6,7","objects":2233}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"321'4421","reported_seq":4509,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:49.520294+0000","last_change":"2024-06-27T21:20:33.673013+0000","last_active":"2024-06-27T21:21:49.520294+0000","last_peered":"2024-06-27T21:21:49.520294+0000","last_clean":"2024-06-27T21:21:49.520294+0000","last_became_active":"2024-06-27T21:20:33.672659+0000","last_became_peered":"2024-06-27T21:20:33.672659+0000","last_unstale":"2024-06-27T21:21:49.520294+0000","last_undegraded":"2024-06-27T21:21:49.520294+0000","last_fullsized":"2024-06-27T21:21:49.520294+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:54.404875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289734656,"num_objects":4421,"num_object_clones":0,"num_object_copies":13263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4421,"num_write_kb":282944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"321'4397","reported_seq":4484,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:49.507044+0000","last_change":"2024-06-27T21:20:33.961841+0000","last_active":"2024-06-27T21:21:49.507044+0000","last_peered":"2024-06-27T21:21:49.507044+0000","last_clean":"2024-06-27T21:21:49.507044+0000","last_became_active":"2024-06-27T21:20:33.961523+0000","last_became_peered":"2024-06-27T21:20:33.961523+0000","last_unstale":"2024-06-27T21:21:49.507044+0000","last_undegraded":"2024-06-27T21:21:49.507044+0000","last_fullsized":"2024-06-27T21:21:49.507044+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288161792,"num_objects":4397,"num_object_clones":0,"num_object_copies":13191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":281408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"321'4426","reported_seq":10940,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:50.277060+0000","last_change":"2024-06-27T21:21:45.495854+0000","last_active":"2024-06-27T21:21:50.277060+0000","last_peered":"2024-06-27T21:21:50.277060+0000","last_clean":"2024-06-27T21:21:50.277060+0000","last_became_active":"2024-06-27T21:21:45.492229+0000","last_became_peered":"2024-06-27T21:21:45.492229+0000","last_unstale":"2024-06-27T21:21:50.277060+0000","last_undegraded":"2024-06-27T21:21:50.277060+0000","last_fullsized":"2024-06-27T21:21:50.277060+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290062336,"num_objects":4426,"num_object_clones":0,"num_object_copies":13278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4426,"num_write_kb":283264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":209846272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":654,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.495081+0000","last_change":"2024-06-27T21:21:13.336248+0000","last_active":"2024-06-27T21:21:45.495081+0000","last_peered":"2024-06-27T21:21:45.495081+0000","last_clean":"2024-06-27T21:21:45.495081+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:21:45.495081+0000","last_undegraded":"2024-06-27T21:21:45.495081+0000","last_fullsized":"2024-06-27T21:21:45.495081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:13.336141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:13.336141+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:13.842380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"321'4458","reported_seq":6170,"reported_epoch":321,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:21:50.393009+0000","last_change":"2024-06-27T21:21:35.679798+0000","last_active":"2024-06-27T21:21:50.393009+0000","last_peered":"2024-06-27T21:21:50.393009+0000","last_clean":"2024-06-27T21:21:33.502606+0000","last_became_active":"2024-06-27T21:21:35.644636+0000","last_became_peered":"2024-06-27T21:21:35.644636+0000","last_unstale":"2024-06-27T21:21:50.393009+0000","last_undegraded":"2024-06-27T21:21:50.393009+0000","last_fullsized":"2024-06-27T21:21:35.482366+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4458,"log_dups_size":0,"ondisk_log_size":4458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292159488,"num_objects":4458,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2832,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4458,"num_write_kb":285312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":810,"num_bytes_recovered":53084160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1],"avail_no_missing":["9","1","7"],"object_location_counts":[{"shards":"1,7,9","objects":2832},{"shards":"1,9","objects":1626}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":239,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.490091+0000","last_change":"2024-06-27T21:20:34.359816+0000","last_active":"2024-06-27T21:21:45.490091+0000","last_peered":"2024-06-27T21:21:45.490091+0000","last_clean":"2024-06-27T21:21:45.490091+0000","last_became_active":"2024-06-27T21:17:58.594952+0000","last_became_peered":"2024-06-27T21:17:58.594952+0000","last_unstale":"2024-06-27T21:21:45.490091+0000","last_undegraded":"2024-06-27T21:21:45.490091+0000","last_fullsized":"2024-06-27T21:21:45.490091+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:34.359736+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:20:34.359736+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:54.904375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"321'4397","reported_seq":10418,"reported_epoch":321,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:21:48.699361+0000","last_change":"2024-06-27T21:21:19.718400+0000","last_active":"2024-06-27T21:21:48.699361+0000","last_peered":"2024-06-27T21:21:48.699361+0000","last_clean":"2024-06-27T21:21:17.587169+0000","last_became_active":"2024-06-27T21:21:19.689946+0000","last_became_peered":"2024-06-27T21:21:19.689946+0000","last_unstale":"2024-06-27T21:21:48.699361+0000","last_undegraded":"2024-06-27T21:21:48.699361+0000","last_fullsized":"2024-06-27T21:21:19.602690+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288161792,"num_objects":4397,"num_object_clones":0,"num_object_copies":13191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":323,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":281408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2960,"num_bytes_recovered":193986560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,0],"avail_no_missing":["1","0","7"],"object_location_counts":[{"shards":"0,1","objects":4074},{"shards":"0,1,7","objects":323}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":659,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.495141+0000","last_change":"2024-06-27T21:20:30.322903+0000","last_active":"2024-06-27T21:21:45.495141+0000","last_peered":"2024-06-27T21:21:45.495141+0000","last_clean":"2024-06-27T21:21:45.495141+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:21:45.495141+0000","last_undegraded":"2024-06-27T21:21:45.495141+0000","last_fullsized":"2024-06-27T21:21:45.495141+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:30.322810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:30.322810+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:56.787406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"322'4576","reported_seq":9467,"reported_epoch":322,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:21:52.314631+0000","last_change":"2024-06-27T21:21:34.626279+0000","last_active":"2024-06-27T21:21:52.314631+0000","last_peered":"2024-06-27T21:21:52.314631+0000","last_clean":"2024-06-27T21:21:17.581403+0000","last_became_active":"2024-06-27T21:21:34.587586+0000","last_became_peered":"2024-06-27T21:21:34.587586+0000","last_unstale":"2024-06-27T21:21:52.314631+0000","last_undegraded":"2024-06-27T21:21:52.314631+0000","last_fullsized":"2024-06-27T21:21:34.482350+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299892736,"num_objects":4576,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2819,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":292864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2391,"num_bytes_recovered":156696576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[3,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3","objects":1757},{"shards":"1,3,7","objects":1756},{"shards":"1,3,7,10","objects":43},{"shards":"1,3,10","objects":1020}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":53,"reported_epoch":321,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:21:48.684987+0000","last_change":"2024-06-27T21:21:48.684153+0000","last_active":"2024-06-27T21:21:48.684987+0000","last_peered":"2024-06-27T21:21:48.684987+0000","last_clean":"2024-06-27T21:21:48.684987+0000","last_became_active":"2024-06-27T21:21:18.602559+0000","last_became_peered":"2024-06-27T21:21:18.602559+0000","last_unstale":"2024-06-27T21:21:48.684987+0000","last_undegraded":"2024-06-27T21:21:48.684987+0000","last_fullsized":"2024-06-27T21:21:48.684987+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:08.502246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:08.502246+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"321'4465","reported_seq":4553,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:50.306924+0000","last_change":"2024-06-27T21:20:33.676832+0000","last_active":"2024-06-27T21:21:50.306924+0000","last_peered":"2024-06-27T21:21:50.306924+0000","last_clean":"2024-06-27T21:21:50.306924+0000","last_became_active":"2024-06-27T21:20:33.676501+0000","last_became_peered":"2024-06-27T21:20:33.676501+0000","last_unstale":"2024-06-27T21:21:50.306924+0000","last_undegraded":"2024-06-27T21:21:50.306924+0000","last_fullsized":"2024-06-27T21:21:50.306924+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292618240,"num_objects":4465,"num_object_clones":0,"num_object_copies":13395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":285760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":388,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.490214+0000","last_change":"2024-06-27T21:21:18.603095+0000","last_active":"2024-06-27T21:21:45.490214+0000","last_peered":"2024-06-27T21:21:45.490214+0000","last_clean":"2024-06-27T21:21:45.490214+0000","last_became_active":"2024-06-27T21:21:18.602789+0000","last_became_peered":"2024-06-27T21:21:18.602789+0000","last_unstale":"2024-06-27T21:21:45.490214+0000","last_undegraded":"2024-06-27T21:21:45.490214+0000","last_fullsized":"2024-06-27T21:21:45.490214+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:32.377138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.377138+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:49.123235+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"321'4412","reported_seq":4499,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:49.521823+0000","last_change":"2024-06-27T21:20:33.961473+0000","last_active":"2024-06-27T21:21:49.521823+0000","last_peered":"2024-06-27T21:21:49.521823+0000","last_clean":"2024-06-27T21:21:49.521823+0000","last_became_active":"2024-06-27T21:20:33.961025+0000","last_became_peered":"2024-06-27T21:20:33.961025+0000","last_unstale":"2024-06-27T21:21:49.521823+0000","last_undegraded":"2024-06-27T21:21:49.521823+0000","last_fullsized":"2024-06-27T21:21:49.521823+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289144832,"num_objects":4412,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":282368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":53,"reported_epoch":321,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:21:47.643416+0000","last_change":"2024-06-27T21:21:47.642499+0000","last_active":"2024-06-27T21:21:47.643416+0000","last_peered":"2024-06-27T21:21:47.643416+0000","last_clean":"2024-06-27T21:21:47.643416+0000","last_became_active":"2024-06-27T21:21:18.602398+0000","last_became_peered":"2024-06-27T21:21:18.602398+0000","last_unstale":"2024-06-27T21:21:47.643416+0000","last_undegraded":"2024-06-27T21:21:47.643416+0000","last_fullsized":"2024-06-27T21:21:47.643416+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:09.515737+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:21:09.515737+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.0","version":"322'4648","reported_seq":4738,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-06-27T21:21:52.272787+0000","last_change":"2024-06-27T21:20:33.961484+0000","last_active":"2024-06-27T21:21:52.272787+0000","last_peered":"2024-06-27T21:21:52.272787+0000","last_clean":"2024-06-27T21:21:52.272787+0000","last_became_active":"2024-06-27T21:20:33.961176+0000","last_became_peered":"2024-06-27T21:20:33.961176+0000","last_unstale":"2024-06-27T21:21:52.272787+0000","last_undegraded":"2024-06-27T21:21:52.272787+0000","last_fullsized":"2024-06-27T21:21:52.272787+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304611328,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":297472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":49,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.491653+0000","last_change":"2024-06-27T21:21:34.494521+0000","last_active":"2024-06-27T21:21:45.491653+0000","last_peered":"2024-06-27T21:21:45.491653+0000","last_clean":"2024-06-27T21:21:45.491653+0000","last_became_active":"2024-06-27T21:21:34.494291+0000","last_became_peered":"2024-06-27T21:21:34.494291+0000","last_unstale":"2024-06-27T21:21:45.491653+0000","last_undegraded":"2024-06-27T21:21:45.491653+0000","last_fullsized":"2024-06-27T21:21:45.491653+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:21:18.604618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:33.574220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1","version":"321'4370","reported_seq":4457,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:49.463278+0000","last_change":"2024-06-27T21:20:33.676743+0000","last_active":"2024-06-27T21:21:49.463278+0000","last_peered":"2024-06-27T21:21:49.463278+0000","last_clean":"2024-06-27T21:21:49.463278+0000","last_became_active":"2024-06-27T21:20:33.676398+0000","last_became_peered":"2024-06-27T21:20:33.676398+0000","last_unstale":"2024-06-27T21:21:49.463278+0000","last_undegraded":"2024-06-27T21:21:49.463278+0000","last_fullsized":"2024-06-27T21:21:49.463278+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:00.774324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286392320,"num_objects":4370,"num_object_clones":0,"num_object_copies":13110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":279680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":463,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.489535+0000","last_change":"2024-06-27T21:21:11.523707+0000","last_active":"2024-06-27T21:21:45.489535+0000","last_peered":"2024-06-27T21:21:45.489535+0000","last_clean":"2024-06-27T21:21:45.489535+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:21:45.489535+0000","last_undegraded":"2024-06-27T21:21:45.489535+0000","last_fullsized":"2024-06-27T21:21:45.489535+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:11.523606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:11.523606+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:24.964399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"321'4305","reported_seq":4392,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:48.622882+0000","last_change":"2024-06-27T21:20:33.674679+0000","last_active":"2024-06-27T21:21:48.622882+0000","last_peered":"2024-06-27T21:21:48.622882+0000","last_clean":"2024-06-27T21:21:48.622882+0000","last_became_active":"2024-06-27T21:20:33.674388+0000","last_became_peered":"2024-06-27T21:20:33.674388+0000","last_unstale":"2024-06-27T21:21:48.622882+0000","last_undegraded":"2024-06-27T21:21:48.622882+0000","last_fullsized":"2024-06-27T21:21:48.622882+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:58.600518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282132480,"num_objects":4305,"num_object_clones":0,"num_object_copies":12915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":275520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":367,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.514948+0000","last_change":"2024-06-27T21:20:31.763820+0000","last_active":"2024-06-27T21:21:45.514948+0000","last_peered":"2024-06-27T21:21:45.514948+0000","last_clean":"2024-06-27T21:21:45.514948+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:21:45.514948+0000","last_undegraded":"2024-06-27T21:21:45.514948+0000","last_fullsized":"2024-06-27T21:21:45.514948+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.763737+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.763737+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:53.584279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"322'4478","reported_seq":4573,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-06-27T21:21:52.272463+0000","last_change":"2024-06-27T21:21:02.542049+0000","last_active":"2024-06-27T21:21:52.272463+0000","last_peered":"2024-06-27T21:21:52.272463+0000","last_clean":"2024-06-27T21:21:52.272463+0000","last_became_active":"2024-06-27T21:21:02.540603+0000","last_became_peered":"2024-06-27T21:21:02.540603+0000","last_unstale":"2024-06-27T21:21:52.272463+0000","last_undegraded":"2024-06-27T21:21:52.272463+0000","last_fullsized":"2024-06-27T21:21:52.272463+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4478,"log_dups_size":0,"ondisk_log_size":4478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293470208,"num_objects":4478,"num_object_clones":0,"num_object_copies":13434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4478,"num_write_kb":286592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":671,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:45.495044+0000","last_change":"2024-06-27T21:21:34.486079+0000","last_active":"2024-06-27T21:21:45.495044+0000","last_peered":"2024-06-27T21:21:45.495044+0000","last_clean":"2024-06-27T21:21:45.495044+0000","last_became_active":"2024-06-27T21:21:34.485764+0000","last_became_peered":"2024-06-27T21:21:34.485764+0000","last_unstale":"2024-06-27T21:21:45.495044+0000","last_undegraded":"2024-06-27T21:21:45.495044+0000","last_fullsized":"2024-06-27T21:21:45.495044+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.336705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.336705+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:49.305396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"321'4459","reported_seq":4551,"reported_epoch":321,"state":"active+clean","last_fresh":"2024-06-27T21:21:50.305655+0000","last_change":"2024-06-27T21:21:02.542379+0000","last_active":"2024-06-27T21:21:50.305655+0000","last_peered":"2024-06-27T21:21:50.305655+0000","last_clean":"2024-06-27T21:21:50.305655+0000","last_became_active":"2024-06-27T21:21:02.541097+0000","last_became_peered":"2024-06-27T21:21:02.541097+0000","last_unstale":"2024-06-27T21:21:50.305655+0000","last_undegraded":"2024-06-27T21:21:50.305655+0000","last_fullsized":"2024-06-27T21:21:50.305655+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":4459,"log_dups_size":0,"ondisk_log_size":4459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:21:53.754646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4459,"num_write_kb":285376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":395,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-06-27T21:21:50.811057+0000","last_change":"2024-06-27T21:21:50.811098+0000","last_active":"2024-06-27T21:21:50.811057+0000","last_peered":"2024-06-27T21:21:50.811057+0000","last_clean":"2024-06-27T21:21:50.811057+0000","last_became_active":"2024-06-27T21:21:34.487122+0000","last_became_peered":"2024-06-27T21:21:34.487122+0000","last_unstale":"2024-06-27T21:21:50.811057+0000","last_undegraded":"2024-06-27T21:21:50.811057+0000","last_fullsized":"2024-06-27T21:21:50.811057+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:50.810976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:50.810976+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:11.015602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4667801600,"num_objects":71225,"num_object_clones":0,"num_object_copies":213675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9888,"num_objects_unfound":0,"num_objects_dirty":71225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71225,"num_write_kb":4558400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12027,"num_bytes_recovered":788201472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14459076608,"data_stored":14459076608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71225,"ondisk_log_size":71225,"up":48,"acting":44,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253014,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2268600,"kb_used_data":1966704,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91476552,"statfs":{"total":95995035648,"available":93671989248,"internally_reserved":0,"allocated":2013904896,"data_stored":2013641045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514651,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1103224,"kb_used_data":963760,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92641928,"statfs":{"total":95995035648,"available":94865334272,"internally_reserved":0,"allocated":986890240,"data_stored":986626389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743583,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033464,"kb_used_data":849584,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92711688,"statfs":{"total":95995035648,"available":94936768512,"internally_reserved":0,"allocated":869974016,"data_stored":869710165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":188283328},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180744,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564408,"statfs":{"total":95995035648,"available":95809953792,"internally_reserved":0,"allocated":1032192,"data_stored":768411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1854008,"kb_used_data":1597936,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91891144,"statfs":{"total":95995035648,"available":94096531456,"internally_reserved":0,"allocated":1636286464,"data_stored":1636022613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":262207946},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528493,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1937080,"kb_used_data":1685168,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91808072,"statfs":{"total":95995035648,"available":94011465728,"internally_reserved":0,"allocated":1725612032,"data_stored":1725348181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757427,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697216,"kb_used_data":579960,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93047936,"statfs":{"total":95995035648,"available":95281086464,"internally_reserved":0,"allocated":593879040,"data_stored":593611707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986359,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1365944,"kb_used_data":1100144,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92379208,"statfs":{"total":95995035648,"available":94596308992,"internally_reserved":0,"allocated":1126547456,"data_stored":1126283605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182588,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1715392,"kb_used_data":1445432,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92029760,"statfs":{"total":95995035648,"available":94238474240,"internally_reserved":0,"allocated":1480122368,"data_stored":1479855035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":544568,"kb_used_data":436856,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93200584,"statfs":{"total":95995035648,"available":95437398016,"internally_reserved":0,"allocated":447340544,"data_stored":447073141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3039792,"kb_used_data":2769648,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90705360,"statfs":{"total":95995035648,"available":92882288640,"internally_reserved":0,"allocated":2836119552,"data_stored":2835855701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1085688,"kb_used_data":865712,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92659464,"statfs":{"total":95995035648,"available":94883291136,"internally_reserved":0,"allocated":886489088,"data_stored":886225237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":893255680,"data_stored":893255680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2863857664,"data_stored":2863857664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180027392,"data_stored":180027392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1494745088,"data_stored":1494745088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1128333312,"data_stored":1128333312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":599064576,"data_stored":599064576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1744830464,"data_stored":1744830464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1644232704,"data_stored":1644232704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":877068288,"data_stored":877068288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1016528896,"data_stored":1016528896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2017132544,"data_stored":2017132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:21:54.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-06-27T21:21:54.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:54.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 74513 74497 65.5727 53.375 0.0407358 0.015244 2024-06-27T21:21:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:54.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:54.739 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:54.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:54.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:54.795 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:21:54.795 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:21:54.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:54.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:54 smithi151 ceph-mon[28965]: 4.3 scrub starts 2024-06-27T21:21:54.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:54 smithi151 ceph-mon[28965]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:21:54.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:54 smithi151 ceph-mon[28965]: osdmap e324: 12 total, 12 up, 12 in 2024-06-27T21:21:54.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:54 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:21:54.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:54 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:21:54.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:54 smithi151 ceph-mon[28965]: 4.b scrub starts 2024-06-27T21:21:54.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:54.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:55.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:54 smithi190 ceph-mon[32247]: 4.3 scrub starts 2024-06-27T21:21:55.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:54 smithi190 ceph-mon[32247]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:21:55.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:54 smithi190 ceph-mon[32247]: osdmap e324: 12 total, 12 up, 12 in 2024-06-27T21:21:55.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:54 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:21:55.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:54 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:21:55.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:54 smithi190 ceph-mon[32247]: 4.b scrub starts 2024-06-27T21:21:55.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:55.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:54 smithi192 ceph-mon[32956]: 4.3 scrub starts 2024-06-27T21:21:55.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:54 smithi192 ceph-mon[32956]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:21:55.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:54 smithi192 ceph-mon[32956]: osdmap e324: 12 total, 12 up, 12 in 2024-06-27T21:21:55.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:54 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:21:55.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:54 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:21:55.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:54 smithi192 ceph-mon[32956]: 4.b scrub starts 2024-06-27T21:21:55.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:55.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:55.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:55.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:55.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:55.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:55.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:55.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:55.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:55.604 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 75347 75331 65.3859 52.125 0.0695894 0.0152876 2024-06-27T21:21:55.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:55.707 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:55.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:55.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:21:55.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:55.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:55.943 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:55 smithi192 ceph-mon[32956]: pgmap v858: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 21 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 828 op/s; 9238/215904 objects misplaced (4.279%); 25 MiB/s, 404 objects/s recovering 2024-06-27T21:21:55.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:55 smithi192 ceph-mon[32956]: 4.9 scrub starts 2024-06-27T21:21:55.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:55 smithi192 ceph-mon[32956]: 4.3 scrub ok 2024-06-27T21:21:55.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:55 smithi192 ceph-mon[32956]: 4.b scrub ok 2024-06-27T21:21:55.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:55 smithi151 ceph-mon[28965]: pgmap v858: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 21 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 828 op/s; 9238/215904 objects misplaced (4.279%); 25 MiB/s, 404 objects/s recovering 2024-06-27T21:21:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:55 smithi151 ceph-mon[28965]: 4.9 scrub starts 2024-06-27T21:21:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:55 smithi151 ceph-mon[28965]: 4.3 scrub ok 2024-06-27T21:21:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:55 smithi151 ceph-mon[28965]: 4.b scrub ok 2024-06-27T21:21:55.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:21:55.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:55.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:21:56.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:55 smithi190 ceph-mon[32247]: pgmap v858: 27 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 21 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 828 op/s; 9238/215904 objects misplaced (4.279%); 25 MiB/s, 404 objects/s recovering 2024-06-27T21:21:56.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:55 smithi190 ceph-mon[32247]: 4.9 scrub starts 2024-06-27T21:21:56.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:55 smithi190 ceph-mon[32247]: 4.3 scrub ok 2024-06-27T21:21:56.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:55 smithi190 ceph-mon[32247]: 4.b scrub ok 2024-06-27T21:21:56.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:56.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:56.191 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:56.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:21:56.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:21:56.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:56.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:56.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:56.508 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:56.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 15 76249 76234 65.2632 56.4375 0.014349 0.0153172 2024-06-27T21:21:56.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:56.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:56.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:56.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:56.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:21:56.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:56.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:21:57.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:57.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:21:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:57.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:21:57.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:57.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:57.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:21:57.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:21:57.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:57.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:57.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 77397 77381 65.35 71.6875 0.0171647 0.0152964 2024-06-27T21:21:57.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:57.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:57.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:57.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:57.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:21:57.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:57 smithi151 ceph-mon[28965]: pgmap v859: 27 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s; 8511/223029 objects misplaced (3.816%); 23 MiB/s, 367 objects/s recovering 2024-06-27T21:21:57.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:57 smithi151 ceph-mon[28965]: 4.9 scrub ok 2024-06-27T21:21:57.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:57.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:21:57.999 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:57 smithi192 ceph-mon[32956]: pgmap v859: 27 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s; 8511/223029 objects misplaced (3.816%); 23 MiB/s, 367 objects/s recovering 2024-06-27T21:21:57.999 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:57 smithi192 ceph-mon[32956]: 4.9 scrub ok 2024-06-27T21:21:58.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:57 smithi190 ceph-mon[32247]: pgmap v859: 27 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s; 8511/223029 objects misplaced (3.816%); 23 MiB/s, 367 objects/s recovering 2024-06-27T21:21:58.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:57 smithi190 ceph-mon[32247]: 4.9 scrub ok 2024-06-27T21:21:58.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:58.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:58.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:58.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:58.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:58.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:58.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:58.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:58.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:58.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 78402 78386 65.3161 62.8125 0.0345713 0.0153046 2024-06-27T21:21:58.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:58.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:58.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:58.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:21:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:58.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:58.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:21:58.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:59.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:59.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:59.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:59.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:59.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:59.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:21:59.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:21:59.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:21:59.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:21:59.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:21:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:21:59.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 79400 79384 65.2773 62.375 0.00837206 0.0153136 2024-06-27T21:21:59.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:21:59.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:21:59.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:21:59.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:21:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:21:59.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:21:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:21:59.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:21:59.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:21:59.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:21:59 smithi151 ceph-mon[28965]: pgmap v860: 27 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 677 op/s; 8256/225093 objects misplaced (3.668%); 16 MiB/s, 253 objects/s recovering 2024-06-27T21:22:00.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:21:59 smithi190 ceph-mon[32247]: pgmap v860: 27 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 677 op/s; 8256/225093 objects misplaced (3.668%); 16 MiB/s, 253 objects/s recovering 2024-06-27T21:22:00.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:00.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:21:59 smithi192 ceph-mon[32956]: pgmap v860: 27 pgs: 2 active+clean+scrubbing, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 677 op/s; 8256/225093 objects misplaced (3.668%); 16 MiB/s, 253 objects/s recovering 2024-06-27T21:22:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:00.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:00.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:00.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:00.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:00.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:00.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:00.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:00.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:00.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 80428 80412 65.2639 64.25 0.00829905 0.0153157 2024-06-27T21:22:00.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:00.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:00.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:00.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:00.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:00.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:00.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:00.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:00 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:22:00.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:00 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:22:01.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:00 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:22:01.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:00 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:22:01.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:00 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:22:01.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:00 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:22:01.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:01.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:01.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:01.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:01.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:01.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:01.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:01.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:01.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:01.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 81207 81191 65.0513 48.6875 0.0168397 0.0153645 2024-06-27T21:22:01.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:01.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:01.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:01.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:01.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:01.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:01.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:02.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:01 smithi190 ceph-mon[32247]: pgmap v861: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 849 op/s; 7869/231576 objects misplaced (3.398%); 18 MiB/s, 280 objects/s recovering 2024-06-27T21:22:02.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:01 smithi192 ceph-mon[32956]: pgmap v861: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 849 op/s; 7869/231576 objects misplaced (3.398%); 18 MiB/s, 280 objects/s recovering 2024-06-27T21:22:02.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:01 smithi151 ceph-mon[28965]: pgmap v861: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 849 op/s; 7869/231576 objects misplaced (3.398%); 18 MiB/s, 280 objects/s recovering 2024-06-27T21:22:02.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:02.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:02.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:02.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:02.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:02.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:02.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:02.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 82256 82240 65.0578 65.5625 0.00607595 0.0153651 2024-06-27T21:22:02.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:02.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.638 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:02.706 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:22:02.707 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 4 pgp_num to 16 2024-06-27T21:22:02.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:02.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:02.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:02.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:02.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:02.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:02.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3600213999' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:22:02.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:02 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:22:02.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1860101622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3600213999' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:22:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:02 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:22:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1860101622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:03.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3600213999' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:22:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:02 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:22:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1860101622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:03.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:03.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:03.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:03.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:03.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:03.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:03.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:03.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:03.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:03.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:22:03.608384+0000 min lat: 0.00102932 max lat: 2.15488 avg lat: 0.0153448 2024-06-27T21:22:03.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:22:03.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 83406 83390 65.1429 71.875 0.0687736 0.0153448 2024-06-27T21:22:03.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:03.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:03.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:03.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:03.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: pgmap v862: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s; 7150/237330 objects misplaced (3.013%); 18 MiB/s, 294 objects/s recovering 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1860101622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:22:03.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:03 smithi151 ceph-mon[28965]: osdmap e325: 12 total, 12 up, 12 in 2024-06-27T21:22:03.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:03.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: pgmap v862: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s; 7150/237330 objects misplaced (3.013%); 18 MiB/s, 294 objects/s recovering 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1860101622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:22:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:03 smithi190 ceph-mon[32247]: osdmap e325: 12 total, 12 up, 12 in 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: pgmap v862: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s; 7150/237330 objects misplaced (3.013%); 18 MiB/s, 294 objects/s recovering 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1860101622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:22:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:03 smithi192 ceph-mon[32956]: osdmap e325: 12 total, 12 up, 12 in 2024-06-27T21:22:04.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:04.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:04.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:04.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:04.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:04.343 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:04.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:04.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:04.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:04.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:04.605 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 84280 84264 65.013 54.625 0.0116874 0.0153768 2024-06-27T21:22:04.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:04.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:04.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:04.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:04.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:04.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:05.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:05.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:05.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:05.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:05.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:05.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:05.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:05.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:05.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:05.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:05.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 85221 85205 64.9373 58.8125 0.0472987 0.0153932 2024-06-27T21:22:05.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:05.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:05.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:05.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:05.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:05.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:05.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:06.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:05 smithi190 ceph-mon[32247]: pgmap v864: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 900 op/s; 6390/242940 objects misplaced (2.630%); 18 MiB/s, 284 objects/s recovering 2024-06-27T21:22:06.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:06.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:05 smithi192 ceph-mon[32956]: pgmap v864: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 900 op/s; 6390/242940 objects misplaced (2.630%); 18 MiB/s, 284 objects/s recovering 2024-06-27T21:22:06.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:06.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:06.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:05 smithi151 ceph-mon[28965]: pgmap v864: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 900 op/s; 6390/242940 objects misplaced (2.630%); 18 MiB/s, 284 objects/s recovering 2024-06-27T21:22:06.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:06.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:06.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:06.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:06.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:06.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:06.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 86133 86117 64.8416 57 0.00768427 0.0154155 2024-06-27T21:22:06.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:06.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:06.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:06.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.815 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:06.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:06.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:06.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:07.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:06 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:07.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:06 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:07.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:06 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:07.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:06 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:07.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:07.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.066 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:07.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:07.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:07.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:06 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:07.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:06 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:07.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:07.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:07.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:07.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:07.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:07.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:07.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 86991 86975 64.708 53.625 0.1556 0.0154487 2024-06-27T21:22:07.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:07.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:07.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:07.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:07.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:07.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:07.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:08.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:07 smithi190 ceph-mon[32247]: pgmap v865: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s; 5507/251826 objects misplaced (2.187%); 19 MiB/s, 300 objects/s recovering 2024-06-27T21:22:08.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:07 smithi190 ceph-mon[32247]: osdmap e326: 12 total, 12 up, 12 in 2024-06-27T21:22:08.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:07 smithi192 ceph-mon[32956]: pgmap v865: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s; 5507/251826 objects misplaced (2.187%); 19 MiB/s, 300 objects/s recovering 2024-06-27T21:22:08.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:07 smithi192 ceph-mon[32956]: osdmap e326: 12 total, 12 up, 12 in 2024-06-27T21:22:08.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:08.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:08.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:07 smithi151 ceph-mon[28965]: pgmap v865: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s; 5507/251826 objects misplaced (2.187%); 19 MiB/s, 300 objects/s recovering 2024-06-27T21:22:08.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:07 smithi151 ceph-mon[28965]: osdmap e326: 12 total, 12 up, 12 in 2024-06-27T21:22:08.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:08.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:08.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:08.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:08.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:08.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:08.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:08.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:08.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 87784 87768 64.5298 49.5625 0.0536541 0.0154898 2024-06-27T21:22:08.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:08.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:08.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:08.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:08.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:08.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:08.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:08.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:08 smithi151 ceph-mon[28965]: osdmap e327: 12 total, 12 up, 12 in 2024-06-27T21:22:08.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:08.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:08.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:09.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:09.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:09.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:08 smithi190 ceph-mon[32247]: osdmap e327: 12 total, 12 up, 12 in 2024-06-27T21:22:09.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:09.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:09.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:08 smithi192 ceph-mon[32956]: osdmap e327: 12 total, 12 up, 12 in 2024-06-27T21:22:09.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:09.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:09.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:09.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:09.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:09.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:09.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:09.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:09.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:09.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 88500 88484 64.2998 44.75 0.0218704 0.0155345 2024-06-27T21:22:09.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:09.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:09.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:09.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:09.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:09.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:10.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:09 smithi190 ceph-mon[32247]: pgmap v868: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 962 op/s; 5237/254664 objects misplaced (2.056%); 20 MiB/s, 318 objects/s recovering 2024-06-27T21:22:10.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:09 smithi190 ceph-mon[32247]: osdmap e328: 12 total, 12 up, 12 in 2024-06-27T21:22:10.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:09 smithi192 ceph-mon[32956]: pgmap v868: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 962 op/s; 5237/254664 objects misplaced (2.056%); 20 MiB/s, 318 objects/s recovering 2024-06-27T21:22:10.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:09 smithi192 ceph-mon[32956]: osdmap e328: 12 total, 12 up, 12 in 2024-06-27T21:22:10.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:10.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:09 smithi151 ceph-mon[28965]: pgmap v868: 27 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 23 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 962 op/s; 5237/254664 objects misplaced (2.056%); 20 MiB/s, 318 objects/s recovering 2024-06-27T21:22:10.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:09 smithi151 ceph-mon[28965]: osdmap e328: 12 total, 12 up, 12 in 2024-06-27T21:22:10.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:10.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:10.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:10.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:10.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:10.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:10.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:10.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:10.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 88695 88679 63.7008 12.1875 0.0101137 0.0155805 2024-06-27T21:22:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:10.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:10.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:10.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:10.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:10.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:10.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:10.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:11.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:11.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:11.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:10 smithi190 ceph-mon[32247]: osdmap e329: 12 total, 12 up, 12 in 2024-06-27T21:22:11.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:11.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:11.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:11.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:10 smithi192 ceph-mon[32956]: osdmap e329: 12 total, 12 up, 12 in 2024-06-27T21:22:11.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:11.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:11.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:11.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:11.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:10 smithi151 ceph-mon[28965]: osdmap e329: 12 total, 12 up, 12 in 2024-06-27T21:22:11.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:11.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:11.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:11.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:11.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:11.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:11.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 89611 89595 63.6274 57.25 0.0772921 0.0156978 2024-06-27T21:22:11.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:11.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:11.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:11.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:11.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:11.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:11.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:12.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:11 smithi190 ceph-mon[32247]: pgmap v870: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 975 op/s; 4779/260502 objects misplaced (1.835%); 17 MiB/s, 268 objects/s recovering 2024-06-27T21:22:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:11 smithi190 ceph-mon[32247]: 4.1 scrub starts 2024-06-27T21:22:12.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:12.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:11 smithi192 ceph-mon[32956]: pgmap v870: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 975 op/s; 4779/260502 objects misplaced (1.835%); 17 MiB/s, 268 objects/s recovering 2024-06-27T21:22:12.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:11 smithi192 ceph-mon[32956]: 4.1 scrub starts 2024-06-27T21:22:12.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:12.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:12.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:11 smithi151 ceph-mon[28965]: pgmap v870: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 975 op/s; 4779/260502 objects misplaced (1.835%); 17 MiB/s, 268 objects/s recovering 2024-06-27T21:22:12.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:11 smithi151 ceph-mon[28965]: 4.1 scrub starts 2024-06-27T21:22:12.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:12.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:12.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:12.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:12.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:12.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:12.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 16 89927 89911 63.1344 19.75 0.100566 0.0157384 2024-06-27T21:22:12.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:12.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:12.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:12.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:12.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:12.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:12.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:13.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:13.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:12 smithi190 ceph-mon[32247]: osdmap e330: 12 total, 12 up, 12 in 2024-06-27T21:22:13.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:13.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:12 smithi192 ceph-mon[32956]: osdmap e330: 12 total, 12 up, 12 in 2024-06-27T21:22:13.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:13.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:13.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:12 smithi151 ceph-mon[28965]: osdmap e330: 12 total, 12 up, 12 in 2024-06-27T21:22:13.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:13.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:13.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:13.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:13.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:13.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:13.606 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 16 90736 90720 62.9946 50.5625 0.0158465 0.0158479 2024-06-27T21:22:13.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:13.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:13.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:13.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:13.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:13.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:13.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:13 smithi190 ceph-mon[32247]: pgmap v873: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 773 op/s; 3689/264492 objects misplaced (1.395%); 23 MiB/s, 365 objects/s recovering 2024-06-27T21:22:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:13 smithi190 ceph-mon[32247]: osdmap e331: 12 total, 12 up, 12 in 2024-06-27T21:22:14.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:14.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:14.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:13 smithi192 ceph-mon[32956]: pgmap v873: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 773 op/s; 3689/264492 objects misplaced (1.395%); 23 MiB/s, 365 objects/s recovering 2024-06-27T21:22:14.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:13 smithi192 ceph-mon[32956]: osdmap e331: 12 total, 12 up, 12 in 2024-06-27T21:22:14.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:14.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:14.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:13 smithi151 ceph-mon[28965]: pgmap v873: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 773 op/s; 3689/264492 objects misplaced (1.395%); 23 MiB/s, 365 objects/s recovering 2024-06-27T21:22:14.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:13 smithi151 ceph-mon[28965]: osdmap e331: 12 total, 12 up, 12 in 2024-06-27T21:22:14.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:14.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:14.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:14.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:14.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:14.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.7081 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 90736 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 62.5193 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 27.5508 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 124.062 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 0 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1000 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 440.813 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 1985 2024-06-27T21:22:14.607 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 0 2024-06-27T21:22:14.608 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0159128 2024-06-27T21:22:14.608 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0535945 2024-06-27T21:22:14.608 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 2.15488 2024-06-27T21:22:14.608 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00102932 2024-06-27T21:22:14.611 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:22:14.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:14.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:14.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:14.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:14.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:14.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:14.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:15.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 4.1 scrub ok 2024-06-27T21:22:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 4.a scrub starts 2024-06-27T21:22:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 4.e scrub starts 2024-06-27T21:22:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 4.a scrub ok 2024-06-27T21:22:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:22:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:22:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: 4.e scrub ok 2024-06-27T21:22:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3559101194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:15.055 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:15.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 4.1 scrub ok 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 4.a scrub starts 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 4.e scrub starts 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 4.a scrub ok 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: 4.e scrub ok 2024-06-27T21:22:15.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3559101194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:15.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:15.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 4.1 scrub ok 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 4.a scrub starts 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 4.e scrub starts 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 4.a scrub ok 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: 4.e scrub ok 2024-06-27T21:22:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3559101194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:15.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:15.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:15.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:15.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:15.452 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:22:15.452 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:22:15.486 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:22:15.486 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:22:15.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:22:15.527 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:22:15.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24508.0:121562 2.c e143/104 2:356cce12:::benchmark_data_smithi192_59818_object29487:head v 143'7617, mlcod=142'7615)", 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:13:47.763576+0000", 2024-06-27T21:22:15.542 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 507.77701357799998, 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.304788487, 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:47.763576+0000", 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:47.763576+0000", 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:47.763576+0000", 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8799999999999998e-07 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:47.763582+0000", 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.993e-06 2024-06-27T21:22:15.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:48.065385+0000", 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30180293000000002 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:48.065392+0000", 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9450000000000004e-06 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:48.065460+0000", 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7948999999999996e-05 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:48.065482+0000", 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1430999999999999e-05 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:48.068327+0000", 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028450279999999999 2024-06-27T21:22:15.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:13:48.068364+0000", 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7823e-05 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24508.0:158556 2.8 2:17998c88:::benchmark_data_smithi192_59818_object66481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:14:41.592527+0000", 2024-06-27T21:22:15.547 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 453.94806265400001, 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22736895100000001, 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24508", 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158556 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.592527+0000", 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.592527+0000", 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.592528+0000", 2024-06-27T21:22:15.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4890000000000001e-06 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.592533+0000", 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9289999999999997e-06 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.592543+0000", 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4599999999999992e-06 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.592551+0000", 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7490000000000004e-06 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.593406+0000", 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00085491500000000004 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.593529+0000", 2024-06-27T21:22:15.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122982 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.627491+0000", 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.033961956000000001 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.627597+0000", 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106038 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,8,9", 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.627726+0000", 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012879300000000001 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.642907+0000", 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015181049 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.643996+0000", 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010886909999999999 2024-06-27T21:22:15.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.645515+0000", 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015193590000000001 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.818844+0000", 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17332890300000001 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.818889+0000", 2024-06-27T21:22:15.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5012999999999997e-05 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:14:41.819896+0000", 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010066249999999999 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:6007 3.1 3:8955d51e:::benchmark_data_smithi192_74314_object6006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:18.168444+0000", 2024-06-27T21:22:15.555 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 357.37214517799998, 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21456740899999999, 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6007 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168444+0000", 2024-06-27T21:22:15.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168444+0000", 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168446+0000", 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3400000000000001e-06 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168507+0000", 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1656999999999996e-05 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168518+0000", 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0944000000000001e-05 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168525+0000", 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9149999999999999e-06 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168576+0000", 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1243999999999999e-05 2024-06-27T21:22:15.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168661+0000", 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4736999999999999e-05 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.168764+0000", 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103038 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.169293+0000", 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052916499999999995 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.169822+0000", 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052837200000000004 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.382897+0000", 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21307504199999999 2024-06-27T21:22:15.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.382954+0000", 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7617000000000001e-05 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:18.383012+0000", 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7337999999999999e-05 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:37692 3.6 3:644ce0da:::benchmark_data_smithi192_74314_object37691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:43.459371+0000", 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 332.081218874, 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21843857799999999, 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.563 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 37692 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459371+0000", 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459371+0000", 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459372+0000", 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.879999999999999e-07 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459399+0000", 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7328e-05 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459410+0000", 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1253999999999999e-05 2024-06-27T21:22:15.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459415+0000", 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.011e-06 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459452+0000", 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6344000000000003e-05 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459540+0000", 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8430000000000005e-05 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.459635+0000", 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4891999999999999e-05 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.460693+0000", 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010580800000000001 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.481894+0000", 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021201212000000001 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677690+0000", 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19579606499999999 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677755+0000", 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4838999999999994e-05 2024-06-27T21:22:15.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677809+0000", 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4135000000000003e-05 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.15132.0:37692 3.6 e200/186 ondisk, result = 0)", 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:43.460931+0000", 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 332.07965867299998, 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216884829, 2024-06-27T21:22:15.569 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.460931+0000", 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.460931+0000", 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.460931+0000", 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5300000000000004e-07 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.460935+0000", 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5870000000000001e-06 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677596+0000", 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216661299 2024-06-27T21:22:15.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677603+0000", 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0419999999999997e-06 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677670+0000", 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7007000000000004e-05 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677678+0000", 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.824e-06 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:43.677816+0000", 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000137417 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.15132.0:41292 3.b e202/186 3:d0ac0269:::benchmark_data_smithi192_74314_object41291:head v 202'2586, mlcod=202'2582)", 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:46.478937+0000", 2024-06-27T21:22:15.573 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.06165274699998, 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23200026300000001, 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478937+0000", 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478937+0000", 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478937+0000", 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.478961+0000", 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3608000000000002e-05 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.709294+0000", 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23033325900000001 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.709304+0000", 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4669999999999993e-06 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.709373+0000", 2024-06-27T21:22:15.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8997000000000004e-05 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.709399+0000", 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6560999999999999e-05 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.710886+0000", 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014868500000000001 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.710937+0000", 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0741000000000001e-05 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:41299 3.1 3:8f342276:::benchmark_data_smithi192_74314_object41298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:46.480825+0000", 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.05976454400002, 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.233950821, 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41299 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480825+0000", 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480825+0000", 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480827+0000", 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.232e-06 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480884+0000", 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6453999999999999e-05 2024-06-27T21:22:15.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480899+0000", 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5441999999999999e-05 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480911+0000", 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2099999999999999e-05 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.480983+0000", 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1391999999999997e-05 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.481113+0000", 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013090500000000001 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.481263+0000", 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014961 2024-06-27T21:22:15.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.482618+0000", 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001354629 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.483187+0000", 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056913999999999999 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714706+0000", 2024-06-27T21:22:15.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23151885 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714747+0000", 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0948999999999997e-05 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714776+0000", 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9118000000000001e-05 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.584 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.15132.0:41299 3.1 e202/186 ondisk, result = 0)", 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:46.482780+0000", 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.057809007, 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.232000664, 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.482780+0000", 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.482780+0000", 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.482781+0000", 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3799999999999996e-07 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.482787+0000", 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2460000000000003e-06 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714648+0000", 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23186122100000001 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714652+0000", 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4179999999999999e-06 2024-06-27T21:22:15.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714691+0000", 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9122000000000003e-05 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714699+0000", 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7780000000000007e-06 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:46.714781+0000", 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1940999999999994e-05 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:63854 3.6 3:681673ca:::benchmark_data_smithi192_74314_object63853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:04.472284+0000", 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.06830506099999, 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21587946399999999, 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63854 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.472284+0000", 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.472284+0000", 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.472285+0000", 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2900000000000003e-07 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.472364+0000", 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9091999999999994e-05 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.472372+0000", 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1939999999999998e-06 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.472378+0000", 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0340000000000002e-06 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.473980+0000", 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016014969999999999 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.474105+0000", 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012459099999999999 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.474234+0000", 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012960399999999999 2024-06-27T21:22:15.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.476880+0000", 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026462930000000001 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.493316+0000", 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016435932 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.688043+0000", 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19472668400000001 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.688112+0000", 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8732999999999994e-05 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.688164+0000", 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2080999999999998e-05 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.596 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:63856 3.1 3:8c287a3c:::benchmark_data_smithi192_74314_object63855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:04.475221+0000", 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.06536859599998, 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21576114599999999, 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.597 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63856 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475221+0000", 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475221+0000", 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475222+0000", 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1939999999999999e-06 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475323+0000", 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010071100000000001 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475336+0000", 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3499000000000001e-05 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475345+0000", 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3140000000000004e-06 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475396+0000", 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1700000000000003e-05 2024-06-27T21:22:15.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475530+0000", 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134017 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.475662+0000", 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013134400000000001 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477543+0000", 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018809219999999999 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.477636+0000", 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3173999999999999e-05 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.690915+0000", 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21327944300000001 2024-06-27T21:22:15.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.690956+0000", 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0689999999999998e-05 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.690982+0000", 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6137999999999999e-05 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:63867 3.6 3:6abe9133:::benchmark_data_smithi192_74314_object63866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:04.490888+0000", 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.04970186700001, 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.215626558, 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 63867 2024-06-27T21:22:15.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.490888+0000", 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.490888+0000", 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.490889+0000", 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3260000000000001e-06 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.490938+0000", 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8662999999999998e-05 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.490950+0000", 2024-06-27T21:22:15.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2795e-05 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.490957+0000", 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3539999999999996e-06 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.491004+0000", 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7456000000000001e-05 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.491103+0000", 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8873e-05 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.491201+0000", 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8010999999999994e-05 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.492940+0000", 2024-06-27T21:22:15.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017392410000000001 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.691287+0000", 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.198346402 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.706377+0000", 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015090345 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.706473+0000", 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5790000000000003e-05 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:04.706514+0000", 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1301999999999999e-05 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.610 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.15132.0:90997 3.e e218/186 3:7de0318a:::benchmark_data_smithi192_74314_object90996:head v 218'5826, mlcod=218'5823)", 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:17:27.753227+0000", 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 287.78736294100003, 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23334509000000001, 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.753227+0000", 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.753227+0000", 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.753227+0000", 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3499999999999995e-07 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.753315+0000", 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8178999999999997e-05 2024-06-27T21:22:15.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.983372+0000", 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23005698399999999 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.983379+0000", 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1840000000000004e-06 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.983480+0000", 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010151500000000001 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.983506+0000", 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6117000000000001e-05 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.986533+0000", 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030264910000000001 2024-06-27T21:22:15.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:17:27.986572+0000", 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8884999999999998e-05 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:174509 3.6 3:6cc3b247:::benchmark_data_smithi192_74314_object67871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:15.082322+0000", 2024-06-27T21:22:15.615 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 180.458267624, 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214522292, 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 174509 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.082322+0000", 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.082322+0000", 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:22:15.618 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:22:15.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:15.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.082323+0000", 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.649e-06 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.082331+0000", 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5560000000000002e-06 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.082343+0000", 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1958e-05 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.082348+0000", 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3120000000000001e-06 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.131470+0000", 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049122167000000001 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.131546+0000", 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5402999999999996e-05 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.131619+0000", 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3452000000000001e-05 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.134907+0000", 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032876849999999998 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.168778+0000", 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.033870892 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.296746+0000", 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.127967989 2024-06-27T21:22:15.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.296783+0000", 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6680000000000001e-05 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:15.296844+0000", 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1549000000000003e-05 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:175453 3.6 3:65c5ffcf:::benchmark_data_smithi192_74314_object68815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:16.787609+0000", 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.75298033799999, 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23380295400000001, 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 175453 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.625 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.787609+0000", 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.787609+0000", 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.787610+0000", 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3149999999999999e-06 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.787616+0000", 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7350000000000001e-06 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.787627+0000", 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0454e-05 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.787633+0000", 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7290000000000003e-06 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.994327+0000", 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206693657 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.994457+0000", 2024-06-27T21:22:15.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012967 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:16.994593+0000", 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000136465 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.001585+0000", 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069918619999999997 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.015380+0000", 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013795042 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.021253+0000", 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058730060000000001 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.021339+0000", 2024-06-27T21:22:15.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6244999999999998e-05 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.021412+0000", 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2774000000000005e-05 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:175617 3.6 3:62f92e0c:::benchmark_data_smithi192_74314_object68979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:17.063952+0000", 2024-06-27T21:22:15.631 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.47663729800001, 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214471411, 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 175617 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.063952+0000", 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.063952+0000", 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.063953+0000", 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1930000000000001e-06 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.063959+0000", 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2259999999999996e-06 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.063967+0000", 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4600000000000003e-06 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.063974+0000", 2024-06-27T21:22:15.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7059999999999998e-06 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.264902+0000", 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20092813000000001 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.265038+0000", 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000136234 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.265183+0000", 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014442200000000001 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.278200+0000", 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013017828 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.278297+0000", 2024-06-27T21:22:15.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6998999999999997e-05 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.278311+0000", 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3524e-05 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.278359+0000", 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8408999999999997e-05 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.278424+0000", 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4280000000000001e-05 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:175740 3.6 3:60415e6c:::benchmark_data_smithi192_74314_object69102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:17.336748+0000", 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 178.20384184900001, 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22485271600000001, 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.638 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 175740 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.336748+0000", 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.336748+0000", 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.336748+0000", 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4499999999999996e-07 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.336752+0000", 2024-06-27T21:22:15.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000001e-06 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.336760+0000", 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0620000000000001e-06 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.336766+0000", 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0820000000000004e-06 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.440873+0000", 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.104106854 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.441038+0000", 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016460499999999999 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.441206+0000", 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016763100000000001 2024-06-27T21:22:15.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.450667+0000", 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0094613279999999998 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.561337+0000", 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.110670385 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.561466+0000", 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012906899999999999 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.561530+0000", 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4060999999999995e-05 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.561600+0000", 2024-06-27T21:22:15.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9924000000000003e-05 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:176025 3.6 3:61171de0:::benchmark_data_smithi192_74314_object69387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:17.902940+0000", 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.637649724, 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.228479034, 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.645 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 176025 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.902940+0000", 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.902940+0000", 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.902941+0000", 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2900000000000002e-07 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.902946+0000", 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8790000000000002e-06 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.902958+0000", 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2189e-05 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.902964+0000", 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2689999999999996e-06 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.924842+0000", 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021878475000000001 2024-06-27T21:22:15.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.924976+0000", 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013354 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.925102+0000", 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012611799999999999 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.932946+0000", 2024-06-27T21:22:15.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0078440290000000006 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:17.942291+0000", 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00934486 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.131295+0000", 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18900420800000001 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.131356+0000", 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0819999999999997e-05 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.131419+0000", 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2718000000000005e-05 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.15132.0:176555 3.6 3:66698adb:::benchmark_data_smithi192_74314_object69917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e256)", 2024-06-27T21:22:15.651 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:19:18.877961+0000", 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 176.662628136, 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21481302499999999, 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.15132", 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 176555 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.877961+0000", 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.877961+0000", 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.877963+0000", 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.623e-06 2024-06-27T21:22:15.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.877972+0000", 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.755e-06 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.877984+0000", 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2368000000000001e-05 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:18.877992+0000", 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9370000000000004e-06 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.058655+0000", 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18066286500000001 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.058727+0000", 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2558999999999994e-05 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.058815+0000", 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7888999999999998e-05 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.070286+0000", 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01147103 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.070303+0000", 2024-06-27T21:22:15.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6404000000000001e-05 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.092645+0000", 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022342153 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.092694+0000", 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9100000000000001e-05 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:19:19.092774+0000", 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0341999999999998e-05 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.5,RELEASE,e:304,from:6,reservation_nonce:0)", 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:09.430120+0000", 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 66.110469623, 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30420182800000001, 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.658 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.430120+0000", 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.430120+0000", 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.430120+0000", 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7100000000000002e-07 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.430124+0000", 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3069999999999998e-06 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.734190+0000", 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30406630099999998 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.734197+0000", 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2350000000000002e-06 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.734280+0000", 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2863999999999998e-05 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.734299+0000", 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9298999999999999e-05 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:09.734322+0000", 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2351e-05 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:47817 4.b e310/282 ondisk, result = 0)", 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:23.580672+0000", 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 51.959917128999997, 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23094853100000001, 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.580672+0000", 2024-06-27T21:22:15.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.580672+0000", 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.580673+0000", 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9800000000000002e-07 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:15.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.580678+0000", 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3900000000000003e-06 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.810087+0000", 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22940909900000001 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.810093+0000", 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6089999999999997e-06 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.811495+0000", 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001401343 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.811503+0000", 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1640000000000001e-06 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:23.811621+0000", 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117928 2024-06-27T21:22:15.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:22:15.667 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:22:15.668 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:22:15.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:22:15.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:22:15.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:22:15.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:22:15.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:22:15.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:15.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:15.750 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:22:15.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:22:15.784 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:22:15.784 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:22:15.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:22:15.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:22:15.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:15.850 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:22:15.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:22:15.871 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: pgmap v875: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 754 op/s; 3411/267294 objects misplaced (1.276%); 20 MiB/s, 320 objects/s recovering 2024-06-27T21:22:15.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: 1.7 deep-scrub starts 2024-06-27T21:22:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: 1.7 deep-scrub ok 2024-06-27T21:22:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: osdmap e332: 12 total, 12 up, 12 in 2024-06-27T21:22:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:15.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:15.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:15.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:15.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:16.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: pgmap v875: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 754 op/s; 3411/267294 objects misplaced (1.276%); 20 MiB/s, 320 objects/s recovering 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: 1.7 deep-scrub starts 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: 1.7 deep-scrub ok 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: osdmap e332: 12 total, 12 up, 12 in 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:16.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: pgmap v875: 27 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 23 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 754 op/s; 3411/267294 objects misplaced (1.276%); 20 MiB/s, 320 objects/s recovering 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: 1.7 deep-scrub starts 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: 1.7 deep-scrub ok 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: osdmap e332: 12 total, 12 up, 12 in 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:16.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:16.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:16.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:16.252 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:16.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:16.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:16.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:16.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:16.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:16.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:16.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:16.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:16.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:22:16.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:16.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:16.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:16.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:17.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:17.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:16 smithi192 ceph-mon[32956]: osdmap e333: 12 total, 12 up, 12 in 2024-06-27T21:22:17.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:16 smithi151 ceph-mon[28965]: osdmap e333: 12 total, 12 up, 12 in 2024-06-27T21:22:17.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:17.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:17.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:17.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:16 smithi190 ceph-mon[32247]: osdmap e333: 12 total, 12 up, 12 in 2024-06-27T21:22:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:17.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:17.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:17.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:17.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:22:17.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:17.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:17.592 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:17.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:17.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:17.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:17.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:17.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:17.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:17.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:18.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:17 smithi192 ceph-mon[32956]: pgmap v878: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 B/s rd, 36 MiB/s wr, 668 op/s; 2756/270465 objects misplaced (1.019%); 14 MiB/s, 221 objects/s recovering 2024-06-27T21:22:18.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:17 smithi151 ceph-mon[28965]: pgmap v878: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 B/s rd, 36 MiB/s wr, 668 op/s; 2756/270465 objects misplaced (1.019%); 14 MiB/s, 221 objects/s recovering 2024-06-27T21:22:18.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:18.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:18.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:18.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:18.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:17 smithi190 ceph-mon[32247]: pgmap v878: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 242 B/s rd, 36 MiB/s wr, 668 op/s; 2756/270465 objects misplaced (1.019%); 14 MiB/s, 221 objects/s recovering 2024-06-27T21:22:18.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:18.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:18.424 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:18.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:18.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:18.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:18.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:18.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:18.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:18.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:18.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:18.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:18.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:22:18.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:19.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:19.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:18 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:22:19.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:18 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:22:19.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:19.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:18 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:22:19.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:18 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:22:19.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:19.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:19.261 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:18 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:22:19.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:18 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:22:19.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:19.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:19.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:19.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:19.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:19.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:19.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:19.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:19.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:19.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:19.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:19.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:19.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:22:19.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:22:19.950 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-27T21:22:19.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:22:20.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:20.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:20.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:19 smithi192 ceph-mon[32956]: pgmap v879: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 27 MiB/s wr, 604 op/s; 2550/269070 objects misplaced (0.948%); 12 MiB/s, 189 objects/s recovering 2024-06-27T21:22:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:20.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:19 smithi151 ceph-mon[28965]: pgmap v879: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 27 MiB/s wr, 604 op/s; 2550/269070 objects misplaced (0.948%); 12 MiB/s, 189 objects/s recovering 2024-06-27T21:22:20.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:20.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:20.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:19 smithi190 ceph-mon[32247]: pgmap v879: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 27 MiB/s wr, 604 op/s; 2550/269070 objects misplaced (0.948%); 12 MiB/s, 189 objects/s recovering 2024-06-27T21:22:20.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:20.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:20.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:20.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:20.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:20.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:20.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:20.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:20.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:20.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:20.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:20.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:20.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:21.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:21.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:21.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:97633 4.f 4:faab62da:::benchmark_data_smithi192_88961_object6894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:21.178274+0000", 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0030702640000000001, 2024-06-27T21:22:21.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031004610000000001, 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 97633 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:21.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178274+0000", 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178274+0000", 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178275+0000", 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.443e-06 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178281+0000", 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0560000000000004e-06 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178298+0000", 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7059e-05 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178307+0000", 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2550000000000003e-06 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178401+0000", 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3648999999999997e-05 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178545+0000", 2024-06-27T21:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014498700000000001 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.178670+0000", 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012405399999999999 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:97635 4.a e333/330 4:50e9bd96:::benchmark_data_smithi192_88961_object6896:head v 333'6170, mlcod=333'6168)", 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:21.179674+0000", 2024-06-27T21:22:21.225 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0016702850000000001, 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017701819999999999, 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.179674+0000", 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.179674+0000", 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.179675+0000", 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2799999999999995e-07 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.179719+0000", 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4447000000000003e-05 2024-06-27T21:22:21.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.179797+0000", 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7883e-05 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.179805+0000", 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6769999999999995e-06 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:21.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181365+0000", 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015604970000000001 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181384+0000", 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8529000000000002e-05 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:97636 4.7 4:e72f58f7:::benchmark_data_smithi192_88961_object6897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-06-27T21:22:21.229 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:21.180597+0000", 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.000747148, 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089446000000000003, 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:22:21.230 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 97636 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.180597+0000", 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.180597+0000", 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.180599+0000", 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.595e-06 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.180604+0000", 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8949999999999997e-06 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.180619+0000", 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5293000000000001e-05 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.180626+0000", 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3660000000000003e-06 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:21.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181320+0000", 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000693514 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181441+0000", 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121434 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:97629 4.4 e333/314)", 2024-06-27T21:22:21.234 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:21.181313+0000", 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.1603999999999998e-05, 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00022560700000000001, 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181313+0000", 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:21.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181313+0000", 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181313+0000", 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5400000000000001e-07 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181319+0000", 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0769999999999996e-06 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181356+0000", 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6251000000000002e-05 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:21.181363+0000", 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8719999999999996e-06 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-06-27T21:22:21.237 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:22:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:21.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:21.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:21.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:21.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:21.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:21.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:21.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:21.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:21.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:21.795 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:22:21.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:21.868 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:21 smithi151 ceph-mon[28965]: pgmap v880: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 B/s rd, 17 MiB/s wr, 725 op/s; 2550/262530 objects misplaced (0.971%); 7.5 MiB/s, 119 objects/s recovering 2024-06-27T21:22:21.868 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1941601415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:22:21.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:21.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:21.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:22:21.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:22.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:22.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:21 smithi192 ceph-mon[32956]: pgmap v880: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 B/s rd, 17 MiB/s wr, 725 op/s; 2550/262530 objects misplaced (0.971%); 7.5 MiB/s, 119 objects/s recovering 2024-06-27T21:22:22.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1941601415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:22:22.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:22.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:22.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:22.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:21 smithi190 ceph-mon[32247]: pgmap v880: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 B/s rd, 17 MiB/s wr, 725 op/s; 2550/262530 objects misplaced (0.971%); 7.5 MiB/s, 119 objects/s recovering 2024-06-27T21:22:22.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1941601415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:22:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.285 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:22.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:22.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:22.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:22.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:22.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:22.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:22.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:22.679 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:22.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:22.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:22.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:22.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:22.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:22.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:22 smithi151 ceph-mon[28965]: 4.5 scrub starts 2024-06-27T21:22:22.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:22 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:22.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1941601415' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:22:22.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:22 smithi151 ceph-mon[28965]: osdmap e334: 12 total, 12 up, 12 in 2024-06-27T21:22:23.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:23.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:22 smithi192 ceph-mon[32956]: 4.5 scrub starts 2024-06-27T21:22:23.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:22 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:23.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1941601415' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:22:23.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:22 smithi192 ceph-mon[32956]: osdmap e334: 12 total, 12 up, 12 in 2024-06-27T21:22:23.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:23.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:23.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:23.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:22 smithi190 ceph-mon[32247]: 4.5 scrub starts 2024-06-27T21:22:23.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:22 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:23.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1941601415' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:22:23.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:22 smithi190 ceph-mon[32247]: osdmap e334: 12 total, 12 up, 12 in 2024-06-27T21:22:23.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:23.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:23.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:23.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:23.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:23.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:23.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:23.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:23.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:23.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:23.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:23 smithi151 ceph-mon[28965]: pgmap v882: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 B/s rd, 7.7 MiB/s wr, 840 op/s; 2120/256737 objects misplaced (0.826%); 11 MiB/s, 179 objects/s recovering 2024-06-27T21:22:23.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:23.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:23.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:24.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:24.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:24.202 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:24.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:24.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:23 smithi190 ceph-mon[32247]: pgmap v882: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 B/s rd, 7.7 MiB/s wr, 840 op/s; 2120/256737 objects misplaced (0.826%); 11 MiB/s, 179 objects/s recovering 2024-06-27T21:22:24.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:23 smithi192 ceph-mon[32956]: pgmap v882: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 142 B/s rd, 7.7 MiB/s wr, 840 op/s; 2120/256737 objects misplaced (0.826%); 11 MiB/s, 179 objects/s recovering 2024-06-27T21:22:24.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:24.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:24.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:24.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:24.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:24.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:24.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:22:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:24.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:24.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:24.859 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:24.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:24.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:24.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:25.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:25.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:25.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:25.252 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:25.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:25.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:25.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:25.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:25.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:25.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:25.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:25.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:25.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:25.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:26.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:26.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:26.095 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:25 smithi192 ceph-mon[32956]: pgmap v883: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 810 op/s; 1812/252168 objects misplaced (0.719%); 7.2 MiB/s, 115 objects/s recovering 2024-06-27T21:22:26.116 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:25 smithi151 ceph-mon[28965]: pgmap v883: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 810 op/s; 1812/252168 objects misplaced (0.719%); 7.2 MiB/s, 115 objects/s recovering 2024-06-27T21:22:26.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:26.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:26.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:25 smithi190 ceph-mon[32247]: pgmap v883: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 810 op/s; 1812/252168 objects misplaced (0.719%); 7.2 MiB/s, 115 objects/s recovering 2024-06-27T21:22:26.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:26.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:26.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:26.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:26.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:26.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:26.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:26.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:26.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:26.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:26.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:26.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:26.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:27.007 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:22:27.008 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:22:27.031 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:22:27.031 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:22:27.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:27.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:22:27.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:22:27.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:27.074 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:22:27.074 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:22:27.095 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:22:27.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:22:27.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:22:27.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:22:27.142 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:22:27.142 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:22:27.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:27.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:22:27.162 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:22:27.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:27.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:22:27.183 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:22:27.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:22:27.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:22:27.254 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:22:27.254 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:22:27.281 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:22:27.282 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:22:27.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:27.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:27.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:27.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:27.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:27.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:27.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:27.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:27.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:27.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:27.859 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:22:27.859 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:22:27.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:27.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:27.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:27.995 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:22:27.995 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:22:27.995 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 336, 2024-06-27T21:22:27.995 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:22:27.810194+0000", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:09:37.640435+0000", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:21:33.506862+0000", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:22:27.996 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 4, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:22:27.997 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:22:27.998 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 11, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 11, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 11, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 11, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 11, 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:22:27.999 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "163", 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "161", 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:22:28.000 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:22:28.001 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:22:28.002 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:22:28.003 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3.2699999809265137, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.2699999809265137, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 3.2699999809265137, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 3.2699999809265137, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.87693524360656738, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.87999999523162842, 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-06-27T21:22:28.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 4, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_2", 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:20:32.017232+0000", 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:22:28.005 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:22:28.006 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "325", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:22:28.007 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:22:28.008 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:22:28.009 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:22:28.010 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.9900000095367432, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.9900000095367432, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.75, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.75, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.87693524360656738, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.87999999523162842, 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.011 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.013 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.016 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:22:28.017 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.1994171142578125, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 327, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.018 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.020 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.021 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:22:28.024 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 104, 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.026 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.027 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:22:28.029 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 314, 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.033 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.035 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.3238067626953125, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 31, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 297, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.037 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.038 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:22:28.039 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2107645547 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/2107645547", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/2107645547", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/2107645547", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/2107645547", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 314, 2024-06-27T21:22:28.043 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.048 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:22:28.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 330, 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:22:28.050 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:22:28.051 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:22:28.052 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.053 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.054 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 314, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.055 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.056 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.057 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.058 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:22:28.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:22:28.060 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 186, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.062 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.064 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:22:28.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 335, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.067 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.068 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.069 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.073 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:22:28.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:22:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.074 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 328, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.075 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.078 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:22:28.080 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.081 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:22:28.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:22:28.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.086 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:22:14.963691+0000", 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.087 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:21:39.019790+0000", 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.088 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:21:25.612782+0000", 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:22:07.672768+0000", 2024-06-27T21:22:28.089 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:22:06.153030+0000", 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:22:14.310742+0000", 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.091 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:21:41.285217+0000", 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.092 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:21:44.534446+0000", 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:22:02.592071+0000", 2024-06-27T21:22:28.093 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:22:09.729464+0000", 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.094 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:21:37.616723+0000", 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:22:28.095 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:21:18.349489+0000", 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-06-27T21:22:28.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.4", 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: 3, 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: 1 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:22:28.097 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:22:28.098 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:22:28.099 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:22:28.100 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:28.100 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:22:28.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:28.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:27 smithi192 ceph-mon[32956]: pgmap v884: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 990 op/s; 1291/241758 objects misplaced (0.534%); 9.2 MiB/s, 146 objects/s recovering 2024-06-27T21:22:28.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:27 smithi192 ceph-mon[32956]: osdmap e335: 12 total, 12 up, 12 in 2024-06-27T21:22:28.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1890299045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:28.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:27 smithi151 ceph-mon[28965]: pgmap v884: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 990 op/s; 1291/241758 objects misplaced (0.534%); 9.2 MiB/s, 146 objects/s recovering 2024-06-27T21:22:28.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:27 smithi151 ceph-mon[28965]: osdmap e335: 12 total, 12 up, 12 in 2024-06-27T21:22:28.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1890299045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:28.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:28.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:28.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:27 smithi190 ceph-mon[32247]: pgmap v884: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s wr, 990 op/s; 1291/241758 objects misplaced (0.534%); 9.2 MiB/s, 146 objects/s recovering 2024-06-27T21:22:28.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:27 smithi190 ceph-mon[32247]: osdmap e335: 12 total, 12 up, 12 in 2024-06-27T21:22:28.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1890299045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:28.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:28.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:28.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:28.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:28.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:28.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:28.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:28.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:28.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:28.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:28.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:28.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:29.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:29.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:29.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:28 smithi151 ceph-mon[28965]: osdmap e336: 12 total, 12 up, 12 in 2024-06-27T21:22:29.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:28 smithi151 ceph-mon[28965]: pgmap v887: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s; 1006/238893 objects misplaced (0.421%); 11 MiB/s, 180 objects/s recovering 2024-06-27T21:22:29.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1333090297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:22:29.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:29.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:28 smithi190 ceph-mon[32247]: osdmap e336: 12 total, 12 up, 12 in 2024-06-27T21:22:29.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:28 smithi190 ceph-mon[32247]: pgmap v887: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s; 1006/238893 objects misplaced (0.421%); 11 MiB/s, 180 objects/s recovering 2024-06-27T21:22:29.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1333090297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:22:29.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:29.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:28 smithi192 ceph-mon[32956]: osdmap e336: 12 total, 12 up, 12 in 2024-06-27T21:22:29.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:28 smithi192 ceph-mon[32956]: pgmap v887: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s; 1006/238893 objects misplaced (0.421%); 11 MiB/s, 180 objects/s recovering 2024-06-27T21:22:29.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1333090297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:22:29.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:29.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:29.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:29.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:22:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:29.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:29.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:29.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:29.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:29.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:29.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:29.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:30.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:30.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:30.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:30.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:30.275 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:30.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:30.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:30.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:30.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:30.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:30.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:30.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:30.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:30.740 INFO:tasks.thrashosds.thrasher:j is {'epoch': 336, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:22:27.810194+0000', 'last_up_change': '2024-06-27T21:09:37.640435+0000', 'last_in_change': '2024-06-27T21:21:33.506862+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '161', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2699999809265137, 'score_stable': 3.2699999809265137, 'optimal_score': 1, 'raw_score_acting': 3.2699999809265137, 'raw_score_stable': 3.2699999809265137, 'primary_affinity_weighted': 0.8769352436065674, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-06-27T21:20:32.017232+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '325', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.8769352436065674, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1994171142578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 327, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3238067626953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 297, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 2107645547}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 2107645547}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 2107645547}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 2107645547}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 2107645547}]}, 'public_addr': '172.21.15.190:6801/2107645547', 'cluster_addr': '172.21.15.190:6803/2107645547', 'heartbeat_back_addr': '172.21.15.190:6807/2107645547', 'heartbeat_front_addr': '172.21.15.190:6805/2107645547', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 330, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 186, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 328, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:22:14.963691+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:21:39.019790+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:21:25.612782+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:22:07.672768+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:22:06.153030+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:22:14.310742+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:21:41.285217+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:21:44.534446+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:22:02.592071+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:22:09.729464+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:21:37.616723+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:21:18.349489+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.4', 'osds': [3, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:22:30.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:22:30.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:30.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:30.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:30.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:30.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:31.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:31.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:31.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:31.208 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:31.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:31 smithi192 ceph-mon[32956]: pgmap v888: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 1006/232794 objects misplaced (0.432%); 8.7 MiB/s, 139 objects/s recovering 2024-06-27T21:22:31.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:31 smithi151 ceph-mon[28965]: pgmap v888: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 1006/232794 objects misplaced (0.432%); 8.7 MiB/s, 139 objects/s recovering 2024-06-27T21:22:31.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:31.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:31.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:31.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:31.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:31 smithi190 ceph-mon[32247]: pgmap v888: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 997 op/s; 1006/232794 objects misplaced (0.432%); 8.7 MiB/s, 139 objects/s recovering 2024-06-27T21:22:31.530 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:31.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:31.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:31.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:31.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:31.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:31.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:31.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:31.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:32.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:32.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:32.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:32.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:32.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:22:32.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:32.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:32.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:32.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:32.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:32.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:32.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:32.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:32.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:32.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:32.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:32.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:33.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:33.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:33.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:33.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:33.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:33.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:33 smithi151 ceph-mon[28965]: pgmap v889: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 966/227460 objects misplaced (0.425%); 6.6 MiB/s, 105 objects/s recovering 2024-06-27T21:22:33.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:33 smithi151 ceph-mon[28965]: osdmap e337: 12 total, 12 up, 12 in 2024-06-27T21:22:33.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:33.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:33.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:33.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:33 smithi192 ceph-mon[32956]: pgmap v889: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 966/227460 objects misplaced (0.425%); 6.6 MiB/s, 105 objects/s recovering 2024-06-27T21:22:33.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:33 smithi192 ceph-mon[32956]: osdmap e337: 12 total, 12 up, 12 in 2024-06-27T21:22:33.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:33.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:33.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:33.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:33.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:33.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:33 smithi190 ceph-mon[32247]: pgmap v889: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 966/227460 objects misplaced (0.425%); 6.6 MiB/s, 105 objects/s recovering 2024-06-27T21:22:33.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:33 smithi190 ceph-mon[32247]: osdmap e337: 12 total, 12 up, 12 in 2024-06-27T21:22:33.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:33.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:33.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:33.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:33.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:33.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:33.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:33.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:33.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:34.039 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:22:34.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:34.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:34.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1094172993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:34.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:34 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:34.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:34.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:34.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1094172993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:34.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:34 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:34.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:34.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:34.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:34.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:34.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1094172993' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:34.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:34 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:22:34.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:34.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:34.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:34.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:34.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:34.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:34.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:34.969 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:34.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:34.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:35.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:35.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:35.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:35 smithi151 ceph-mon[28965]: pgmap v891: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 519/222459 objects misplaced (0.233%); 6.7 MiB/s, 107 objects/s recovering 2024-06-27T21:22:35.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:35 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:35.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:35 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:22:35.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:35 smithi151 ceph-mon[28965]: osdmap e338: 12 total, 12 up, 12 in 2024-06-27T21:22:35.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:35.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:35.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:35 smithi192 ceph-mon[32956]: pgmap v891: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 519/222459 objects misplaced (0.233%); 6.7 MiB/s, 107 objects/s recovering 2024-06-27T21:22:35.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:35 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:35.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:35 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:22:35.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:35 smithi192 ceph-mon[32956]: osdmap e338: 12 total, 12 up, 12 in 2024-06-27T21:22:35.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:35.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:35.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:35 smithi190 ceph-mon[32247]: pgmap v891: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s; 519/222459 objects misplaced (0.233%); 6.7 MiB/s, 107 objects/s recovering 2024-06-27T21:22:35.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:35 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:35.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:35 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:22:35.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:35 smithi190 ceph-mon[32247]: osdmap e338: 12 total, 12 up, 12 in 2024-06-27T21:22:35.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:35.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:35.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:35.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:35.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:35.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:35.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:35.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:35.869 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:22:35.870 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:22:35.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:35.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:35.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:36.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:36.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:36.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:36.230 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:36.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:36.262 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:36.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:36 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:22:36.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:36.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:36.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:36.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:36.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:36 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:22:36.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:36 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:22:36.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:36.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:36.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:36.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:36.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:22:36.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:36.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:36.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:36.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:37.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:37.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:37.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:37.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:37.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:37.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:37.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:37 smithi151 ceph-mon[28965]: pgmap v893: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 519/211344 objects misplaced (0.246%); 3.8 MiB/s, 60 objects/s recovering 2024-06-27T21:22:37.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:37.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:37.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:37 smithi190 ceph-mon[32247]: pgmap v893: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 519/211344 objects misplaced (0.246%); 3.8 MiB/s, 60 objects/s recovering 2024-06-27T21:22:37.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:37.542 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:37 smithi192 ceph-mon[32956]: pgmap v893: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 519/211344 objects misplaced (0.246%); 3.8 MiB/s, 60 objects/s recovering 2024-06-27T21:22:37.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:37.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:37.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:37.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:37.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:37.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:37.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:37.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:37.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:22:38.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:38.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:38.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:38.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:38.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:38.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:38.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:38.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:38.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:38.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:38.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:38.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:38.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:38.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:38.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:38.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:38.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:38.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:39.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:39.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:39.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:39.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:39.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:39.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:39.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:39.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:39 smithi151 ceph-mon[28965]: pgmap v894: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 30/208485 objects misplaced (0.014%); 7.6 MiB/s, 121 objects/s recovering 2024-06-27T21:22:39.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:39 smithi151 ceph-mon[28965]: osdmap e339: 12 total, 12 up, 12 in 2024-06-27T21:22:39.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:39.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:39 smithi190 ceph-mon[32247]: pgmap v894: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 30/208485 objects misplaced (0.014%); 7.6 MiB/s, 121 objects/s recovering 2024-06-27T21:22:39.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:39 smithi190 ceph-mon[32247]: osdmap e339: 12 total, 12 up, 12 in 2024-06-27T21:22:39.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:39 smithi192 ceph-mon[32956]: pgmap v894: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 30/208485 objects misplaced (0.014%); 7.6 MiB/s, 121 objects/s recovering 2024-06-27T21:22:39.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:39 smithi192 ceph-mon[32956]: osdmap e339: 12 total, 12 up, 12 in 2024-06-27T21:22:39.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:39.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:39.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:39.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:39.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:39.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:39.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:39.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:39.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:40.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:40.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:40.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:40.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:40.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:40.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:40.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:40.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:40 smithi151 ceph-mon[28965]: osdmap e340: 12 total, 12 up, 12 in 2024-06-27T21:22:40.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:40.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:40.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:40.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:40 smithi190 ceph-mon[32247]: osdmap e340: 12 total, 12 up, 12 in 2024-06-27T21:22:40.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:40.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:40.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:40 smithi192 ceph-mon[32956]: osdmap e340: 12 total, 12 up, 12 in 2024-06-27T21:22:40.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:40.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:40.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:40.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:40.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:40.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:40.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:40.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:40.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:41.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:41.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:41.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:41.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:41.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:41.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:41.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:41.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:41.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:41 smithi151 ceph-mon[28965]: pgmap v897: 27 pgs: 1 creating+peering, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-06-27T21:22:41.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:41 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-27T21:22:41.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:41 smithi151 ceph-mon[28965]: osdmap e341: 12 total, 12 up, 12 in 2024-06-27T21:22:41.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:41 smithi190 ceph-mon[32247]: pgmap v897: 27 pgs: 1 creating+peering, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-06-27T21:22:41.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:41 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-27T21:22:41.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:41 smithi190 ceph-mon[32247]: osdmap e341: 12 total, 12 up, 12 in 2024-06-27T21:22:41.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:41.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:41 smithi192 ceph-mon[32956]: pgmap v897: 27 pgs: 1 creating+peering, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-06-27T21:22:41.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:41 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-27T21:22:41.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:41 smithi192 ceph-mon[32956]: osdmap e341: 12 total, 12 up, 12 in 2024-06-27T21:22:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:41.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:41.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:41.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:41.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:41.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:41.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:41.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:41.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:42.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:42.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:42.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:42.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:42.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:42.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:42.456 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:42.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:42.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:42.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:42.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:42.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:42.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:42.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:42.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:42.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:43.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:43.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:43.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:43.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:43 smithi151 ceph-mon[28965]: pgmap v899: 27 pgs: 1 creating+peering, 26 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-06-27T21:22:43.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:43 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:22:43.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:43.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:43.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:43.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:43.466 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:43.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:43 smithi190 ceph-mon[32247]: pgmap v899: 27 pgs: 1 creating+peering, 26 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-06-27T21:22:43.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:43 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:22:43.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:43 smithi192 ceph-mon[32956]: pgmap v899: 27 pgs: 1 creating+peering, 26 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 5.4 MiB/s, 86 objects/s recovering 2024-06-27T21:22:43.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:43 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:22:43.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:43.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:43.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:43.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:43.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:43.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:43.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:43.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:43.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:44.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:44.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:44.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:44.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:44.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:44.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:44.394 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:44.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:44.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:44 smithi151 ceph-mon[28965]: osdmap e342: 12 total, 12 up, 12 in 2024-06-27T21:22:44.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:22:44.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:22:44.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:22:44.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:44.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:44.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:44.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:44.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:44.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:44 smithi190 ceph-mon[32247]: osdmap e342: 12 total, 12 up, 12 in 2024-06-27T21:22:44.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:22:44.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:22:44.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:22:44.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:44.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:44.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:44 smithi192 ceph-mon[32956]: osdmap e342: 12 total, 12 up, 12 in 2024-06-27T21:22:44.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:22:44.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:22:44.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:22:44.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:44.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:44.693 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:44.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:44.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:44.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:44.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:44.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:44.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:45.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:45.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:45.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:45.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:22:45.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:22:45.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:22:45.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:22:45.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:22:45.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:22:45.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:22:45.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:22:45.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:22:45.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:22:45.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:22:45.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:22:45.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:45.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:22:45.279 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:22:45.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:45.310 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:22:45.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:22:45.346 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:22:45.346 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:22:45.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:22:45.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:22:45.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:45.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:45.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:22:45.413 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:22:45.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:22:45.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:22:45.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:45.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:45.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:45.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:45.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:45 smithi151 ceph-mon[28965]: pgmap v901: 27 pgs: 27 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 326 KiB/s, 5 objects/s recovering 2024-06-27T21:22:45.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2057638526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:45.700 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:22:45.700 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:22:45.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:45 smithi190 ceph-mon[32247]: pgmap v901: 27 pgs: 27 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 326 KiB/s, 5 objects/s recovering 2024-06-27T21:22:45.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2057638526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:45.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:45.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:45.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:45 smithi192 ceph-mon[32956]: pgmap v901: 27 pgs: 27 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 326 KiB/s, 5 objects/s recovering 2024-06-27T21:22:45.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2057638526' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:45.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:45.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:45.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:46.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:46.044 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:46.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:46.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:46.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:46.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:46.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:46.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:46.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:46.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:46.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:22:46.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:22:46.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:46 smithi190 ceph-mon[32247]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:22:46.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:46 smithi190 ceph-mon[32247]: osdmap e343: 12 total, 12 up, 12 in 2024-06-27T21:22:46.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:46.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:22:46.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:22:46.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:46 smithi192 ceph-mon[32956]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:22:46.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:46 smithi192 ceph-mon[32956]: osdmap e343: 12 total, 12 up, 12 in 2024-06-27T21:22:46.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:22:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:22:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:46 smithi151 ceph-mon[28965]: from='client.25462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:22:46.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:46 smithi151 ceph-mon[28965]: osdmap e343: 12 total, 12 up, 12 in 2024-06-27T21:22:46.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:46.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:46.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:47.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:47.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:47.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:47.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:47.200 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":901,"stamp":"2024-06-27T21:22:43.973738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4229497400,"num_objects":64531,"num_object_clones":0,"num_object_copies":193593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116945,"num_write_kb":5807105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21917,"num_bytes_recovered":1436623936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117957,"ondisk_log_size":117957,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15561636,"kb_used_data":12712032,"kb_used_omap":18,"kb_used_meta":2849325,"kb_avail":1109380188,"statfs":{"total":1151940427776,"available":1136005312512,"internally_reserved":0,"allocated":13017120768,"data_stored":13013712868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":2917709114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-325320704,"num_objects":-4964,"num_object_clones":0,"num_object_copies":-14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-30,"num_objects_unfound":0,"num_objects_dirty":-4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.881027"},"pg_stats":[{"pgid":"4.e","version":"340'7161","reported_seq":13851,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.499166+0000","last_change":"2024-06-27T21:22:14.344819+0000","last_active":"2024-06-27T21:22:39.499166+0000","last_peered":"2024-06-27T21:22:39.499166+0000","last_clean":"2024-06-27T21:22:39.499166+0000","last_became_active":"2024-06-27T21:22:10.500450+0000","last_became_peered":"2024-06-27T21:22:10.500450+0000","last_unstale":"2024-06-27T21:22:39.499166+0000","last_undegraded":"2024-06-27T21:22:39.499166+0000","last_fullsized":"2024-06-27T21:22:39.499166+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5664","last_scrub_stamp":"2024-06-27T21:22:14.344672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:22:14.344672+0000","objects_scrubbed":5664,"log_size":7161,"log_dups_size":0,"ondisk_log_size":7161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:17.741063+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273088512,"num_objects":4167,"num_object_clones":0,"num_object_copies":12501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7161,"num_write_kb":362496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3236,"num_bytes_recovered":212074496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.b","version":"341'7315","reported_seq":7500,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.761638+0000","last_change":"2024-06-27T21:21:55.489846+0000","last_active":"2024-06-27T21:22:40.761638+0000","last_peered":"2024-06-27T21:22:40.761638+0000","last_clean":"2024-06-27T21:22:40.761638+0000","last_became_active":"2024-06-27T21:20:33.673842+0000","last_became_peered":"2024-06-27T21:20:33.673842+0000","last_unstale":"2024-06-27T21:22:40.761638+0000","last_undegraded":"2024-06-27T21:22:40.761638+0000","last_fullsized":"2024-06-27T21:22:40.761638+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'4766","last_scrub_stamp":"2024-06-27T21:21:55.489644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:21:55.489644+0000","objects_scrubbed":4722,"log_size":7315,"log_dups_size":0,"ondisk_log_size":7315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:57.065141+0000","scrub_duration":1413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7315,"num_write_kb":364160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"341'7394","reported_seq":14291,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.329550+0000","last_change":"2024-06-27T21:22:14.177071+0000","last_active":"2024-06-27T21:22:40.329550+0000","last_peered":"2024-06-27T21:22:40.329550+0000","last_clean":"2024-06-27T21:22:40.329550+0000","last_became_active":"2024-06-27T21:22:12.758092+0000","last_became_peered":"2024-06-27T21:22:12.758092+0000","last_unstale":"2024-06-27T21:22:40.329550+0000","last_undegraded":"2024-06-27T21:22:40.329550+0000","last_fullsized":"2024-06-27T21:22:40.329550+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5742","last_scrub_stamp":"2024-06-27T21:22:14.176937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:22:14.176937+0000","objects_scrubbed":5742,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:34.975665+0000","scrub_duration":576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268042240,"num_objects":4090,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7394,"num_write_kb":367488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3342,"num_bytes_recovered":219021312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"340'7221","reported_seq":7403,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.533493+0000","last_change":"2024-06-27T21:21:56.082645+0000","last_active":"2024-06-27T21:22:39.533493+0000","last_peered":"2024-06-27T21:22:39.533493+0000","last_clean":"2024-06-27T21:22:39.533493+0000","last_became_active":"2024-06-27T21:20:33.672659+0000","last_became_peered":"2024-06-27T21:20:33.672659+0000","last_unstale":"2024-06-27T21:22:39.533493+0000","last_undegraded":"2024-06-27T21:22:39.533493+0000","last_fullsized":"2024-06-27T21:22:39.533493+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'4744","last_scrub_stamp":"2024-06-27T21:21:56.082516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:21:56.082516+0000","objects_scrubbed":4701,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:00.512148+0000","scrub_duration":1599,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268238848,"num_objects":4093,"num_object_clones":0,"num_object_copies":12279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":362048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"340'7171","reported_seq":7351,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.522717+0000","last_change":"2024-06-27T21:21:52.432536+0000","last_active":"2024-06-27T21:22:39.522717+0000","last_peered":"2024-06-27T21:22:39.522717+0000","last_clean":"2024-06-27T21:22:39.522717+0000","last_became_active":"2024-06-27T21:20:33.961523+0000","last_became_peered":"2024-06-27T21:20:33.961523+0000","last_unstale":"2024-06-27T21:22:39.522717+0000","last_undegraded":"2024-06-27T21:22:39.522717+0000","last_fullsized":"2024-06-27T21:22:39.522717+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'4571","last_scrub_stamp":"2024-06-27T21:21:52.432356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:21:52.432356+0000","objects_scrubbed":4547,"log_size":7171,"log_dups_size":0,"ondisk_log_size":7171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:02.019247+0000","scrub_duration":978,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270073856,"num_objects":4121,"num_object_clones":0,"num_object_copies":12363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7171,"num_write_kb":361344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"341'7191","reported_seq":13771,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.391896+0000","last_change":"2024-06-27T21:22:21.813603+0000","last_active":"2024-06-27T21:22:40.391896+0000","last_peered":"2024-06-27T21:22:40.391896+0000","last_clean":"2024-06-27T21:22:40.391896+0000","last_became_active":"2024-06-27T21:21:45.492229+0000","last_became_peered":"2024-06-27T21:21:45.492229+0000","last_unstale":"2024-06-27T21:22:40.391896+0000","last_undegraded":"2024-06-27T21:22:40.391896+0000","last_fullsized":"2024-06-27T21:22:40.391896+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":2149,"log_size":7191,"log_dups_size":0,"ondisk_log_size":7191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:41.459478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263782400,"num_objects":4025,"num_object_clones":0,"num_object_copies":12075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":358912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":209846272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":701,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.135582+0000","last_change":"2024-06-27T21:22:14.275501+0000","last_active":"2024-06-27T21:22:39.135582+0000","last_peered":"2024-06-27T21:22:39.135582+0000","last_clean":"2024-06-27T21:22:39.135582+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:22:39.135582+0000","last_undegraded":"2024-06-27T21:22:39.135582+0000","last_fullsized":"2024-06-27T21:22:39.135582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:22:14.275429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:22:14.275429+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:35.446216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"341'7236","reported_seq":14658,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.368807+0000","last_change":"2024-06-27T21:22:27.855646+0000","last_active":"2024-06-27T21:22:40.368807+0000","last_peered":"2024-06-27T21:22:40.368807+0000","last_clean":"2024-06-27T21:22:40.368807+0000","last_became_active":"2024-06-27T21:22:27.848953+0000","last_became_peered":"2024-06-27T21:22:27.848953+0000","last_unstale":"2024-06-27T21:22:40.368807+0000","last_undegraded":"2024-06-27T21:22:40.368807+0000","last_fullsized":"2024-06-27T21:22:40.368807+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:40.547401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267124736,"num_objects":4076,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":361984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3642,"num_bytes_recovered":238682112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":288,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.104623+0000","last_change":"2024-06-27T21:22:17.551528+0000","last_active":"2024-06-27T21:22:40.104623+0000","last_peered":"2024-06-27T21:22:40.104623+0000","last_clean":"2024-06-27T21:22:40.104623+0000","last_became_active":"2024-06-27T21:17:58.594952+0000","last_became_peered":"2024-06-27T21:17:58.594952+0000","last_unstale":"2024-06-27T21:22:40.104623+0000","last_undegraded":"2024-06-27T21:22:40.104623+0000","last_fullsized":"2024-06-27T21:22:40.104623+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:22:17.551456+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:22:17.551456+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:18.093219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"342'7588","reported_seq":14305,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-06-27T21:22:43.758217+0000","last_change":"2024-06-27T21:21:53.677396+0000","last_active":"2024-06-27T21:22:43.758217+0000","last_peered":"2024-06-27T21:22:43.758217+0000","last_clean":"2024-06-27T21:22:43.758217+0000","last_became_active":"2024-06-27T21:21:53.673619+0000","last_became_peered":"2024-06-27T21:21:53.673619+0000","last_unstale":"2024-06-27T21:22:43.758217+0000","last_undegraded":"2024-06-27T21:22:43.758217+0000","last_fullsized":"2024-06-27T21:22:43.758217+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7588,"log_dups_size":0,"ondisk_log_size":7588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:46.905617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257818624,"num_objects":3934,"num_object_clones":0,"num_object_copies":11802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":368704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3283,"num_bytes_recovered":215154688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":705,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.132943+0000","last_change":"2024-06-27T21:22:00.073009+0000","last_active":"2024-06-27T21:22:39.132943+0000","last_peered":"2024-06-27T21:22:39.132943+0000","last_clean":"2024-06-27T21:22:39.132943+0000","last_became_active":"2024-06-27T21:14:55.718546+0000","last_became_peered":"2024-06-27T21:14:55.718546+0000","last_unstale":"2024-06-27T21:22:39.132943+0000","last_undegraded":"2024-06-27T21:22:39.132943+0000","last_fullsized":"2024-06-27T21:22:39.132943+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:22:00.072911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:22:00.072911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:27.484458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"342'7435","reported_seq":18015,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-06-27T21:22:43.220885+0000","last_change":"2024-06-27T21:22:39.116586+0000","last_active":"2024-06-27T21:22:43.220885+0000","last_peered":"2024-06-27T21:22:43.220885+0000","last_clean":"2024-06-27T21:22:43.220885+0000","last_became_active":"2024-06-27T21:22:39.102008+0000","last_became_peered":"2024-06-27T21:22:39.102008+0000","last_unstale":"2024-06-27T21:22:43.220885+0000","last_undegraded":"2024-06-27T21:22:43.220885+0000","last_fullsized":"2024-06-27T21:22:43.220885+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7435,"log_dups_size":0,"ondisk_log_size":7435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:44.666754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253427712,"num_objects":3867,"num_object_clones":0,"num_object_copies":11601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7435,"num_write_kb":361664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5210,"num_bytes_recovered":341442560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":99,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.167013+0000","last_change":"2024-06-27T21:21:53.725857+0000","last_active":"2024-06-27T21:22:39.167013+0000","last_peered":"2024-06-27T21:22:39.167013+0000","last_clean":"2024-06-27T21:22:39.167013+0000","last_became_active":"2024-06-27T21:21:18.602559+0000","last_became_peered":"2024-06-27T21:21:18.602559+0000","last_unstale":"2024-06-27T21:22:39.167013+0000","last_undegraded":"2024-06-27T21:22:39.167013+0000","last_fullsized":"2024-06-27T21:22:39.167013+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:53.725753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:53.725753+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:19.909886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"341'7326","reported_seq":7456,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.364858+0000","last_change":"2024-06-27T21:20:33.676832+0000","last_active":"2024-06-27T21:22:40.364858+0000","last_peered":"2024-06-27T21:22:40.364858+0000","last_clean":"2024-06-27T21:22:40.364858+0000","last_became_active":"2024-06-27T21:20:33.676501+0000","last_became_peered":"2024-06-27T21:20:33.676501+0000","last_unstale":"2024-06-27T21:22:40.364858+0000","last_undegraded":"2024-06-27T21:22:40.364858+0000","last_fullsized":"2024-06-27T21:22:40.364858+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7326,"log_dups_size":0,"ondisk_log_size":7326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:40.547312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264634368,"num_objects":4038,"num_object_clones":0,"num_object_copies":12114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":363648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,11],"acting":[9,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":436,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.104689+0000","last_change":"2024-06-27T21:22:14.628532+0000","last_active":"2024-06-27T21:22:40.104689+0000","last_peered":"2024-06-27T21:22:40.104689+0000","last_clean":"2024-06-27T21:22:40.104689+0000","last_became_active":"2024-06-27T21:21:18.602789+0000","last_became_peered":"2024-06-27T21:21:18.602789+0000","last_unstale":"2024-06-27T21:22:40.104689+0000","last_undegraded":"2024-06-27T21:22:40.104689+0000","last_fullsized":"2024-06-27T21:22:40.104689+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:22:14.628428+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:22:14.628428+0000","last_clean_scrub_stamp":"2024-06-27T21:22:14.628428+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:34.302214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"340'7239","reported_seq":7421,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.521420+0000","last_change":"2024-06-27T21:21:54.579950+0000","last_active":"2024-06-27T21:22:39.521420+0000","last_peered":"2024-06-27T21:22:39.521420+0000","last_clean":"2024-06-27T21:22:39.521420+0000","last_became_active":"2024-06-27T21:20:33.961025+0000","last_became_peered":"2024-06-27T21:20:33.961025+0000","last_unstale":"2024-06-27T21:22:39.521420+0000","last_undegraded":"2024-06-27T21:22:39.521420+0000","last_fullsized":"2024-06-27T21:22:39.521420+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'4638","last_scrub_stamp":"2024-06-27T21:21:54.579827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:21:54.579827+0000","objects_scrubbed":4609,"log_size":7239,"log_dups_size":0,"ondisk_log_size":7239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:59.806190+0000","scrub_duration":1127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271908864,"num_objects":4149,"num_object_clones":0,"num_object_copies":12447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":364416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":99,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.167056+0000","last_change":"2024-06-27T21:21:52.769965+0000","last_active":"2024-06-27T21:22:39.167056+0000","last_peered":"2024-06-27T21:22:39.167056+0000","last_clean":"2024-06-27T21:22:39.167056+0000","last_became_active":"2024-06-27T21:21:18.602398+0000","last_became_peered":"2024-06-27T21:21:18.602398+0000","last_unstale":"2024-06-27T21:22:39.167056+0000","last_undegraded":"2024-06-27T21:22:39.167056+0000","last_fullsized":"2024-06-27T21:22:39.167056+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:52.769856+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:19.344541+0000","last_clean_scrub_stamp":"2024-06-27T21:21:52.769856+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:19.000088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.0","version":"341'7503","reported_seq":7633,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:42.356202+0000","last_change":"2024-06-27T21:20:33.961484+0000","last_active":"2024-06-27T21:22:42.356202+0000","last_peered":"2024-06-27T21:22:42.356202+0000","last_clean":"2024-06-27T21:22:42.356202+0000","last_became_active":"2024-06-27T21:20:33.961176+0000","last_became_peered":"2024-06-27T21:20:33.961176+0000","last_unstale":"2024-06-27T21:22:42.356202+0000","last_undegraded":"2024-06-27T21:22:42.356202+0000","last_fullsized":"2024-06-27T21:22:42.356202+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7503,"log_dups_size":0,"ondisk_log_size":7503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:43.987334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265748480,"num_objects":4055,"num_object_clones":0,"num_object_copies":12165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":369856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":87,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.167055+0000","last_change":"2024-06-27T21:21:34.494521+0000","last_active":"2024-06-27T21:22:39.167055+0000","last_peered":"2024-06-27T21:22:39.167055+0000","last_clean":"2024-06-27T21:22:39.167055+0000","last_became_active":"2024-06-27T21:21:34.494291+0000","last_became_peered":"2024-06-27T21:21:34.494291+0000","last_unstale":"2024-06-27T21:22:39.167055+0000","last_undegraded":"2024-06-27T21:22:39.167055+0000","last_fullsized":"2024-06-27T21:22:39.167055+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:21:18.604618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:41.763291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1","version":"340'7201","reported_seq":7394,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.552124+0000","last_change":"2024-06-27T21:22:13.108908+0000","last_active":"2024-06-27T21:22:39.552124+0000","last_peered":"2024-06-27T21:22:39.552124+0000","last_clean":"2024-06-27T21:22:39.552124+0000","last_became_active":"2024-06-27T21:20:33.676398+0000","last_became_peered":"2024-06-27T21:20:33.676398+0000","last_unstale":"2024-06-27T21:22:39.552124+0000","last_undegraded":"2024-06-27T21:22:39.552124+0000","last_fullsized":"2024-06-27T21:22:39.552124+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5605","last_scrub_stamp":"2024-06-27T21:22:13.108655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:22:13.108655+0000","objects_scrubbed":5556,"log_size":7201,"log_dups_size":0,"ondisk_log_size":7201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:20.479287+0000","scrub_duration":2636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266272768,"num_objects":4063,"num_object_clones":0,"num_object_copies":12189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":360448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":501,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.119743+0000","last_change":"2024-06-27T21:21:11.523707+0000","last_active":"2024-06-27T21:22:39.119743+0000","last_peered":"2024-06-27T21:22:39.119743+0000","last_clean":"2024-06-27T21:22:39.119743+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:22:39.119743+0000","last_undegraded":"2024-06-27T21:22:39.119743+0000","last_fullsized":"2024-06-27T21:22:39.119743+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:11.523606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:11.523606+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:42.574394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.f","version":"342'7383","reported_seq":7513,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-06-27T21:22:43.762175+0000","last_change":"2024-06-27T21:20:33.674679+0000","last_active":"2024-06-27T21:22:43.762175+0000","last_peered":"2024-06-27T21:22:43.762175+0000","last_clean":"2024-06-27T21:22:43.762175+0000","last_became_active":"2024-06-27T21:20:33.674388+0000","last_became_peered":"2024-06-27T21:20:33.674388+0000","last_unstale":"2024-06-27T21:22:43.762175+0000","last_undegraded":"2024-06-27T21:22:43.762175+0000","last_fullsized":"2024-06-27T21:22:43.762175+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7383,"log_dups_size":0,"ondisk_log_size":7383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:46.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256442368,"num_objects":3913,"num_object_clones":0,"num_object_copies":11739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":361472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":407,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.111537+0000","last_change":"2024-06-27T21:20:31.763820+0000","last_active":"2024-06-27T21:22:40.111537+0000","last_peered":"2024-06-27T21:22:40.111537+0000","last_clean":"2024-06-27T21:22:40.111537+0000","last_became_active":"2024-06-27T21:14:55.869341+0000","last_became_peered":"2024-06-27T21:14:55.869341+0000","last_unstale":"2024-06-27T21:22:40.111537+0000","last_undegraded":"2024-06-27T21:22:40.111537+0000","last_fullsized":"2024-06-27T21:22:40.111537+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:20:31.763737+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:20:31.763737+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:40.547441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"341'7265","reported_seq":7400,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:42.375582+0000","last_change":"2024-06-27T21:21:02.542049+0000","last_active":"2024-06-27T21:22:42.375582+0000","last_peered":"2024-06-27T21:22:42.375582+0000","last_clean":"2024-06-27T21:22:42.375582+0000","last_became_active":"2024-06-27T21:21:02.540603+0000","last_became_peered":"2024-06-27T21:21:02.540603+0000","last_unstale":"2024-06-27T21:22:42.375582+0000","last_undegraded":"2024-06-27T21:22:42.375582+0000","last_fullsized":"2024-06-27T21:22:42.375582+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7265,"log_dups_size":0,"ondisk_log_size":7265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:43.987311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252379136,"num_objects":3851,"num_object_clones":0,"num_object_copies":11553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":355712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":717,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-06-27T21:22:39.132946+0000","last_change":"2024-06-27T21:21:51.072163+0000","last_active":"2024-06-27T21:22:39.132946+0000","last_peered":"2024-06-27T21:22:39.132946+0000","last_clean":"2024-06-27T21:22:39.132946+0000","last_became_active":"2024-06-27T21:21:34.485764+0000","last_became_peered":"2024-06-27T21:21:34.485764+0000","last_unstale":"2024-06-27T21:22:39.132946+0000","last_undegraded":"2024-06-27T21:22:39.132946+0000","last_fullsized":"2024-06-27T21:22:39.132946+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:51.072093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:51.072093+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:59.581088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"341'7316","reported_seq":7450,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.406827+0000","last_change":"2024-06-27T21:21:02.542379+0000","last_active":"2024-06-27T21:22:40.406827+0000","last_peered":"2024-06-27T21:22:40.406827+0000","last_clean":"2024-06-27T21:22:40.406827+0000","last_became_active":"2024-06-27T21:21:02.541097+0000","last_became_peered":"2024-06-27T21:21:02.541097+0000","last_unstale":"2024-06-27T21:22:40.406827+0000","last_undegraded":"2024-06-27T21:22:40.406827+0000","last_fullsized":"2024-06-27T21:22:40.406827+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":282,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:20:32.657175+0000","last_clean_scrub_stamp":"2024-06-27T21:20:32.657175+0000","objects_scrubbed":0,"log_size":7316,"log_dups_size":0,"ondisk_log_size":7316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:22:40.547357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520280,"num_objects":4022,"num_object_clones":0,"num_object_copies":12066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7316,"num_write_kb":362753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":433,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-06-27T21:22:40.112591+0000","last_change":"2024-06-27T21:21:50.811098+0000","last_active":"2024-06-27T21:22:40.112591+0000","last_peered":"2024-06-27T21:22:40.112591+0000","last_clean":"2024-06-27T21:22:40.112591+0000","last_became_active":"2024-06-27T21:21:34.487122+0000","last_became_peered":"2024-06-27T21:21:34.487122+0000","last_unstale":"2024-06-27T21:22:40.112591+0000","last_undegraded":"2024-06-27T21:22:40.112591+0000","last_fullsized":"2024-06-27T21:22:40.112591+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:21:50.810976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:21:50.810976+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:23:11.015602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4228907032,"num_objects":64529,"num_object_clones":0,"num_object_copies":193587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116945,"num_write_kb":5807105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21915,"num_bytes_recovered":1436221440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12606255104,"data_stored":12606242888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116945,"ondisk_log_size":116945,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253024,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2157996,"kb_used_data":1842208,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91587156,"statfs":{"total":95995035648,"available":93785247744,"internally_reserved":0,"allocated":1886420992,"data_stored":1886140777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514661,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368348,"kb_used_data":1192788,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92376804,"statfs":{"total":95995035648,"available":94593847296,"internally_reserved":0,"allocated":1221414912,"data_stored":1221139237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743593,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975916,"kb_used_data":786468,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92769236,"statfs":{"total":95995035648,"available":94995697664,"internally_reserved":0,"allocated":805343232,"data_stored":805058945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":193984960},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180792,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564360,"statfs":{"total":95995035648,"available":95809904640,"internally_reserved":0,"allocated":1093632,"data_stored":808919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1045680,"kb_used_data":802080,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92699472,"statfs":{"total":95995035648,"available":94924259328,"internally_reserved":0,"allocated":821329920,"data_stored":821049705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":249428426},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1918768,"kb_used_data":1576544,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91826384,"statfs":{"total":95995035648,"available":94030217216,"internally_reserved":0,"allocated":1614381056,"data_stored":1614100841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757438,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":629440,"kb_used_data":503860,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93115712,"statfs":{"total":95995035648,"available":95350489088,"internally_reserved":0,"allocated":515952640,"data_stored":515664375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986370,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1270780,"kb_used_data":1004976,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92474372,"statfs":{"total":95995035648,"available":94693756928,"internally_reserved":0,"allocated":1029095424,"data_stored":1028806639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182598,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546240,"kb_used_data":1272116,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92198912,"statfs":{"total":95995035648,"available":94411685888,"internally_reserved":0,"allocated":1302646784,"data_stored":1302358519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":811196,"kb_used_data":667572,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92933956,"statfs":{"total":95995035648,"available":95164370944,"internally_reserved":0,"allocated":683593728,"data_stored":683305463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2585784,"kb_used_data":2265776,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91159368,"statfs":{"total":95995035648,"available":93347192832,"internally_reserved":0,"allocated":2320154624,"data_stored":2319865839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738580,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070696,"kb_used_data":796576,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92674456,"statfs":{"total":95995035648,"available":94898642944,"internally_reserved":0,"allocated":815693824,"data_stored":815413609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":805437440,"data_stored":805437440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2290094080,"data_stored":2290090008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":509476864,"data_stored":509476864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1293615104,"data_stored":1293615104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1025380352,"data_stored":1025376280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":510656512,"data_stored":510656512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1590689792,"data_stored":1590689792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":815595520,"data_stored":815595520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":795283456,"data_stored":795279384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1085800448,"data_stored":1085800448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1884225536,"data_stored":1884225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:22:47.203 INFO:tasks.thrashosds.thrasher:Setting 4.4 to [7, 6, 9] 2024-06-27T21:22:47.203 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.4', '7', '6', '9'] 2024-06-27T21:22:47.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap 4.4 7 6 9 2024-06-27T21:22:47.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:47.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:47.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:47.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:47.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:47.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:47.667 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:47.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:47.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:47 smithi151 ceph-mon[28965]: pgmap v903: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:22:47.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:47 smithi190 ceph-mon[32247]: pgmap v903: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:22:47.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:47.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:47 smithi192 ceph-mon[32956]: pgmap v903: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:22:47.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:47.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:47.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:47.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:47.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:48.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:48.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:48.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:48.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:48.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:22:48.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:48.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:48.407 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:48.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:48.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:48.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:48.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:48.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:48.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:48.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:48.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:48.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:48.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:48.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:49.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:49.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:49.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:49.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:49.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:49.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:49.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:49.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:49.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:49.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:49.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:49.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:49 smithi190 ceph-mon[32247]: pgmap v904: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-06-27T21:22:49.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:49.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:49 smithi192 ceph-mon[32956]: pgmap v904: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-06-27T21:22:49.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:49.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:49 smithi151 ceph-mon[28965]: pgmap v904: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s 2024-06-27T21:22:49.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:49.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:50.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:50.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:50.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:50.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:50.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:50.283 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:50.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:50.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:50.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:50.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:50.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:50.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:50.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:50.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:50.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:50.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:50.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:50.972 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:51.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:51.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:51.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:51.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:51.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:51.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:51.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:51.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:51.516 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:22:51.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:51.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:51.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:51.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:51.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:51 smithi192 ceph-mon[32956]: pgmap v905: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-06-27T21:22:51.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2821455745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:22:51.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:22:51.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:51.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:51.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:51 smithi151 ceph-mon[28965]: pgmap v905: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-06-27T21:22:51.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2821455745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:22:51.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:22:51.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:51.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:51.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:51 smithi190 ceph-mon[32247]: pgmap v905: 27 pgs: 27 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-06-27T21:22:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2821455745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:22:52.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:22:52.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:52.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:52.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:52.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:52.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:52.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:52.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:52.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:52.485 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:22:52.485 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:127479 4.4 4.e5c08354 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:52.482326+0000", 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0028709220000000001, 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028903539999999999, 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:22:52.486 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:22:52.487 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-06-27T21:22:52.487 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:22:52.487 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:22:52.487 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:22:52.487 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127479 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.482326+0000", 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.482326+0000", 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.482327+0000", 2024-06-27T21:22:52.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.482333+0000", 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.356e-06 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.482343+0000", 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0685e-05 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.482350+0000", 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5390000000000004e-06 2024-06-27T21:22:52.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:52.485224+0000", 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028737400000000001 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:52.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:22:52.493 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:22:52.493 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-06-27T21:22:52.493 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:22:52.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:52.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:52.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:52.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:52.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:52 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:52.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:52 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:22:52.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:52 smithi192 ceph-mon[32956]: osdmap e344: 12 total, 12 up, 12 in 2024-06-27T21:22:52.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:52 smithi192 ceph-mon[32956]: 4.4 scrub starts 2024-06-27T21:22:52.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:52.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:52.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:52 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:52.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:52 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:22:52.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:52 smithi151 ceph-mon[28965]: osdmap e344: 12 total, 12 up, 12 in 2024-06-27T21:22:52.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:52 smithi151 ceph-mon[28965]: 4.4 scrub starts 2024-06-27T21:22:52.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:52.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:52.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:53.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:52 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:22:53.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:52 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:22:53.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:52 smithi190 ceph-mon[32247]: osdmap e344: 12 total, 12 up, 12 in 2024-06-27T21:22:53.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:52 smithi190 ceph-mon[32247]: 4.4 scrub starts 2024-06-27T21:22:53.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:53.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:53.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:53.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:53.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:53.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:53.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:53.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:53.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:53.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:22:53.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:53.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:53.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:53.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:53 smithi192 ceph-mon[32956]: pgmap v907: 27 pgs: 27 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-06-27T21:22:53.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:53 smithi192 ceph-mon[32956]: 4.4 scrub ok 2024-06-27T21:22:53.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:53.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:53.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:53 smithi151 ceph-mon[28965]: pgmap v907: 27 pgs: 27 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-06-27T21:22:53.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:53 smithi151 ceph-mon[28965]: 4.4 scrub ok 2024-06-27T21:22:53.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:54.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:53 smithi190 ceph-mon[32247]: pgmap v907: 27 pgs: 27 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-06-27T21:22:54.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:53 smithi190 ceph-mon[32247]: 4.4 scrub ok 2024-06-27T21:22:54.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:54.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:54.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:54.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:54.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:54.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:54.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:54.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:54.559 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:54.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:54.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:54.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:54 smithi151 ceph-mon[28965]: 4.5 scrub starts 2024-06-27T21:22:54.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:54.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:54.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:54.910 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:54.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:55.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:54 smithi190 ceph-mon[32247]: 4.5 scrub starts 2024-06-27T21:22:55.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:55.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:55.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:55.042 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:54 smithi192 ceph-mon[32956]: 4.5 scrub starts 2024-06-27T21:22:55.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:55.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:55.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:55.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:55.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:55.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:55.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:55.547 INFO:teuthology.orchestra.run.smithi151.stderr:set 4.4 pg_upmap mapping to [7,6,9] 2024-06-27T21:22:55.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:55.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:55.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:55.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:55.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:55.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:55.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:55.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:55 smithi151 ceph-mon[28965]: pgmap v908: 27 pgs: 27 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s 2024-06-27T21:22:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:55 smithi151 ceph-mon[28965]: 4.5 scrub ok 2024-06-27T21:22:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:55 smithi151 ceph-mon[28965]: 4.7 scrub starts 2024-06-27T21:22:55.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/241896721' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.4", "id": [7, 6, 9]}]: dispatch 2024-06-27T21:22:56.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:55 smithi190 ceph-mon[32247]: pgmap v908: 27 pgs: 27 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s 2024-06-27T21:22:56.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:55 smithi190 ceph-mon[32247]: 4.5 scrub ok 2024-06-27T21:22:56.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:55 smithi190 ceph-mon[32247]: 4.7 scrub starts 2024-06-27T21:22:56.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/241896721' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.4", "id": [7, 6, 9]}]: dispatch 2024-06-27T21:22:56.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:56.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:22:56.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:55 smithi192 ceph-mon[32956]: pgmap v908: 27 pgs: 27 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s 2024-06-27T21:22:56.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:55 smithi192 ceph-mon[32956]: 4.5 scrub ok 2024-06-27T21:22:56.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:55 smithi192 ceph-mon[32956]: 4.7 scrub starts 2024-06-27T21:22:56.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/241896721' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.4", "id": [7, 6, 9]}]: dispatch 2024-06-27T21:22:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:56.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:56.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:56.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:56.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:56.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:56.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:56.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:56.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:22:56.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:22:56.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:56.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:56.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/241896721' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.4", "id": [7, 6, 9]}]': finished 2024-06-27T21:22:56.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:56 smithi151 ceph-mon[28965]: osdmap e345: 12 total, 12 up, 12 in 2024-06-27T21:22:56.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:56 smithi151 ceph-mon[28965]: 4.2 scrub starts 2024-06-27T21:22:56.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:56 smithi151 ceph-mon[28965]: 4.2 scrub ok 2024-06-27T21:22:56.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:56.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:56.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:56.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:56.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:56.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/241896721' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.4", "id": [7, 6, 9]}]': finished 2024-06-27T21:22:56.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:56 smithi192 ceph-mon[32956]: osdmap e345: 12 total, 12 up, 12 in 2024-06-27T21:22:56.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:56 smithi192 ceph-mon[32956]: 4.2 scrub starts 2024-06-27T21:22:56.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:56 smithi192 ceph-mon[32956]: 4.2 scrub ok 2024-06-27T21:22:57.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/241896721' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.4", "id": [7, 6, 9]}]': finished 2024-06-27T21:22:57.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:56 smithi190 ceph-mon[32247]: osdmap e345: 12 total, 12 up, 12 in 2024-06-27T21:22:57.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:56 smithi190 ceph-mon[32247]: 4.2 scrub starts 2024-06-27T21:22:57.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:56 smithi190 ceph-mon[32247]: 4.2 scrub ok 2024-06-27T21:22:57.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:57.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:57.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:57.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:57.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:57.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:57.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:57.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:57.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:57.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:57.607 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:57 smithi151 ceph-mon[28965]: pgmap v910: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-06-27T21:22:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:57 smithi151 ceph-mon[28965]: osdmap e346: 12 total, 12 up, 12 in 2024-06-27T21:22:57.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:57 smithi151 ceph-mon[28965]: 4.7 scrub ok 2024-06-27T21:22:57.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:57.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:57.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:22:57.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:22:57.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:57.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:22:57.771 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:22:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:57.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:57.799 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:22:57.799 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:22:57.821 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:22:57.821 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:22:57.844 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:22:57.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:22:57.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:22:57.866 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:22:57.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:22:57.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:22:57.896 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:22:57.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:57.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:22:57.922 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:22:57.923 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:22:57.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:22:57.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:22:57.980 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:22:57.981 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:22:58.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:57 smithi190 ceph-mon[32247]: pgmap v910: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-06-27T21:22:58.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:57 smithi190 ceph-mon[32247]: osdmap e346: 12 total, 12 up, 12 in 2024-06-27T21:22:58.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:57 smithi190 ceph-mon[32247]: 4.7 scrub ok 2024-06-27T21:22:58.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:22:58.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:22:58.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:22:58.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:57 smithi192 ceph-mon[32956]: pgmap v910: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-06-27T21:22:58.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:57 smithi192 ceph-mon[32956]: osdmap e346: 12 total, 12 up, 12 in 2024-06-27T21:22:58.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:57 smithi192 ceph-mon[32956]: 4.7 scrub ok 2024-06-27T21:22:58.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:22:58.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:22:58.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:22:58.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:22:58.172 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:22:58.202 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:22:58.202 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:22:58.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:22:58.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:22:58.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:58.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:22:58.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:22:58.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:22:58.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:58.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:58.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:58 smithi151 ceph-mon[28965]: osdmap e347: 12 total, 12 up, 12 in 2024-06-27T21:22:58.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/38091963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:58.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:58.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:58.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:22:58.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:58.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:22:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:58 smithi190 ceph-mon[32247]: osdmap e347: 12 total, 12 up, 12 in 2024-06-27T21:22:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/38091963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:59.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:59.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:58 smithi192 ceph-mon[32956]: osdmap e347: 12 total, 12 up, 12 in 2024-06-27T21:22:59.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/38091963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:22:59.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:59.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:22:59.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:22:59.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:22:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:22:59.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:22:59.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:59.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:59.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:59.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:59.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:22:59.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:22:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:22:59.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:22:59.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:22:59.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:22:59.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:59 smithi151 ceph-mon[28965]: pgmap v913: 27 pgs: 27 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s 2024-06-27T21:22:59.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:59 smithi151 ceph-mon[28965]: 4.b scrub starts 2024-06-27T21:22:59.867 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:22:59 smithi151 ceph-mon[28965]: 4.c scrub starts 2024-06-27T21:22:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:59.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:22:59.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:22:59.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:22:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:22:59.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:00.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:59 smithi190 ceph-mon[32247]: pgmap v913: 27 pgs: 27 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s 2024-06-27T21:23:00.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:59 smithi190 ceph-mon[32247]: 4.b scrub starts 2024-06-27T21:23:00.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:22:59 smithi190 ceph-mon[32247]: 4.c scrub starts 2024-06-27T21:23:00.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:59 smithi192 ceph-mon[32956]: pgmap v913: 27 pgs: 27 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s 2024-06-27T21:23:00.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:59 smithi192 ceph-mon[32956]: 4.b scrub starts 2024-06-27T21:23:00.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:22:59 smithi192 ceph-mon[32956]: 4.c scrub starts 2024-06-27T21:23:00.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:00.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:00.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:00.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:00.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:00.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:00.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:23:00.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:00.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:00.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:00.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:00.822 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:00.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:00.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:00.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:00.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:00 smithi151 ceph-mon[28965]: 4.b scrub ok 2024-06-27T21:23:00.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:00 smithi151 ceph-mon[28965]: 4.0 scrub starts 2024-06-27T21:23:00.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:00 smithi151 ceph-mon[28965]: 4.c scrub ok 2024-06-27T21:23:00.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:01.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:00 smithi190 ceph-mon[32247]: 4.b scrub ok 2024-06-27T21:23:01.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:00 smithi190 ceph-mon[32247]: 4.0 scrub starts 2024-06-27T21:23:01.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:00 smithi190 ceph-mon[32247]: 4.c scrub ok 2024-06-27T21:23:01.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:01.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:00 smithi192 ceph-mon[32956]: 4.b scrub ok 2024-06-27T21:23:01.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:00 smithi192 ceph-mon[32956]: 4.0 scrub starts 2024-06-27T21:23:01.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:00 smithi192 ceph-mon[32956]: 4.c scrub ok 2024-06-27T21:23:01.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:01.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:01.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:01.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:01.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:23:01.306 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:23:01.306 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:23:01.307 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-27T21:23:01.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4 2024-06-27T21:23:01.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:01.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:01.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:01.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:01.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:01.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:01.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:01 smithi190 ceph-mon[32247]: pgmap v914: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 9465/151950 objects misplaced (6.229%); 608 KiB/s, 9 objects/s recovering 2024-06-27T21:23:01.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:01 smithi190 ceph-mon[32247]: 4.0 scrub ok 2024-06-27T21:23:01.700 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:01 smithi190 systemd[1]: Stopping Ceph osd.4 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:23:01.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:23:01.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:01.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:01.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:01.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:01.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:01.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:01.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:01 smithi151 ceph-mon[28965]: pgmap v914: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 9465/151950 objects misplaced (6.229%); 608 KiB/s, 9 objects/s recovering 2024-06-27T21:23:01.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:01 smithi151 ceph-mon[28965]: 4.0 scrub ok 2024-06-27T21:23:02.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[36726]: 2024-06-27T21:23:01.700+0000 7fe62fcb6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:23:02.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[36726]: 2024-06-27T21:23:01.700+0000 7fe62fcb6640 -1 osd.4 347 *** Got signal Terminated *** 2024-06-27T21:23:02.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[36726]: 2024-06-27T21:23:01.700+0000 7fe62fcb6640 -1 osd.4 347 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:23:02.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:02.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:02.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:01 smithi192 ceph-mon[32956]: pgmap v914: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 9465/151950 objects misplaced (6.229%); 608 KiB/s, 9 objects/s recovering 2024-06-27T21:23:02.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:01 smithi192 ceph-mon[32956]: 4.0 scrub ok 2024-06-27T21:23:02.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:02.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:02.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:02.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:02.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:02.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:02.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:02.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:02.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:02.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:02.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:02.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:02.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:02 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:23:02.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:02 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:23:02.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:02 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:23:02.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:02 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:23:02.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:02 smithi151 ceph-mon[28965]: osd.4 marked itself down and dead 2024-06-27T21:23:02.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:02.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:02.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:02 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:23:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:02 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:23:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:02 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:23:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:02 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:23:03.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:02 smithi190 ceph-mon[32247]: osd.4 marked itself down and dead 2024-06-27T21:23:03.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:03.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:02 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:23:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:02 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:23:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:02 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:23:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:02 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:23:03.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:02 smithi192 ceph-mon[32956]: osd.4 marked itself down and dead 2024-06-27T21:23:03.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:03.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:03.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:03.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:03.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:03.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:03.482 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:03 smithi190 podman[96676]: 2024-06-27 21:23:03.184380561 +0000 UTC m=+1.616106543 container died 9fc8767f70e210910e9fb0e876caca540258250affac1ffa4bc32c523eb6df5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:23:03.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:03.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:03.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:03.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:03.681 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:23:03.681 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:23:03.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:03 smithi190 ceph-mon[32247]: pgmap v915: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 404 op/s; 9465/149742 objects misplaced (6.321%); 568 KiB/s, 8 objects/s recovering 2024-06-27T21:23:03.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:03 smithi190 ceph-mon[32247]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-27T21:23:03.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:03 smithi190 ceph-mon[32247]: osdmap e348: 12 total, 11 up, 12 in 2024-06-27T21:23:03.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:03 smithi190 ceph-mon[32247]: 4.3 scrub starts 2024-06-27T21:23:03.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:03 smithi190 podman[96676]: 2024-06-27 21:23:03.675927922 +0000 UTC m=+2.107653912 container cleanup 9fc8767f70e210910e9fb0e876caca540258250affac1ffa4bc32c523eb6df5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux ) 2024-06-27T21:23:03.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:03 smithi190 bash[96676]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4 2024-06-27T21:23:03.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:03.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:03.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:03 smithi151 ceph-mon[28965]: pgmap v915: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 404 op/s; 9465/149742 objects misplaced (6.321%); 568 KiB/s, 8 objects/s recovering 2024-06-27T21:23:03.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:03 smithi151 ceph-mon[28965]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-27T21:23:03.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:03 smithi151 ceph-mon[28965]: osdmap e348: 12 total, 11 up, 12 in 2024-06-27T21:23:03.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:03 smithi151 ceph-mon[28965]: 4.3 scrub starts 2024-06-27T21:23:03.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:03.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:03.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:03 smithi192 ceph-mon[32956]: pgmap v915: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 404 op/s; 9465/149742 objects misplaced (6.321%); 568 KiB/s, 8 objects/s recovering 2024-06-27T21:23:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:03 smithi192 ceph-mon[32956]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-27T21:23:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:03 smithi192 ceph-mon[32956]: osdmap e348: 12 total, 11 up, 12 in 2024-06-27T21:23:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:03 smithi192 ceph-mon[32956]: 4.3 scrub starts 2024-06-27T21:23:04.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:04.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:04.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:04.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:04.262 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:03 smithi190 podman[96724]: 2024-06-27 21:23:03.986407268 +0000 UTC m=+0.797966497 container remove 9fc8767f70e210910e9fb0e876caca540258250affac1ffa4bc32c523eb6df5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux ) 2024-06-27T21:23:04.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:04.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:04.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:04.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:04.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:04.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:04.616 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:04.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:04 smithi190 ceph-mon[32247]: 4.3 scrub ok 2024-06-27T21:23:04.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:04 smithi190 ceph-mon[32247]: osdmap e349: 12 total, 11 up, 12 in 2024-06-27T21:23:04.648 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:04 smithi190 ceph-mon[32247]: 4.9 scrub starts 2024-06-27T21:23:04.648 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 podman[96826]: 2024-06-27 21:23:04.282582349 +0000 UTC m=+0.023765619 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:23:04.648 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 podman[96826]: 2024-06-27 21:23:04.40421071 +0000 UTC m=+0.145393965 container create 0676277a4d9c5d0eb618cb56b5f754cb8108450536e4313e303da05f585ff3d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-27T21:23:04.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:04.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:04.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:04 smithi151 ceph-mon[28965]: 4.3 scrub ok 2024-06-27T21:23:04.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:04 smithi151 ceph-mon[28965]: osdmap e349: 12 total, 11 up, 12 in 2024-06-27T21:23:04.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:04 smithi151 ceph-mon[28965]: 4.9 scrub starts 2024-06-27T21:23:04.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:04.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:04.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:04.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:04.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:05.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 podman[96826]: 2024-06-27 21:23:04.648827531 +0000 UTC m=+0.390010801 container init 0676277a4d9c5d0eb618cb56b5f754cb8108450536e4313e303da05f585ff3d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625) 2024-06-27T21:23:05.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 podman[96826]: 2024-06-27 21:23:04.655743423 +0000 UTC m=+0.396926679 container start 0676277a4d9c5d0eb618cb56b5f754cb8108450536e4313e303da05f585ff3d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:23:05.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 podman[96826]: 2024-06-27 21:23:04.774186522 +0000 UTC m=+0.515369787 container attach 0676277a4d9c5d0eb618cb56b5f754cb8108450536e4313e303da05f585ff3d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:23:05.013 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 conmon[96839]: conmon 0676277a4d9c5d0eb618 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-0676277a4d9c5d0eb618cb56b5f754cb8108450536e4313e303da05f585ff3d2.scope/container/memory.events 2024-06-27T21:23:05.013 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:04 smithi190 podman[96826]: 2024-06-27 21:23:04.889380755 +0000 UTC m=+0.630564013 container died 0676277a4d9c5d0eb618cb56b5f754cb8108450536e4313e303da05f585ff3d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240625, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-27T21:23:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:04 smithi192 ceph-mon[32956]: 4.3 scrub ok 2024-06-27T21:23:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:04 smithi192 ceph-mon[32956]: osdmap e349: 12 total, 11 up, 12 in 2024-06-27T21:23:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:04 smithi192 ceph-mon[32956]: 4.9 scrub starts 2024-06-27T21:23:05.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:05.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:05.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:05.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:05.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:05.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:05.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:23:05.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:05.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:05.599 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:23:05.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4.service' 2024-06-27T21:23:05.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:05.660 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:05 smithi190 ceph-mon[32247]: pgmap v918: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 8817/146931 objects misplaced (6.001%); 2.7 MiB/s, 42 objects/s recovering 2024-06-27T21:23:05.660 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:05 smithi190 ceph-mon[32247]: 4.9 scrub ok 2024-06-27T21:23:05.660 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2780920389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:05.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:23:05.662 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-27T21:23:05.662 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-06-27T21:23:05.662 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi190.front.sepia.ceph.com 2024-06-27T21:23:05.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -n osd.4 --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-06-27T21:23:05.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:05.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:05 smithi151 ceph-mon[28965]: pgmap v918: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 8817/146931 objects misplaced (6.001%); 2.7 MiB/s, 42 objects/s recovering 2024-06-27T21:23:05.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:05 smithi151 ceph-mon[28965]: 4.9 scrub ok 2024-06-27T21:23:05.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2780920389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:05.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:05.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:05.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:05.933 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/osd.4/config 2024-06-27T21:23:05.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:05.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:06.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:05 smithi192 ceph-mon[32956]: pgmap v918: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 8817/146931 objects misplaced (6.001%); 2.7 MiB/s, 42 objects/s recovering 2024-06-27T21:23:06.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:05 smithi192 ceph-mon[32956]: 4.9 scrub ok 2024-06-27T21:23:06.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2780920389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:06.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:06.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:06.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:06.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:06.342 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:06.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:06.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:06.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:06.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:06.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:06.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:06.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:06.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:06.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:06.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:06 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:23:06.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2780920389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:23:06.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:06 smithi151 ceph-mon[28965]: osdmap e350: 12 total, 11 up, 12 in 2024-06-27T21:23:06.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:06.987 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:06.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:06.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:07.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:06 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:23:07.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2780920389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:23:07.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:06 smithi190 ceph-mon[32247]: osdmap e350: 12 total, 11 up, 12 in 2024-06-27T21:23:07.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:06 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:23:07.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2780920389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:23:07.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:06 smithi192 ceph-mon[32956]: osdmap e350: 12 total, 11 up, 12 in 2024-06-27T21:23:07.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:07.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:07.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:07.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:07.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:07.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:07.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:07.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:07.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:07.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:07.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:07.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:07.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:07 smithi151 ceph-mon[28965]: pgmap v920: 27 pgs: 3 active+undersized, 1 active+recovering+undersized+remapped, 3 active+undersized+degraded, 20 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 8693/140505 objects degraded (6.187%); 8817/140505 objects misplaced (6.275%); 2.2 MiB/s, 36 objects/s recovering 2024-06-27T21:23:07.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:07 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 8693/140505 objects degraded (6.187%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:07.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:07.895 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:07.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:07.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:08.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:07 smithi190 ceph-mon[32247]: pgmap v920: 27 pgs: 3 active+undersized, 1 active+recovering+undersized+remapped, 3 active+undersized+degraded, 20 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 8693/140505 objects degraded (6.187%); 8817/140505 objects misplaced (6.275%); 2.2 MiB/s, 36 objects/s recovering 2024-06-27T21:23:08.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:07 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 8693/140505 objects degraded (6.187%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:08.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:08.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:07 smithi192 ceph-mon[32956]: pgmap v920: 27 pgs: 3 active+undersized, 1 active+recovering+undersized+remapped, 3 active+undersized+degraded, 20 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 8693/140505 objects degraded (6.187%); 8817/140505 objects misplaced (6.275%); 2.2 MiB/s, 36 objects/s recovering 2024-06-27T21:23:08.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:07 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 8693/140505 objects degraded (6.187%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:08.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:08.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:08.173 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-27T21:23:08.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:08.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:08.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:08.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:08.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:08.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:23:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:08.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:08.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:08.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:08.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:08.799 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:23:08.800 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:23:08.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:08.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:23:08.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:23:08.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:23:08.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:23:08.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:08.915 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:23:08.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:23:08.915 INFO:teuthology.orchestra.run.smithi151.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-27T21:23:08.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:08.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:08.942 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:23:08.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:23:08.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:23:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:23:08.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:23:08.989 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:23:09.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:23:09.010 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:23:09.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:09.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:23:09.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:23:09.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:09.071 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:23:09.072 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:23:09.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:23:09.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:23:09.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:09.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:09.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:09.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:09.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:09.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:09.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:09.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:09.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:09.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:09.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:09.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:4.d 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:4.e 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:4.c 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:4.b 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:1.a 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:1.3 2024-06-27T21:23:09.937 INFO:teuthology.orchestra.run.smithi190.stdout:1.2 2024-06-27T21:23:09.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:09 smithi190 ceph-mon[32247]: pgmap v921: 27 pgs: 3 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 19 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 11421/139362 objects degraded (8.195%); 8817/139362 objects misplaced (6.327%); 2.2 MiB/s, 36 objects/s recovering 2024-06-27T21:23:09.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:09.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:09.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:09 smithi190 ceph-mon[32247]: osdmap e351: 12 total, 11 up, 12 in 2024-06-27T21:23:09.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1932270199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:09.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:09.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:09 smithi151 ceph-mon[28965]: pgmap v921: 27 pgs: 3 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 19 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 11421/139362 objects degraded (8.195%); 8817/139362 objects misplaced (6.327%); 2.2 MiB/s, 36 objects/s recovering 2024-06-27T21:23:09.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:09.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:09.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:09 smithi151 ceph-mon[28965]: osdmap e351: 12 total, 11 up, 12 in 2024-06-27T21:23:09.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1932270199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:09.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:09.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:10.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:09 smithi192 ceph-mon[32956]: pgmap v921: 27 pgs: 3 active+undersized, 1 active+recovering+undersized+remapped, 4 active+undersized+degraded, 19 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 576 op/s; 11421/139362 objects degraded (8.195%); 8817/139362 objects misplaced (6.327%); 2.2 MiB/s, 36 objects/s recovering 2024-06-27T21:23:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:09 smithi192 ceph-mon[32956]: osdmap e351: 12 total, 11 up, 12 in 2024-06-27T21:23:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1932270199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:10.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:10.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:10.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:10.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:10.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:10.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:10.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:10.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:10.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:10.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:10.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:10.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:10.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:10.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:10.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -n osd.4 --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 4.d --file /var/log/ceph/exp.4.d.4 2024-06-27T21:23:10.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:10.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:10.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:11.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:11.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:11.161 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/osd.4/config 2024-06-27T21:23:11.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:11.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:11.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.370 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:11.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:11.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:11.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:11.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:11.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:11.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:11 smithi151 ceph-mon[28965]: pgmap v923: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 11195/133596 objects degraded (8.380%); 8123/133596 objects misplaced (6.080%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:23:11.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:11 smithi151 ceph-mon[28965]: 4.8 scrub starts 2024-06-27T21:23:11.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:11.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:11.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:11.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:11.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:11.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:12.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:11 smithi190 ceph-mon[32247]: pgmap v923: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 11195/133596 objects degraded (8.380%); 8123/133596 objects misplaced (6.080%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:23:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:11 smithi190 ceph-mon[32247]: 4.8 scrub starts 2024-06-27T21:23:12.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:11 smithi192 ceph-mon[32956]: pgmap v923: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 11195/133596 objects degraded (8.380%); 8123/133596 objects misplaced (6.080%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:23:12.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:11 smithi192 ceph-mon[32956]: 4.8 scrub starts 2024-06-27T21:23:12.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:12.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:12.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:12.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:12.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:12.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:12.444 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-27T21:23:12.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:12.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:12.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:12.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:12.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:12.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:12 smithi190 ceph-mon[32247]: 4.8 scrub ok 2024-06-27T21:23:12.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:12.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:12.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:12.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:12 smithi151 ceph-mon[28965]: 4.8 scrub ok 2024-06-27T21:23:12.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:13.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:13.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:12 smithi192 ceph-mon[32956]: 4.8 scrub ok 2024-06-27T21:23:13.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:13.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:13.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:13.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:13.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.406 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:13.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:13.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:13.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:13.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:13.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:13.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:13.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:13.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:13.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:13.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:13 smithi151 ceph-mon[28965]: pgmap v924: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 10717/129420 objects degraded (8.281%); 8123/129420 objects misplaced (6.276%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:23:13.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:13 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 10717/129420 objects degraded (8.281%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:13 smithi190 ceph-mon[32247]: pgmap v924: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 10717/129420 objects degraded (8.281%); 8123/129420 objects misplaced (6.276%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:23:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:13 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 10717/129420 objects degraded (8.281%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:14.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:14.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:13 smithi192 ceph-mon[32956]: pgmap v924: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 10717/129420 objects degraded (8.281%); 8123/129420 objects misplaced (6.276%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:23:14.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:13 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 10717/129420 objects degraded (8.281%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:14.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:14.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:14.172 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:14.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:14.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:14.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:14.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:14.464 INFO:teuthology.orchestra.run.smithi190.stderr:Exporting 4.d info 4.d( v 347'8364 (0'0,347'8364] local-lis/les=301/302 n=2974 ec=282/282 lis/c=301/301 les/c/f=302/302/0 sis=301) 2024-06-27T21:23:14.521 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0004acc:::benchmark_data_smithi192_88961_object85358:head# 2024-06-27T21:23:14.521 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0005339:::benchmark_data_smithi192_88961_object45375:head# 2024-06-27T21:23:14.521 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b000b22e:::benchmark_data_smithi192_88961_object59051:head# 2024-06-27T21:23:14.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b000c10b:::benchmark_data_smithi192_88961_object59216:head# 2024-06-27T21:23:14.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b001fa71:::benchmark_data_smithi192_88961_object64829:head# 2024-06-27T21:23:14.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0031f54:::benchmark_data_smithi192_88961_object70895:head# 2024-06-27T21:23:14.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b004b9ce:::benchmark_data_smithi192_88961_object54652:head# 2024-06-27T21:23:14.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b006a44d:::benchmark_data_smithi192_88961_object72024:head# 2024-06-27T21:23:14.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00af01b:::benchmark_data_smithi192_88961_object69933:head# 2024-06-27T21:23:14.524 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00bdffe:::benchmark_data_smithi192_88961_object51483:head# 2024-06-27T21:23:14.525 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00d6665:::benchmark_data_smithi192_88961_object68732:head# 2024-06-27T21:23:14.525 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00e0b6c:::benchmark_data_smithi192_88961_object76654:head# 2024-06-27T21:23:14.526 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00f6de9:::benchmark_data_smithi192_88961_object50818:head# 2024-06-27T21:23:14.526 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00f7d2b:::benchmark_data_smithi192_88961_object83888:head# 2024-06-27T21:23:14.526 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b00f8b78:::benchmark_data_smithi192_88961_object48522:head# 2024-06-27T21:23:14.527 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b010175c:::benchmark_data_smithi192_88961_object60439:head# 2024-06-27T21:23:14.527 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0102d12:::benchmark_data_smithi192_88961_object62841:head# 2024-06-27T21:23:14.528 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b010f0df:::benchmark_data_smithi192_88961_object71054:head# 2024-06-27T21:23:14.528 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0110fe3:::benchmark_data_smithi192_88961_object67333:head# 2024-06-27T21:23:14.528 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b011cb76:::benchmark_data_smithi192_88961_object65053:head# 2024-06-27T21:23:14.529 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0124027:::benchmark_data_smithi192_88961_object76317:head# 2024-06-27T21:23:14.529 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b01518b6:::benchmark_data_smithi192_88961_object57985:head# 2024-06-27T21:23:14.529 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b015a981:::benchmark_data_smithi192_88961_object61380:head# 2024-06-27T21:23:14.530 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0162506:::benchmark_data_smithi192_88961_object79453:head# 2024-06-27T21:23:14.531 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0169fbf:::benchmark_data_smithi192_88961_object86519:head# 2024-06-27T21:23:14.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:14.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0193e0b:::benchmark_data_smithi192_88961_object85630:head# 2024-06-27T21:23:14.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b019697d:::benchmark_data_smithi192_88961_object90588:head# 2024-06-27T21:23:14.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b01dc879:::benchmark_data_smithi192_88961_object77247:head# 2024-06-27T21:23:14.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b020788d:::benchmark_data_smithi192_88961_object59514:head# 2024-06-27T21:23:14.533 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b020aa7a:::benchmark_data_smithi192_88961_object74975:head# 2024-06-27T21:23:14.533 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0222bc4:::benchmark_data_smithi192_88961_object75710:head# 2024-06-27T21:23:14.533 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0232d80:::benchmark_data_smithi192_88961_object72595:head# 2024-06-27T21:23:14.534 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0258b13:::benchmark_data_smithi192_88961_object53116:head# 2024-06-27T21:23:14.534 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b026131a:::benchmark_data_smithi192_88961_object53908:head# 2024-06-27T21:23:14.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b026a8aa:::benchmark_data_smithi192_88961_object90567:head# 2024-06-27T21:23:14.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b026f576:::benchmark_data_smithi192_88961_object49343:head# 2024-06-27T21:23:14.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0277be1:::benchmark_data_smithi192_88961_object69249:head# 2024-06-27T21:23:14.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b027b239:::benchmark_data_smithi192_88961_object52351:head# 2024-06-27T21:23:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b029ab65:::benchmark_data_smithi192_88961_object83322:head# 2024-06-27T21:23:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02a3f49:::benchmark_data_smithi192_88961_object73312:head# 2024-06-27T21:23:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02bb6f5:::benchmark_data_smithi192_88961_object62462:head# 2024-06-27T21:23:14.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02bf647:::benchmark_data_smithi192_88961_object76587:head# 2024-06-27T21:23:14.538 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02c38eb:::benchmark_data_smithi192_88961_object55655:head# 2024-06-27T21:23:14.538 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02c69eb:::benchmark_data_smithi192_88961_object90462:head# 2024-06-27T21:23:14.538 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02d7de4:::benchmark_data_smithi192_88961_object66053:head# 2024-06-27T21:23:14.539 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02f2ae9:::benchmark_data_smithi192_88961_object74456:head# 2024-06-27T21:23:14.539 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02f3dbe:::benchmark_data_smithi192_88961_object47966:head# 2024-06-27T21:23:14.540 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02fb9b8:::benchmark_data_smithi192_88961_object81127:head# 2024-06-27T21:23:14.541 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b02fcdfa:::benchmark_data_smithi192_88961_object54092:head# 2024-06-27T21:23:14.541 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b030227a:::benchmark_data_smithi192_88961_object68770:head# 2024-06-27T21:23:14.542 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b031ca18:::benchmark_data_smithi192_88961_object76405:head# 2024-06-27T21:23:14.543 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0336e12:::benchmark_data_smithi192_88961_object63493:head# 2024-06-27T21:23:14.543 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0368d87:::benchmark_data_smithi192_88961_object83041:head# 2024-06-27T21:23:14.544 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b038fb94:::benchmark_data_smithi192_88961_object68018:head# 2024-06-27T21:23:14.544 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b03e2c9e:::benchmark_data_smithi192_88961_object77409:head# 2024-06-27T21:23:14.545 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b03e3eb5:::benchmark_data_smithi192_88961_object82517:head# 2024-06-27T21:23:14.545 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b040096c:::benchmark_data_smithi192_88961_object65993:head# 2024-06-27T21:23:14.546 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b040d733:::benchmark_data_smithi192_88961_object46624:head# 2024-06-27T21:23:14.547 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0419461:::benchmark_data_smithi192_88961_object49242:head# 2024-06-27T21:23:14.547 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0447a44:::benchmark_data_smithi192_88961_object83262:head# 2024-06-27T21:23:14.547 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b04c0d46:::benchmark_data_smithi192_88961_object82385:head# 2024-06-27T21:23:14.547 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b04c948a:::benchmark_data_smithi192_88961_object76433:head# 2024-06-27T21:23:14.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b04d5e81:::benchmark_data_smithi192_88961_object85682:head# 2024-06-27T21:23:14.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b050c9f8:::benchmark_data_smithi192_88961_object57758:head# 2024-06-27T21:23:14.549 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b052809e:::benchmark_data_smithi192_88961_object80264:head# 2024-06-27T21:23:14.549 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0537b1f:::benchmark_data_smithi192_88961_object77353:head# 2024-06-27T21:23:14.550 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b053d5c3:::benchmark_data_smithi192_88961_object67875:head# 2024-06-27T21:23:14.550 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b054f09e:::benchmark_data_smithi192_88961_object53749:head# 2024-06-27T21:23:14.550 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0566aae:::benchmark_data_smithi192_88961_object54703:head# 2024-06-27T21:23:14.551 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b056e5de:::benchmark_data_smithi192_88961_object62602:head# 2024-06-27T21:23:14.551 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b05870a0:::benchmark_data_smithi192_88961_object45374:head# 2024-06-27T21:23:14.552 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b05ad45f:::benchmark_data_smithi192_88961_object49744:head# 2024-06-27T21:23:14.552 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b05bb117:::benchmark_data_smithi192_88961_object62031:head# 2024-06-27T21:23:14.552 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b05e04c3:::benchmark_data_smithi192_88961_object45313:head# 2024-06-27T21:23:14.553 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b05e050a:::benchmark_data_smithi192_88961_object79931:head# 2024-06-27T21:23:14.554 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b06212ef:::benchmark_data_smithi192_88961_object50685:head# 2024-06-27T21:23:14.554 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0634a19:::benchmark_data_smithi192_88961_object77450:head# 2024-06-27T21:23:14.555 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b063b9e6:::benchmark_data_smithi192_88961_object87881:head# 2024-06-27T21:23:14.555 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0667678:::benchmark_data_smithi192_88961_object81132:head# 2024-06-27T21:23:14.556 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b06a7adc:::benchmark_data_smithi192_88961_object55965:head# 2024-06-27T21:23:14.556 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b06aa327:::benchmark_data_smithi192_88961_object45848:head# 2024-06-27T21:23:14.557 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b06d102c:::benchmark_data_smithi192_88961_object52892:head# 2024-06-27T21:23:14.557 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b06d2376:::benchmark_data_smithi192_88961_object50294:head# 2024-06-27T21:23:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:14.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:14.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b070ba0f:::benchmark_data_smithi192_88961_object68354:head# 2024-06-27T21:23:14.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b076b237:::benchmark_data_smithi192_88961_object58124:head# 2024-06-27T21:23:14.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b079993a:::benchmark_data_smithi192_88961_object85058:head# 2024-06-27T21:23:14.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b07a86fa:::benchmark_data_smithi192_88961_object44181:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b07cbc11:::benchmark_data_smithi192_88961_object72944:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b07d1cad:::benchmark_data_smithi192_88961_object69610:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b07ec854:::benchmark_data_smithi192_88961_object58772:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b080b09f:::benchmark_data_smithi192_88961_object66274:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b080c6bc:::benchmark_data_smithi192_88961_object51998:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b081f53d:::benchmark_data_smithi192_88961_object88188:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b083c9c9:::benchmark_data_smithi192_88961_object83161:head# 2024-06-27T21:23:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b084a43e:::benchmark_data_smithi192_88961_object59453:head# 2024-06-27T21:23:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0856df6:::benchmark_data_smithi192_88961_object88697:head# 2024-06-27T21:23:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0859ec2:::benchmark_data_smithi192_88961_object57447:head# 2024-06-27T21:23:14.567 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0866c9b:::benchmark_data_smithi192_88961_object47928:head# 2024-06-27T21:23:14.567 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b086800c:::benchmark_data_smithi192_88961_object54678:head# 2024-06-27T21:23:14.567 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b088718f:::benchmark_data_smithi192_88961_object55409:head# 2024-06-27T21:23:14.567 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b08a4988:::benchmark_data_smithi192_88961_object46705:head# 2024-06-27T21:23:14.568 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b08a4ea2:::benchmark_data_smithi192_88961_object76100:head# 2024-06-27T21:23:14.568 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b08dfa38:::benchmark_data_smithi192_88961_object58537:head# 2024-06-27T21:23:14.569 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b08fc276:::benchmark_data_smithi192_88961_object56803:head# 2024-06-27T21:23:14.569 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b08fe8a8:::benchmark_data_smithi192_88961_object82886:head# 2024-06-27T21:23:14.569 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0925c94:::benchmark_data_smithi192_88961_object89131:head# 2024-06-27T21:23:14.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b092facb:::benchmark_data_smithi192_88961_object44040:head# 2024-06-27T21:23:14.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0930b2e:::benchmark_data_smithi192_88961_object56964:head# 2024-06-27T21:23:14.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0935913:::benchmark_data_smithi192_88961_object44000:head# 2024-06-27T21:23:14.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b09478f7:::benchmark_data_smithi192_88961_object61956:head# 2024-06-27T21:23:14.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b094cb19:::benchmark_data_smithi192_88961_object49398:head# 2024-06-27T21:23:14.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b097c219:::benchmark_data_smithi192_88961_object48727:head# 2024-06-27T21:23:14.572 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b097defc:::benchmark_data_smithi192_88961_object82111:head# 2024-06-27T21:23:14.572 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b09bf574:::benchmark_data_smithi192_88961_object58933:head# 2024-06-27T21:23:14.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b09c20bb:::benchmark_data_smithi192_88961_object70897:head# 2024-06-27T21:23:14.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b09cc095:::benchmark_data_smithi192_88961_object45954:head# 2024-06-27T21:23:14.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b09eb751:::benchmark_data_smithi192_88961_object54221:head# 2024-06-27T21:23:14.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b09eef1d:::benchmark_data_smithi192_88961_object60021:head# 2024-06-27T21:23:14.575 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0a20962:::benchmark_data_smithi192_88961_object71088:head# 2024-06-27T21:23:14.575 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0a34672:::benchmark_data_smithi192_88961_object52125:head# 2024-06-27T21:23:14.575 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0a3acaa:::benchmark_data_smithi192_88961_object50120:head# 2024-06-27T21:23:14.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0a6481e:::benchmark_data_smithi192_88961_object59503:head# 2024-06-27T21:23:14.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0a9ae0e:::benchmark_data_smithi192_88961_object71268:head# 2024-06-27T21:23:14.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0aa66e6:::benchmark_data_smithi192_88961_object88933:head# 2024-06-27T21:23:14.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ab3a2b:::benchmark_data_smithi192_88961_object55273:head# 2024-06-27T21:23:14.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ab91ed:::benchmark_data_smithi192_88961_object54663:head# 2024-06-27T21:23:14.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ab9ce9:::benchmark_data_smithi192_88961_object45293:head# 2024-06-27T21:23:14.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0abbad3:::benchmark_data_smithi192_88961_object59220:head# 2024-06-27T21:23:14.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ac17c2:::benchmark_data_smithi192_88961_object61178:head# 2024-06-27T21:23:14.580 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0adc187:::benchmark_data_smithi192_88961_object84046:head# 2024-06-27T21:23:14.580 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0adce32:::benchmark_data_smithi192_88961_object76217:head# 2024-06-27T21:23:14.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0af0a68:::benchmark_data_smithi192_88961_object64779:head# 2024-06-27T21:23:14.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0af599d:::benchmark_data_smithi192_88961_object81791:head# 2024-06-27T21:23:14.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0b0f442:::benchmark_data_smithi192_88961_object78979:head# 2024-06-27T21:23:14.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0b3f864:::benchmark_data_smithi192_88961_object77297:head# 2024-06-27T21:23:14.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0b545e2:::benchmark_data_smithi192_88961_object70879:head# 2024-06-27T21:23:14.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0bfa327:::benchmark_data_smithi192_88961_object60146:head# 2024-06-27T21:23:14.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0bfb53a:::benchmark_data_smithi192_88961_object71444:head# 2024-06-27T21:23:14.584 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c02f76:::benchmark_data_smithi192_88961_object59769:head# 2024-06-27T21:23:14.584 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c25a8e:::benchmark_data_smithi192_88961_object63645:head# 2024-06-27T21:23:14.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c32434:::benchmark_data_smithi192_88961_object66634:head# 2024-06-27T21:23:14.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c35834:::benchmark_data_smithi192_88961_object60336:head# 2024-06-27T21:23:14.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c4538a:::benchmark_data_smithi192_88961_object82054:head# 2024-06-27T21:23:14.586 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c52f0d:::benchmark_data_smithi192_88961_object54066:head# 2024-06-27T21:23:14.586 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c5b7a3:::benchmark_data_smithi192_88961_object54218:head# 2024-06-27T21:23:14.587 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c6264c:::benchmark_data_smithi192_88961_object57775:head# 2024-06-27T21:23:14.587 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0c7ddfc:::benchmark_data_smithi192_88961_object79801:head# 2024-06-27T21:23:14.588 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0caecff:::benchmark_data_smithi192_88961_object86282:head# 2024-06-27T21:23:14.588 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0cb59f8:::benchmark_data_smithi192_88961_object56186:head# 2024-06-27T21:23:14.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0cc8206:::benchmark_data_smithi192_88961_object68155:head# 2024-06-27T21:23:14.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ce4e75:::benchmark_data_smithi192_88961_object72133:head# 2024-06-27T21:23:14.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0cea05d:::benchmark_data_smithi192_88961_object60007:head# 2024-06-27T21:23:14.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0cf854f:::benchmark_data_smithi192_88961_object48471:head# 2024-06-27T21:23:14.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d0d652:::benchmark_data_smithi192_88961_object48811:head# 2024-06-27T21:23:14.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d17ec5:::benchmark_data_smithi192_88961_object46795:head# 2024-06-27T21:23:14.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d21b06:::benchmark_data_smithi192_88961_object44191:head# 2024-06-27T21:23:14.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d25f94:::benchmark_data_smithi192_88961_object45583:head# 2024-06-27T21:23:14.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d304fb:::benchmark_data_smithi192_88961_object55653:head# 2024-06-27T21:23:14.592 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d3f6fa:::benchmark_data_smithi192_88961_object61376:head# 2024-06-27T21:23:14.592 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d42e03:::benchmark_data_smithi192_88961_object68596:head# 2024-06-27T21:23:14.593 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d42f29:::benchmark_data_smithi192_88961_object76017:head# 2024-06-27T21:23:14.593 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d5b7f9:::benchmark_data_smithi192_88961_object52881:head# 2024-06-27T21:23:14.593 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d627b4:::benchmark_data_smithi192_88961_object86940:head# 2024-06-27T21:23:14.594 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d8f0c9:::benchmark_data_smithi192_88961_object78776:head# 2024-06-27T21:23:14.595 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0d8f802:::benchmark_data_smithi192_88961_object50200:head# 2024-06-27T21:23:14.595 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0da9f94:::benchmark_data_smithi192_88961_object50623:head# 2024-06-27T21:23:14.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0dc75c1:::benchmark_data_smithi192_88961_object60856:head# 2024-06-27T21:23:14.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0dcd16f:::benchmark_data_smithi192_88961_object60727:head# 2024-06-27T21:23:14.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0dce0af:::benchmark_data_smithi192_88961_object45562:head# 2024-06-27T21:23:14.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0dda33e:::benchmark_data_smithi192_88961_object60395:head# 2024-06-27T21:23:14.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ddf097:::benchmark_data_smithi192_88961_object72344:head# 2024-06-27T21:23:14.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0e10463:::benchmark_data_smithi192_88961_object59671:head# 2024-06-27T21:23:14.600 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0e30646:::benchmark_data_smithi192_88961_object57366:head# 2024-06-27T21:23:14.600 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0e62f7e:::benchmark_data_smithi192_88961_object64801:head# 2024-06-27T21:23:14.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0e86cc7:::benchmark_data_smithi192_88961_object44363:head# 2024-06-27T21:23:14.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0e9d62e:::benchmark_data_smithi192_88961_object50543:head# 2024-06-27T21:23:14.602 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ec0a6b:::benchmark_data_smithi192_88961_object48343:head# 2024-06-27T21:23:14.602 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ec1822:::benchmark_data_smithi192_88961_object71179:head# 2024-06-27T21:23:14.603 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0ec81b6:::benchmark_data_smithi192_88961_object70335:head# 2024-06-27T21:23:14.603 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0edb2ff:::benchmark_data_smithi192_88961_object52981:head# 2024-06-27T21:23:14.603 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f19f78:::benchmark_data_smithi192_88961_object75435:head# 2024-06-27T21:23:14.604 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f34c09:::benchmark_data_smithi192_88961_object73778:head# 2024-06-27T21:23:14.604 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f428c9:::benchmark_data_smithi192_88961_object58437:head# 2024-06-27T21:23:14.605 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f42a3b:::benchmark_data_smithi192_88961_object54722:head# 2024-06-27T21:23:14.605 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f63a02:::benchmark_data_smithi192_88961_object87933:head# 2024-06-27T21:23:14.605 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f6ae8e:::benchmark_data_smithi192_88961_object55092:head# 2024-06-27T21:23:14.607 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f8d9eb:::benchmark_data_smithi192_88961_object68925:head# 2024-06-27T21:23:14.608 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0f91a50:::benchmark_data_smithi192_88961_object89697:head# 2024-06-27T21:23:14.608 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0fb4012:::benchmark_data_smithi192_88961_object80167:head# 2024-06-27T21:23:14.608 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0fc4d00:::benchmark_data_smithi192_88961_object55376:head# 2024-06-27T21:23:14.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0fdeb08:::benchmark_data_smithi192_88961_object56757:head# 2024-06-27T21:23:14.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b0feda33:::benchmark_data_smithi192_88961_object90458:head# 2024-06-27T21:23:14.612 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b10102ea:::benchmark_data_smithi192_88961_object67630:head# 2024-06-27T21:23:14.612 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b10105e9:::benchmark_data_smithi192_88961_object46282:head# 2024-06-27T21:23:14.612 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b102c04e:::benchmark_data_smithi192_88961_object44841:head# 2024-06-27T21:23:14.613 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1053508:::benchmark_data_smithi192_88961_object80730:head# 2024-06-27T21:23:14.613 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b10703f8:::benchmark_data_smithi192_88961_object67617:head# 2024-06-27T21:23:14.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b107957d:::benchmark_data_smithi192_88961_object59645:head# 2024-06-27T21:23:14.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b10949ab:::benchmark_data_smithi192_88961_object83314:head# 2024-06-27T21:23:14.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b10b9dcd:::benchmark_data_smithi192_88961_object85851:head# 2024-06-27T21:23:14.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b10fd701:::benchmark_data_smithi192_88961_object47261:head# 2024-06-27T21:23:14.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b110023d:::benchmark_data_smithi192_88961_object88703:head# 2024-06-27T21:23:14.616 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1101901:::benchmark_data_smithi192_88961_object53439:head# 2024-06-27T21:23:14.616 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1110da9:::benchmark_data_smithi192_88961_object65131:head# 2024-06-27T21:23:14.617 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1124a80:::benchmark_data_smithi192_88961_object86292:head# 2024-06-27T21:23:14.617 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1127213:::benchmark_data_smithi192_88961_object56107:head# 2024-06-27T21:23:14.617 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b11293ed:::benchmark_data_smithi192_88961_object58119:head# 2024-06-27T21:23:14.618 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b114053f:::benchmark_data_smithi192_88961_object60697:head# 2024-06-27T21:23:14.618 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1140856:::benchmark_data_smithi192_88961_object70896:head# 2024-06-27T21:23:14.618 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b114df6f:::benchmark_data_smithi192_88961_object69723:head# 2024-06-27T21:23:14.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b11711f8:::benchmark_data_smithi192_88961_object69486:head# 2024-06-27T21:23:14.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b118a5a9:::benchmark_data_smithi192_88961_object85492:head# 2024-06-27T21:23:14.620 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b11b0316:::benchmark_data_smithi192_88961_object77252:head# 2024-06-27T21:23:14.620 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b11bfb72:::benchmark_data_smithi192_88961_object73345:head# 2024-06-27T21:23:14.621 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b11d7125:::benchmark_data_smithi192_88961_object63921:head# 2024-06-27T21:23:14.621 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b11ebc93:::benchmark_data_smithi192_88961_object71323:head# 2024-06-27T21:23:14.621 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1208458:::benchmark_data_smithi192_88961_object60846:head# 2024-06-27T21:23:14.622 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b123f6c5:::benchmark_data_smithi192_88961_object45947:head# 2024-06-27T21:23:14.622 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1241cd1:::benchmark_data_smithi192_88961_object88955:head# 2024-06-27T21:23:14.623 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1250798:::benchmark_data_smithi192_88961_object89154:head# 2024-06-27T21:23:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1257812:::benchmark_data_smithi192_88961_object73949:head# 2024-06-27T21:23:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1262110:::benchmark_data_smithi192_88961_object82044:head# 2024-06-27T21:23:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1278ce2:::benchmark_data_smithi192_88961_object45611:head# 2024-06-27T21:23:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b127fc4a:::benchmark_data_smithi192_88961_object52284:head# 2024-06-27T21:23:14.625 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1283f7f:::benchmark_data_smithi192_88961_object66524:head# 2024-06-27T21:23:14.625 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1297e25:::benchmark_data_smithi192_88961_object61193:head# 2024-06-27T21:23:14.626 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b12b254a:::benchmark_data_smithi192_88961_object46656:head# 2024-06-27T21:23:14.626 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b12c2ae5:::benchmark_data_smithi192_88961_object58887:head# 2024-06-27T21:23:14.626 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b12d6204:::benchmark_data_smithi192_88961_object85598:head# 2024-06-27T21:23:14.627 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b12e7237:::benchmark_data_smithi192_88961_object85261:head# 2024-06-27T21:23:14.627 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b131ab89:::benchmark_data_smithi192_88961_object83884:head# 2024-06-27T21:23:14.628 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1327fc7:::benchmark_data_smithi192_88961_object46753:head# 2024-06-27T21:23:14.628 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b133181a:::benchmark_data_smithi192_88961_object54763:head# 2024-06-27T21:23:14.628 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1347dd4:::benchmark_data_smithi192_88961_object65598:head# 2024-06-27T21:23:14.629 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b13602ea:::benchmark_data_smithi192_88961_object57342:head# 2024-06-27T21:23:14.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b13603d2:::benchmark_data_smithi192_88961_object74141:head# 2024-06-27T21:23:14.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1393989:::benchmark_data_smithi192_88961_object86017:head# 2024-06-27T21:23:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b13a6d53:::benchmark_data_smithi192_88961_object85427:head# 2024-06-27T21:23:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b13e1ef0:::benchmark_data_smithi192_88961_object59487:head# 2024-06-27T21:23:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b13f6c8f:::benchmark_data_smithi192_88961_object47727:head# 2024-06-27T21:23:14.633 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b140028f:::benchmark_data_smithi192_88961_object45132:head# 2024-06-27T21:23:14.634 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1400380:::benchmark_data_smithi192_88961_object72672:head# 2024-06-27T21:23:14.634 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b140cc59:::benchmark_data_smithi192_88961_object78387:head# 2024-06-27T21:23:14.635 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b141e507:::benchmark_data_smithi192_88961_object44523:head# 2024-06-27T21:23:14.635 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b142ebc2:::benchmark_data_smithi192_88961_object68153:head# 2024-06-27T21:23:14.635 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b142f2b1:::benchmark_data_smithi192_88961_object50022:head# 2024-06-27T21:23:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1450851:::benchmark_data_smithi192_88961_object68086:head# 2024-06-27T21:23:14.637 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b146bcda:::benchmark_data_smithi192_88961_object57789:head# 2024-06-27T21:23:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b14795a7:::benchmark_data_smithi192_88961_object64889:head# 2024-06-27T21:23:14.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b148892a:::benchmark_data_smithi192_88961_object62164:head# 2024-06-27T21:23:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b14932ea:::benchmark_data_smithi192_88961_object50202:head# 2024-06-27T21:23:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b14af43c:::benchmark_data_smithi192_88961_object64321:head# 2024-06-27T21:23:14.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b14d8d8e:::benchmark_data_smithi192_88961_object84379:head# 2024-06-27T21:23:14.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b14f5769:::benchmark_data_smithi192_88961_object70004:head# 2024-06-27T21:23:14.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b15062e5:::benchmark_data_smithi192_88961_object71122:head# 2024-06-27T21:23:14.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b150b88b:::benchmark_data_smithi192_88961_object51254:head# 2024-06-27T21:23:14.641 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b150c84a:::benchmark_data_smithi192_88961_object44315:head# 2024-06-27T21:23:14.642 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b154acca:::benchmark_data_smithi192_88961_object81084:head# 2024-06-27T21:23:14.642 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1550b02:::benchmark_data_smithi192_88961_object84747:head# 2024-06-27T21:23:14.642 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1579e4d:::benchmark_data_smithi192_88961_object75574:head# 2024-06-27T21:23:14.643 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1588f92:::benchmark_data_smithi192_88961_object65244:head# 2024-06-27T21:23:14.643 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1593688:::benchmark_data_smithi192_88961_object62004:head# 2024-06-27T21:23:14.644 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b159a64c:::benchmark_data_smithi192_88961_object81318:head# 2024-06-27T21:23:14.645 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b15cacd9:::benchmark_data_smithi192_88961_object75091:head# 2024-06-27T21:23:14.645 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b15dabfa:::benchmark_data_smithi192_88961_object77207:head# 2024-06-27T21:23:14.646 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b15db2d0:::benchmark_data_smithi192_88961_object47663:head# 2024-06-27T21:23:14.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b15f2bdb:::benchmark_data_smithi192_88961_object73089:head# 2024-06-27T21:23:14.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b161c131:::benchmark_data_smithi192_88961_object65433:head# 2024-06-27T21:23:14.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b162e7d9:::benchmark_data_smithi192_88961_object59389:head# 2024-06-27T21:23:14.650 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1639b55:::benchmark_data_smithi192_88961_object57254:head# 2024-06-27T21:23:14.650 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1639c07:::benchmark_data_smithi192_88961_object47444:head# 2024-06-27T21:23:14.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b16529a3:::benchmark_data_smithi192_88961_object70043:head# 2024-06-27T21:23:14.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b16be8a2:::benchmark_data_smithi192_88961_object63657:head# 2024-06-27T21:23:14.652 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b16c7d93:::benchmark_data_smithi192_88961_object90537:head# 2024-06-27T21:23:14.652 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b16f63a0:::benchmark_data_smithi192_88961_object71545:head# 2024-06-27T21:23:14.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b16faf6c:::benchmark_data_smithi192_88961_object81832:head# 2024-06-27T21:23:14.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b16fd924:::benchmark_data_smithi192_88961_object45617:head# 2024-06-27T21:23:14.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17100db:::benchmark_data_smithi192_88961_object49399:head# 2024-06-27T21:23:14.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17161fa:::benchmark_data_smithi192_88961_object72587:head# 2024-06-27T21:23:14.654 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b171e503:::benchmark_data_smithi192_88961_object60175:head# 2024-06-27T21:23:14.654 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1723ea9:::benchmark_data_smithi192_88961_object52290:head# 2024-06-27T21:23:14.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b173a61f:::benchmark_data_smithi192_88961_object60937:head# 2024-06-27T21:23:14.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b174c0c2:::benchmark_data_smithi192_88961_object67847:head# 2024-06-27T21:23:14.656 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1791e02:::benchmark_data_smithi192_88961_object84252:head# 2024-06-27T21:23:14.656 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17b2aaf:::benchmark_data_smithi192_88961_object71532:head# 2024-06-27T21:23:14.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17bd424:::benchmark_data_smithi192_88961_object71106:head# 2024-06-27T21:23:14.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17d853f:::benchmark_data_smithi192_88961_object51701:head# 2024-06-27T21:23:14.658 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17da211:::benchmark_data_smithi192_88961_object88394:head# 2024-06-27T21:23:14.658 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17ee8ed:::benchmark_data_smithi192_88961_object82080:head# 2024-06-27T21:23:14.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b17f53a3:::benchmark_data_smithi192_88961_object46228:head# 2024-06-27T21:23:14.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1804ab5:::benchmark_data_smithi192_88961_object60998:head# 2024-06-27T21:23:14.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b180f6af:::benchmark_data_smithi192_88961_object64930:head# 2024-06-27T21:23:14.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b182a9ad:::benchmark_data_smithi192_88961_object45864:head# 2024-06-27T21:23:14.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b184552b:::benchmark_data_smithi192_88961_object70191:head# 2024-06-27T21:23:14.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:14.661 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b184d560:::benchmark_data_smithi192_88961_object52408:head# 2024-06-27T21:23:14.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b184f88a:::benchmark_data_smithi192_88961_object84253:head# 2024-06-27T21:23:14.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1865f89:::benchmark_data_smithi192_88961_object87107:head# 2024-06-27T21:23:14.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1875258:::benchmark_data_smithi192_88961_object49353:head# 2024-06-27T21:23:14.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b18853ed:::benchmark_data_smithi192_88961_object46815:head# 2024-06-27T21:23:14.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b188a51a:::benchmark_data_smithi192_88961_object63374:head# 2024-06-27T21:23:14.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b188eff1:::benchmark_data_smithi192_88961_object75602:head# 2024-06-27T21:23:14.664 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b18a8838:::benchmark_data_smithi192_88961_object87673:head# 2024-06-27T21:23:14.664 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b18a8ba2:::benchmark_data_smithi192_88961_object61109:head# 2024-06-27T21:23:14.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b18c6683:::benchmark_data_smithi192_88961_object45393:head# 2024-06-27T21:23:14.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b191dccf:::benchmark_data_smithi192_88961_object57009:head# 2024-06-27T21:23:14.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b193c308:::benchmark_data_smithi192_88961_object50664:head# 2024-06-27T21:23:14.666 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b19480e2:::benchmark_data_smithi192_88961_object82696:head# 2024-06-27T21:23:14.666 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1959a16:::benchmark_data_smithi192_88961_object57098:head# 2024-06-27T21:23:14.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b19af94f:::benchmark_data_smithi192_88961_object53724:head# 2024-06-27T21:23:14.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b19b1d1b:::benchmark_data_smithi192_88961_object79526:head# 2024-06-27T21:23:14.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1a1bc11:::benchmark_data_smithi192_88961_object80557:head# 2024-06-27T21:23:14.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1a4818b:::benchmark_data_smithi192_88961_object68256:head# 2024-06-27T21:23:14.671 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1a496ae:::benchmark_data_smithi192_88961_object63847:head# 2024-06-27T21:23:14.671 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1a56fe8:::benchmark_data_smithi192_88961_object62884:head# 2024-06-27T21:23:14.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1a6c31c:::benchmark_data_smithi192_88961_object75921:head# 2024-06-27T21:23:14.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1a73d17:::benchmark_data_smithi192_88961_object66911:head# 2024-06-27T21:23:14.674 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1ac09c8:::benchmark_data_smithi192_88961_object71265:head# 2024-06-27T21:23:14.674 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1adeb5e:::benchmark_data_smithi192_88961_object72697:head# 2024-06-27T21:23:14.674 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1ae09cb:::benchmark_data_smithi192_88961_object79214:head# 2024-06-27T21:23:14.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1af04c4:::benchmark_data_smithi192_88961_object47540:head# 2024-06-27T21:23:14.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b3b3f6:::benchmark_data_smithi192_88961_object82038:head# 2024-06-27T21:23:14.676 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b3b67a:::benchmark_data_smithi192_88961_object82243:head# 2024-06-27T21:23:14.677 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b3da8a:::benchmark_data_smithi192_88961_object53398:head# 2024-06-27T21:23:14.677 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b40f9c:::benchmark_data_smithi192_88961_object86523:head# 2024-06-27T21:23:14.678 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b5f472:::benchmark_data_smithi192_88961_object84607:head# 2024-06-27T21:23:14.679 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b755a2:::benchmark_data_smithi192_88961_object86874:head# 2024-06-27T21:23:14.679 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1b94c90:::benchmark_data_smithi192_88961_object85191:head# 2024-06-27T21:23:14.680 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1bb5374:::benchmark_data_smithi192_88961_object46134:head# 2024-06-27T21:23:14.680 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1be3b35:::benchmark_data_smithi192_88961_object69651:head# 2024-06-27T21:23:14.680 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1be5161:::benchmark_data_smithi192_88961_object55051:head# 2024-06-27T21:23:14.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1bf8281:::benchmark_data_smithi192_88961_object83684:head# 2024-06-27T21:23:14.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c09795:::benchmark_data_smithi192_88961_object88040:head# 2024-06-27T21:23:14.683 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c0acc5:::benchmark_data_smithi192_88961_object46359:head# 2024-06-27T21:23:14.685 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c135a9:::benchmark_data_smithi192_88961_object49974:head# 2024-06-27T21:23:14.686 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c36eb8:::benchmark_data_smithi192_88961_object80536:head# 2024-06-27T21:23:14.688 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c3f089:::benchmark_data_smithi192_88961_object55977:head# 2024-06-27T21:23:14.689 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c4b9f6:::benchmark_data_smithi192_88961_object76116:head# 2024-06-27T21:23:14.689 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c6eacf:::benchmark_data_smithi192_88961_object89409:head# 2024-06-27T21:23:14.690 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1c8bae1:::benchmark_data_smithi192_88961_object86648:head# 2024-06-27T21:23:14.690 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1cac732:::benchmark_data_smithi192_88961_object87641:head# 2024-06-27T21:23:14.691 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1cb5414:::benchmark_data_smithi192_88961_object59212:head# 2024-06-27T21:23:14.691 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1cca20c:::benchmark_data_smithi192_88961_object47591:head# 2024-06-27T21:23:14.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1cca438:::benchmark_data_smithi192_88961_object70100:head# 2024-06-27T21:23:14.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d02577:::benchmark_data_smithi192_88961_object66944:head# 2024-06-27T21:23:14.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d17869:::benchmark_data_smithi192_88961_object67906:head# 2024-06-27T21:23:14.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d1b02f:::benchmark_data_smithi192_88961_object49638:head# 2024-06-27T21:23:14.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d1bf47:::benchmark_data_smithi192_88961_object83727:head# 2024-06-27T21:23:14.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d2b16b:::benchmark_data_smithi192_88961_object45113:head# 2024-06-27T21:23:14.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d44e8d:::benchmark_data_smithi192_88961_object78645:head# 2024-06-27T21:23:14.694 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d52687:::benchmark_data_smithi192_88961_object81188:head# 2024-06-27T21:23:14.695 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d6a1e9:::benchmark_data_smithi192_88961_object49235:head# 2024-06-27T21:23:14.695 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d89932:::benchmark_data_smithi192_88961_object82281:head# 2024-06-27T21:23:14.695 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1d97bf6:::benchmark_data_smithi192_88961_object48783:head# 2024-06-27T21:23:14.697 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1db28ce:::benchmark_data_smithi192_88961_object69691:head# 2024-06-27T21:23:14.697 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1dc05f8:::benchmark_data_smithi192_88961_object51189:head# 2024-06-27T21:23:14.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1df91bd:::benchmark_data_smithi192_88961_object89970:head# 2024-06-27T21:23:14.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1dff40e:::benchmark_data_smithi192_88961_object57933:head# 2024-06-27T21:23:14.699 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e0defa:::benchmark_data_smithi192_88961_object74527:head# 2024-06-27T21:23:14.699 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e20806:::benchmark_data_smithi192_88961_object57323:head# 2024-06-27T21:23:14.700 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e29bd5:::benchmark_data_smithi192_88961_object46439:head# 2024-06-27T21:23:14.701 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e2e5d7:::benchmark_data_smithi192_88961_object43767:head# 2024-06-27T21:23:14.701 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e59f12:::benchmark_data_smithi192_88961_object64224:head# 2024-06-27T21:23:14.701 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e60e3a:::benchmark_data_smithi192_88961_object86751:head# 2024-06-27T21:23:14.702 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e67aaa:::benchmark_data_smithi192_88961_object47192:head# 2024-06-27T21:23:14.702 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e6c131:::benchmark_data_smithi192_88961_object69255:head# 2024-06-27T21:23:14.702 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e77c5a:::benchmark_data_smithi192_88961_object55894:head# 2024-06-27T21:23:14.702 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e7b61c:::benchmark_data_smithi192_88961_object71257:head# 2024-06-27T21:23:14.703 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1e94af3:::benchmark_data_smithi192_88961_object71333:head# 2024-06-27T21:23:14.703 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1ed21e0:::benchmark_data_smithi192_88961_object62580:head# 2024-06-27T21:23:14.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1ef06f7:::benchmark_data_smithi192_88961_object49525:head# 2024-06-27T21:23:14.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1ef2d00:::benchmark_data_smithi192_88961_object62055:head# 2024-06-27T21:23:14.705 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f222c8:::benchmark_data_smithi192_88961_object51775:head# 2024-06-27T21:23:14.705 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f2937e:::benchmark_data_smithi192_88961_object86993:head# 2024-06-27T21:23:14.705 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f43bcc:::benchmark_data_smithi192_88961_object46426:head# 2024-06-27T21:23:14.706 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f5f36e:::benchmark_data_smithi192_88961_object79637:head# 2024-06-27T21:23:14.707 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f6fcbd:::benchmark_data_smithi192_88961_object87112:head# 2024-06-27T21:23:14.707 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f87799:::benchmark_data_smithi192_88961_object48197:head# 2024-06-27T21:23:14.707 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1f8e5ac:::benchmark_data_smithi192_88961_object86622:head# 2024-06-27T21:23:14.707 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1fa7fd3:::benchmark_data_smithi192_88961_object85023:head# 2024-06-27T21:23:14.709 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1faa763:::benchmark_data_smithi192_88961_object87556:head# 2024-06-27T21:23:14.709 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1fccb97:::benchmark_data_smithi192_88961_object59273:head# 2024-06-27T21:23:14.709 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1fea38a:::benchmark_data_smithi192_88961_object89158:head# 2024-06-27T21:23:14.710 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b1ff3419:::benchmark_data_smithi192_88961_object84058:head# 2024-06-27T21:23:14.710 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b204e3ca:::benchmark_data_smithi192_88961_object52405:head# 2024-06-27T21:23:14.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2050aa5:::benchmark_data_smithi192_88961_object50541:head# 2024-06-27T21:23:14.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2053c55:::benchmark_data_smithi192_88961_object75286:head# 2024-06-27T21:23:14.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b20735b6:::benchmark_data_smithi192_88961_object54932:head# 2024-06-27T21:23:14.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b208205b:::benchmark_data_smithi192_88961_object45868:head# 2024-06-27T21:23:14.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b208c067:::benchmark_data_smithi192_88961_object79331:head# 2024-06-27T21:23:14.713 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b20b8a48:::benchmark_data_smithi192_88961_object55976:head# 2024-06-27T21:23:14.714 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b20d84e4:::benchmark_data_smithi192_88961_object73335:head# 2024-06-27T21:23:14.714 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b20ef4b7:::benchmark_data_smithi192_88961_object48215:head# 2024-06-27T21:23:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:14 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:23:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:14 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:23:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:14.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:14.717 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b20f8503:::benchmark_data_smithi192_88961_object54933:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b20fbcf4:::benchmark_data_smithi192_88961_object60230:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2102350:::benchmark_data_smithi192_88961_object72478:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b210cd6d:::benchmark_data_smithi192_88961_object64714:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b21165cd:::benchmark_data_smithi192_88961_object87739:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2147782:::benchmark_data_smithi192_88961_object46924:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b214a0ac:::benchmark_data_smithi192_88961_object52684:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b216f034:::benchmark_data_smithi192_88961_object83318:head# 2024-06-27T21:23:14.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2173c38:::benchmark_data_smithi192_88961_object76154:head# 2024-06-27T21:23:14.719 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b217f364:::benchmark_data_smithi192_88961_object49396:head# 2024-06-27T21:23:14.719 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2182a99:::benchmark_data_smithi192_88961_object54911:head# 2024-06-27T21:23:14.719 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b218b2e8:::benchmark_data_smithi192_88961_object69419:head# 2024-06-27T21:23:14.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b21bb050:::benchmark_data_smithi192_88961_object49400:head# 2024-06-27T21:23:14.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b21bf403:::benchmark_data_smithi192_88961_object70689:head# 2024-06-27T21:23:14.721 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b21bf621:::benchmark_data_smithi192_88961_object48777:head# 2024-06-27T21:23:14.721 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b21e973b:::benchmark_data_smithi192_88961_object75492:head# 2024-06-27T21:23:14.722 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b21fce42:::benchmark_data_smithi192_88961_object45017:head# 2024-06-27T21:23:14.722 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2212da1:::benchmark_data_smithi192_88961_object80348:head# 2024-06-27T21:23:14.722 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b225e205:::benchmark_data_smithi192_88961_object46876:head# 2024-06-27T21:23:14.724 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b226284f:::benchmark_data_smithi192_88961_object49352:head# 2024-06-27T21:23:14.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22714db:::benchmark_data_smithi192_88961_object46372:head# 2024-06-27T21:23:14.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2272dc5:::benchmark_data_smithi192_88961_object74137:head# 2024-06-27T21:23:14.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b227ffc3:::benchmark_data_smithi192_88961_object70071:head# 2024-06-27T21:23:14.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22933f5:::benchmark_data_smithi192_88961_object82723:head# 2024-06-27T21:23:14.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22b9f68:::benchmark_data_smithi192_88961_object46869:head# 2024-06-27T21:23:14.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22bc78d:::benchmark_data_smithi192_88961_object53874:head# 2024-06-27T21:23:14.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22c94a5:::benchmark_data_smithi192_88961_object82857:head# 2024-06-27T21:23:14.729 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22c9b2d:::benchmark_data_smithi192_88961_object50235:head# 2024-06-27T21:23:14.730 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22cde6d:::benchmark_data_smithi192_88961_object68329:head# 2024-06-27T21:23:14.730 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22e8b0e:::benchmark_data_smithi192_88961_object73250:head# 2024-06-27T21:23:14.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22f196a:::benchmark_data_smithi192_88961_object58765:head# 2024-06-27T21:23:14.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22f3d57:::benchmark_data_smithi192_88961_object73403:head# 2024-06-27T21:23:14.732 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b22ff9ca:::benchmark_data_smithi192_88961_object56622:head# 2024-06-27T21:23:14.732 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b233b1ac:::benchmark_data_smithi192_88961_object62793:head# 2024-06-27T21:23:14.734 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2379098:::benchmark_data_smithi192_88961_object46755:head# 2024-06-27T21:23:14.734 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b237f75b:::benchmark_data_smithi192_88961_object61765:head# 2024-06-27T21:23:14.734 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b239e80b:::benchmark_data_smithi192_88961_object90654:head# 2024-06-27T21:23:14.734 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b23b4c82:::benchmark_data_smithi192_88961_object69635:head# 2024-06-27T21:23:14.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b23b6960:::benchmark_data_smithi192_88961_object82855:head# 2024-06-27T21:23:14.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b23f6561:::benchmark_data_smithi192_88961_object82286:head# 2024-06-27T21:23:14.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b23f94ab:::benchmark_data_smithi192_88961_object60382:head# 2024-06-27T21:23:14.736 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b241670d:::benchmark_data_smithi192_88961_object75063:head# 2024-06-27T21:23:14.736 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b24168b5:::benchmark_data_smithi192_88961_object47523:head# 2024-06-27T21:23:14.737 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2432052:::benchmark_data_smithi192_88961_object54786:head# 2024-06-27T21:23:14.737 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b244fa0b:::benchmark_data_smithi192_88961_object62451:head# 2024-06-27T21:23:14.738 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2455125:::benchmark_data_smithi192_88961_object85022:head# 2024-06-27T21:23:14.738 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b245d121:::benchmark_data_smithi192_88961_object72713:head# 2024-06-27T21:23:14.739 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2467c08:::benchmark_data_smithi192_88961_object66572:head# 2024-06-27T21:23:14.739 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2491cbf:::benchmark_data_smithi192_88961_object79356:head# 2024-06-27T21:23:14.740 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b24c2db4:::benchmark_data_smithi192_88961_object50434:head# 2024-06-27T21:23:14.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b24fc09c:::benchmark_data_smithi192_88961_object89498:head# 2024-06-27T21:23:14.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b250ab9a:::benchmark_data_smithi192_88961_object81699:head# 2024-06-27T21:23:14.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2513f20:::benchmark_data_smithi192_88961_object55355:head# 2024-06-27T21:23:14.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2518b7e:::benchmark_data_smithi192_88961_object90194:head# 2024-06-27T21:23:14.742 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b252d58c:::benchmark_data_smithi192_88961_object64615:head# 2024-06-27T21:23:14.742 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2534dc3:::benchmark_data_smithi192_88961_object57440:head# 2024-06-27T21:23:14.744 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b253b730:::benchmark_data_smithi192_88961_object77471:head# 2024-06-27T21:23:14.744 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b25511bb:::benchmark_data_smithi192_88961_object44749:head# 2024-06-27T21:23:14.744 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2558ca3:::benchmark_data_smithi192_88961_object69145:head# 2024-06-27T21:23:14.748 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b25809a2:::benchmark_data_smithi192_88961_object88246:head# 2024-06-27T21:23:14.748 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2597126:::benchmark_data_smithi192_88961_object86910:head# 2024-06-27T21:23:14.748 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b25c5e1d:::benchmark_data_smithi192_88961_object62082:head# 2024-06-27T21:23:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b25ca0e7:::benchmark_data_smithi192_88961_object75728:head# 2024-06-27T21:23:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b25dbd0c:::benchmark_data_smithi192_88961_object79773:head# 2024-06-27T21:23:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b25ed63b:::benchmark_data_smithi192_88961_object76407:head# 2024-06-27T21:23:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2615d8e:::benchmark_data_smithi192_88961_object87121:head# 2024-06-27T21:23:14.749 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b263bf64:::benchmark_data_smithi192_88961_object48648:head# 2024-06-27T21:23:14.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2650a1e:::benchmark_data_smithi192_88961_object68652:head# 2024-06-27T21:23:14.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b265bc2c:::benchmark_data_smithi192_88961_object61568:head# 2024-06-27T21:23:14.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b26648e2:::benchmark_data_smithi192_88961_object79224:head# 2024-06-27T21:23:14.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b266503f:::benchmark_data_smithi192_88961_object69390:head# 2024-06-27T21:23:14.751 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b26b8cd1:::benchmark_data_smithi192_88961_object86783:head# 2024-06-27T21:23:14.751 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b26bc08b:::benchmark_data_smithi192_88961_object47741:head# 2024-06-27T21:23:14.751 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2707448:::benchmark_data_smithi192_88961_object64015:head# 2024-06-27T21:23:14.751 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b272a984:::benchmark_data_smithi192_88961_object43135:head# 2024-06-27T21:23:14.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b274314c:::benchmark_data_smithi192_88961_object71577:head# 2024-06-27T21:23:14.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2744e12:::benchmark_data_smithi192_88961_object53168:head# 2024-06-27T21:23:14.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2749880:::benchmark_data_smithi192_88961_object61907:head# 2024-06-27T21:23:14.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b27dfca2:::benchmark_data_smithi192_88961_object62568:head# 2024-06-27T21:23:14.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b27e1a61:::benchmark_data_smithi192_88961_object56489:head# 2024-06-27T21:23:14.754 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b28021b8:::benchmark_data_smithi192_88961_object83906:head# 2024-06-27T21:23:14.754 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b281766a:::benchmark_data_smithi192_88961_object63621:head# 2024-06-27T21:23:14.755 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2839428:::benchmark_data_smithi192_88961_object43744:head# 2024-06-27T21:23:14.755 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2864010:::benchmark_data_smithi192_88961_object46703:head# 2024-06-27T21:23:14.756 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b286989c:::benchmark_data_smithi192_88961_object81179:head# 2024-06-27T21:23:14.756 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b286b77d:::benchmark_data_smithi192_88961_object59014:head# 2024-06-27T21:23:14.757 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b28a34da:::benchmark_data_smithi192_88961_object60233:head# 2024-06-27T21:23:14.757 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b28a5b20:::benchmark_data_smithi192_88961_object72971:head# 2024-06-27T21:23:14.757 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b28c0edf:::benchmark_data_smithi192_88961_object48263:head# 2024-06-27T21:23:14.758 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29181e3:::benchmark_data_smithi192_88961_object88374:head# 2024-06-27T21:23:14.758 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2927309:::benchmark_data_smithi192_88961_object72878:head# 2024-06-27T21:23:14.759 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2927aa7:::benchmark_data_smithi192_88961_object87879:head# 2024-06-27T21:23:14.759 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b293ef59:::benchmark_data_smithi192_88961_object62746:head# 2024-06-27T21:23:14.760 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b295152a:::benchmark_data_smithi192_88961_object73968:head# 2024-06-27T21:23:14.760 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2962895:::benchmark_data_smithi192_88961_object61733:head# 2024-06-27T21:23:14.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2970791:::benchmark_data_smithi192_88961_object77518:head# 2024-06-27T21:23:14.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2982e27:::benchmark_data_smithi192_88961_object88003:head# 2024-06-27T21:23:14.762 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29924f4:::benchmark_data_smithi192_88961_object66336:head# 2024-06-27T21:23:14.762 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2993711:::benchmark_data_smithi192_88961_object80013:head# 2024-06-27T21:23:14.763 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29b53a1:::benchmark_data_smithi192_88961_object48199:head# 2024-06-27T21:23:14.763 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29cd88b:::benchmark_data_smithi192_88961_object78596:head# 2024-06-27T21:23:14.763 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29e40d2:::benchmark_data_smithi192_88961_object83720:head# 2024-06-27T21:23:14.764 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29f8d23:::benchmark_data_smithi192_88961_object86106:head# 2024-06-27T21:23:14.764 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b29f9a2d:::benchmark_data_smithi192_88961_object59896:head# 2024-06-27T21:23:14.764 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2a0995f:::benchmark_data_smithi192_88961_object49523:head# 2024-06-27T21:23:14.765 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2a331c5:::benchmark_data_smithi192_88961_object48747:head# 2024-06-27T21:23:14.765 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2a5691a:::benchmark_data_smithi192_88961_object46996:head# 2024-06-27T21:23:14.765 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2a6232b:::benchmark_data_smithi192_88961_object83462:head# 2024-06-27T21:23:14.766 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2a76049:::benchmark_data_smithi192_88961_object44051:head# 2024-06-27T21:23:14.766 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2a7aefe:::benchmark_data_smithi192_88961_object84088:head# 2024-06-27T21:23:14.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ab6997:::benchmark_data_smithi192_88961_object68205:head# 2024-06-27T21:23:14.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ab7c8c:::benchmark_data_smithi192_88961_object49536:head# 2024-06-27T21:23:14.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ac8e6a:::benchmark_data_smithi192_88961_object84800:head# 2024-06-27T21:23:14.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ae3bbc:::benchmark_data_smithi192_88961_object76425:head# 2024-06-27T21:23:14.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2b3de70:::benchmark_data_smithi192_88961_object52647:head# 2024-06-27T21:23:14.769 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2b64a5e:::benchmark_data_smithi192_88961_object43530:head# 2024-06-27T21:23:14.769 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2b70eee:::benchmark_data_smithi192_88961_object65599:head# 2024-06-27T21:23:14.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2b7c1d4:::benchmark_data_smithi192_88961_object60620:head# 2024-06-27T21:23:14.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2b905de:::benchmark_data_smithi192_88961_object47772:head# 2024-06-27T21:23:14.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2b99b45:::benchmark_data_smithi192_88961_object49767:head# 2024-06-27T21:23:14.771 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2baa057:::benchmark_data_smithi192_88961_object53482:head# 2024-06-27T21:23:14.771 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2bc75f1:::benchmark_data_smithi192_88961_object56952:head# 2024-06-27T21:23:14.771 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2be7819:::benchmark_data_smithi192_88961_object57041:head# 2024-06-27T21:23:14.772 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2c797c2:::benchmark_data_smithi192_88961_object86365:head# 2024-06-27T21:23:14.772 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2c7c958:::benchmark_data_smithi192_88961_object87921:head# 2024-06-27T21:23:14.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2c839a2:::benchmark_data_smithi192_88961_object64528:head# 2024-06-27T21:23:14.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2c989c2:::benchmark_data_smithi192_88961_object78191:head# 2024-06-27T21:23:14.774 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2c9b611:::benchmark_data_smithi192_88961_object58272:head# 2024-06-27T21:23:14.774 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ca01a1:::benchmark_data_smithi192_88961_object53931:head# 2024-06-27T21:23:14.775 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cb9ac7:::benchmark_data_smithi192_88961_object55724:head# 2024-06-27T21:23:14.775 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cba18c:::benchmark_data_smithi192_88961_object74058:head# 2024-06-27T21:23:14.776 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cbcd0a:::benchmark_data_smithi192_88961_object53893:head# 2024-06-27T21:23:14.776 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cbe581:::benchmark_data_smithi192_88961_object69204:head# 2024-06-27T21:23:14.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cc873a:::benchmark_data_smithi192_88961_object49847:head# 2024-06-27T21:23:14.778 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cd1d5f:::benchmark_data_smithi192_88961_object61690:head# 2024-06-27T21:23:14.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cd2943:::benchmark_data_smithi192_88961_object88245:head# 2024-06-27T21:23:14.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cd8bf5:::benchmark_data_smithi192_88961_object54688:head# 2024-06-27T21:23:14.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ce5968:::benchmark_data_smithi192_88961_object49563:head# 2024-06-27T21:23:14.780 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2cff831:::benchmark_data_smithi192_88961_object78960:head# 2024-06-27T21:23:14.780 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d1c5fb:::benchmark_data_smithi192_88961_object60982:head# 2024-06-27T21:23:14.782 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d20e69:::benchmark_data_smithi192_88961_object68043:head# 2024-06-27T21:23:14.782 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d232eb:::benchmark_data_smithi192_88961_object60513:head# 2024-06-27T21:23:14.782 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d2fd58:::benchmark_data_smithi192_88961_object72255:head# 2024-06-27T21:23:14.783 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d2ff8d:::benchmark_data_smithi192_88961_object55543:head# 2024-06-27T21:23:14.783 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d68bf9:::benchmark_data_smithi192_88961_object90397:head# 2024-06-27T21:23:14.783 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d69075:::benchmark_data_smithi192_88961_object55342:head# 2024-06-27T21:23:14.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d6fd8f:::benchmark_data_smithi192_88961_object81530:head# 2024-06-27T21:23:14.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d7ef7b:::benchmark_data_smithi192_88961_object80633:head# 2024-06-27T21:23:14.785 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d88271:::benchmark_data_smithi192_88961_object81281:head# 2024-06-27T21:23:14.785 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d8cc71:::benchmark_data_smithi192_88961_object75843:head# 2024-06-27T21:23:14.787 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2d9e047:::benchmark_data_smithi192_88961_object57481:head# 2024-06-27T21:23:14.787 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2daf348:::benchmark_data_smithi192_88961_object43622:head# 2024-06-27T21:23:14.787 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2db0687:::benchmark_data_smithi192_88961_object88828:head# 2024-06-27T21:23:14.788 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2dc7b32:::benchmark_data_smithi192_88961_object51559:head# 2024-06-27T21:23:14.788 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2dcdfa9:::benchmark_data_smithi192_88961_object62872:head# 2024-06-27T21:23:14.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2dd7492:::benchmark_data_smithi192_88961_object62535:head# 2024-06-27T21:23:14.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ddd99f:::benchmark_data_smithi192_88961_object53906:head# 2024-06-27T21:23:14.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e0183a:::benchmark_data_smithi192_88961_object56993:head# 2024-06-27T21:23:14.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e2af94:::benchmark_data_smithi192_88961_object60166:head# 2024-06-27T21:23:14.793 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e38b47:::benchmark_data_smithi192_88961_object57796:head# 2024-06-27T21:23:14.793 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e3db39:::benchmark_data_smithi192_88961_object66245:head# 2024-06-27T21:23:14.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e5f9a8:::benchmark_data_smithi192_88961_object54318:head# 2024-06-27T21:23:14.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e656e6:::benchmark_data_smithi192_88961_object69757:head# 2024-06-27T21:23:14.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e6817c:::benchmark_data_smithi192_88961_object74126:head# 2024-06-27T21:23:14.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e6a9ed:::benchmark_data_smithi192_88961_object70838:head# 2024-06-27T21:23:14.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e85037:::benchmark_data_smithi192_88961_object57883:head# 2024-06-27T21:23:14.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e9021b:::benchmark_data_smithi192_88961_object71622:head# 2024-06-27T21:23:14.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2e94b1d:::benchmark_data_smithi192_88961_object58118:head# 2024-06-27T21:23:14.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2eb76f4:::benchmark_data_smithi192_88961_object57053:head# 2024-06-27T21:23:14.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ecbfe2:::benchmark_data_smithi192_88961_object63319:head# 2024-06-27T21:23:14.798 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ecce78:::benchmark_data_smithi192_88961_object88586:head# 2024-06-27T21:23:14.798 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ecfae0:::benchmark_data_smithi192_88961_object71819:head# 2024-06-27T21:23:14.799 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ed7391:::benchmark_data_smithi192_88961_object75770:head# 2024-06-27T21:23:14.799 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2eda202:::benchmark_data_smithi192_88961_object77896:head# 2024-06-27T21:23:14.800 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2ee8b01:::benchmark_data_smithi192_88961_object57408:head# 2024-06-27T21:23:14.802 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2eec7f2:::benchmark_data_smithi192_88961_object53087:head# 2024-06-27T21:23:14.802 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2f26658:::benchmark_data_smithi192_88961_object58479:head# 2024-06-27T21:23:14.802 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2f37ce4:::benchmark_data_smithi192_88961_object65715:head# 2024-06-27T21:23:14.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2f70a43:::benchmark_data_smithi192_88961_object73486:head# 2024-06-27T21:23:14.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2f7905c:::benchmark_data_smithi192_88961_object61076:head# 2024-06-27T21:23:14.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2f7d265:::benchmark_data_smithi192_88961_object56988:head# 2024-06-27T21:23:14.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2f8b907:::benchmark_data_smithi192_88961_object55254:head# 2024-06-27T21:23:14.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2fa2065:::benchmark_data_smithi192_88961_object63373:head# 2024-06-27T21:23:14.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2fb5c35:::benchmark_data_smithi192_88961_object45368:head# 2024-06-27T21:23:14.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2fc9d71:::benchmark_data_smithi192_88961_object84204:head# 2024-06-27T21:23:14.807 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2fea75f:::benchmark_data_smithi192_88961_object83399:head# 2024-06-27T21:23:14.807 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b2fee9ec:::benchmark_data_smithi192_88961_object75665:head# 2024-06-27T21:23:14.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3022042:::benchmark_data_smithi192_88961_object80152:head# 2024-06-27T21:23:14.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3031cce:::benchmark_data_smithi192_88961_object78607:head# 2024-06-27T21:23:14.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3035353:::benchmark_data_smithi192_88961_object52114:head# 2024-06-27T21:23:14.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30431dc:::benchmark_data_smithi192_88961_object87942:head# 2024-06-27T21:23:14.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b306e790:::benchmark_data_smithi192_88961_object87895:head# 2024-06-27T21:23:14.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b306ec99:::benchmark_data_smithi192_88961_object85016:head# 2024-06-27T21:23:14.810 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3078ce6:::benchmark_data_smithi192_88961_object48219:head# 2024-06-27T21:23:14.810 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b307a785:::benchmark_data_smithi192_88961_object63364:head# 2024-06-27T21:23:14.811 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3083d49:::benchmark_data_smithi192_88961_object74296:head# 2024-06-27T21:23:14.811 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b308f927:::benchmark_data_smithi192_88961_object43097:head# 2024-06-27T21:23:14.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30adaec:::benchmark_data_smithi192_88961_object52487:head# 2024-06-27T21:23:14.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30b414a:::benchmark_data_smithi192_88961_object71166:head# 2024-06-27T21:23:14.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30c17c6:::benchmark_data_smithi192_88961_object68827:head# 2024-06-27T21:23:14.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30c1c22:::benchmark_data_smithi192_88961_object67221:head# 2024-06-27T21:23:14.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30c516a:::benchmark_data_smithi192_88961_object86830:head# 2024-06-27T21:23:14.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30ccc07:::benchmark_data_smithi192_88961_object46470:head# 2024-06-27T21:23:14.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30cd534:::benchmark_data_smithi192_88961_object54030:head# 2024-06-27T21:23:14.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b30e0447:::benchmark_data_smithi192_88961_object51005:head# 2024-06-27T21:23:14.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b311459f:::benchmark_data_smithi192_88961_object73334:head# 2024-06-27T21:23:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b313fad7:::benchmark_data_smithi192_88961_object81309:head# 2024-06-27T21:23:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b315f8f9:::benchmark_data_smithi192_88961_object47633:head# 2024-06-27T21:23:14.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3174ae4:::benchmark_data_smithi192_88961_object82552:head# 2024-06-27T21:23:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b31768ac:::benchmark_data_smithi192_88961_object71661:head# 2024-06-27T21:23:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b318095d:::benchmark_data_smithi192_88961_object74594:head# 2024-06-27T21:23:14.819 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3192686:::benchmark_data_smithi192_88961_object47703:head# 2024-06-27T21:23:14.819 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b31c2219:::benchmark_data_smithi192_88961_object80691:head# 2024-06-27T21:23:14.819 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b31c9520:::benchmark_data_smithi192_88961_object82413:head# 2024-06-27T21:23:14.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b320471e:::benchmark_data_smithi192_88961_object85658:head# 2024-06-27T21:23:14.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b322270f:::benchmark_data_smithi192_88961_object47758:head# 2024-06-27T21:23:14.821 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b324bd28:::benchmark_data_smithi192_88961_object87849:head# 2024-06-27T21:23:14.821 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b324f3e1:::benchmark_data_smithi192_88961_object71585:head# 2024-06-27T21:23:14.822 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3255ccf:::benchmark_data_smithi192_88961_object58050:head# 2024-06-27T21:23:14.822 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b325d033:::benchmark_data_smithi192_88961_object80457:head# 2024-06-27T21:23:14.823 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3270e05:::benchmark_data_smithi192_88961_object89070:head# 2024-06-27T21:23:14.823 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b327542d:::benchmark_data_smithi192_88961_object75353:head# 2024-06-27T21:23:14.823 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b328a87c:::benchmark_data_smithi192_88961_object60322:head# 2024-06-27T21:23:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b32a8d4b:::benchmark_data_smithi192_88961_object69413:head# 2024-06-27T21:23:14.826 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b32c9708:::benchmark_data_smithi192_88961_object46605:head# 2024-06-27T21:23:14.826 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b32f6a2d:::benchmark_data_smithi192_88961_object53042:head# 2024-06-27T21:23:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b331603d:::benchmark_data_smithi192_88961_object65283:head# 2024-06-27T21:23:14.827 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b334341f:::benchmark_data_smithi192_88961_object65248:head# 2024-06-27T21:23:14.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b334f4c5:::benchmark_data_smithi192_88961_object77289:head# 2024-06-27T21:23:14.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3359929:::benchmark_data_smithi192_88961_object79952:head# 2024-06-27T21:23:14.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b336f1c6:::benchmark_data_smithi192_88961_object59071:head# 2024-06-27T21:23:14.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b33b96f8:::benchmark_data_smithi192_88961_object65424:head# 2024-06-27T21:23:14.829 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b33c27f3:::benchmark_data_smithi192_88961_object45913:head# 2024-06-27T21:23:14.829 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b33d30cc:::benchmark_data_smithi192_88961_object68414:head# 2024-06-27T21:23:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b34053c5:::benchmark_data_smithi192_88961_object67206:head# 2024-06-27T21:23:14.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3412708:::benchmark_data_smithi192_88961_object80614:head# 2024-06-27T21:23:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b341a50d:::benchmark_data_smithi192_88961_object89848:head# 2024-06-27T21:23:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b341e998:::benchmark_data_smithi192_88961_object52145:head# 2024-06-27T21:23:14.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b343a44f:::benchmark_data_smithi192_88961_object55478:head# 2024-06-27T21:23:14.832 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b344c758:::benchmark_data_smithi192_88961_object58754:head# 2024-06-27T21:23:14.835 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b344f335:::benchmark_data_smithi192_88961_object87578:head# 2024-06-27T21:23:14.835 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3453cec:::benchmark_data_smithi192_88961_object83729:head# 2024-06-27T21:23:14.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3455d3c:::benchmark_data_smithi192_88961_object65510:head# 2024-06-27T21:23:14.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b348fe1b:::benchmark_data_smithi192_88961_object51578:head# 2024-06-27T21:23:14.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b349f9e6:::benchmark_data_smithi192_88961_object71688:head# 2024-06-27T21:23:14.837 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b34fad76:::benchmark_data_smithi192_88961_object74804:head# 2024-06-27T21:23:14.837 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b350c18a:::benchmark_data_smithi192_88961_object64287:head# 2024-06-27T21:23:14.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3510b8b:::benchmark_data_smithi192_88961_object73030:head# 2024-06-27T21:23:14.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3515545:::benchmark_data_smithi192_88961_object87041:head# 2024-06-27T21:23:14.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3523925:::benchmark_data_smithi192_88961_object58894:head# 2024-06-27T21:23:14.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b353cb70:::benchmark_data_smithi192_88961_object45066:head# 2024-06-27T21:23:14.841 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b35710c8:::benchmark_data_smithi192_88961_object50121:head# 2024-06-27T21:23:14.841 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b357d4c6:::benchmark_data_smithi192_88961_object61276:head# 2024-06-27T21:23:14.842 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b35a6654:::benchmark_data_smithi192_88961_object69467:head# 2024-06-27T21:23:14.842 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b35b2428:::benchmark_data_smithi192_88961_object45781:head# 2024-06-27T21:23:14.843 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b35b2862:::benchmark_data_smithi192_88961_object67115:head# 2024-06-27T21:23:14.844 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b35e2734:::benchmark_data_smithi192_88961_object60309:head# 2024-06-27T21:23:14.844 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b35eb9e7:::benchmark_data_smithi192_88961_object77431:head# 2024-06-27T21:23:14.845 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b36057e0:::benchmark_data_smithi192_88961_object87134:head# 2024-06-27T21:23:14.845 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b361b6f8:::benchmark_data_smithi192_88961_object80558:head# 2024-06-27T21:23:14.846 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3620934:::benchmark_data_smithi192_88961_object66267:head# 2024-06-27T21:23:14.846 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b362d6a3:::benchmark_data_smithi192_88961_object56297:head# 2024-06-27T21:23:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:14.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:14.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b365bf07:::benchmark_data_smithi192_88961_object82980:head# 2024-06-27T21:23:14.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3662597:::benchmark_data_smithi192_88961_object75989:head# 2024-06-27T21:23:14.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b366d503:::benchmark_data_smithi192_88961_object81631:head# 2024-06-27T21:23:14.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b366e749:::benchmark_data_smithi192_88961_object86894:head# 2024-06-27T21:23:14.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b36bea62:::benchmark_data_smithi192_88961_object83081:head# 2024-06-27T21:23:14.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b36bf392:::benchmark_data_smithi192_88961_object65931:head# 2024-06-27T21:23:14.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b36ffb9f:::benchmark_data_smithi192_88961_object72005:head# 2024-06-27T21:23:14.851 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3700771:::benchmark_data_smithi192_88961_object58803:head# 2024-06-27T21:23:14.851 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3707ada:::benchmark_data_smithi192_88961_object43941:head# 2024-06-27T21:23:14.851 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b370d613:::benchmark_data_smithi192_88961_object85735:head# 2024-06-27T21:23:14.852 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b37160a7:::benchmark_data_smithi192_88961_object66440:head# 2024-06-27T21:23:14.852 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b371699c:::benchmark_data_smithi192_88961_object67000:head# 2024-06-27T21:23:14.852 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3737abc:::benchmark_data_smithi192_88961_object54122:head# 2024-06-27T21:23:14.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b374b9c0:::benchmark_data_smithi192_88961_object72388:head# 2024-06-27T21:23:14.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3774a79:::benchmark_data_smithi192_88961_object62891:head# 2024-06-27T21:23:14.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b37dcb5f:::benchmark_data_smithi192_88961_object47357:head# 2024-06-27T21:23:14.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b38230d1:::benchmark_data_smithi192_88961_object45098:head# 2024-06-27T21:23:14.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b382e0c5:::benchmark_data_smithi192_88961_object84019:head# 2024-06-27T21:23:14.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3862083:::benchmark_data_smithi192_88961_object44034:head# 2024-06-27T21:23:14.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3879b3c:::benchmark_data_smithi192_88961_object76314:head# 2024-06-27T21:23:14.856 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b389c65c:::benchmark_data_smithi192_88961_object56182:head# 2024-06-27T21:23:14.856 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b38b90af:::benchmark_data_smithi192_88961_object75105:head# 2024-06-27T21:23:14.858 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b38bcbf1:::benchmark_data_smithi192_88961_object83836:head# 2024-06-27T21:23:14.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b38df052:::benchmark_data_smithi192_88961_object75296:head# 2024-06-27T21:23:14.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b38fa24f:::benchmark_data_smithi192_88961_object50136:head# 2024-06-27T21:23:14.860 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b392757d:::benchmark_data_smithi192_88961_object81904:head# 2024-06-27T21:23:14.860 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b393ac1e:::benchmark_data_smithi192_88961_object73984:head# 2024-06-27T21:23:14.860 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3997731:::benchmark_data_smithi192_88961_object49765:head# 2024-06-27T21:23:14.861 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b39c370e:::benchmark_data_smithi192_88961_object74028:head# 2024-06-27T21:23:14.862 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b39d24ef:::benchmark_data_smithi192_88961_object83863:head# 2024-06-27T21:23:14.862 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b39d9a49:::benchmark_data_smithi192_88961_object68091:head# 2024-06-27T21:23:14.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b39df4ae:::benchmark_data_smithi192_88961_object68597:head# 2024-06-27T21:23:14.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b39e38e2:::benchmark_data_smithi192_88961_object65461:head# 2024-06-27T21:23:14.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a42665:::benchmark_data_smithi192_88961_object89606:head# 2024-06-27T21:23:14.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a52f0f:::benchmark_data_smithi192_88961_object85508:head# 2024-06-27T21:23:14.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a55ac1:::benchmark_data_smithi192_88961_object84933:head# 2024-06-27T21:23:14.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a67ffc:::benchmark_data_smithi192_88961_object48949:head# 2024-06-27T21:23:14.865 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a7b4b1:::benchmark_data_smithi192_88961_object69071:head# 2024-06-27T21:23:14.867 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a7ba39:::benchmark_data_smithi192_88961_object85126:head# 2024-06-27T21:23:14.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a7d460:::benchmark_data_smithi192_88961_object65658:head# 2024-06-27T21:23:14.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3a83dbd:::benchmark_data_smithi192_88961_object69606:head# 2024-06-27T21:23:14.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3aa3d11:::benchmark_data_smithi192_88961_object52497:head# 2024-06-27T21:23:14.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3ab573f:::benchmark_data_smithi192_88961_object81998:head# 2024-06-27T21:23:14.869 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3ad9a56:::benchmark_data_smithi192_88961_object68004:head# 2024-06-27T21:23:14.869 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3ae39d7:::benchmark_data_smithi192_88961_object57406:head# 2024-06-27T21:23:14.870 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3ae435b:::benchmark_data_smithi192_88961_object70847:head# 2024-06-27T21:23:14.870 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b0078c:::benchmark_data_smithi192_88961_object55293:head# 2024-06-27T21:23:14.870 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b070a3:::benchmark_data_smithi192_88961_object87734:head# 2024-06-27T21:23:14.871 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b34e61:::benchmark_data_smithi192_88961_object50287:head# 2024-06-27T21:23:14.871 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b38b70:::benchmark_data_smithi192_88961_object47940:head# 2024-06-27T21:23:14.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b41a5f:::benchmark_data_smithi192_88961_object56248:head# 2024-06-27T21:23:14.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b5a5f3:::benchmark_data_smithi192_88961_object68537:head# 2024-06-27T21:23:14.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3b81e18:::benchmark_data_smithi192_88961_object61478:head# 2024-06-27T21:23:14.873 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3bad26a:::benchmark_data_smithi192_88961_object62553:head# 2024-06-27T21:23:14.873 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3bbff02:::benchmark_data_smithi192_88961_object49528:head# 2024-06-27T21:23:14.874 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3bc8cd3:::benchmark_data_smithi192_88961_object48627:head# 2024-06-27T21:23:14.874 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c2bb89:::benchmark_data_smithi192_88961_object58484:head# 2024-06-27T21:23:14.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c340ca:::benchmark_data_smithi192_88961_object77655:head# 2024-06-27T21:23:14.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c3a58b:::benchmark_data_smithi192_88961_object51970:head# 2024-06-27T21:23:14.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c4885e:::benchmark_data_smithi192_88961_object86762:head# 2024-06-27T21:23:14.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c51bea:::benchmark_data_smithi192_88961_object82468:head# 2024-06-27T21:23:14.877 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c6fe84:::benchmark_data_smithi192_88961_object53677:head# 2024-06-27T21:23:14.877 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c7abae:::benchmark_data_smithi192_88961_object81008:head# 2024-06-27T21:23:14.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c91911:::benchmark_data_smithi192_88961_object48251:head# 2024-06-27T21:23:14.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c93f64:::benchmark_data_smithi192_88961_object55371:head# 2024-06-27T21:23:14.879 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3c9433a:::benchmark_data_smithi192_88961_object55827:head# 2024-06-27T21:23:14.880 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3ca2c6c:::benchmark_data_smithi192_88961_object75437:head# 2024-06-27T21:23:14.880 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cb19ad:::benchmark_data_smithi192_88961_object66835:head# 2024-06-27T21:23:14.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cb40fa:::benchmark_data_smithi192_88961_object75303:head# 2024-06-27T21:23:14.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cb58e3:::benchmark_data_smithi192_88961_object63264:head# 2024-06-27T21:23:14.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cb6852:::benchmark_data_smithi192_88961_object58026:head# 2024-06-27T21:23:14.882 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cc9903:::benchmark_data_smithi192_88961_object51135:head# 2024-06-27T21:23:14.882 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cd1c6c:::benchmark_data_smithi192_88961_object85418:head# 2024-06-27T21:23:14.883 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cf7603:::benchmark_data_smithi192_88961_object70752:head# 2024-06-27T21:23:14.883 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3cf7cc4:::benchmark_data_smithi192_88961_object90201:head# 2024-06-27T21:23:14.885 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d188b8:::benchmark_data_smithi192_88961_object76883:head# 2024-06-27T21:23:14.885 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d24217:::benchmark_data_smithi192_88961_object50188:head# 2024-06-27T21:23:14.885 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d33cbc:::benchmark_data_smithi192_88961_object53462:head# 2024-06-27T21:23:14.886 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d3c629:::benchmark_data_smithi192_88961_object53326:head# 2024-06-27T21:23:14.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d4795f:::benchmark_data_smithi192_88961_object76322:head# 2024-06-27T21:23:14.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d59c97:::benchmark_data_smithi192_88961_object84254:head# 2024-06-27T21:23:14.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d797c0:::benchmark_data_smithi192_88961_object56937:head# 2024-06-27T21:23:14.888 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d8ec1f:::benchmark_data_smithi192_88961_object73784:head# 2024-06-27T21:23:14.888 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3d9fd12:::benchmark_data_smithi192_88961_object52527:head# 2024-06-27T21:23:14.889 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3db6310:::benchmark_data_smithi192_88961_object78774:head# 2024-06-27T21:23:14.889 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3dcf2fb:::benchmark_data_smithi192_88961_object59612:head# 2024-06-27T21:23:14.890 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3deacee:::benchmark_data_smithi192_88961_object65282:head# 2024-06-27T21:23:14.890 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3df086e:::benchmark_data_smithi192_88961_object88252:head# 2024-06-27T21:23:14.891 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e0b1ff:::benchmark_data_smithi192_88961_object72808:head# 2024-06-27T21:23:14.891 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e19950:::benchmark_data_smithi192_88961_object46845:head# 2024-06-27T21:23:14.892 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e29c27:::benchmark_data_smithi192_88961_object53072:head# 2024-06-27T21:23:14.892 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e3e7a3:::benchmark_data_smithi192_88961_object62132:head# 2024-06-27T21:23:14.893 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e40842:::benchmark_data_smithi192_88961_object56790:head# 2024-06-27T21:23:14.893 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e4d4bc:::benchmark_data_smithi192_88961_object78508:head# 2024-06-27T21:23:14.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e54b67:::benchmark_data_smithi192_88961_object45725:head# 2024-06-27T21:23:14.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e66641:::benchmark_data_smithi192_88961_object57659:head# 2024-06-27T21:23:14.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e78543:::benchmark_data_smithi192_88961_object80639:head# 2024-06-27T21:23:14.895 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3e8333d:::benchmark_data_smithi192_88961_object70947:head# 2024-06-27T21:23:14.895 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3ea60ae:::benchmark_data_smithi192_88961_object87476:head# 2024-06-27T21:23:14.895 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3eaaf67:::benchmark_data_smithi192_88961_object78042:head# 2024-06-27T21:23:14.896 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3eb5a00:::benchmark_data_smithi192_88961_object53673:head# 2024-06-27T21:23:14.896 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3edf535:::benchmark_data_smithi192_88961_object51569:head# 2024-06-27T21:23:14.897 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3f2cc56:::benchmark_data_smithi192_88961_object52389:head# 2024-06-27T21:23:14.897 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3f4ae8e:::benchmark_data_smithi192_88961_object88973:head# 2024-06-27T21:23:14.897 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3f63454:::benchmark_data_smithi192_88961_object73387:head# 2024-06-27T21:23:14.898 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3f902c5:::benchmark_data_smithi192_88961_object59991:head# 2024-06-27T21:23:14.898 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3f9de23:::benchmark_data_smithi192_88961_object65672:head# 2024-06-27T21:23:14.898 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3fbb0ca:::benchmark_data_smithi192_88961_object47445:head# 2024-06-27T21:23:14.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b3fdb89f:::benchmark_data_smithi192_88961_object90070:head# 2024-06-27T21:23:14.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4008d43:::benchmark_data_smithi192_88961_object46709:head# 2024-06-27T21:23:14.900 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b402d5fc:::benchmark_data_smithi192_88961_object60211:head# 2024-06-27T21:23:14.900 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40642f7:::benchmark_data_smithi192_88961_object59924:head# 2024-06-27T21:23:14.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b406b6b1:::benchmark_data_smithi192_88961_object58505:head# 2024-06-27T21:23:14.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b406eb18:::benchmark_data_smithi192_88961_object57281:head# 2024-06-27T21:23:14.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40741e8:::benchmark_data_smithi192_88961_object66280:head# 2024-06-27T21:23:14.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40a924f:::benchmark_data_smithi192_88961_object82097:head# 2024-06-27T21:23:14.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40b65bc:::benchmark_data_smithi192_88961_object68575:head# 2024-06-27T21:23:14.903 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40cf3b6:::benchmark_data_smithi192_88961_object87572:head# 2024-06-27T21:23:14.903 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40ef503:::benchmark_data_smithi192_88961_object59123:head# 2024-06-27T21:23:14.903 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40f78eb:::benchmark_data_smithi192_88961_object45674:head# 2024-06-27T21:23:14.904 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b40fc315:::benchmark_data_smithi192_88961_object88867:head# 2024-06-27T21:23:14.904 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b41109aa:::benchmark_data_smithi192_88961_object71971:head# 2024-06-27T21:23:14.904 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b412e9e4:::benchmark_data_smithi192_88961_object77625:head# 2024-06-27T21:23:14.905 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b413ee25:::benchmark_data_smithi192_88961_object48357:head# 2024-06-27T21:23:14.905 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4144113:::benchmark_data_smithi192_88961_object68514:head# 2024-06-27T21:23:14.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b415ed41:::benchmark_data_smithi192_88961_object65116:head# 2024-06-27T21:23:14.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4167466:::benchmark_data_smithi192_88961_object50579:head# 2024-06-27T21:23:14.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b416e755:::benchmark_data_smithi192_88961_object54595:head# 2024-06-27T21:23:14.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b41806a1:::benchmark_data_smithi192_88961_object43722:head# 2024-06-27T21:23:14.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b41caae1:::benchmark_data_smithi192_88961_object58726:head# 2024-06-27T21:23:14.908 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b41d5824:::benchmark_data_smithi192_88961_object50464:head# 2024-06-27T21:23:14.908 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b41e5784:::benchmark_data_smithi192_88961_object68238:head# 2024-06-27T21:23:14.909 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b41eeb5d:::benchmark_data_smithi192_88961_object61526:head# 2024-06-27T21:23:14.909 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b420089c:::benchmark_data_smithi192_88961_object64096:head# 2024-06-27T21:23:14.910 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4206309:::benchmark_data_smithi192_88961_object53205:head# 2024-06-27T21:23:14.910 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42088cb:::benchmark_data_smithi192_88961_object70020:head# 2024-06-27T21:23:14.911 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b424582c:::benchmark_data_smithi192_88961_object85241:head# 2024-06-27T21:23:14.911 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4260513:::benchmark_data_smithi192_88961_object84878:head# 2024-06-27T21:23:14.911 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42695d8:::benchmark_data_smithi192_88961_object73252:head# 2024-06-27T21:23:14.912 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4277970:::benchmark_data_smithi192_88961_object57699:head# 2024-06-27T21:23:14.912 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4298ffd:::benchmark_data_smithi192_88961_object61819:head# 2024-06-27T21:23:14.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b429df2d:::benchmark_data_smithi192_88961_object73675:head# 2024-06-27T21:23:14.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42a3cbe:::benchmark_data_smithi192_88961_object88210:head# 2024-06-27T21:23:14.914 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42b752b:::benchmark_data_smithi192_88961_object64939:head# 2024-06-27T21:23:14.914 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42be6e8:::benchmark_data_smithi192_88961_object76261:head# 2024-06-27T21:23:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42d1784:::benchmark_data_smithi192_88961_object53138:head# 2024-06-27T21:23:14.915 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b42d1fd8:::benchmark_data_smithi192_88961_object73425:head# 2024-06-27T21:23:14.916 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43078a0:::benchmark_data_smithi192_88961_object57452:head# 2024-06-27T21:23:14.916 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b432715c:::benchmark_data_smithi192_88961_object72758:head# 2024-06-27T21:23:14.917 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b432ffd8:::benchmark_data_smithi192_88961_object62103:head# 2024-06-27T21:23:14.917 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4371055:::benchmark_data_smithi192_88961_object74874:head# 2024-06-27T21:23:14.917 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b439210b:::benchmark_data_smithi192_88961_object45272:head# 2024-06-27T21:23:14.918 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b439c5b4:::benchmark_data_smithi192_88961_object82828:head# 2024-06-27T21:23:14.918 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43a0c36:::benchmark_data_smithi192_88961_object49323:head# 2024-06-27T21:23:14.919 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43a5b01:::benchmark_data_smithi192_88961_object64827:head# 2024-06-27T21:23:14.920 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43a62f1:::benchmark_data_smithi192_88961_object47326:head# 2024-06-27T21:23:14.920 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43c3c01:::benchmark_data_smithi192_88961_object87394:head# 2024-06-27T21:23:14.920 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43d6903:::benchmark_data_smithi192_88961_object58212:head# 2024-06-27T21:23:14.921 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43db4d7:::benchmark_data_smithi192_88961_object58345:head# 2024-06-27T21:23:14.921 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b43e6e7f:::benchmark_data_smithi192_88961_object77233:head# 2024-06-27T21:23:14.922 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4409837:::benchmark_data_smithi192_88961_object77327:head# 2024-06-27T21:23:14.922 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b44203da:::benchmark_data_smithi192_88961_object80582:head# 2024-06-27T21:23:14.923 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4424054:::benchmark_data_smithi192_88961_object88721:head# 2024-06-27T21:23:14.923 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b442451f:::benchmark_data_smithi192_88961_object82522:head# 2024-06-27T21:23:14.924 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b444dbe3:::benchmark_data_smithi192_88961_object80310:head# 2024-06-27T21:23:14.924 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b44646e4:::benchmark_data_smithi192_88961_object75027:head# 2024-06-27T21:23:14.924 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4482e30:::benchmark_data_smithi192_88961_object80744:head# 2024-06-27T21:23:14.925 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b449bf9b:::benchmark_data_smithi192_88961_object73926:head# 2024-06-27T21:23:14.925 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b449eca6:::benchmark_data_smithi192_88961_object51508:head# 2024-06-27T21:23:14.925 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b44a5955:::benchmark_data_smithi192_88961_object66871:head# 2024-06-27T21:23:14.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b44ac2e8:::benchmark_data_smithi192_88961_object85802:head# 2024-06-27T21:23:14.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b44c699a:::benchmark_data_smithi192_88961_object59314:head# 2024-06-27T21:23:14.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b44eb691:::benchmark_data_smithi192_88961_object81229:head# 2024-06-27T21:23:14.927 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4506366:::benchmark_data_smithi192_88961_object57597:head# 2024-06-27T21:23:14.927 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b451c4b9:::benchmark_data_smithi192_88961_object78346:head# 2024-06-27T21:23:14.928 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4534d5d:::benchmark_data_smithi192_88961_object81138:head# 2024-06-27T21:23:14.928 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b453f62b:::benchmark_data_smithi192_88961_object84198:head# 2024-06-27T21:23:14.928 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b456ac9f:::benchmark_data_smithi192_88961_object79418:head# 2024-06-27T21:23:14.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b45855ad:::benchmark_data_smithi192_88961_object80455:head# 2024-06-27T21:23:14.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b459818f:::benchmark_data_smithi192_88961_object42965:head# 2024-06-27T21:23:14.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b459d171:::benchmark_data_smithi192_88961_object58773:head# 2024-06-27T21:23:14.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b45a1da2:::benchmark_data_smithi192_88961_object79159:head# 2024-06-27T21:23:14.931 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b45b7f43:::benchmark_data_smithi192_88961_object45333:head# 2024-06-27T21:23:14.931 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b45fc96b:::benchmark_data_smithi192_88961_object71495:head# 2024-06-27T21:23:14.932 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b462adc5:::benchmark_data_smithi192_88961_object55469:head# 2024-06-27T21:23:14.933 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b46543d5:::benchmark_data_smithi192_88961_object61922:head# 2024-06-27T21:23:14.933 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b466198e:::benchmark_data_smithi192_88961_object82503:head# 2024-06-27T21:23:14.934 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4665010:::benchmark_data_smithi192_88961_object53860:head# 2024-06-27T21:23:14.934 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b46936b6:::benchmark_data_smithi192_88961_object59735:head# 2024-06-27T21:23:14.934 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b46bb098:::benchmark_data_smithi192_88961_object59131:head# 2024-06-27T21:23:14.935 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b46cc33e:::benchmark_data_smithi192_88961_object62213:head# 2024-06-27T21:23:14.935 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b46d7a9a:::benchmark_data_smithi192_88961_object86397:head# 2024-06-27T21:23:14.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b46ebc11:::benchmark_data_smithi192_88961_object77521:head# 2024-06-27T21:23:14.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4724f19:::benchmark_data_smithi192_88961_object73406:head# 2024-06-27T21:23:14.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b473d382:::benchmark_data_smithi192_88961_object58876:head# 2024-06-27T21:23:14.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4747ad7:::benchmark_data_smithi192_88961_object52157:head# 2024-06-27T21:23:14.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b47599f7:::benchmark_data_smithi192_88961_object59272:head# 2024-06-27T21:23:14.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b475ab65:::benchmark_data_smithi192_88961_object74574:head# 2024-06-27T21:23:14.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4774652:::benchmark_data_smithi192_88961_object76961:head# 2024-06-27T21:23:14.938 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4775315:::benchmark_data_smithi192_88961_object67723:head# 2024-06-27T21:23:14.938 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b47915f4:::benchmark_data_smithi192_88961_object50673:head# 2024-06-27T21:23:14.939 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b47a1f58:::benchmark_data_smithi192_88961_object56603:head# 2024-06-27T21:23:14.939 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b47c8498:::benchmark_data_smithi192_88961_object89703:head# 2024-06-27T21:23:14.939 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b47dbf7c:::benchmark_data_smithi192_88961_object86898:head# 2024-06-27T21:23:14.940 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b47e2462:::benchmark_data_smithi192_88961_object60804:head# 2024-06-27T21:23:14.940 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b48077ab:::benchmark_data_smithi192_88961_object78077:head# 2024-06-27T21:23:14.940 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b481d0d8:::benchmark_data_smithi192_88961_object72530:head# 2024-06-27T21:23:14.941 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b484eeb8:::benchmark_data_smithi192_88961_object90132:head# 2024-06-27T21:23:14.941 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4853d93:::benchmark_data_smithi192_88961_object86615:head# 2024-06-27T21:23:14.942 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4861d03:::benchmark_data_smithi192_88961_object46652:head# 2024-06-27T21:23:14.942 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b486c844:::benchmark_data_smithi192_88961_object80214:head# 2024-06-27T21:23:14.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b489d27d:::benchmark_data_smithi192_88961_object62764:head# 2024-06-27T21:23:14.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b48aff28:::benchmark_data_smithi192_88961_object63788:head# 2024-06-27T21:23:14.944 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4900402:::benchmark_data_smithi192_88961_object61030:head# 2024-06-27T21:23:14.944 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b491d455:::benchmark_data_smithi192_88961_object58107:head# 2024-06-27T21:23:14.944 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b49b50d6:::benchmark_data_smithi192_88961_object49791:head# 2024-06-27T21:23:14.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b49c8983:::benchmark_data_smithi192_88961_object89701:head# 2024-06-27T21:23:14.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b49d69a7:::benchmark_data_smithi192_88961_object82310:head# 2024-06-27T21:23:14.946 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b49d9a22:::benchmark_data_smithi192_88961_object51291:head# 2024-06-27T21:23:14.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:14 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:23:14.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:14 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:23:14.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:14.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:14.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b49e314b:::benchmark_data_smithi192_88961_object44884:head# 2024-06-27T21:23:14.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b49fb69f:::benchmark_data_smithi192_88961_object70827:head# 2024-06-27T21:23:14.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a0e141:::benchmark_data_smithi192_88961_object57869:head# 2024-06-27T21:23:14.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a2085d:::benchmark_data_smithi192_88961_object62761:head# 2024-06-27T21:23:14.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:14.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a2c0ca:::benchmark_data_smithi192_88961_object81888:head# 2024-06-27T21:23:14.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a40664:::benchmark_data_smithi192_88961_object49944:head# 2024-06-27T21:23:14.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a4d660:::benchmark_data_smithi192_88961_object78558:head# 2024-06-27T21:23:14.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a5eb19:::benchmark_data_smithi192_88961_object76678:head# 2024-06-27T21:23:14.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a69c40:::benchmark_data_smithi192_88961_object43876:head# 2024-06-27T21:23:14.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a6d17a:::benchmark_data_smithi192_88961_object75338:head# 2024-06-27T21:23:14.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a88bbb:::benchmark_data_smithi192_88961_object55544:head# 2024-06-27T21:23:14.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a901d1:::benchmark_data_smithi192_88961_object79169:head# 2024-06-27T21:23:14.951 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a9666e:::benchmark_data_smithi192_88961_object81755:head# 2024-06-27T21:23:14.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4a9facb:::benchmark_data_smithi192_88961_object54041:head# 2024-06-27T21:23:14.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4ab193d:::benchmark_data_smithi192_88961_object50259:head# 2024-06-27T21:23:14.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4acbd78:::benchmark_data_smithi192_88961_object68093:head# 2024-06-27T21:23:14.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b3b11b:::benchmark_data_smithi192_88961_object67287:head# 2024-06-27T21:23:14.953 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b4876c:::benchmark_data_smithi192_88961_object75605:head# 2024-06-27T21:23:14.953 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b5641e:::benchmark_data_smithi192_88961_object88701:head# 2024-06-27T21:23:14.954 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b66c66:::benchmark_data_smithi192_88961_object83207:head# 2024-06-27T21:23:14.954 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b6b63e:::benchmark_data_smithi192_88961_object60383:head# 2024-06-27T21:23:14.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b6c3e3:::benchmark_data_smithi192_88961_object68772:head# 2024-06-27T21:23:14.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b91468:::benchmark_data_smithi192_88961_object90656:head# 2024-06-27T21:23:14.956 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b91f14:::benchmark_data_smithi192_88961_object81303:head# 2024-06-27T21:23:14.957 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4b9a481:::benchmark_data_smithi192_88961_object75229:head# 2024-06-27T21:23:14.957 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4ba2dbd:::benchmark_data_smithi192_88961_object62606:head# 2024-06-27T21:23:14.962 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4bca496:::benchmark_data_smithi192_88961_object64391:head# 2024-06-27T21:23:14.962 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4bdc4f5:::benchmark_data_smithi192_88961_object49615:head# 2024-06-27T21:23:14.962 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4be0da0:::benchmark_data_smithi192_88961_object50531:head# 2024-06-27T21:23:14.963 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c0a587:::benchmark_data_smithi192_88961_object66477:head# 2024-06-27T21:23:14.963 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c2a74c:::benchmark_data_smithi192_88961_object61266:head# 2024-06-27T21:23:14.963 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c2ed22:::benchmark_data_smithi192_88961_object67458:head# 2024-06-27T21:23:14.963 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c57c3b:::benchmark_data_smithi192_88961_object46472:head# 2024-06-27T21:23:14.963 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c5eba5:::benchmark_data_smithi192_88961_object63720:head# 2024-06-27T21:23:14.964 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c66219:::benchmark_data_smithi192_88961_object77452:head# 2024-06-27T21:23:14.965 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c6d71e:::benchmark_data_smithi192_88961_object85678:head# 2024-06-27T21:23:14.965 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c7250b:::benchmark_data_smithi192_88961_object84446:head# 2024-06-27T21:23:14.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c72c3f:::benchmark_data_smithi192_88961_object47959:head# 2024-06-27T21:23:14.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4c8356d:::benchmark_data_smithi192_88961_object48332:head# 2024-06-27T21:23:14.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4ca79f5:::benchmark_data_smithi192_88961_object63700:head# 2024-06-27T21:23:14.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4cb91f3:::benchmark_data_smithi192_88961_object49451:head# 2024-06-27T21:23:14.967 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4cc46ac:::benchmark_data_smithi192_88961_object76216:head# 2024-06-27T21:23:14.968 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4cd7a52:::benchmark_data_smithi192_88961_object58832:head# 2024-06-27T21:23:14.968 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4cdc222:::benchmark_data_smithi192_88961_object69323:head# 2024-06-27T21:23:14.968 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4cf9ec6:::benchmark_data_smithi192_88961_object44794:head# 2024-06-27T21:23:14.969 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d16aea:::benchmark_data_smithi192_88961_object58949:head# 2024-06-27T21:23:14.969 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d2010f:::benchmark_data_smithi192_88961_object60297:head# 2024-06-27T21:23:14.969 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d23477:::benchmark_data_smithi192_88961_object82067:head# 2024-06-27T21:23:14.970 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d447cb:::benchmark_data_smithi192_88961_object60737:head# 2024-06-27T21:23:14.970 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d4b2c5:::benchmark_data_smithi192_88961_object45675:head# 2024-06-27T21:23:14.971 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d55366:::benchmark_data_smithi192_88961_object69209:head# 2024-06-27T21:23:14.971 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d5ee01:::benchmark_data_smithi192_88961_object45471:head# 2024-06-27T21:23:14.971 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d61901:::benchmark_data_smithi192_88961_object43438:head# 2024-06-27T21:23:14.972 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d84fd4:::benchmark_data_smithi192_88961_object53951:head# 2024-06-27T21:23:14.972 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d93111:::benchmark_data_smithi192_88961_object68854:head# 2024-06-27T21:23:14.973 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4d99741:::benchmark_data_smithi192_88961_object57033:head# 2024-06-27T21:23:14.973 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4dabb23:::benchmark_data_smithi192_88961_object83353:head# 2024-06-27T21:23:14.973 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4df41e1:::benchmark_data_smithi192_88961_object62762:head# 2024-06-27T21:23:14.974 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4dfd639:::benchmark_data_smithi192_88961_object72503:head# 2024-06-27T21:23:14.974 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4e3622f:::benchmark_data_smithi192_88961_object75564:head# 2024-06-27T21:23:14.975 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4e498fd:::benchmark_data_smithi192_88961_object81854:head# 2024-06-27T21:23:14.975 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4e6c440:::benchmark_data_smithi192_88961_object53739:head# 2024-06-27T21:23:14.975 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4e88633:::benchmark_data_smithi192_88961_object83460:head# 2024-06-27T21:23:14.976 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4e8c9c4:::benchmark_data_smithi192_88961_object47441:head# 2024-06-27T21:23:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:14.977 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4eb98c2:::benchmark_data_smithi192_88961_object52427:head# 2024-06-27T21:23:14.977 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4ecd47a:::benchmark_data_smithi192_88961_object53440:head# 2024-06-27T21:23:14.978 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4ee71e6:::benchmark_data_smithi192_88961_object52492:head# 2024-06-27T21:23:14.978 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4ee7ee2:::benchmark_data_smithi192_88961_object63272:head# 2024-06-27T21:23:14.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:14.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:14.979 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4eedf72:::benchmark_data_smithi192_88961_object60515:head# 2024-06-27T21:23:14.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f0ef23:::benchmark_data_smithi192_88961_object78761:head# 2024-06-27T21:23:14.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f172d5:::benchmark_data_smithi192_88961_object48280:head# 2024-06-27T21:23:14.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f3e482:::benchmark_data_smithi192_88961_object65022:head# 2024-06-27T21:23:14.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f410a3:::benchmark_data_smithi192_88961_object68780:head# 2024-06-27T21:23:14.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f49582:::benchmark_data_smithi192_88961_object78616:head# 2024-06-27T21:23:14.981 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f6f180:::benchmark_data_smithi192_88961_object75908:head# 2024-06-27T21:23:14.981 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f80a15:::benchmark_data_smithi192_88961_object75872:head# 2024-06-27T21:23:14.981 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f87f3d:::benchmark_data_smithi192_88961_object49094:head# 2024-06-27T21:23:14.981 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f8fdfd:::benchmark_data_smithi192_88961_object66162:head# 2024-06-27T21:23:14.982 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4f99e57:::benchmark_data_smithi192_88961_object85724:head# 2024-06-27T21:23:14.982 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4fb3648:::benchmark_data_smithi192_88961_object80271:head# 2024-06-27T21:23:14.982 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4fb3c4b:::benchmark_data_smithi192_88961_object75443:head# 2024-06-27T21:23:14.983 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4fc62c0:::benchmark_data_smithi192_88961_object45217:head# 2024-06-27T21:23:14.983 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4fd84cd:::benchmark_data_smithi192_88961_object80638:head# 2024-06-27T21:23:14.984 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b4fe956e:::benchmark_data_smithi192_88961_object69042:head# 2024-06-27T21:23:14.984 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5001821:::benchmark_data_smithi192_88961_object49630:head# 2024-06-27T21:23:14.984 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b505f941:::benchmark_data_smithi192_88961_object60848:head# 2024-06-27T21:23:14.985 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b507ef2f:::benchmark_data_smithi192_88961_object83677:head# 2024-06-27T21:23:14.986 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b50a44e8:::benchmark_data_smithi192_88961_object74308:head# 2024-06-27T21:23:14.986 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b50a9886:::benchmark_data_smithi192_88961_object79465:head# 2024-06-27T21:23:14.986 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b50b05ee:::benchmark_data_smithi192_88961_object78917:head# 2024-06-27T21:23:14.987 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b50f3f46:::benchmark_data_smithi192_88961_object55889:head# 2024-06-27T21:23:14.987 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b510f298:::benchmark_data_smithi192_88961_object61826:head# 2024-06-27T21:23:14.988 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b51e1e33:::benchmark_data_smithi192_88961_object83192:head# 2024-06-27T21:23:14.988 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b51ea504:::benchmark_data_smithi192_88961_object71729:head# 2024-06-27T21:23:14.988 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b51ef08a:::benchmark_data_smithi192_88961_object48151:head# 2024-06-27T21:23:14.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b520b680:::benchmark_data_smithi192_88961_object87119:head# 2024-06-27T21:23:14.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b521f6d3:::benchmark_data_smithi192_88961_object52020:head# 2024-06-27T21:23:14.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5226c56:::benchmark_data_smithi192_88961_object83281:head# 2024-06-27T21:23:14.990 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b522fae0:::benchmark_data_smithi192_88961_object52044:head# 2024-06-27T21:23:14.990 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5255039:::benchmark_data_smithi192_88961_object60253:head# 2024-06-27T21:23:14.991 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52658e2:::benchmark_data_smithi192_88961_object85399:head# 2024-06-27T21:23:14.991 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b526eddf:::benchmark_data_smithi192_88961_object55251:head# 2024-06-27T21:23:14.992 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b526efa0:::benchmark_data_smithi192_88961_object84754:head# 2024-06-27T21:23:14.992 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5296894:::benchmark_data_smithi192_88961_object49493:head# 2024-06-27T21:23:14.992 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b529cf3a:::benchmark_data_smithi192_88961_object54728:head# 2024-06-27T21:23:14.993 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52a4d34:::benchmark_data_smithi192_88961_object78704:head# 2024-06-27T21:23:14.993 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52b0359:::benchmark_data_smithi192_88961_object69088:head# 2024-06-27T21:23:14.994 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52b852b:::benchmark_data_smithi192_88961_object90041:head# 2024-06-27T21:23:14.994 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52c8849:::benchmark_data_smithi192_88961_object65601:head# 2024-06-27T21:23:14.994 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52d72c1:::benchmark_data_smithi192_88961_object58634:head# 2024-06-27T21:23:14.995 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52d808a:::benchmark_data_smithi192_88961_object58172:head# 2024-06-27T21:23:14.995 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b52daeac:::benchmark_data_smithi192_88961_object80646:head# 2024-06-27T21:23:14.995 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5331f4a:::benchmark_data_smithi192_88961_object88138:head# 2024-06-27T21:23:14.996 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5334d37:::benchmark_data_smithi192_88961_object63942:head# 2024-06-27T21:23:14.996 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b534dab2:::benchmark_data_smithi192_88961_object84300:head# 2024-06-27T21:23:14.996 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5352355:::benchmark_data_smithi192_88961_object62751:head# 2024-06-27T21:23:14.997 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b535c14a:::benchmark_data_smithi192_88961_object52970:head# 2024-06-27T21:23:14.997 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b536873f:::benchmark_data_smithi192_88961_object88980:head# 2024-06-27T21:23:14.998 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5376920:::benchmark_data_smithi192_88961_object68785:head# 2024-06-27T21:23:14.998 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b538924e:::benchmark_data_smithi192_88961_object76581:head# 2024-06-27T21:23:14.999 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b538b5b1:::benchmark_data_smithi192_88961_object73271:head# 2024-06-27T21:23:14.999 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53977de:::benchmark_data_smithi192_88961_object82840:head# 2024-06-27T21:23:14.999 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53b47ae:::benchmark_data_smithi192_88961_object80980:head# 2024-06-27T21:23:15.000 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53b93c6:::benchmark_data_smithi192_88961_object49043:head# 2024-06-27T21:23:15.000 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53bbf25:::benchmark_data_smithi192_88961_object49182:head# 2024-06-27T21:23:15.000 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53c791f:::benchmark_data_smithi192_88961_object75243:head# 2024-06-27T21:23:15.001 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53c7c44:::benchmark_data_smithi192_88961_object66819:head# 2024-06-27T21:23:15.001 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53cb51c:::benchmark_data_smithi192_88961_object75844:head# 2024-06-27T21:23:15.001 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53cd96a:::benchmark_data_smithi192_88961_object75853:head# 2024-06-27T21:23:15.002 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53cf9cd:::benchmark_data_smithi192_88961_object69436:head# 2024-06-27T21:23:15.002 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b53d9210:::benchmark_data_smithi192_88961_object66253:head# 2024-06-27T21:23:15.003 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b541c326:::benchmark_data_smithi192_88961_object50401:head# 2024-06-27T21:23:15.003 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5426e9e:::benchmark_data_smithi192_88961_object57768:head# 2024-06-27T21:23:15.003 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b542f04f:::benchmark_data_smithi192_88961_object45735:head# 2024-06-27T21:23:15.004 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5438b9c:::benchmark_data_smithi192_88961_object47854:head# 2024-06-27T21:23:15.004 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54415b2:::benchmark_data_smithi192_88961_object90200:head# 2024-06-27T21:23:15.004 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5448993:::benchmark_data_smithi192_88961_object68744:head# 2024-06-27T21:23:15.005 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5454619:::benchmark_data_smithi192_88961_object60106:head# 2024-06-27T21:23:15.005 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b546b723:::benchmark_data_smithi192_88961_object82802:head# 2024-06-27T21:23:15.006 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5471c4a:::benchmark_data_smithi192_88961_object66699:head# 2024-06-27T21:23:15.006 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5480026:::benchmark_data_smithi192_88961_object47964:head# 2024-06-27T21:23:15.006 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54aa8fb:::benchmark_data_smithi192_88961_object84448:head# 2024-06-27T21:23:15.007 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54b65d0:::benchmark_data_smithi192_88961_object72958:head# 2024-06-27T21:23:15.007 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54c6af7:::benchmark_data_smithi192_88961_object80651:head# 2024-06-27T21:23:15.008 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54ee090:::benchmark_data_smithi192_88961_object84560:head# 2024-06-27T21:23:15.008 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54f364e:::benchmark_data_smithi192_88961_object44785:head# 2024-06-27T21:23:15.009 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b54fa8e6:::benchmark_data_smithi192_88961_object64452:head# 2024-06-27T21:23:15.009 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b550250e:::benchmark_data_smithi192_88961_object45295:head# 2024-06-27T21:23:15.009 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b551241f:::benchmark_data_smithi192_88961_object60681:head# 2024-06-27T21:23:15.010 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b553ad6b:::benchmark_data_smithi192_88961_object61285:head# 2024-06-27T21:23:15.010 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b553b381:::benchmark_data_smithi192_88961_object54572:head# 2024-06-27T21:23:15.011 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b553f24c:::benchmark_data_smithi192_88961_object46483:head# 2024-06-27T21:23:15.011 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b556bb22:::benchmark_data_smithi192_88961_object76927:head# 2024-06-27T21:23:15.012 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b55927e2:::benchmark_data_smithi192_88961_object75912:head# 2024-06-27T21:23:15.012 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b559b8c3:::benchmark_data_smithi192_88961_object77442:head# 2024-06-27T21:23:15.012 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b55ca7f2:::benchmark_data_smithi192_88961_object48745:head# 2024-06-27T21:23:15.013 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b55f4011:::benchmark_data_smithi192_88961_object78522:head# 2024-06-27T21:23:15.013 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5605008:::benchmark_data_smithi192_88961_object56666:head# 2024-06-27T21:23:15.013 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56319be:::benchmark_data_smithi192_88961_object89814:head# 2024-06-27T21:23:15.014 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5634849:::benchmark_data_smithi192_88961_object73331:head# 2024-06-27T21:23:15.014 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5681d12:::benchmark_data_smithi192_88961_object81420:head# 2024-06-27T21:23:15.015 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b568d519:::benchmark_data_smithi192_88961_object47029:head# 2024-06-27T21:23:15.015 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5692c93:::benchmark_data_smithi192_88961_object63564:head# 2024-06-27T21:23:15.015 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b569333c:::benchmark_data_smithi192_88961_object86913:head# 2024-06-27T21:23:15.016 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56a705c:::benchmark_data_smithi192_88961_object55693:head# 2024-06-27T21:23:15.017 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56ad5af:::benchmark_data_smithi192_88961_object55208:head# 2024-06-27T21:23:15.017 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56b4a83:::benchmark_data_smithi192_88961_object75777:head# 2024-06-27T21:23:15.017 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56c176c:::benchmark_data_smithi192_88961_object86801:head# 2024-06-27T21:23:15.018 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56eadcb:::benchmark_data_smithi192_88961_object88146:head# 2024-06-27T21:23:15.018 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56ed0ad:::benchmark_data_smithi192_88961_object52315:head# 2024-06-27T21:23:15.019 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56f294c:::benchmark_data_smithi192_88961_object44618:head# 2024-06-27T21:23:15.019 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b56fbd64:::benchmark_data_smithi192_88961_object58649:head# 2024-06-27T21:23:15.019 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5703701:::benchmark_data_smithi192_88961_object71676:head# 2024-06-27T21:23:15.020 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b57317b9:::benchmark_data_smithi192_88961_object49689:head# 2024-06-27T21:23:15.020 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5755838:::benchmark_data_smithi192_88961_object75298:head# 2024-06-27T21:23:15.021 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b57583b4:::benchmark_data_smithi192_88961_object80955:head# 2024-06-27T21:23:15.021 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b577ac68:::benchmark_data_smithi192_88961_object75592:head# 2024-06-27T21:23:15.021 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5782654:::benchmark_data_smithi192_88961_object87520:head# 2024-06-27T21:23:15.022 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b57ba9fa:::benchmark_data_smithi192_88961_object85590:head# 2024-06-27T21:23:15.022 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b580236c:::benchmark_data_smithi192_88961_object45399:head# 2024-06-27T21:23:15.022 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b581ce91:::benchmark_data_smithi192_88961_object44461:head# 2024-06-27T21:23:15.023 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b581e5c2:::benchmark_data_smithi192_88961_object73264:head# 2024-06-27T21:23:15.026 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b582e97b:::benchmark_data_smithi192_88961_object87201:head# 2024-06-27T21:23:15.027 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58555f2:::benchmark_data_smithi192_88961_object60505:head# 2024-06-27T21:23:15.027 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5867969:::benchmark_data_smithi192_88961_object73679:head# 2024-06-27T21:23:15.028 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5870363:::benchmark_data_smithi192_88961_object48391:head# 2024-06-27T21:23:15.028 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b587856b:::benchmark_data_smithi192_88961_object70680:head# 2024-06-27T21:23:15.029 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b588108f:::benchmark_data_smithi192_88961_object83788:head# 2024-06-27T21:23:15.029 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b588f7bc:::benchmark_data_smithi192_88961_object75319:head# 2024-06-27T21:23:15.029 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58943bc:::benchmark_data_smithi192_88961_object74869:head# 2024-06-27T21:23:15.030 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5896717:::benchmark_data_smithi192_88961_object70091:head# 2024-06-27T21:23:15.030 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58d0241:::benchmark_data_smithi192_88961_object62588:head# 2024-06-27T21:23:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58e033c:::benchmark_data_smithi192_88961_object87517:head# 2024-06-27T21:23:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58e57f4:::benchmark_data_smithi192_88961_object52608:head# 2024-06-27T21:23:15.032 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58e588a:::benchmark_data_smithi192_88961_object74272:head# 2024-06-27T21:23:15.032 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b58fe230:::benchmark_data_smithi192_88961_object77127:head# 2024-06-27T21:23:15.032 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59016ef:::benchmark_data_smithi192_88961_object76253:head# 2024-06-27T21:23:15.033 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5919ece:::benchmark_data_smithi192_88961_object47601:head# 2024-06-27T21:23:15.033 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b591a6df:::benchmark_data_smithi192_88961_object71712:head# 2024-06-27T21:23:15.034 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b596964a:::benchmark_data_smithi192_88961_object64338:head# 2024-06-27T21:23:15.034 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5980e9f:::benchmark_data_smithi192_88961_object49144:head# 2024-06-27T21:23:15.034 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59a661e:::benchmark_data_smithi192_88961_object71475:head# 2024-06-27T21:23:15.035 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59a7bfc:::benchmark_data_smithi192_88961_object66016:head# 2024-06-27T21:23:15.035 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59bdff2:::benchmark_data_smithi192_88961_object83494:head# 2024-06-27T21:23:15.035 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59e8ea9:::benchmark_data_smithi192_88961_object42973:head# 2024-06-27T21:23:15.036 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59f1b8b:::benchmark_data_smithi192_88961_object81847:head# 2024-06-27T21:23:15.036 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b59f1f2c:::benchmark_data_smithi192_88961_object82122:head# 2024-06-27T21:23:15.037 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a168fa:::benchmark_data_smithi192_88961_object66686:head# 2024-06-27T21:23:15.037 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a38beb:::benchmark_data_smithi192_88961_object85398:head# 2024-06-27T21:23:15.037 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a3bad2:::benchmark_data_smithi192_88961_object86721:head# 2024-06-27T21:23:15.038 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a44a28:::benchmark_data_smithi192_88961_object54929:head# 2024-06-27T21:23:15.039 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a7c817:::benchmark_data_smithi192_88961_object55442:head# 2024-06-27T21:23:15.039 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a8a52a:::benchmark_data_smithi192_88961_object53373:head# 2024-06-27T21:23:15.039 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5a93226:::benchmark_data_smithi192_88961_object66855:head# 2024-06-27T21:23:15.040 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ab11ad:::benchmark_data_smithi192_88961_object53600:head# 2024-06-27T21:23:15.040 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ac5594:::benchmark_data_smithi192_88961_object63718:head# 2024-06-27T21:23:15.040 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ad15e1:::benchmark_data_smithi192_88961_object69900:head# 2024-06-27T21:23:15.041 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5b35e8e:::benchmark_data_smithi192_88961_object87793:head# 2024-06-27T21:23:15.041 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5b4cd0a:::benchmark_data_smithi192_88961_object63894:head# 2024-06-27T21:23:15.042 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5b61ee6:::benchmark_data_smithi192_88961_object89281:head# 2024-06-27T21:23:15.042 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5b6b5d7:::benchmark_data_smithi192_88961_object77505:head# 2024-06-27T21:23:15.043 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5b9041d:::benchmark_data_smithi192_88961_object81781:head# 2024-06-27T21:23:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:14 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:23:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:14 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:23:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:15.044 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5bb912d:::benchmark_data_smithi192_88961_object63880:head# 2024-06-27T21:23:15.044 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5bc0f10:::benchmark_data_smithi192_88961_object50181:head# 2024-06-27T21:23:15.044 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5bc47cb:::benchmark_data_smithi192_88961_object59023:head# 2024-06-27T21:23:15.045 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5bd8ac8:::benchmark_data_smithi192_88961_object51460:head# 2024-06-27T21:23:15.045 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5c4697c:::benchmark_data_smithi192_88961_object66450:head# 2024-06-27T21:23:15.045 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5c9553f:::benchmark_data_smithi192_88961_object75729:head# 2024-06-27T21:23:15.046 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5cd205a:::benchmark_data_smithi192_88961_object64904:head# 2024-06-27T21:23:15.046 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5cedc37:::benchmark_data_smithi192_88961_object58567:head# 2024-06-27T21:23:15.047 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5cf5059:::benchmark_data_smithi192_88961_object51503:head# 2024-06-27T21:23:15.047 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5d041ca:::benchmark_data_smithi192_88961_object89846:head# 2024-06-27T21:23:15.047 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5d26909:::benchmark_data_smithi192_88961_object63206:head# 2024-06-27T21:23:15.048 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5d79a0d:::benchmark_data_smithi192_88961_object57687:head# 2024-06-27T21:23:15.048 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5d827ab:::benchmark_data_smithi192_88961_object73172:head# 2024-06-27T21:23:15.049 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5d8d5c1:::benchmark_data_smithi192_88961_object82699:head# 2024-06-27T21:23:15.049 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5d915ec:::benchmark_data_smithi192_88961_object79785:head# 2024-06-27T21:23:15.050 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5da80d0:::benchmark_data_smithi192_88961_object50527:head# 2024-06-27T21:23:15.050 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5db26c9:::benchmark_data_smithi192_88961_object81524:head# 2024-06-27T21:23:15.051 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5db4afe:::benchmark_data_smithi192_88961_object71353:head# 2024-06-27T21:23:15.051 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5db7f9b:::benchmark_data_smithi192_88961_object62632:head# 2024-06-27T21:23:15.051 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5dd90f3:::benchmark_data_smithi192_88961_object73249:head# 2024-06-27T21:23:15.052 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ddaeee:::benchmark_data_smithi192_88961_object61779:head# 2024-06-27T21:23:15.052 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e00c0d:::benchmark_data_smithi192_88961_object72149:head# 2024-06-27T21:23:15.053 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e1925c:::benchmark_data_smithi192_88961_object50408:head# 2024-06-27T21:23:15.053 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e3e83f:::benchmark_data_smithi192_88961_object68353:head# 2024-06-27T21:23:15.053 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e51496:::benchmark_data_smithi192_88961_object71443:head# 2024-06-27T21:23:15.054 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e5422e:::benchmark_data_smithi192_88961_object43887:head# 2024-06-27T21:23:15.054 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e66b07:::benchmark_data_smithi192_88961_object59723:head# 2024-06-27T21:23:15.055 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e68d89:::benchmark_data_smithi192_88961_object54969:head# 2024-06-27T21:23:15.055 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5e87dc4:::benchmark_data_smithi192_88961_object53387:head# 2024-06-27T21:23:15.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5eb4b4f:::benchmark_data_smithi192_88961_object84701:head# 2024-06-27T21:23:15.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ec06f8:::benchmark_data_smithi192_88961_object55607:head# 2024-06-27T21:23:15.057 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ee09e3:::benchmark_data_smithi192_88961_object85937:head# 2024-06-27T21:23:15.057 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5ef4d51:::benchmark_data_smithi192_88961_object69225:head# 2024-06-27T21:23:15.057 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5efe235:::benchmark_data_smithi192_88961_object72638:head# 2024-06-27T21:23:15.058 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5f10f48:::benchmark_data_smithi192_88961_object78669:head# 2024-06-27T21:23:15.058 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5f18564:::benchmark_data_smithi192_88961_object63334:head# 2024-06-27T21:23:15.058 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5f2173d:::benchmark_data_smithi192_88961_object49614:head# 2024-06-27T21:23:15.059 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5f576e7:::benchmark_data_smithi192_88961_object67082:head# 2024-06-27T21:23:15.059 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5f68908:::benchmark_data_smithi192_88961_object74178:head# 2024-06-27T21:23:15.060 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5f9fe34:::benchmark_data_smithi192_88961_object84476:head# 2024-06-27T21:23:15.060 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5fc1dad:::benchmark_data_smithi192_88961_object49518:head# 2024-06-27T21:23:15.060 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5fdeba1:::benchmark_data_smithi192_88961_object83354:head# 2024-06-27T21:23:15.061 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b5fe2dff:::benchmark_data_smithi192_88961_object89143:head# 2024-06-27T21:23:15.061 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b60026c9:::benchmark_data_smithi192_88961_object81204:head# 2024-06-27T21:23:15.061 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6004745:::benchmark_data_smithi192_88961_object62456:head# 2024-06-27T21:23:15.062 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6036b13:::benchmark_data_smithi192_88961_object85849:head# 2024-06-27T21:23:15.062 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b603bf4d:::benchmark_data_smithi192_88961_object50676:head# 2024-06-27T21:23:15.063 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6060bfa:::benchmark_data_smithi192_88961_object86712:head# 2024-06-27T21:23:15.063 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6068a35:::benchmark_data_smithi192_88961_object55302:head# 2024-06-27T21:23:15.064 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b607404c:::benchmark_data_smithi192_88961_object47511:head# 2024-06-27T21:23:15.064 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b60cd7b5:::benchmark_data_smithi192_88961_object80690:head# 2024-06-27T21:23:15.065 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b60e97aa:::benchmark_data_smithi192_88961_object88520:head# 2024-06-27T21:23:15.065 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b60f21e0:::benchmark_data_smithi192_88961_object65425:head# 2024-06-27T21:23:15.065 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b610a48c:::benchmark_data_smithi192_88961_object56766:head# 2024-06-27T21:23:15.066 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61390c8:::benchmark_data_smithi192_88961_object83206:head# 2024-06-27T21:23:15.066 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b613fcf7:::benchmark_data_smithi192_88961_object43323:head# 2024-06-27T21:23:15.067 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61a093e:::benchmark_data_smithi192_88961_object63009:head# 2024-06-27T21:23:15.067 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61c14d4:::benchmark_data_smithi192_88961_object79498:head# 2024-06-27T21:23:15.068 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61cb3a5:::benchmark_data_smithi192_88961_object51309:head# 2024-06-27T21:23:15.068 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61e97a9:::benchmark_data_smithi192_88961_object83067:head# 2024-06-27T21:23:15.069 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61ebb12:::benchmark_data_smithi192_88961_object54123:head# 2024-06-27T21:23:15.069 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b61f8c8c:::benchmark_data_smithi192_88961_object45886:head# 2024-06-27T21:23:15.070 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b622cb6c:::benchmark_data_smithi192_88961_object79475:head# 2024-06-27T21:23:15.070 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b622d273:::benchmark_data_smithi192_88961_object83168:head# 2024-06-27T21:23:15.070 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b622f55e:::benchmark_data_smithi192_88961_object78730:head# 2024-06-27T21:23:15.071 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6250b67:::benchmark_data_smithi192_88961_object47686:head# 2024-06-27T21:23:15.073 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6251526:::benchmark_data_smithi192_88961_object65543:head# 2024-06-27T21:23:15.073 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6258c3f:::benchmark_data_smithi192_88961_object67911:head# 2024-06-27T21:23:15.074 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b625a9f0:::benchmark_data_smithi192_88961_object88336:head# 2024-06-27T21:23:15.074 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6270e37:::benchmark_data_smithi192_88961_object58675:head# 2024-06-27T21:23:15.076 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b627460a:::benchmark_data_smithi192_88961_object87174:head# 2024-06-27T21:23:15.076 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b627636e:::benchmark_data_smithi192_88961_object70307:head# 2024-06-27T21:23:15.076 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6289a0b:::benchmark_data_smithi192_88961_object76743:head# 2024-06-27T21:23:15.077 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b62abfae:::benchmark_data_smithi192_88961_object56789:head# 2024-06-27T21:23:15.077 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b62ae722:::benchmark_data_smithi192_88961_object58776:head# 2024-06-27T21:23:15.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:15.080 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b62e4487:::benchmark_data_smithi192_88961_object43565:head# 2024-06-27T21:23:15.080 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b62f74a5:::benchmark_data_smithi192_88961_object63066:head# 2024-06-27T21:23:15.081 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b631bdb9:::benchmark_data_smithi192_88961_object65962:head# 2024-06-27T21:23:15.081 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b631c34c:::benchmark_data_smithi192_88961_object90254:head# 2024-06-27T21:23:15.082 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b631f04d:::benchmark_data_smithi192_88961_object90679:head# 2024-06-27T21:23:15.083 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6325f24:::benchmark_data_smithi192_88961_object55860:head# 2024-06-27T21:23:15.083 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6329650:::benchmark_data_smithi192_88961_object55062:head# 2024-06-27T21:23:15.086 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6364a25:::benchmark_data_smithi192_88961_object80506:head# 2024-06-27T21:23:15.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6370243:::benchmark_data_smithi192_88961_object57327:head# 2024-06-27T21:23:15.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63aa41e:::benchmark_data_smithi192_88961_object84688:head# 2024-06-27T21:23:15.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63bc387:::benchmark_data_smithi192_88961_object86594:head# 2024-06-27T21:23:15.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63d2ed3:::benchmark_data_smithi192_88961_object54624:head# 2024-06-27T21:23:15.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63d4713:::benchmark_data_smithi192_88961_object87693:head# 2024-06-27T21:23:15.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63ea525:::benchmark_data_smithi192_88961_object70597:head# 2024-06-27T21:23:15.088 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63ebc91:::benchmark_data_smithi192_88961_object51081:head# 2024-06-27T21:23:15.088 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63ee422:::benchmark_data_smithi192_88961_object65086:head# 2024-06-27T21:23:15.089 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b63f8046:::benchmark_data_smithi192_88961_object87081:head# 2024-06-27T21:23:15.089 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6406790:::benchmark_data_smithi192_88961_object49683:head# 2024-06-27T21:23:15.090 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6458e76:::benchmark_data_smithi192_88961_object81775:head# 2024-06-27T21:23:15.091 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b646d2f0:::benchmark_data_smithi192_88961_object64985:head# 2024-06-27T21:23:15.091 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b64829fe:::benchmark_data_smithi192_88961_object43037:head# 2024-06-27T21:23:15.091 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b64aa763:::benchmark_data_smithi192_88961_object57553:head# 2024-06-27T21:23:15.091 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b64ca395:::benchmark_data_smithi192_88961_object54836:head# 2024-06-27T21:23:15.092 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b64d094a:::benchmark_data_smithi192_88961_object43674:head# 2024-06-27T21:23:15.092 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b64e7b5d:::benchmark_data_smithi192_88961_object62758:head# 2024-06-27T21:23:15.092 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b64fdfbc:::benchmark_data_smithi192_88961_object68784:head# 2024-06-27T21:23:15.093 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6508dbe:::benchmark_data_smithi192_88961_object66279:head# 2024-06-27T21:23:15.093 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b650fe33:::benchmark_data_smithi192_88961_object64369:head# 2024-06-27T21:23:15.093 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65634a9:::benchmark_data_smithi192_88961_object47242:head# 2024-06-27T21:23:15.095 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65651ef:::benchmark_data_smithi192_88961_object58518:head# 2024-06-27T21:23:15.095 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65a8fea:::benchmark_data_smithi192_88961_object76651:head# 2024-06-27T21:23:15.095 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65b5bb3:::benchmark_data_smithi192_88961_object75701:head# 2024-06-27T21:23:15.096 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65ca8fe:::benchmark_data_smithi192_88961_object45666:head# 2024-06-27T21:23:15.097 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65d18d1:::benchmark_data_smithi192_88961_object43877:head# 2024-06-27T21:23:15.097 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65d44d3:::benchmark_data_smithi192_88961_object64502:head# 2024-06-27T21:23:15.099 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65d5258:::benchmark_data_smithi192_88961_object59354:head# 2024-06-27T21:23:15.100 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65dca54:::benchmark_data_smithi192_88961_object75197:head# 2024-06-27T21:23:15.102 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65e8ca3:::benchmark_data_smithi192_88961_object59144:head# 2024-06-27T21:23:15.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:15.104 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b65f2081:::benchmark_data_smithi192_88961_object78973:head# 2024-06-27T21:23:15.104 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6601ce4:::benchmark_data_smithi192_88961_object56048:head# 2024-06-27T21:23:15.104 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6605830:::benchmark_data_smithi192_88961_object59788:head# 2024-06-27T21:23:15.104 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b66101ee:::benchmark_data_smithi192_88961_object61456:head# 2024-06-27T21:23:15.104 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b663ca78:::benchmark_data_smithi192_88961_object70763:head# 2024-06-27T21:23:15.105 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6658c84:::benchmark_data_smithi192_88961_object89513:head# 2024-06-27T21:23:15.105 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b665a1af:::benchmark_data_smithi192_88961_object63346:head# 2024-06-27T21:23:15.105 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b665ccc8:::benchmark_data_smithi192_88961_object84846:head# 2024-06-27T21:23:15.106 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b667441a:::benchmark_data_smithi192_88961_object81469:head# 2024-06-27T21:23:15.106 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b668cd5c:::benchmark_data_smithi192_88961_object85976:head# 2024-06-27T21:23:15.107 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b668f1aa:::benchmark_data_smithi192_88961_object79207:head# 2024-06-27T21:23:15.107 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6698348:::benchmark_data_smithi192_88961_object59970:head# 2024-06-27T21:23:15.107 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b66f8093:::benchmark_data_smithi192_88961_object45300:head# 2024-06-27T21:23:15.108 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b672bfdf:::benchmark_data_smithi192_88961_object46147:head# 2024-06-27T21:23:15.108 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6730b89:::benchmark_data_smithi192_88961_object73328:head# 2024-06-27T21:23:15.108 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67439a2:::benchmark_data_smithi192_88961_object50745:head# 2024-06-27T21:23:15.108 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67473aa:::benchmark_data_smithi192_88961_object61137:head# 2024-06-27T21:23:15.109 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b676288b:::benchmark_data_smithi192_88961_object84017:head# 2024-06-27T21:23:15.109 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67678d9:::benchmark_data_smithi192_88961_object86808:head# 2024-06-27T21:23:15.110 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b676b1d4:::benchmark_data_smithi192_88961_object57140:head# 2024-06-27T21:23:15.110 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67797f4:::benchmark_data_smithi192_88961_object48540:head# 2024-06-27T21:23:15.111 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67a38f5:::benchmark_data_smithi192_88961_object89263:head# 2024-06-27T21:23:15.111 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67dd177:::benchmark_data_smithi192_88961_object43385:head# 2024-06-27T21:23:15.112 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67e6423:::benchmark_data_smithi192_88961_object44096:head# 2024-06-27T21:23:15.112 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b67f9365:::benchmark_data_smithi192_88961_object71184:head# 2024-06-27T21:23:15.113 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6805fe6:::benchmark_data_smithi192_88961_object88403:head# 2024-06-27T21:23:15.113 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6819963:::benchmark_data_smithi192_88961_object60926:head# 2024-06-27T21:23:15.113 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6825fe1:::benchmark_data_smithi192_88961_object68437:head# 2024-06-27T21:23:15.114 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b686c95e:::benchmark_data_smithi192_88961_object78814:head# 2024-06-27T21:23:15.114 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b687deac:::benchmark_data_smithi192_88961_object86716:head# 2024-06-27T21:23:15.115 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b689a5dc:::benchmark_data_smithi192_88961_object57538:head# 2024-06-27T21:23:15.115 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b68a9c3c:::benchmark_data_smithi192_88961_object59189:head# 2024-06-27T21:23:15.115 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b68cf7c7:::benchmark_data_smithi192_88961_object90039:head# 2024-06-27T21:23:15.116 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b68fdc7b:::benchmark_data_smithi192_88961_object79004:head# 2024-06-27T21:23:15.116 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b690a1f1:::benchmark_data_smithi192_88961_object48802:head# 2024-06-27T21:23:15.116 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b690aa47:::benchmark_data_smithi192_88961_object51943:head# 2024-06-27T21:23:15.117 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b691a89c:::benchmark_data_smithi192_88961_object56582:head# 2024-06-27T21:23:15.117 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6929b65:::benchmark_data_smithi192_88961_object82869:head# 2024-06-27T21:23:15.118 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b693dfb9:::benchmark_data_smithi192_88961_object76115:head# 2024-06-27T21:23:15.118 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b694b6a6:::benchmark_data_smithi192_88961_object49171:head# 2024-06-27T21:23:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b694d16a:::benchmark_data_smithi192_88961_object89615:head# 2024-06-27T21:23:15.119 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69752ad:::benchmark_data_smithi192_88961_object46644:head# 2024-06-27T21:23:15.120 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b697c178:::benchmark_data_smithi192_88961_object83896:head# 2024-06-27T21:23:15.120 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69803d9:::benchmark_data_smithi192_88961_object63979:head# 2024-06-27T21:23:15.121 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69a1faa:::benchmark_data_smithi192_88961_object76395:head# 2024-06-27T21:23:15.122 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69c67b6:::benchmark_data_smithi192_88961_object70378:head# 2024-06-27T21:23:15.122 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69cd768:::benchmark_data_smithi192_88961_object84726:head# 2024-06-27T21:23:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69f3135:::benchmark_data_smithi192_88961_object63871:head# 2024-06-27T21:23:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69f5ce3:::benchmark_data_smithi192_88961_object58780:head# 2024-06-27T21:23:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b69fcd21:::benchmark_data_smithi192_88961_object58150:head# 2024-06-27T21:23:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a46598:::benchmark_data_smithi192_88961_object43464:head# 2024-06-27T21:23:15.124 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a6015d:::benchmark_data_smithi192_88961_object72706:head# 2024-06-27T21:23:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a606e5:::benchmark_data_smithi192_88961_object83339:head# 2024-06-27T21:23:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a61ae0:::benchmark_data_smithi192_88961_object82762:head# 2024-06-27T21:23:15.125 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a6f2cf:::benchmark_data_smithi192_88961_object52295:head# 2024-06-27T21:23:15.126 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a723f0:::benchmark_data_smithi192_88961_object56427:head# 2024-06-27T21:23:15.126 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a7f0fd:::benchmark_data_smithi192_88961_object87851:head# 2024-06-27T21:23:15.127 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6a90d63:::benchmark_data_smithi192_88961_object49095:head# 2024-06-27T21:23:15.127 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6aada4d:::benchmark_data_smithi192_88961_object59522:head# 2024-06-27T21:23:15.128 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6ac7dfc:::benchmark_data_smithi192_88961_object61655:head# 2024-06-27T21:23:15.128 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b04c41:::benchmark_data_smithi192_88961_object70420:head# 2024-06-27T21:23:15.129 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b23c66:::benchmark_data_smithi192_88961_object54553:head# 2024-06-27T21:23:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b23ff6:::benchmark_data_smithi192_88961_object73722:head# 2024-06-27T21:23:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b4a2e4:::benchmark_data_smithi192_88961_object87955:head# 2024-06-27T21:23:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b5122e:::benchmark_data_smithi192_88961_object44195:head# 2024-06-27T21:23:15.131 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b5b0ac:::benchmark_data_smithi192_88961_object83621:head# 2024-06-27T21:23:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b7c014:::benchmark_data_smithi192_88961_object75808:head# 2024-06-27T21:23:15.132 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6b8ede2:::benchmark_data_smithi192_88961_object65481:head# 2024-06-27T21:23:15.133 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6ba644f:::benchmark_data_smithi192_88961_object78202:head# 2024-06-27T21:23:15.133 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6bb2b43:::benchmark_data_smithi192_88961_object80057:head# 2024-06-27T21:23:15.134 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6bd0268:::benchmark_data_smithi192_88961_object69961:head# 2024-06-27T21:23:15.134 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6beb63e:::benchmark_data_smithi192_88961_object88627:head# 2024-06-27T21:23:15.134 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6bf1fab:::benchmark_data_smithi192_88961_object77527:head# 2024-06-27T21:23:15.135 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6bfb12b:::benchmark_data_smithi192_88961_object60019:head# 2024-06-27T21:23:15.135 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c1944c:::benchmark_data_smithi192_88961_object50261:head# 2024-06-27T21:23:15.136 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c2de8f:::benchmark_data_smithi192_88961_object79949:head# 2024-06-27T21:23:15.136 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c49a64:::benchmark_data_smithi192_88961_object90625:head# 2024-06-27T21:23:15.139 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c4d8c8:::benchmark_data_smithi192_88961_object71589:head# 2024-06-27T21:23:15.140 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c4da84:::benchmark_data_smithi192_88961_object71237:head# 2024-06-27T21:23:15.140 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c59d6d:::benchmark_data_smithi192_88961_object69877:head# 2024-06-27T21:23:15.141 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c77981:::benchmark_data_smithi192_88961_object83957:head# 2024-06-27T21:23:15.141 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c7a5b2:::benchmark_data_smithi192_88961_object60675:head# 2024-06-27T21:23:15.144 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c8028d:::benchmark_data_smithi192_88961_object77368:head# 2024-06-27T21:23:15.144 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6c82f17:::benchmark_data_smithi192_88961_object83363:head# 2024-06-27T21:23:15.144 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6cb82bd:::benchmark_data_smithi192_88961_object70429:head# 2024-06-27T21:23:15.145 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6cbe570:::benchmark_data_smithi192_88961_object60209:head# 2024-06-27T21:23:15.145 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6cbf49a:::benchmark_data_smithi192_88961_object75051:head# 2024-06-27T21:23:15.146 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6cd03d2:::benchmark_data_smithi192_88961_object52794:head# 2024-06-27T21:23:15.146 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6cd2308:::benchmark_data_smithi192_88961_object70613:head# 2024-06-27T21:23:15.146 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6ce0b75:::benchmark_data_smithi192_88961_object59282:head# 2024-06-27T21:23:15.147 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6d01f0c:::benchmark_data_smithi192_88961_object68629:head# 2024-06-27T21:23:15.147 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6d0e177:::benchmark_data_smithi192_88961_object46855:head# 2024-06-27T21:23:15.148 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6d2353e:::benchmark_data_smithi192_88961_object69316:head# 2024-06-27T21:23:15.148 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6d60c12:::benchmark_data_smithi192_88961_object78543:head# 2024-06-27T21:23:15.149 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6d74f52:::benchmark_data_smithi192_88961_object80694:head# 2024-06-27T21:23:15.149 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6d8795a:::benchmark_data_smithi192_88961_object84516:head# 2024-06-27T21:23:15.149 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6dbf552:::benchmark_data_smithi192_88961_object80079:head# 2024-06-27T21:23:15.150 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6dd9400:::benchmark_data_smithi192_88961_object50473:head# 2024-06-27T21:23:15.150 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6dfb172:::benchmark_data_smithi192_88961_object85980:head# 2024-06-27T21:23:15.151 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6e185f5:::benchmark_data_smithi192_88961_object73017:head# 2024-06-27T21:23:15.151 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6e267c9:::benchmark_data_smithi192_88961_object44183:head# 2024-06-27T21:23:15.151 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6e2d0f5:::benchmark_data_smithi192_88961_object63379:head# 2024-06-27T21:23:15.152 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6e63f1f:::benchmark_data_smithi192_88961_object83240:head# 2024-06-27T21:23:15.152 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6e84745:::benchmark_data_smithi192_88961_object63029:head# 2024-06-27T21:23:15.153 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6e8bc7d:::benchmark_data_smithi192_88961_object77623:head# 2024-06-27T21:23:15.153 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6eb0589:::benchmark_data_smithi192_88961_object62074:head# 2024-06-27T21:23:15.153 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6eb990e:::benchmark_data_smithi192_88961_object78769:head# 2024-06-27T21:23:15.154 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6eebd34:::benchmark_data_smithi192_88961_object62294:head# 2024-06-27T21:23:15.155 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f23c00:::benchmark_data_smithi192_88961_object73465:head# 2024-06-27T21:23:15.156 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f2c81d:::benchmark_data_smithi192_88961_object85035:head# 2024-06-27T21:23:15.156 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f2f67e:::benchmark_data_smithi192_88961_object87464:head# 2024-06-27T21:23:15.156 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f322c0:::benchmark_data_smithi192_88961_object46403:head# 2024-06-27T21:23:15.157 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f43dd5:::benchmark_data_smithi192_88961_object67316:head# 2024-06-27T21:23:15.157 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f76b52:::benchmark_data_smithi192_88961_object87152:head# 2024-06-27T21:23:15.157 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6f8bab9:::benchmark_data_smithi192_88961_object60951:head# 2024-06-27T21:23:15.157 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6fbf4b8:::benchmark_data_smithi192_88961_object66159:head# 2024-06-27T21:23:15.158 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6fc6747:::benchmark_data_smithi192_88961_object80992:head# 2024-06-27T21:23:15.159 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6fc7398:::benchmark_data_smithi192_88961_object89339:head# 2024-06-27T21:23:15.159 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6fd748b:::benchmark_data_smithi192_88961_object81968:head# 2024-06-27T21:23:15.159 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6ff7a5b:::benchmark_data_smithi192_88961_object65540:head# 2024-06-27T21:23:15.160 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b6ff8b6a:::benchmark_data_smithi192_88961_object54204:head# 2024-06-27T21:23:15.160 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b700ad0f:::benchmark_data_smithi192_88961_object46218:head# 2024-06-27T21:23:15.160 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b700c975:::benchmark_data_smithi192_88961_object72113:head# 2024-06-27T21:23:15.160 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7014924:::benchmark_data_smithi192_88961_object84422:head# 2024-06-27T21:23:15.161 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7020058:::benchmark_data_smithi192_88961_object75179:head# 2024-06-27T21:23:15.161 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b70235c5:::benchmark_data_smithi192_88961_object52502:head# 2024-06-27T21:23:15.162 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b702eedc:::benchmark_data_smithi192_88961_object63005:head# 2024-06-27T21:23:15.163 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b703ca8d:::benchmark_data_smithi192_88961_object72832:head# 2024-06-27T21:23:15.163 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7045576:::benchmark_data_smithi192_88961_object50033:head# 2024-06-27T21:23:15.163 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b70d2bf5:::benchmark_data_smithi192_88961_object77238:head# 2024-06-27T21:23:15.163 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b70e0c2a:::benchmark_data_smithi192_88961_object48513:head# 2024-06-27T21:23:15.164 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b70ee090:::benchmark_data_smithi192_88961_object79926:head# 2024-06-27T21:23:15.164 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7130a85:::benchmark_data_smithi192_88961_object48853:head# 2024-06-27T21:23:15.164 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b71389b7:::benchmark_data_smithi192_88961_object55557:head# 2024-06-27T21:23:15.164 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7177aa6:::benchmark_data_smithi192_88961_object62560:head# 2024-06-27T21:23:15.165 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b718a494:::benchmark_data_smithi192_88961_object48067:head# 2024-06-27T21:23:15.165 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b71910ae:::benchmark_data_smithi192_88961_object51662:head# 2024-06-27T21:23:15.166 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b719427a:::benchmark_data_smithi192_88961_object65205:head# 2024-06-27T21:23:15.166 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b719ec01:::benchmark_data_smithi192_88961_object85061:head# 2024-06-27T21:23:15.166 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b71c664e:::benchmark_data_smithi192_88961_object72157:head# 2024-06-27T21:23:15.167 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b71cbe85:::benchmark_data_smithi192_88961_object50804:head# 2024-06-27T21:23:15.167 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b71dc7f2:::benchmark_data_smithi192_88961_object66473:head# 2024-06-27T21:23:15.168 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7201894:::benchmark_data_smithi192_88961_object82666:head# 2024-06-27T21:23:15.168 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7207bba:::benchmark_data_smithi192_88961_object73645:head# 2024-06-27T21:23:15.169 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7211fa7:::benchmark_data_smithi192_88961_object62337:head# 2024-06-27T21:23:15.169 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b724573a:::benchmark_data_smithi192_88961_object82575:head# 2024-06-27T21:23:15.169 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b725fd72:::benchmark_data_smithi192_88961_object83346:head# 2024-06-27T21:23:15.170 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7274ac5:::benchmark_data_smithi192_88961_object84462:head# 2024-06-27T21:23:15.170 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b728824c:::benchmark_data_smithi192_88961_object73656:head# 2024-06-27T21:23:15.171 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7295ba5:::benchmark_data_smithi192_88961_object49686:head# 2024-06-27T21:23:15.171 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b72c0141:::benchmark_data_smithi192_88961_object61338:head# 2024-06-27T21:23:15.171 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b72c2c4a:::benchmark_data_smithi192_88961_object45212:head# 2024-06-27T21:23:15.171 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b72c2ecc:::benchmark_data_smithi192_88961_object82457:head# 2024-06-27T21:23:15.172 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b72c754e:::benchmark_data_smithi192_88961_object77650:head# 2024-06-27T21:23:15.173 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b72d0060:::benchmark_data_smithi192_88961_object67787:head# 2024-06-27T21:23:15.173 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b72f3374:::benchmark_data_smithi192_88961_object68369:head# 2024-06-27T21:23:15.173 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b733331c:::benchmark_data_smithi192_88961_object55670:head# 2024-06-27T21:23:15.174 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b734dd69:::benchmark_data_smithi192_88961_object65024:head# 2024-06-27T21:23:15.174 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b736635f:::benchmark_data_smithi192_88961_object65364:head# 2024-06-27T21:23:15.175 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b738a558:::benchmark_data_smithi192_88961_object88418:head# 2024-06-27T21:23:15.175 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b738b4b1:::benchmark_data_smithi192_88961_object58864:head# 2024-06-27T21:23:15.176 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b73a6ec8:::benchmark_data_smithi192_88961_object43741:head# 2024-06-27T21:23:15.176 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b73c56b2:::benchmark_data_smithi192_88961_object60636:head# 2024-06-27T21:23:15.176 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b73c9484:::benchmark_data_smithi192_88961_object77030:head# 2024-06-27T21:23:15.177 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b73cfdde:::benchmark_data_smithi192_88961_object63419:head# 2024-06-27T21:23:15.178 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b73e131f:::benchmark_data_smithi192_88961_object83158:head# 2024-06-27T21:23:15.178 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b741ece0:::benchmark_data_smithi192_88961_object87583:head# 2024-06-27T21:23:15.179 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b742be32:::benchmark_data_smithi192_88961_object45040:head# 2024-06-27T21:23:15.179 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7440a26:::benchmark_data_smithi192_88961_object79907:head# 2024-06-27T21:23:15.179 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b744435a:::benchmark_data_smithi192_88961_object50482:head# 2024-06-27T21:23:15.180 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b74606a8:::benchmark_data_smithi192_88961_object89092:head# 2024-06-27T21:23:15.180 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b746a648:::benchmark_data_smithi192_88961_object49662:head# 2024-06-27T21:23:15.180 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b746be58:::benchmark_data_smithi192_88961_object54338:head# 2024-06-27T21:23:15.181 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7481316:::benchmark_data_smithi192_88961_object56000:head# 2024-06-27T21:23:15.181 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b748c9ad:::benchmark_data_smithi192_88961_object70474:head# 2024-06-27T21:23:15.182 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7498daf:::benchmark_data_smithi192_88961_object79274:head# 2024-06-27T21:23:15.185 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b749c4d0:::benchmark_data_smithi192_88961_object62210:head# 2024-06-27T21:23:15.185 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b74a4155:::benchmark_data_smithi192_88961_object65413:head# 2024-06-27T21:23:15.185 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b74c6a13:::benchmark_data_smithi192_88961_object65340:head# 2024-06-27T21:23:15.185 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b74cbb64:::benchmark_data_smithi192_88961_object82835:head# 2024-06-27T21:23:15.186 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7505aaf:::benchmark_data_smithi192_88961_object68539:head# 2024-06-27T21:23:15.186 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b750afd1:::benchmark_data_smithi192_88961_object70780:head# 2024-06-27T21:23:15.186 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b754a57f:::benchmark_data_smithi192_88961_object70788:head# 2024-06-27T21:23:15.186 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b754abdc:::benchmark_data_smithi192_88961_object52737:head# 2024-06-27T21:23:15.187 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b75598bc:::benchmark_data_smithi192_88961_object88456:head# 2024-06-27T21:23:15.187 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7561b63:::benchmark_data_smithi192_88961_object50924:head# 2024-06-27T21:23:15.188 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b75ab834:::benchmark_data_smithi192_88961_object59849:head# 2024-06-27T21:23:15.188 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b75bc7bc:::benchmark_data_smithi192_88961_object56362:head# 2024-06-27T21:23:15.189 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b75bd1f3:::benchmark_data_smithi192_88961_object66101:head# 2024-06-27T21:23:15.189 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b75c694d:::benchmark_data_smithi192_88961_object66040:head# 2024-06-27T21:23:15.190 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b75dbbbe:::benchmark_data_smithi192_88961_object44504:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7615173:::benchmark_data_smithi192_88961_object80937:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b76376c2:::benchmark_data_smithi192_88961_object64030:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7699694:::benchmark_data_smithi192_88961_object85888:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b76a8921:::benchmark_data_smithi192_88961_object60798:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b76b5d73:::benchmark_data_smithi192_88961_object87161:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b771086e:::benchmark_data_smithi192_88961_object50852:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7742d3d:::benchmark_data_smithi192_88961_object77926:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77683eb:::benchmark_data_smithi192_88961_object81643:head# 2024-06-27T21:23:15.195 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7773363:::benchmark_data_smithi192_88961_object46070:head# 2024-06-27T21:23:15.196 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b777df05:::benchmark_data_smithi192_88961_object52731:head# 2024-06-27T21:23:15.196 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77804c5:::benchmark_data_smithi192_88961_object62102:head# 2024-06-27T21:23:15.196 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b778a992:::benchmark_data_smithi192_88961_object89584:head# 2024-06-27T21:23:15.196 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7790487:::benchmark_data_smithi192_88961_object46714:head# 2024-06-27T21:23:15.197 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77947a6:::benchmark_data_smithi192_88961_object43023:head# 2024-06-27T21:23:15.197 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77abcce:::benchmark_data_smithi192_88961_object74241:head# 2024-06-27T21:23:15.197 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77b30fa:::benchmark_data_smithi192_88961_object79143:head# 2024-06-27T21:23:15.198 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77c9fcc:::benchmark_data_smithi192_88961_object81895:head# 2024-06-27T21:23:15.198 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77cf000:::benchmark_data_smithi192_88961_object57132:head# 2024-06-27T21:23:15.199 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77cf7bc:::benchmark_data_smithi192_88961_object86385:head# 2024-06-27T21:23:15.199 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77fa32d:::benchmark_data_smithi192_88961_object87396:head# 2024-06-27T21:23:15.199 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77feaf1:::benchmark_data_smithi192_88961_object71154:head# 2024-06-27T21:23:15.200 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b77fee99:::benchmark_data_smithi192_88961_object44426:head# 2024-06-27T21:23:15.203 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b78236a0:::benchmark_data_smithi192_88961_object67596:head# 2024-06-27T21:23:15.203 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b782c960:::benchmark_data_smithi192_88961_object45436:head# 2024-06-27T21:23:15.203 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7835fac:::benchmark_data_smithi192_88961_object62228:head# 2024-06-27T21:23:15.204 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7855dc0:::benchmark_data_smithi192_88961_object70459:head# 2024-06-27T21:23:15.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:15.205 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b785bc74:::benchmark_data_smithi192_88961_object47282:head# 2024-06-27T21:23:15.206 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b786b2de:::benchmark_data_smithi192_88961_object47459:head# 2024-06-27T21:23:15.206 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b788e52d:::benchmark_data_smithi192_88961_object68133:head# 2024-06-27T21:23:15.208 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b78b93f9:::benchmark_data_smithi192_88961_object78575:head# 2024-06-27T21:23:15.208 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b78bae93:::benchmark_data_smithi192_88961_object89748:head# 2024-06-27T21:23:15.208 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b78c721c:::benchmark_data_smithi192_88961_object72418:head# 2024-06-27T21:23:15.208 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b78f3e81:::benchmark_data_smithi192_88961_object90400:head# 2024-06-27T21:23:15.209 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b78ffc63:::benchmark_data_smithi192_88961_object73318:head# 2024-06-27T21:23:15.210 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b790372e:::benchmark_data_smithi192_88961_object43499:head# 2024-06-27T21:23:15.211 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b79102c1:::benchmark_data_smithi192_88961_object52785:head# 2024-06-27T21:23:15.211 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b79197a5:::benchmark_data_smithi192_88961_object76790:head# 2024-06-27T21:23:15.211 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b792998d:::benchmark_data_smithi192_88961_object86611:head# 2024-06-27T21:23:15.211 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7935703:::benchmark_data_smithi192_88961_object73479:head# 2024-06-27T21:23:15.212 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b793c11d:::benchmark_data_smithi192_88961_object81746:head# 2024-06-27T21:23:15.212 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7972306:::benchmark_data_smithi192_88961_object78840:head# 2024-06-27T21:23:15.213 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b798d2b3:::benchmark_data_smithi192_88961_object59398:head# 2024-06-27T21:23:15.213 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7998dde:::benchmark_data_smithi192_88961_object74357:head# 2024-06-27T21:23:15.214 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b79ad291:::benchmark_data_smithi192_88961_object54220:head# 2024-06-27T21:23:15.214 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b79cc2f6:::benchmark_data_smithi192_88961_object81654:head# 2024-06-27T21:23:15.215 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b79d829c:::benchmark_data_smithi192_88961_object88406:head# 2024-06-27T21:23:15.215 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7a07a8e:::benchmark_data_smithi192_88961_object87291:head# 2024-06-27T21:23:15.215 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7a0d0b3:::benchmark_data_smithi192_88961_object70900:head# 2024-06-27T21:23:15.216 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7a3a5b2:::benchmark_data_smithi192_88961_object87612:head# 2024-06-27T21:23:15.216 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7a90f22:::benchmark_data_smithi192_88961_object48044:head# 2024-06-27T21:23:15.217 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7aa01f3:::benchmark_data_smithi192_88961_object79104:head# 2024-06-27T21:23:15.217 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7aa1f67:::benchmark_data_smithi192_88961_object72506:head# 2024-06-27T21:23:15.218 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7acd029:::benchmark_data_smithi192_88961_object46480:head# 2024-06-27T21:23:15.218 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ae1df6:::benchmark_data_smithi192_88961_object51683:head# 2024-06-27T21:23:15.218 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ae39a3:::benchmark_data_smithi192_88961_object51490:head# 2024-06-27T21:23:15.219 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ae3d62:::benchmark_data_smithi192_88961_object81440:head# 2024-06-27T21:23:15.219 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b07f0d:::benchmark_data_smithi192_88961_object48455:head# 2024-06-27T21:23:15.220 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b15578:::benchmark_data_smithi192_88961_object60480:head# 2024-06-27T21:23:15.220 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b18a72:::benchmark_data_smithi192_88961_object49651:head# 2024-06-27T21:23:15.221 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b3a7d1:::benchmark_data_smithi192_88961_object73430:head# 2024-06-27T21:23:15.221 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b4812b:::benchmark_data_smithi192_88961_object76655:head# 2024-06-27T21:23:15.222 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b4fcb3:::benchmark_data_smithi192_88961_object52216:head# 2024-06-27T21:23:15.222 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b51bbe:::benchmark_data_smithi192_88961_object46498:head# 2024-06-27T21:23:15.222 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b5623f:::benchmark_data_smithi192_88961_object88967:head# 2024-06-27T21:23:15.223 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b56e49:::benchmark_data_smithi192_88961_object75450:head# 2024-06-27T21:23:15.223 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b61ceb:::benchmark_data_smithi192_88961_object57617:head# 2024-06-27T21:23:15.224 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b78367:::benchmark_data_smithi192_88961_object54637:head# 2024-06-27T21:23:15.224 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b82df0:::benchmark_data_smithi192_88961_object68610:head# 2024-06-27T21:23:15.224 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b89e88:::benchmark_data_smithi192_88961_object47875:head# 2024-06-27T21:23:15.225 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7b95743:::benchmark_data_smithi192_88961_object67193:head# 2024-06-27T21:23:15.225 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ba131b:::benchmark_data_smithi192_88961_object88731:head# 2024-06-27T21:23:15.226 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7bc3e8a:::benchmark_data_smithi192_88961_object79177:head# 2024-06-27T21:23:15.226 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7bdb28f:::benchmark_data_smithi192_88961_object84510:head# 2024-06-27T21:23:15.226 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7bdc26f:::benchmark_data_smithi192_88961_object90214:head# 2024-06-27T21:23:15.227 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7bdf096:::benchmark_data_smithi192_88961_object71777:head# 2024-06-27T21:23:15.228 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7be808f:::benchmark_data_smithi192_88961_object73767:head# 2024-06-27T21:23:15.230 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c01fd2:::benchmark_data_smithi192_88961_object76466:head# 2024-06-27T21:23:15.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:15.230 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c166e8:::benchmark_data_smithi192_88961_object54424:head# 2024-06-27T21:23:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:15.232 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c172d7:::benchmark_data_smithi192_88961_object48100:head# 2024-06-27T21:23:15.232 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c19b9c:::benchmark_data_smithi192_88961_object56707:head# 2024-06-27T21:23:15.232 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c3a7e4:::benchmark_data_smithi192_88961_object87012:head# 2024-06-27T21:23:15.232 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c4687e:::benchmark_data_smithi192_88961_object83358:head# 2024-06-27T21:23:15.232 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7c70c8c:::benchmark_data_smithi192_88961_object60018:head# 2024-06-27T21:23:15.233 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7cc7230:::benchmark_data_smithi192_88961_object58983:head# 2024-06-27T21:23:15.233 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7cfade6:::benchmark_data_smithi192_88961_object86675:head# 2024-06-27T21:23:15.234 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7cfc5c4:::benchmark_data_smithi192_88961_object72488:head# 2024-06-27T21:23:15.234 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d0a812:::benchmark_data_smithi192_88961_object68803:head# 2024-06-27T21:23:15.234 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d1da73:::benchmark_data_smithi192_88961_object73167:head# 2024-06-27T21:23:15.234 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d21ab8:::benchmark_data_smithi192_88961_object66284:head# 2024-06-27T21:23:15.235 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d24867:::benchmark_data_smithi192_88961_object45405:head# 2024-06-27T21:23:15.235 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d42812:::benchmark_data_smithi192_88961_object77584:head# 2024-06-27T21:23:15.236 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d49771:::benchmark_data_smithi192_88961_object46281:head# 2024-06-27T21:23:15.236 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d6fef4:::benchmark_data_smithi192_88961_object64451:head# 2024-06-27T21:23:15.236 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d71efa:::benchmark_data_smithi192_88961_object51355:head# 2024-06-27T21:23:15.238 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d74eb3:::benchmark_data_smithi192_88961_object47993:head# 2024-06-27T21:23:15.238 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d7f2f9:::benchmark_data_smithi192_88961_object44676:head# 2024-06-27T21:23:15.238 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7d9c0cd:::benchmark_data_smithi192_88961_object78290:head# 2024-06-27T21:23:15.239 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7da4a91:::benchmark_data_smithi192_88961_object83312:head# 2024-06-27T21:23:15.241 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7daafe8:::benchmark_data_smithi192_88961_object52883:head# 2024-06-27T21:23:15.241 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7de3424:::benchmark_data_smithi192_88961_object60603:head# 2024-06-27T21:23:15.241 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e12cda:::benchmark_data_smithi192_88961_object50627:head# 2024-06-27T21:23:15.243 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e2377f:::benchmark_data_smithi192_88961_object67102:head# 2024-06-27T21:23:15.244 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e3d6e6:::benchmark_data_smithi192_88961_object53459:head# 2024-06-27T21:23:15.244 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e4301a:::benchmark_data_smithi192_88961_object53164:head# 2024-06-27T21:23:15.245 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e57b16:::benchmark_data_smithi192_88961_object67134:head# 2024-06-27T21:23:15.246 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e913fc:::benchmark_data_smithi192_88961_object83288:head# 2024-06-27T21:23:15.246 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7e9f80e:::benchmark_data_smithi192_88961_object53830:head# 2024-06-27T21:23:15.246 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ea40c1:::benchmark_data_smithi192_88961_object77600:head# 2024-06-27T21:23:15.247 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ec65a0:::benchmark_data_smithi192_88961_object76585:head# 2024-06-27T21:23:15.247 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ed2188:::benchmark_data_smithi192_88961_object84722:head# 2024-06-27T21:23:15.247 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ee4842:::benchmark_data_smithi192_88961_object81644:head# 2024-06-27T21:23:15.248 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7efdd13:::benchmark_data_smithi192_88961_object45711:head# 2024-06-27T21:23:15.248 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f0a630:::benchmark_data_smithi192_88961_object88966:head# 2024-06-27T21:23:15.249 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f28009:::benchmark_data_smithi192_88961_object45343:head# 2024-06-27T21:23:15.249 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f4a1e5:::benchmark_data_smithi192_88961_object62087:head# 2024-06-27T21:23:15.250 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f608bb:::benchmark_data_smithi192_88961_object66419:head# 2024-06-27T21:23:15.250 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f6d1c3:::benchmark_data_smithi192_88961_object71119:head# 2024-06-27T21:23:15.251 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f7ced2:::benchmark_data_smithi192_88961_object65198:head# 2024-06-27T21:23:15.252 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7f9b5b4:::benchmark_data_smithi192_88961_object74131:head# 2024-06-27T21:23:15.252 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7fd06b5:::benchmark_data_smithi192_88961_object86877:head# 2024-06-27T21:23:15.252 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7fd7729:::benchmark_data_smithi192_88961_object47904:head# 2024-06-27T21:23:15.253 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7fe7862:::benchmark_data_smithi192_88961_object85577:head# 2024-06-27T21:23:15.253 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b7ffe51f:::benchmark_data_smithi192_88961_object52159:head# 2024-06-27T21:23:15.254 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b801687b:::benchmark_data_smithi192_88961_object53783:head# 2024-06-27T21:23:15.254 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b802cad6:::benchmark_data_smithi192_88961_object51393:head# 2024-06-27T21:23:15.255 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8036d78:::benchmark_data_smithi192_88961_object63685:head# 2024-06-27T21:23:15.255 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8071999:::benchmark_data_smithi192_88961_object65416:head# 2024-06-27T21:23:15.255 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b807e1dc:::benchmark_data_smithi192_88961_object61205:head# 2024-06-27T21:23:15.256 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b809a232:::benchmark_data_smithi192_88961_object48979:head# 2024-06-27T21:23:15.256 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b809ae59:::benchmark_data_smithi192_88961_object83415:head# 2024-06-27T21:23:15.257 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b809d2b0:::benchmark_data_smithi192_88961_object55830:head# 2024-06-27T21:23:15.258 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b80a43c2:::benchmark_data_smithi192_88961_object59448:head# 2024-06-27T21:23:15.258 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b80eba65:::benchmark_data_smithi192_88961_object58731:head# 2024-06-27T21:23:15.258 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b80ee5c8:::benchmark_data_smithi192_88961_object60196:head# 2024-06-27T21:23:15.259 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b80f0ebb:::benchmark_data_smithi192_88961_object60403:head# 2024-06-27T21:23:15.259 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b81038b7:::benchmark_data_smithi192_88961_object78823:head# 2024-06-27T21:23:15.260 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8123154:::benchmark_data_smithi192_88961_object77321:head# 2024-06-27T21:23:15.260 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8124233:::benchmark_data_smithi192_88961_object50968:head# 2024-06-27T21:23:15.261 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b81250c2:::benchmark_data_smithi192_88961_object78824:head# 2024-06-27T21:23:15.261 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b813dbee:::benchmark_data_smithi192_88961_object45987:head# 2024-06-27T21:23:15.261 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b815aea2:::benchmark_data_smithi192_88961_object67676:head# 2024-06-27T21:23:15.262 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b817260e:::benchmark_data_smithi192_88961_object66210:head# 2024-06-27T21:23:15.262 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b81a4002:::benchmark_data_smithi192_88961_object43764:head# 2024-06-27T21:23:15.263 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b81c4550:::benchmark_data_smithi192_88961_object62747:head# 2024-06-27T21:23:15.264 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b81d41a9:::benchmark_data_smithi192_88961_object87534:head# 2024-06-27T21:23:15.264 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b81e3a6e:::benchmark_data_smithi192_88961_object89068:head# 2024-06-27T21:23:15.264 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8200055:::benchmark_data_smithi192_88961_object44871:head# 2024-06-27T21:23:15.265 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8214ffc:::benchmark_data_smithi192_88961_object68130:head# 2024-06-27T21:23:15.265 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82197f3:::benchmark_data_smithi192_88961_object45933:head# 2024-06-27T21:23:15.265 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b822f8fb:::benchmark_data_smithi192_88961_object83152:head# 2024-06-27T21:23:15.271 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8230c94:::benchmark_data_smithi192_88961_object73935:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82510ed:::benchmark_data_smithi192_88961_object44959:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82663b5:::benchmark_data_smithi192_88961_object67626:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b828b72c:::benchmark_data_smithi192_88961_object70427:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b828ea40:::benchmark_data_smithi192_88961_object62993:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8295785:::benchmark_data_smithi192_88961_object90380:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82a36cc:::benchmark_data_smithi192_88961_object67197:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82a6d49:::benchmark_data_smithi192_88961_object65819:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82e65da:::benchmark_data_smithi192_88961_object61749:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b82f2ae4:::benchmark_data_smithi192_88961_object46356:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8318ab2:::benchmark_data_smithi192_88961_object79853:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b836f15f:::benchmark_data_smithi192_88961_object46170:head# 2024-06-27T21:23:15.272 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b83855a9:::benchmark_data_smithi192_88961_object67177:head# 2024-06-27T21:23:15.273 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b83a4f8d:::benchmark_data_smithi192_88961_object78916:head# 2024-06-27T21:23:15.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b83ac178:::benchmark_data_smithi192_88961_object85586:head# 2024-06-27T21:23:15.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b83c6581:::benchmark_data_smithi192_88961_object78201:head# 2024-06-27T21:23:15.275 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b83d74b2:::benchmark_data_smithi192_88961_object53277:head# 2024-06-27T21:23:15.275 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b83d9582:::benchmark_data_smithi192_88961_object46575:head# 2024-06-27T21:23:15.276 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8402895:::benchmark_data_smithi192_88961_object83792:head# 2024-06-27T21:23:15.278 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8414a0b:::benchmark_data_smithi192_88961_object72688:head# 2024-06-27T21:23:15.279 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8423c29:::benchmark_data_smithi192_88961_object86714:head# 2024-06-27T21:23:15.279 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b84357eb:::benchmark_data_smithi192_88961_object45193:head# 2024-06-27T21:23:15.280 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8439cc3:::benchmark_data_smithi192_88961_object68137:head# 2024-06-27T21:23:15.280 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8465c9c:::benchmark_data_smithi192_88961_object43769:head# 2024-06-27T21:23:15.280 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b84693b2:::benchmark_data_smithi192_88961_object69743:head# 2024-06-27T21:23:15.281 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b847d589:::benchmark_data_smithi192_88961_object48894:head# 2024-06-27T21:23:15.281 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8480ea6:::benchmark_data_smithi192_88961_object76574:head# 2024-06-27T21:23:15.282 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b84d3a40:::benchmark_data_smithi192_88961_object50169:head# 2024-06-27T21:23:15.282 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b84da017:::benchmark_data_smithi192_88961_object48213:head# 2024-06-27T21:23:15.283 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b84e9206:::benchmark_data_smithi192_88961_object61033:head# 2024-06-27T21:23:15.283 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8504a9f:::benchmark_data_smithi192_88961_object81321:head# 2024-06-27T21:23:15.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b851be47:::benchmark_data_smithi192_88961_object60198:head# 2024-06-27T21:23:15.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b852237f:::benchmark_data_smithi192_88961_object76352:head# 2024-06-27T21:23:15.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b852ccb1:::benchmark_data_smithi192_88961_object47488:head# 2024-06-27T21:23:15.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8531f66:::benchmark_data_smithi192_88961_object51785:head# 2024-06-27T21:23:15.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b853b060:::benchmark_data_smithi192_88961_object55674:head# 2024-06-27T21:23:15.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8540741:::benchmark_data_smithi192_88961_object81901:head# 2024-06-27T21:23:15.287 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8541367:::benchmark_data_smithi192_88961_object65763:head# 2024-06-27T21:23:15.289 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8553135:::benchmark_data_smithi192_88961_object68701:head# 2024-06-27T21:23:15.289 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8564ca9:::benchmark_data_smithi192_88961_object65627:head# 2024-06-27T21:23:15.290 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b856ad88:::benchmark_data_smithi192_88961_object72302:head# 2024-06-27T21:23:15.290 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8594544:::benchmark_data_smithi192_88961_object73773:head# 2024-06-27T21:23:15.291 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8599bbc:::benchmark_data_smithi192_88961_object52907:head# 2024-06-27T21:23:15.291 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b85a06a5:::benchmark_data_smithi192_88961_object44178:head# 2024-06-27T21:23:15.292 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b85b3fa8:::benchmark_data_smithi192_88961_object50485:head# 2024-06-27T21:23:15.292 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b85ce39f:::benchmark_data_smithi192_88961_object82006:head# 2024-06-27T21:23:15.292 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8604fa6:::benchmark_data_smithi192_88961_object44886:head# 2024-06-27T21:23:15.293 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b860ea61:::benchmark_data_smithi192_88961_object69752:head# 2024-06-27T21:23:15.294 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8615e71:::benchmark_last_metadata:head# 2024-06-27T21:23:15.294 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b862894a:::benchmark_data_smithi192_88961_object81224:head# 2024-06-27T21:23:15.294 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8647723:::benchmark_data_smithi192_88961_object84816:head# 2024-06-27T21:23:15.294 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8673606:::benchmark_data_smithi192_88961_object68240:head# 2024-06-27T21:23:15.295 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b86876ea:::benchmark_data_smithi192_88961_object66315:head# 2024-06-27T21:23:15.295 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b868efae:::benchmark_data_smithi192_88961_object84248:head# 2024-06-27T21:23:15.296 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b86988db:::benchmark_data_smithi192_88961_object55076:head# 2024-06-27T21:23:15.296 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b86c5546:::benchmark_data_smithi192_88961_object54946:head# 2024-06-27T21:23:15.299 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b86cc07c:::benchmark_data_smithi192_88961_object50288:head# 2024-06-27T21:23:15.299 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b86cca07:::benchmark_data_smithi192_88961_object70981:head# 2024-06-27T21:23:15.300 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87282f4:::benchmark_data_smithi192_88961_object87727:head# 2024-06-27T21:23:15.300 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b872b512:::benchmark_data_smithi192_88961_object79694:head# 2024-06-27T21:23:15.301 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b873531b:::benchmark_data_smithi192_88961_object79299:head# 2024-06-27T21:23:15.301 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8735c60:::benchmark_data_smithi192_88961_object83196:head# 2024-06-27T21:23:15.302 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b873c342:::benchmark_data_smithi192_88961_object72614:head# 2024-06-27T21:23:15.302 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8740295:::benchmark_data_smithi192_88961_object86159:head# 2024-06-27T21:23:15.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b877b681:::benchmark_data_smithi192_88961_object52228:head# 2024-06-27T21:23:15.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b878c1f8:::benchmark_data_smithi192_88961_object64247:head# 2024-06-27T21:23:15.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b879ccee:::benchmark_data_smithi192_88961_object77403:head# 2024-06-27T21:23:15.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87a44f4:::benchmark_data_smithi192_88961_object68036:head# 2024-06-27T21:23:15.304 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87b02a6:::benchmark_data_smithi192_88961_object66060:head# 2024-06-27T21:23:15.305 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87b3942:::benchmark_data_smithi192_88961_object88800:head# 2024-06-27T21:23:15.305 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87d968f:::benchmark_data_smithi192_88961_object72858:head# 2024-06-27T21:23:15.305 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87df77e:::benchmark_data_smithi192_88961_object51950:head# 2024-06-27T21:23:15.306 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87ec49a:::benchmark_data_smithi192_88961_object64771:head# 2024-06-27T21:23:15.306 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b87fda5a:::benchmark_data_smithi192_88961_object84205:head# 2024-06-27T21:23:15.307 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b880c406:::benchmark_data_smithi192_88961_object65780:head# 2024-06-27T21:23:15.307 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b880f5fb:::benchmark_data_smithi192_88961_object67547:head# 2024-06-27T21:23:15.307 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b882ecd6:::benchmark_data_smithi192_88961_object88068:head# 2024-06-27T21:23:15.308 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b883a906:::benchmark_data_smithi192_88961_object73760:head# 2024-06-27T21:23:15.308 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b885086c:::benchmark_data_smithi192_88961_object78340:head# 2024-06-27T21:23:15.308 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8893b45:::benchmark_data_smithi192_88961_object80918:head# 2024-06-27T21:23:15.309 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b88aafea:::benchmark_data_smithi192_88961_object44254:head# 2024-06-27T21:23:15.309 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b88ac095:::benchmark_data_smithi192_88961_object44811:head# 2024-06-27T21:23:15.310 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b88c4b2b:::benchmark_data_smithi192_88961_object76794:head# 2024-06-27T21:23:15.310 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b88c955e:::benchmark_data_smithi192_88961_object83810:head# 2024-06-27T21:23:15.310 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b88d38c1:::benchmark_data_smithi192_88961_object48439:head# 2024-06-27T21:23:15.311 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b890b240:::benchmark_data_smithi192_88961_object56198:head# 2024-06-27T21:23:15.311 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b890d11f:::benchmark_data_smithi192_88961_object51757:head# 2024-06-27T21:23:15.312 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b892d813:::benchmark_data_smithi192_88961_object45488:head# 2024-06-27T21:23:15.312 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8980f92:::benchmark_data_smithi192_88961_object51388:head# 2024-06-27T21:23:15.313 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8989eef:::benchmark_data_smithi192_88961_object65390:head# 2024-06-27T21:23:15.313 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8997640:::benchmark_data_smithi192_88961_object50686:head# 2024-06-27T21:23:15.313 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b89a1498:::benchmark_data_smithi192_88961_object69177:head# 2024-06-27T21:23:15.314 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b89a1d58:::benchmark_data_smithi192_88961_object78437:head# 2024-06-27T21:23:15.314 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b89b1c02:::benchmark_data_smithi192_88961_object51590:head# 2024-06-27T21:23:15.314 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b89d8d20:::benchmark_data_smithi192_88961_object55684:head# 2024-06-27T21:23:15.315 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b89f09cf:::benchmark_data_smithi192_88961_object66483:head# 2024-06-27T21:23:15.316 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b89fcb58:::benchmark_data_smithi192_88961_object86936:head# 2024-06-27T21:23:15.316 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8a14b8a:::benchmark_data_smithi192_88961_object75900:head# 2024-06-27T21:23:15.317 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8a2b568:::benchmark_data_smithi192_88961_object68373:head# 2024-06-27T21:23:15.317 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8a5622b:::benchmark_data_smithi192_88961_object82563:head# 2024-06-27T21:23:15.318 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8a7e9b7:::benchmark_data_smithi192_88961_object86268:head# 2024-06-27T21:23:15.318 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8a8b36f:::benchmark_data_smithi192_88961_object87843:head# 2024-06-27T21:23:15.319 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8a9326e:::benchmark_data_smithi192_88961_object50752:head# 2024-06-27T21:23:15.319 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8ab3c7c:::benchmark_data_smithi192_88961_object57111:head# 2024-06-27T21:23:15.319 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8ac78f0:::benchmark_data_smithi192_88961_object75898:head# 2024-06-27T21:23:15.320 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8ae0eef:::benchmark_data_smithi192_88961_object90343:head# 2024-06-27T21:23:15.320 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8aeef9e:::benchmark_data_smithi192_88961_object69168:head# 2024-06-27T21:23:15.321 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b0b9e6:::benchmark_data_smithi192_88961_object56218:head# 2024-06-27T21:23:15.321 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b0d489:::benchmark_data_smithi192_88961_object58909:head# 2024-06-27T21:23:15.322 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b15aa5:::benchmark_data_smithi192_88961_object85135:head# 2024-06-27T21:23:15.322 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b16752:::benchmark_data_smithi192_88961_object43762:head# 2024-06-27T21:23:15.322 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b27f19:::benchmark_data_smithi192_88961_object64671:head# 2024-06-27T21:23:15.323 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b423c7:::benchmark_data_smithi192_88961_object79688:head# 2024-06-27T21:23:15.324 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b52f07:::benchmark_data_smithi192_88961_object81540:head# 2024-06-27T21:23:15.324 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8b750a8:::benchmark_data_smithi192_88961_object76588:head# 2024-06-27T21:23:15.324 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8bbb4fc:::benchmark_data_smithi192_88961_object47547:head# 2024-06-27T21:23:15.325 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8bbdc99:::benchmark_data_smithi192_88961_object77119:head# 2024-06-27T21:23:15.325 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8bd52b0:::benchmark_data_smithi192_88961_object48302:head# 2024-06-27T21:23:15.325 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8bd866f:::benchmark_data_smithi192_88961_object53480:head# 2024-06-27T21:23:15.326 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8c23d66:::benchmark_data_smithi192_88961_object67521:head# 2024-06-27T21:23:15.326 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8c2bc8f:::benchmark_data_smithi192_88961_object82090:head# 2024-06-27T21:23:15.326 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8c4def5:::benchmark_data_smithi192_88961_object51494:head# 2024-06-27T21:23:15.326 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8c550c0:::benchmark_data_smithi192_88961_object75009:head# 2024-06-27T21:23:15.327 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8c7b6fc:::benchmark_data_smithi192_88961_object49278:head# 2024-06-27T21:23:15.327 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8c7fc7a:::benchmark_data_smithi192_88961_object57037:head# 2024-06-27T21:23:15.328 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8caf73b:::benchmark_data_smithi192_88961_object74450:head# 2024-06-27T21:23:15.328 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8cd75c9:::benchmark_data_smithi192_88961_object51962:head# 2024-06-27T21:23:15.328 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d0b594:::benchmark_data_smithi192_88961_object50165:head# 2024-06-27T21:23:15.329 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d13655:::benchmark_data_smithi192_88961_object52750:head# 2024-06-27T21:23:15.329 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d1b8bb:::benchmark_data_smithi192_88961_object71383:head# 2024-06-27T21:23:15.329 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d1c01e:::benchmark_data_smithi192_88961_object54751:head# 2024-06-27T21:23:15.330 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d1c5c2:::benchmark_data_smithi192_88961_object71481:head# 2024-06-27T21:23:15.330 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d27b67:::benchmark_data_smithi192_88961_object80452:head# 2024-06-27T21:23:15.331 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d7a494:::benchmark_data_smithi192_88961_object58311:head# 2024-06-27T21:23:15.331 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8d870c6:::benchmark_data_smithi192_88961_object46183:head# 2024-06-27T21:23:15.332 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8db140c:::benchmark_data_smithi192_88961_object64561:head# 2024-06-27T21:23:15.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8db4e06:::benchmark_data_smithi192_88961_object84486:head# 2024-06-27T21:23:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8dedb84:::benchmark_data_smithi192_88961_object55303:head# 2024-06-27T21:23:15.333 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8e009b6:::benchmark_data_smithi192_88961_object83316:head# 2024-06-27T21:23:15.334 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8e2753e:::benchmark_data_smithi192_88961_object56347:head# 2024-06-27T21:23:15.334 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8e43200:::benchmark_data_smithi192_88961_object80664:head# 2024-06-27T21:23:15.335 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8e651ba:::benchmark_data_smithi192_88961_object52862:head# 2024-06-27T21:23:15.336 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8e6f7a2:::benchmark_data_smithi192_88961_object86112:head# 2024-06-27T21:23:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8eb263d:::benchmark_data_smithi192_88961_object83381:head# 2024-06-27T21:23:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8eb5c4b:::benchmark_data_smithi192_88961_object52898:head# 2024-06-27T21:23:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8ed7c11:::benchmark_data_smithi192_88961_object55338:head# 2024-06-27T21:23:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8edbd52:::benchmark_data_smithi192_88961_object58946:head# 2024-06-27T21:23:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8ee1170:::benchmark_data_smithi192_88961_object45311:head# 2024-06-27T21:23:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8eead08:::benchmark_data_smithi192_88961_object70138:head# 2024-06-27T21:23:15.338 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8f15296:::benchmark_data_smithi192_88961_object56504:head# 2024-06-27T21:23:15.338 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8f6d086:::benchmark_data_smithi192_88961_object85057:head# 2024-06-27T21:23:15.339 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8f8157e:::benchmark_data_smithi192_88961_object82297:head# 2024-06-27T21:23:15.339 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8f89b70:::benchmark_data_smithi192_88961_object80859:head# 2024-06-27T21:23:15.339 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8f9f457:::benchmark_data_smithi192_88961_object80288:head# 2024-06-27T21:23:15.340 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8fa6151:::benchmark_data_smithi192_88961_object88514:head# 2024-06-27T21:23:15.340 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8fa8bf5:::benchmark_data_smithi192_88961_object76399:head# 2024-06-27T21:23:15.341 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b8fb19ec:::benchmark_data_smithi192_88961_object72315:head# 2024-06-27T21:23:15.341 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b902c49b:::benchmark_data_smithi192_88961_object62396:head# 2024-06-27T21:23:15.342 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b903a531:::benchmark_data_smithi192_88961_object61024:head# 2024-06-27T21:23:15.342 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b903c2b7:::benchmark_data_smithi192_88961_object55799:head# 2024-06-27T21:23:15.342 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b906a862:::benchmark_data_smithi192_88961_object59575:head# 2024-06-27T21:23:15.343 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9073805:::benchmark_data_smithi192_88961_object45326:head# 2024-06-27T21:23:15.343 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9074bc9:::benchmark_data_smithi192_88961_object90186:head# 2024-06-27T21:23:15.344 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9085d1a:::benchmark_data_smithi192_88961_object79780:head# 2024-06-27T21:23:15.344 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b908cf32:::benchmark_data_smithi192_88961_object72869:head# 2024-06-27T21:23:15.345 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9093b30:::benchmark_data_smithi192_88961_object65867:head# 2024-06-27T21:23:15.345 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b909493f:::benchmark_data_smithi192_88961_object64629:head# 2024-06-27T21:23:15.346 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b90b1947:::benchmark_data_smithi192_88961_object69895:head# 2024-06-27T21:23:15.347 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b90b5802:::benchmark_data_smithi192_88961_object77425:head# 2024-06-27T21:23:15.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b90e1dbf:::benchmark_data_smithi192_88961_object76538:head# 2024-06-27T21:23:15.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b90ffb18:::benchmark_data_smithi192_88961_object82177:head# 2024-06-27T21:23:15.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9105f78:::benchmark_data_smithi192_88961_object49076:head# 2024-06-27T21:23:15.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9108362:::benchmark_data_smithi192_88961_object87676:head# 2024-06-27T21:23:15.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91142d2:::benchmark_data_smithi192_88961_object57297:head# 2024-06-27T21:23:15.349 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b914eb55:::benchmark_data_smithi192_88961_object69270:head# 2024-06-27T21:23:15.349 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9186d49:::benchmark_data_smithi192_88961_object61639:head# 2024-06-27T21:23:15.349 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91ba582:::benchmark_data_smithi192_88961_object57570:head# 2024-06-27T21:23:15.350 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91ba7e5:::benchmark_data_smithi192_88961_object65280:head# 2024-06-27T21:23:15.350 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91d0053:::benchmark_data_smithi192_88961_object90613:head# 2024-06-27T21:23:15.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91d34b0:::benchmark_data_smithi192_88961_object77078:head# 2024-06-27T21:23:15.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91d9696:::benchmark_data_smithi192_88961_object82361:head# 2024-06-27T21:23:15.352 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b91f2b5d:::benchmark_data_smithi192_88961_object84427:head# 2024-06-27T21:23:15.352 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9202abe:::benchmark_data_smithi192_88961_object45969:head# 2024-06-27T21:23:15.352 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b92258b7:::benchmark_data_smithi192_88961_object82736:head# 2024-06-27T21:23:15.353 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9234bb0:::benchmark_data_smithi192_88961_object90029:head# 2024-06-27T21:23:15.353 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b923b033:::benchmark_data_smithi192_88961_object59268:head# 2024-06-27T21:23:15.354 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b924ba6c:::benchmark_data_smithi192_88961_object52032:head# 2024-06-27T21:23:15.355 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9266534:::benchmark_data_smithi192_88961_object49428:head# 2024-06-27T21:23:15.355 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9267067:::benchmark_data_smithi192_88961_object74928:head# 2024-06-27T21:23:15.356 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b928c0ee:::benchmark_data_smithi192_88961_object89206:head# 2024-06-27T21:23:15.356 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b92d936a:::benchmark_data_smithi192_88961_object64205:head# 2024-06-27T21:23:15.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:15.357 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b92f01a2:::benchmark_data_smithi192_88961_object84172:head# 2024-06-27T21:23:15.357 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b92ffade:::benchmark_data_smithi192_88961_object76855:head# 2024-06-27T21:23:15.358 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9337603:::benchmark_data_smithi192_88961_object60155:head# 2024-06-27T21:23:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:15.359 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9339f3e:::benchmark_data_smithi192_88961_object79713:head# 2024-06-27T21:23:15.359 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b934e7e2:::benchmark_data_smithi192_88961_object65362:head# 2024-06-27T21:23:15.359 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9372e72:::benchmark_data_smithi192_88961_object87299:head# 2024-06-27T21:23:15.359 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9380d51:::benchmark_data_smithi192_88961_object67466:head# 2024-06-27T21:23:15.360 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9381383:::benchmark_data_smithi192_88961_object48437:head# 2024-06-27T21:23:15.360 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b938936a:::benchmark_data_smithi192_88961_object61079:head# 2024-06-27T21:23:15.360 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b93b634c:::benchmark_data_smithi192_88961_object85476:head# 2024-06-27T21:23:15.361 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b93bf00e:::benchmark_data_smithi192_88961_object90646:head# 2024-06-27T21:23:15.361 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b943c417:::benchmark_data_smithi192_88961_object78697:head# 2024-06-27T21:23:15.361 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b943e2be:::benchmark_data_smithi192_88961_object82697:head# 2024-06-27T21:23:15.362 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b943f118:::benchmark_data_smithi192_88961_object85788:head# 2024-06-27T21:23:15.362 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9456b82:::benchmark_data_smithi192_88961_object77565:head# 2024-06-27T21:23:15.362 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b945c97f:::benchmark_data_smithi192_88961_object80475:head# 2024-06-27T21:23:15.363 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b94ab563:::benchmark_data_smithi192_88961_object54503:head# 2024-06-27T21:23:15.363 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b94bc138:::benchmark_data_smithi192_88961_object49289:head# 2024-06-27T21:23:15.364 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b94dff16:::benchmark_data_smithi192_88961_object59552:head# 2024-06-27T21:23:15.364 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b94e2e02:::benchmark_data_smithi192_88961_object87659:head# 2024-06-27T21:23:15.365 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b94e4686:::benchmark_data_smithi192_88961_object54165:head# 2024-06-27T21:23:15.365 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b94e6d62:::benchmark_data_smithi192_88961_object52504:head# 2024-06-27T21:23:15.365 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9505dd5:::benchmark_data_smithi192_88961_object72060:head# 2024-06-27T21:23:15.366 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b950be96:::benchmark_data_smithi192_88961_object44779:head# 2024-06-27T21:23:15.366 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9510b79:::benchmark_data_smithi192_88961_object61781:head# 2024-06-27T21:23:15.367 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9527cf7:::benchmark_data_smithi192_88961_object86030:head# 2024-06-27T21:23:15.367 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b954e5d0:::benchmark_data_smithi192_88961_object82022:head# 2024-06-27T21:23:15.367 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9553f29:::benchmark_data_smithi192_88961_object56701:head# 2024-06-27T21:23:15.368 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b95be180:::benchmark_data_smithi192_88961_object87940:head# 2024-06-27T21:23:15.368 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b95cb431:::benchmark_data_smithi192_88961_object47399:head# 2024-06-27T21:23:15.369 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b95d4bc3:::benchmark_data_smithi192_88961_object49280:head# 2024-06-27T21:23:15.369 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b95df2e0:::benchmark_data_smithi192_88961_object50806:head# 2024-06-27T21:23:15.370 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b95f0fec:::benchmark_data_smithi192_88961_object69795:head# 2024-06-27T21:23:15.370 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b961b071:::benchmark_data_smithi192_88961_object74341:head# 2024-06-27T21:23:15.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9635748:::benchmark_data_smithi192_88961_object44244:head# 2024-06-27T21:23:15.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96432d8:::benchmark_data_smithi192_88961_object60102:head# 2024-06-27T21:23:15.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b965c1bc:::benchmark_data_smithi192_88961_object68450:head# 2024-06-27T21:23:15.372 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9663e19:::benchmark_data_smithi192_88961_object73419:head# 2024-06-27T21:23:15.374 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9667bdf:::benchmark_data_smithi192_88961_object60640:head# 2024-06-27T21:23:15.374 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b969b363:::benchmark_data_smithi192_88961_object55802:head# 2024-06-27T21:23:15.374 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96a4daf:::benchmark_data_smithi192_88961_object54070:head# 2024-06-27T21:23:15.374 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96bffc7:::benchmark_data_smithi192_88961_object73418:head# 2024-06-27T21:23:15.374 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96d45b6:::benchmark_data_smithi192_88961_object45890:head# 2024-06-27T21:23:15.375 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96d5744:::benchmark_data_smithi192_88961_object86717:head# 2024-06-27T21:23:15.375 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96e54af:::benchmark_data_smithi192_88961_object50542:head# 2024-06-27T21:23:15.375 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b96f67bf:::benchmark_data_smithi192_88961_object85965:head# 2024-06-27T21:23:15.376 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b971128e:::benchmark_data_smithi192_88961_object61252:head# 2024-06-27T21:23:15.377 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9726128:::benchmark_data_smithi192_88961_object70438:head# 2024-06-27T21:23:15.379 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b973621b:::benchmark_data_smithi192_88961_object72196:head# 2024-06-27T21:23:15.379 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9762d04:::benchmark_data_smithi192_88961_object52481:head# 2024-06-27T21:23:15.379 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9772026:::benchmark_data_smithi192_88961_object71697:head# 2024-06-27T21:23:15.379 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9775f8b:::benchmark_data_smithi192_88961_object85632:head# 2024-06-27T21:23:15.380 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b977d018:::benchmark_data_smithi192_88961_object65936:head# 2024-06-27T21:23:15.380 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b97a4706:::benchmark_data_smithi192_88961_object57032:head# 2024-06-27T21:23:15.380 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b97c72d8:::benchmark_data_smithi192_88961_object50276:head# 2024-06-27T21:23:15.381 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b97cfc67:::benchmark_data_smithi192_88961_object76225:head# 2024-06-27T21:23:15.382 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b97daf75:::benchmark_data_smithi192_88961_object66300:head# 2024-06-27T21:23:15.382 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b97f0c3f:::benchmark_data_smithi192_88961_object43133:head# 2024-06-27T21:23:15.382 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9821bba:::benchmark_data_smithi192_88961_object49548:head# 2024-06-27T21:23:15.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9841a29:::benchmark_data_smithi192_88961_object46807:head# 2024-06-27T21:23:15.384 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b985a263:::benchmark_data_smithi192_88961_object67790:head# 2024-06-27T21:23:15.384 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9884e88:::benchmark_data_smithi192_88961_object89424:head# 2024-06-27T21:23:15.385 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b989d7b9:::benchmark_data_smithi192_88961_object67155:head# 2024-06-27T21:23:15.385 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b98bc02c:::benchmark_data_smithi192_88961_object46389:head# 2024-06-27T21:23:15.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b98c9bc0:::benchmark_data_smithi192_88961_object47159:head# 2024-06-27T21:23:15.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b98d0a49:::benchmark_data_smithi192_88961_object71919:head# 2024-06-27T21:23:15.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99190c0:::benchmark_data_smithi192_88961_object55014:head# 2024-06-27T21:23:15.387 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9926549:::benchmark_data_smithi192_88961_object52144:head# 2024-06-27T21:23:15.387 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b994eebf:::benchmark_data_smithi192_88961_object55828:head# 2024-06-27T21:23:15.388 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9961ba7:::benchmark_data_smithi192_88961_object50646:head# 2024-06-27T21:23:15.388 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99a1f72:::benchmark_data_smithi192_88961_object47309:head# 2024-06-27T21:23:15.388 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99c1e1b:::benchmark_data_smithi192_88961_object82193:head# 2024-06-27T21:23:15.389 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99d860c:::benchmark_data_smithi192_88961_object62880:head# 2024-06-27T21:23:15.389 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99deef8:::benchmark_data_smithi192_88961_object89717:head# 2024-06-27T21:23:15.389 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99ec9a7:::benchmark_data_smithi192_88961_object77882:head# 2024-06-27T21:23:15.390 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b99f43f1:::benchmark_data_smithi192_88961_object56370:head# 2024-06-27T21:23:15.390 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9a34df5:::benchmark_data_smithi192_88961_object62931:head# 2024-06-27T21:23:15.391 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9a46cc1:::benchmark_data_smithi192_88961_object83631:head# 2024-06-27T21:23:15.391 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9a50050:::benchmark_data_smithi192_88961_object52238:head# 2024-06-27T21:23:15.391 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9a51590:::benchmark_data_smithi192_88961_object62025:head# 2024-06-27T21:23:15.392 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9a67338:::benchmark_data_smithi192_88961_object72778:head# 2024-06-27T21:23:15.392 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9a780d8:::benchmark_data_smithi192_88961_object56557:head# 2024-06-27T21:23:15.393 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9ab3fc5:::benchmark_data_smithi192_88961_object69247:head# 2024-06-27T21:23:15.393 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9adb02f:::benchmark_data_smithi192_88961_object76099:head# 2024-06-27T21:23:15.393 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9adc995:::benchmark_data_smithi192_88961_object54087:head# 2024-06-27T21:23:15.394 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9ae0aee:::benchmark_data_smithi192_88961_object66379:head# 2024-06-27T21:23:15.394 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b1aa52:::benchmark_data_smithi192_88961_object63756:head# 2024-06-27T21:23:15.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b31747:::benchmark_data_smithi192_88961_object50708:head# 2024-06-27T21:23:15.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b3817c:::benchmark_data_smithi192_88961_object45546:head# 2024-06-27T21:23:15.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b4d075:::benchmark_data_smithi192_88961_object50476:head# 2024-06-27T21:23:15.396 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b65b49:::benchmark_data_smithi192_88961_object75714:head# 2024-06-27T21:23:15.396 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b6b9a6:::benchmark_data_smithi192_88961_object76180:head# 2024-06-27T21:23:15.397 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b7e8e0:::benchmark_data_smithi192_88961_object65445:head# 2024-06-27T21:23:15.397 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b91b42:::benchmark_data_smithi192_88961_object48336:head# 2024-06-27T21:23:15.398 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b946c6:::benchmark_data_smithi192_88961_object47443:head# 2024-06-27T21:23:15.398 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9b969e3:::benchmark_data_smithi192_88961_object79779:head# 2024-06-27T21:23:15.399 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9bb1213:::benchmark_data_smithi192_88961_object66642:head# 2024-06-27T21:23:15.399 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9bb140a:::benchmark_data_smithi192_88961_object79112:head# 2024-06-27T21:23:15.399 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9bc8491:::benchmark_data_smithi192_88961_object59372:head# 2024-06-27T21:23:15.400 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9be3217:::benchmark_data_smithi192_88961_object56260:head# 2024-06-27T21:23:15.400 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9be4eb1:::benchmark_data_smithi192_88961_object72368:head# 2024-06-27T21:23:15.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c0f86c:::benchmark_data_smithi192_88961_object81779:head# 2024-06-27T21:23:15.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c2dc6f:::benchmark_data_smithi192_88961_object65414:head# 2024-06-27T21:23:15.402 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c3b0ef:::benchmark_data_smithi192_88961_object86634:head# 2024-06-27T21:23:15.402 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c6a591:::benchmark_data_smithi192_88961_object57456:head# 2024-06-27T21:23:15.402 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c6fd17:::benchmark_data_smithi192_88961_object65264:head# 2024-06-27T21:23:15.403 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c7ae75:::benchmark_data_smithi192_88961_object90037:head# 2024-06-27T21:23:15.403 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c859f7:::benchmark_data_smithi192_88961_object59286:head# 2024-06-27T21:23:15.404 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9c8f2c8:::benchmark_data_smithi192_88961_object72367:head# 2024-06-27T21:23:15.404 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9cac8c5:::benchmark_data_smithi192_88961_object79786:head# 2024-06-27T21:23:15.404 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9cb708a:::benchmark_data_smithi192_88961_object50576:head# 2024-06-27T21:23:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9cbf62c:::benchmark_data_smithi192_88961_object45327:head# 2024-06-27T21:23:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9ccae14:::benchmark_data_smithi192_88961_object76335:head# 2024-06-27T21:23:15.405 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9cf86e6:::benchmark_data_smithi192_88961_object67434:head# 2024-06-27T21:23:15.406 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9cfd8ad:::benchmark_data_smithi192_88961_object75123:head# 2024-06-27T21:23:15.406 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9d01e54:::benchmark_data_smithi192_88961_object60983:head# 2024-06-27T21:23:15.407 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9d13948:::benchmark_data_smithi192_88961_object66655:head# 2024-06-27T21:23:15.407 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9d60f57:::benchmark_data_smithi192_88961_object80930:head# 2024-06-27T21:23:15.408 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9d7321e:::benchmark_data_smithi192_88961_object60282:head# 2024-06-27T21:23:15.408 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9ddd625:::benchmark_data_smithi192_88961_object48238:head# 2024-06-27T21:23:15.408 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9df9674:::benchmark_data_smithi192_88961_object62812:head# 2024-06-27T21:23:15.409 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e044f2:::benchmark_data_smithi192_88961_object81047:head# 2024-06-27T21:23:15.409 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e0f6bb:::benchmark_data_smithi192_88961_object44367:head# 2024-06-27T21:23:15.410 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e2445c:::benchmark_data_smithi192_88961_object44384:head# 2024-06-27T21:23:15.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e2c5e6:::benchmark_data_smithi192_88961_object70229:head# 2024-06-27T21:23:15.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e5b57b:::benchmark_data_smithi192_88961_object90182:head# 2024-06-27T21:23:15.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e6d5b9:::benchmark_data_smithi192_88961_object51990:head# 2024-06-27T21:23:15.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9e9866f:::benchmark_data_smithi192_88961_object67375:head# 2024-06-27T21:23:15.412 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9ee81a0:::benchmark_data_smithi192_88961_object54375:head# 2024-06-27T21:23:15.412 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f03454:::benchmark_data_smithi192_88961_object89013:head# 2024-06-27T21:23:15.413 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f209a9:::benchmark_data_smithi192_88961_object60705:head# 2024-06-27T21:23:15.413 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f27d60:::benchmark_data_smithi192_88961_object89098:head# 2024-06-27T21:23:15.413 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f2e4fa:::benchmark_data_smithi192_88961_object82576:head# 2024-06-27T21:23:15.414 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f41272:::benchmark_data_smithi192_88961_object86771:head# 2024-06-27T21:23:15.414 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f46f18:::benchmark_data_smithi192_88961_object50768:head# 2024-06-27T21:23:15.415 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f549aa:::benchmark_data_smithi192_88961_object72039:head# 2024-06-27T21:23:15.415 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f623b1:::benchmark_data_smithi192_88961_object52314:head# 2024-06-27T21:23:15.415 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f708e4:::benchmark_data_smithi192_88961_object86015:head# 2024-06-27T21:23:15.416 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9f74513:::benchmark_data_smithi192_88961_object81005:head# 2024-06-27T21:23:15.416 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9faa516:::benchmark_data_smithi192_88961_object48955:head# 2024-06-27T21:23:15.416 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9faa955:::benchmark_data_smithi192_88961_object82807:head# 2024-06-27T21:23:15.417 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9fde356:::benchmark_data_smithi192_88961_object71794:head# 2024-06-27T21:23:15.417 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:b9ffc5e5:::benchmark_data_smithi192_88961_object43025:head# 2024-06-27T21:23:15.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba005285:::benchmark_data_smithi192_88961_object55899:head# 2024-06-27T21:23:15.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba02aadc:::benchmark_data_smithi192_88961_object88426:head# 2024-06-27T21:23:15.419 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba0437eb:::benchmark_data_smithi192_88961_object88217:head# 2024-06-27T21:23:15.419 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba04e076:::benchmark_data_smithi192_88961_object46534:head# 2024-06-27T21:23:15.419 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba05da2b:::benchmark_data_smithi192_88961_object69683:head# 2024-06-27T21:23:15.420 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba06e31b:::benchmark_data_smithi192_88961_object51527:head# 2024-06-27T21:23:15.420 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba073a95:::benchmark_data_smithi192_88961_object46202:head# 2024-06-27T21:23:15.421 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba08107f:::benchmark_data_smithi192_88961_object73633:head# 2024-06-27T21:23:15.421 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba0ae05b:::benchmark_data_smithi192_88961_object77542:head# 2024-06-27T21:23:15.422 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba0c2deb:::benchmark_data_smithi192_88961_object60973:head# 2024-06-27T21:23:15.422 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba0d1950:::benchmark_data_smithi192_88961_object74979:head# 2024-06-27T21:23:15.423 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba13d211:::benchmark_data_smithi192_88961_object47963:head# 2024-06-27T21:23:15.423 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba14b746:::benchmark_data_smithi192_88961_object58385:head# 2024-06-27T21:23:15.424 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba14cee1:::benchmark_data_smithi192_88961_object61693:head# 2024-06-27T21:23:15.424 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba15faa4:::benchmark_data_smithi192_88961_object53289:head# 2024-06-27T21:23:15.425 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba161d1f:::benchmark_data_smithi192_88961_object59145:head# 2024-06-27T21:23:15.426 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba17ed8e:::benchmark_data_smithi192_88961_object74694:head# 2024-06-27T21:23:15.426 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba1868b4:::benchmark_data_smithi192_88961_object68198:head# 2024-06-27T21:23:15.427 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba1a0a84:::benchmark_data_smithi192_88961_object67201:head# 2024-06-27T21:23:15.427 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba1a5e61:::benchmark_data_smithi192_88961_object57964:head# 2024-06-27T21:23:15.428 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba1b48e3:::benchmark_data_smithi192_88961_object46810:head# 2024-06-27T21:23:15.428 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba1eaab7:::benchmark_data_smithi192_88961_object84188:head# 2024-06-27T21:23:15.428 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba216337:::benchmark_data_smithi192_88961_object89101:head# 2024-06-27T21:23:15.429 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba22f2ee:::benchmark_data_smithi192_88961_object66380:head# 2024-06-27T21:23:15.429 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba24cbcf:::benchmark_data_smithi192_88961_object71751:head# 2024-06-27T21:23:15.430 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2502b6:::benchmark_data_smithi192_88961_object66744:head# 2024-06-27T21:23:15.430 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba283a13:::benchmark_data_smithi192_88961_object45477:head# 2024-06-27T21:23:15.431 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2846ee:::benchmark_data_smithi192_88961_object76361:head# 2024-06-27T21:23:15.431 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba28ccbc:::benchmark_data_smithi192_88961_object84147:head# 2024-06-27T21:23:15.431 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba298094:::benchmark_data_smithi192_88961_object70673:head# 2024-06-27T21:23:15.432 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2a31f1:::benchmark_data_smithi192_88961_object81088:head# 2024-06-27T21:23:15.432 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2b00e7:::benchmark_data_smithi192_88961_object49281:head# 2024-06-27T21:23:15.433 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2c0982:::benchmark_data_smithi192_88961_object48446:head# 2024-06-27T21:23:15.434 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2dd4d5:::benchmark_data_smithi192_88961_object87376:head# 2024-06-27T21:23:15.434 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba2ebd84:::benchmark_data_smithi192_88961_object45127:head# 2024-06-27T21:23:15.435 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3072d1:::benchmark_data_smithi192_88961_object82664:head# 2024-06-27T21:23:15.436 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3301e0:::benchmark_data_smithi192_88961_object59944:head# 2024-06-27T21:23:15.436 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3358dd:::benchmark_data_smithi192_88961_object43241:head# 2024-06-27T21:23:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba34ee05:::benchmark_data_smithi192_88961_object60379:head# 2024-06-27T21:23:15.437 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba35c462:::benchmark_data_smithi192_88961_object58742:head# 2024-06-27T21:23:15.438 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba36c98f:::benchmark_data_smithi192_88961_object56460:head# 2024-06-27T21:23:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba37157f:::benchmark_data_smithi192_88961_object69037:head# 2024-06-27T21:23:15.439 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3774e2:::benchmark_data_smithi192_88961_object81410:head# 2024-06-27T21:23:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba38b1f6:::benchmark_data_smithi192_88961_object84614:head# 2024-06-27T21:23:15.440 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3add6d:::benchmark_data_smithi192_88961_object50509:head# 2024-06-27T21:23:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3d50bc:::benchmark_data_smithi192_88961_object60321:head# 2024-06-27T21:23:15.441 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba3f7abc:::benchmark_data_smithi192_88961_object76298:head# 2024-06-27T21:23:15.442 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4068de:::benchmark_data_smithi192_88961_object58429:head# 2024-06-27T21:23:15.442 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba438835:::benchmark_data_smithi192_88961_object54372:head# 2024-06-27T21:23:15.442 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba45e3b5:::benchmark_data_smithi192_88961_object47735:head# 2024-06-27T21:23:15.443 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba46521e:::benchmark_data_smithi192_88961_object76994:head# 2024-06-27T21:23:15.443 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4697f7:::benchmark_data_smithi192_88961_object64134:head# 2024-06-27T21:23:15.444 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba46eabf:::benchmark_data_smithi192_88961_object84161:head# 2024-06-27T21:23:15.444 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba48407b:::benchmark_data_smithi192_88961_object84298:head# 2024-06-27T21:23:15.445 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba48aaa7:::benchmark_data_smithi192_88961_object74866:head# 2024-06-27T21:23:15.445 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba492b09:::benchmark_data_smithi192_88961_object62463:head# 2024-06-27T21:23:15.446 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4942ab:::benchmark_data_smithi192_88961_object68870:head# 2024-06-27T21:23:15.446 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4a016d:::benchmark_data_smithi192_88961_object71089:head# 2024-06-27T21:23:15.446 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4b3bf1:::benchmark_data_smithi192_88961_object45527:head# 2024-06-27T21:23:15.447 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4b59bf:::benchmark_data_smithi192_88961_object81757:head# 2024-06-27T21:23:15.447 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4b6dd0:::benchmark_data_smithi192_88961_object76155:head# 2024-06-27T21:23:15.447 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4e6cca:::benchmark_data_smithi192_88961_object81956:head# 2024-06-27T21:23:15.448 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba4f698d:::benchmark_data_smithi192_88961_object62972:head# 2024-06-27T21:23:15.449 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba501b34:::benchmark_data_smithi192_88961_object90404:head# 2024-06-27T21:23:15.449 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba50c038:::benchmark_data_smithi192_88961_object45203:head# 2024-06-27T21:23:15.450 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba50ef17:::benchmark_data_smithi192_88961_object72820:head# 2024-06-27T21:23:15.450 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba52d16a:::benchmark_data_smithi192_88961_object47031:head# 2024-06-27T21:23:15.451 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba54ab4e:::benchmark_data_smithi192_88961_object50426:head# 2024-06-27T21:23:15.451 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba554c31:::benchmark_data_smithi192_88961_object76378:head# 2024-06-27T21:23:15.451 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba576dfc:::benchmark_data_smithi192_88961_object89387:head# 2024-06-27T21:23:15.452 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba595558:::benchmark_data_smithi192_88961_object76838:head# 2024-06-27T21:23:15.452 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba5bdaec:::benchmark_data_smithi192_88961_object48816:head# 2024-06-27T21:23:15.452 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba5ca36e:::benchmark_data_smithi192_88961_object69418:head# 2024-06-27T21:23:15.453 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba5e4b4d:::benchmark_data_smithi192_88961_object43116:head# 2024-06-27T21:23:15.453 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba61906f:::benchmark_data_smithi192_88961_object84053:head# 2024-06-27T21:23:15.453 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba637918:::benchmark_data_smithi192_88961_object85721:head# 2024-06-27T21:23:15.454 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba66e577:::benchmark_data_smithi192_88961_object88213:head# 2024-06-27T21:23:15.454 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba67fda4:::benchmark_data_smithi192_88961_object76028:head# 2024-06-27T21:23:15.457 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba6ac071:::benchmark_data_smithi192_88961_object63710:head# 2024-06-27T21:23:15.457 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba6bb220:::benchmark_data_smithi192_88961_object65933:head# 2024-06-27T21:23:15.458 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba6f9861:::benchmark_data_smithi192_88961_object62958:head# 2024-06-27T21:23:15.458 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba74c177:::benchmark_data_smithi192_88961_object47778:head# 2024-06-27T21:23:15.459 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba74ee8f:::benchmark_data_smithi192_88961_object89977:head# 2024-06-27T21:23:15.459 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba76db83:::benchmark_data_smithi192_88961_object61387:head# 2024-06-27T21:23:15.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:15.460 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba77c716:::benchmark_data_smithi192_88961_object86083:head# 2024-06-27T21:23:15.460 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba7b08c1:::benchmark_data_smithi192_88961_object53419:head# 2024-06-27T21:23:15.462 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba7c7b96:::benchmark_data_smithi192_88961_object69428:head# 2024-06-27T21:23:15.463 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba7e5dd6:::benchmark_data_smithi192_88961_object46087:head# 2024-06-27T21:23:15.464 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba7f5b41:::benchmark_data_smithi192_88961_object79667:head# 2024-06-27T21:23:15.465 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba80e1e9:::benchmark_data_smithi192_88961_object85997:head# 2024-06-27T21:23:15.465 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba8157cf:::benchmark_data_smithi192_88961_object72281:head# 2024-06-27T21:23:15.465 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba818d28:::benchmark_data_smithi192_88961_object71799:head# 2024-06-27T21:23:15.466 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba81d4ef:::benchmark_data_smithi192_88961_object62520:head# 2024-06-27T21:23:15.466 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba81ddf0:::benchmark_data_smithi192_88961_object79922:head# 2024-06-27T21:23:15.467 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba81eddb:::benchmark_data_smithi192_88961_object84501:head# 2024-06-27T21:23:15.468 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba8256db:::benchmark_data_smithi192_88961_object59492:head# 2024-06-27T21:23:15.468 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba845bd5:::benchmark_data_smithi192_88961_object46108:head# 2024-06-27T21:23:15.469 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba85dca5:::benchmark_data_smithi192_88961_object79582:head# 2024-06-27T21:23:15.470 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba866a64:::benchmark_data_smithi192_88961_object90269:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba8985c9:::benchmark_data_smithi192_88961_object58561:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba89961a:::benchmark_data_smithi192_88961_object83478:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba89ce7d:::benchmark_data_smithi192_88961_object87384:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba8a3a83:::benchmark_data_smithi192_88961_object54035:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba8ddc1e:::benchmark_data_smithi192_88961_object65869:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba9147dd:::benchmark_data_smithi192_88961_object69942:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba9172ab:::benchmark_data_smithi192_88961_object51725:head# 2024-06-27T21:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba929103:::benchmark_data_smithi192_88961_object65908:head# 2024-06-27T21:23:15.474 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba932543:::benchmark_data_smithi192_88961_object44829:head# 2024-06-27T21:23:15.474 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba94f94c:::benchmark_data_smithi192_88961_object60241:head# 2024-06-27T21:23:15.475 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba970cd6:::benchmark_data_smithi192_88961_object74678:head# 2024-06-27T21:23:15.475 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba99742a:::benchmark_data_smithi192_88961_object47139:head# 2024-06-27T21:23:15.475 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba9ac536:::benchmark_data_smithi192_88961_object44730:head# 2024-06-27T21:23:15.476 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba9bec95:::benchmark_data_smithi192_88961_object73728:head# 2024-06-27T21:23:15.477 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:ba9ea348:::benchmark_data_smithi192_88961_object85013:head# 2024-06-27T21:23:15.477 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa0ce56:::benchmark_data_smithi192_88961_object46604:head# 2024-06-27T21:23:15.478 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa1dca6:::benchmark_data_smithi192_88961_object59582:head# 2024-06-27T21:23:15.478 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa265cc:::benchmark_data_smithi192_88961_object57249:head# 2024-06-27T21:23:15.478 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa3219b:::benchmark_data_smithi192_88961_object54571:head# 2024-06-27T21:23:15.479 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa3a69e:::benchmark_data_smithi192_88961_object77951:head# 2024-06-27T21:23:15.479 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa5962d:::benchmark_data_smithi192_88961_object66633:head# 2024-06-27T21:23:15.481 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa6a6d9:::benchmark_data_smithi192_88961_object78738:head# 2024-06-27T21:23:15.482 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa748d2:::benchmark_data_smithi192_88961_object70275:head# 2024-06-27T21:23:15.482 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa7e1ce:::benchmark_data_smithi192_88961_object84779:head# 2024-06-27T21:23:15.483 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baa97ff8:::benchmark_data_smithi192_88961_object85596:head# 2024-06-27T21:23:15.483 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baaa5229:::benchmark_data_smithi192_88961_object48155:head# 2024-06-27T21:23:15.483 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baac74b0:::benchmark_data_smithi192_88961_object43157:head# 2024-06-27T21:23:15.484 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baaca298:::benchmark_data_smithi192_88961_object79355:head# 2024-06-27T21:23:15.484 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baadf722:::benchmark_data_smithi192_88961_object51176:head# 2024-06-27T21:23:15.485 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab058bd:::benchmark_data_smithi192_88961_object47404:head# 2024-06-27T21:23:15.486 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab333a0:::benchmark_data_smithi192_88961_object48174:head# 2024-06-27T21:23:15.486 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab33e71:::benchmark_data_smithi192_88961_object73690:head# 2024-06-27T21:23:15.486 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab37bec:::benchmark_data_smithi192_88961_object79913:head# 2024-06-27T21:23:15.487 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab3ebe7:::benchmark_data_smithi192_88961_object74221:head# 2024-06-27T21:23:15.487 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab4489a:::benchmark_data_smithi192_88961_object85123:head# 2024-06-27T21:23:15.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab7f872:::benchmark_data_smithi192_88961_object78910:head# 2024-06-27T21:23:15.489 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bab8877d:::benchmark_data_smithi192_88961_object49644:head# 2024-06-27T21:23:15.489 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:babaf580:::benchmark_data_smithi192_88961_object56277:head# 2024-06-27T21:23:15.490 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:babb97e6:::benchmark_data_smithi192_88961_object79987:head# 2024-06-27T21:23:15.490 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:babd5288:::benchmark_data_smithi192_88961_object71559:head# 2024-06-27T21:23:15.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:15.491 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac03983:::benchmark_data_smithi192_88961_object69694:head# 2024-06-27T21:23:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:15.493 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac12523:::benchmark_data_smithi192_88961_object50229:head# 2024-06-27T21:23:15.493 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac1df33:::benchmark_data_smithi192_88961_object85480:head# 2024-06-27T21:23:15.493 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac4298b:::benchmark_data_smithi192_88961_object87577:head# 2024-06-27T21:23:15.493 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac4ea1b:::benchmark_data_smithi192_88961_object59129:head# 2024-06-27T21:23:15.496 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac55a30:::benchmark_data_smithi192_88961_object76700:head# 2024-06-27T21:23:15.496 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac5e37a:::benchmark_data_smithi192_88961_object74756:head# 2024-06-27T21:23:15.496 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac612cb:::benchmark_data_smithi192_88961_object71874:head# 2024-06-27T21:23:15.498 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac69683:::benchmark_data_smithi192_88961_object54126:head# 2024-06-27T21:23:15.498 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac72758:::benchmark_data_smithi192_88961_object66167:head# 2024-06-27T21:23:15.498 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac76958:::benchmark_data_smithi192_88961_object52253:head# 2024-06-27T21:23:15.499 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bac81b22:::benchmark_data_smithi192_88961_object88902:head# 2024-06-27T21:23:15.500 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bacbc825:::benchmark_data_smithi192_88961_object48159:head# 2024-06-27T21:23:15.500 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bacd1aa3:::benchmark_data_smithi192_88961_object55007:head# 2024-06-27T21:23:15.500 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bace35fa:::benchmark_data_smithi192_88961_object54523:head# 2024-06-27T21:23:15.501 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad2e891:::benchmark_data_smithi192_88961_object75251:head# 2024-06-27T21:23:15.501 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad44c60:::benchmark_data_smithi192_88961_object63844:head# 2024-06-27T21:23:15.501 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad4bc81:::benchmark_data_smithi192_88961_object44964:head# 2024-06-27T21:23:15.501 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad55f7f:::benchmark_data_smithi192_88961_object50452:head# 2024-06-27T21:23:15.502 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad5b567:::benchmark_data_smithi192_88961_object56531:head# 2024-06-27T21:23:15.502 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad6621f:::benchmark_data_smithi192_88961_object64184:head# 2024-06-27T21:23:15.503 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad6df76:::benchmark_data_smithi192_88961_object72004:head# 2024-06-27T21:23:15.504 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad72c9c:::benchmark_data_smithi192_88961_object44199:head# 2024-06-27T21:23:15.504 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bad83b10:::benchmark_data_smithi192_88961_object75352:head# 2024-06-27T21:23:15.504 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:badc9e67:::benchmark_data_smithi192_88961_object69865:head# 2024-06-27T21:23:15.505 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baddd0a8:::benchmark_data_smithi192_88961_object89171:head# 2024-06-27T21:23:15.506 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:badeb8e2:::benchmark_data_smithi192_88961_object69574:head# 2024-06-27T21:23:15.506 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bae02167:::benchmark_data_smithi192_88961_object69885:head# 2024-06-27T21:23:15.506 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bae2e1fd:::benchmark_data_smithi192_88961_object86846:head# 2024-06-27T21:23:15.507 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bae2f055:::benchmark_data_smithi192_88961_object87246:head# 2024-06-27T21:23:15.507 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bae5002b:::benchmark_data_smithi192_88961_object50951:head# 2024-06-27T21:23:15.507 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bae80500:::benchmark_data_smithi192_88961_object85099:head# 2024-06-27T21:23:15.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bae82141:::benchmark_data_smithi192_88961_object63477:head# 2024-06-27T21:23:15.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baeb9f4e:::benchmark_data_smithi192_88961_object50851:head# 2024-06-27T21:23:15.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baed29a6:::benchmark_data_smithi192_88961_object49561:head# 2024-06-27T21:23:15.510 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baefaa26:::benchmark_data_smithi192_88961_object49286:head# 2024-06-27T21:23:15.510 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baf2760e:::benchmark_data_smithi192_88961_object61455:head# 2024-06-27T21:23:15.510 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baf31a24:::benchmark_data_smithi192_88961_object61304:head# 2024-06-27T21:23:15.511 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baf58dd9:::benchmark_data_smithi192_88961_object87358:head# 2024-06-27T21:23:15.512 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baf5ee7f:::benchmark_data_smithi192_88961_object45515:head# 2024-06-27T21:23:15.512 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baf905c2:::benchmark_data_smithi192_88961_object61282:head# 2024-06-27T21:23:15.513 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:baf92bda:::benchmark_data_smithi192_88961_object52897:head# 2024-06-27T21:23:15.513 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bafc8ae8:::benchmark_data_smithi192_88961_object55524:head# 2024-06-27T21:23:15.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bafea1c5:::benchmark_data_smithi192_88961_object67619:head# 2024-06-27T21:23:15.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb02010b:::benchmark_data_smithi192_88961_object85887:head# 2024-06-27T21:23:15.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0265b0:::benchmark_data_smithi192_88961_object60407:head# 2024-06-27T21:23:15.515 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb02a715:::benchmark_data_smithi192_88961_object64438:head# 2024-06-27T21:23:15.517 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb038d16:::benchmark_data_smithi192_88961_object44717:head# 2024-06-27T21:23:15.517 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb071985:::benchmark_data_smithi192_88961_object80123:head# 2024-06-27T21:23:15.518 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0bbd55:::benchmark_data_smithi192_88961_object46513:head# 2024-06-27T21:23:15.518 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0c2101:::benchmark_data_smithi192_88961_object81379:head# 2024-06-27T21:23:15.519 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0e4c45:::benchmark_data_smithi192_88961_object47652:head# 2024-06-27T21:23:15.521 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0e7ff7:::benchmark_data_smithi192_88961_object56450:head# 2024-06-27T21:23:15.521 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0fca03:::benchmark_data_smithi192_88961_object52201:head# 2024-06-27T21:23:15.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0fcf35:::benchmark_data_smithi192_88961_object86826:head# 2024-06-27T21:23:15.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb0fe035:::benchmark_data_smithi192_88961_object75108:head# 2024-06-27T21:23:15.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb119eb9:::benchmark_data_smithi192_88961_object54199:head# 2024-06-27T21:23:15.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb14241f:::benchmark_data_smithi192_88961_object59122:head# 2024-06-27T21:23:15.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb143a31:::benchmark_data_smithi192_88961_object82941:head# 2024-06-27T21:23:15.524 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb14ac6e:::benchmark_data_smithi192_88961_object71617:head# 2024-06-27T21:23:15.525 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1531fb:::benchmark_data_smithi192_88961_object81394:head# 2024-06-27T21:23:15.525 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb155faf:::benchmark_data_smithi192_88961_object64210:head# 2024-06-27T21:23:15.526 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1636b0:::benchmark_data_smithi192_88961_object89857:head# 2024-06-27T21:23:15.526 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1692eb:::benchmark_data_smithi192_88961_object45235:head# 2024-06-27T21:23:15.527 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1801e0:::benchmark_data_smithi192_88961_object59541:head# 2024-06-27T21:23:15.527 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1b3615:::benchmark_data_smithi192_88961_object87961:head# 2024-06-27T21:23:15.528 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1d5314:::benchmark_data_smithi192_88961_object76119:head# 2024-06-27T21:23:15.528 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1e4a0a:::benchmark_data_smithi192_88961_object79354:head# 2024-06-27T21:23:15.529 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb1f53a7:::benchmark_data_smithi192_88961_object60303:head# 2024-06-27T21:23:15.529 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb200cfc:::benchmark_data_smithi192_88961_object73287:head# 2024-06-27T21:23:15.529 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb20a07e:::benchmark_data_smithi192_88961_object75643:head# 2024-06-27T21:23:15.530 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb23ef5a:::benchmark_data_smithi192_88961_object68936:head# 2024-06-27T21:23:15.530 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb249f1c:::benchmark_data_smithi192_88961_object45381:head# 2024-06-27T21:23:15.531 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb26b129:::benchmark_data_smithi192_88961_object81996:head# 2024-06-27T21:23:15.531 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb26d66c:::benchmark_data_smithi192_88961_object66730:head# 2024-06-27T21:23:15.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb2802b4:::benchmark_data_smithi192_88961_object54160:head# 2024-06-27T21:23:15.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb2858c3:::benchmark_data_smithi192_88961_object56935:head# 2024-06-27T21:23:15.533 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb28a690:::benchmark_data_smithi192_88961_object59685:head# 2024-06-27T21:23:15.534 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb2b6fc6:::benchmark_data_smithi192_88961_object43803:head# 2024-06-27T21:23:15.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb2f72b7:::benchmark_data_smithi192_88961_object60293:head# 2024-06-27T21:23:15.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb32eb2b:::benchmark_data_smithi192_88961_object71411:head# 2024-06-27T21:23:15.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb332cdb:::benchmark_data_smithi192_88961_object73662:head# 2024-06-27T21:23:15.538 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb33863f:::benchmark_data_smithi192_88961_object59711:head# 2024-06-27T21:23:15.538 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb348869:::benchmark_data_smithi192_88961_object85154:head# 2024-06-27T21:23:15.538 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb34c3a5:::benchmark_data_smithi192_88961_object87183:head# 2024-06-27T21:23:15.539 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3520cb:::benchmark_data_smithi192_88961_object55739:head# 2024-06-27T21:23:15.539 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb359cd5:::benchmark_data_smithi192_88961_object61261:head# 2024-06-27T21:23:15.540 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb36c058:::benchmark_data_smithi192_88961_object47193:head# 2024-06-27T21:23:15.540 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3832c9:::benchmark_data_smithi192_88961_object46222:head# 2024-06-27T21:23:15.543 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3b3068:::benchmark_data_smithi192_88961_object56691:head# 2024-06-27T21:23:15.543 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3c274e:::benchmark_data_smithi192_88961_object64693:head# 2024-06-27T21:23:15.544 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3d1c36:::benchmark_data_smithi192_88961_object59324:head# 2024-06-27T21:23:15.544 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3f1666:::benchmark_data_smithi192_88961_object49194:head# 2024-06-27T21:23:15.544 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb3f92bf:::benchmark_data_smithi192_88961_object47446:head# 2024-06-27T21:23:15.545 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb45befc:::benchmark_data_smithi192_88961_object59279:head# 2024-06-27T21:23:15.545 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb4771c6:::benchmark_data_smithi192_88961_object70483:head# 2024-06-27T21:23:15.546 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb48ecbb:::benchmark_data_smithi192_88961_object58253:head# 2024-06-27T21:23:15.546 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb4946e6:::benchmark_data_smithi192_88961_object87873:head# 2024-06-27T21:23:15.547 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb4ac95c:::benchmark_data_smithi192_88961_object83210:head# 2024-06-27T21:23:15.547 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb4b3ff3:::benchmark_data_smithi192_88961_object61443:head# 2024-06-27T21:23:15.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb4e1b88:::benchmark_data_smithi192_88961_object69760:head# 2024-06-27T21:23:15.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb4eed1a:::benchmark_data_smithi192_88961_object50015:head# 2024-06-27T21:23:15.550 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb50376f:::benchmark_data_smithi192_88961_object58616:head# 2024-06-27T21:23:15.551 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb51887f:::benchmark_data_smithi192_88961_object76518:head# 2024-06-27T21:23:15.551 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb53a3eb:::benchmark_data_smithi192_88961_object47934:head# 2024-06-27T21:23:15.551 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb548bb2:::benchmark_data_smithi192_88961_object66822:head# 2024-06-27T21:23:15.552 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb5579f3:::benchmark_data_smithi192_88961_object87290:head# 2024-06-27T21:23:15.553 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb56d518:::benchmark_data_smithi192_88961_object53846:head# 2024-06-27T21:23:15.553 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb5946f8:::benchmark_data_smithi192_88961_object53790:head# 2024-06-27T21:23:15.553 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb5f6188:::benchmark_data_smithi192_88961_object72291:head# 2024-06-27T21:23:15.554 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb62fae8:::benchmark_data_smithi192_88961_object47280:head# 2024-06-27T21:23:15.554 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb6731bf:::benchmark_data_smithi192_88961_object68882:head# 2024-06-27T21:23:15.554 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb682011:::benchmark_data_smithi192_88961_object90558:head# 2024-06-27T21:23:15.555 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb68f7a5:::benchmark_data_smithi192_88961_object65351:head# 2024-06-27T21:23:15.556 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb690318:::benchmark_data_smithi192_88961_object79430:head# 2024-06-27T21:23:15.556 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb697b08:::benchmark_data_smithi192_88961_object73510:head# 2024-06-27T21:23:15.556 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb6b799b:::benchmark_data_smithi192_88961_object59710:head# 2024-06-27T21:23:15.557 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb6d35ed:::benchmark_data_smithi192_88961_object71462:head# 2024-06-27T21:23:15.558 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb6f1091:::benchmark_data_smithi192_88961_object55318:head# 2024-06-27T21:23:15.558 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb72e5b6:::benchmark_data_smithi192_88961_object65691:head# 2024-06-27T21:23:15.559 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb72fd11:::benchmark_data_smithi192_88961_object47286:head# 2024-06-27T21:23:15.559 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb74f951:::benchmark_data_smithi192_88961_object57021:head# 2024-06-27T21:23:15.559 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb75ef69:::benchmark_data_smithi192_88961_object70366:head# 2024-06-27T21:23:15.560 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb771999:::benchmark_data_smithi192_88961_object86077:head# 2024-06-27T21:23:15.560 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb7c1cd7:::benchmark_data_smithi192_88961_object72354:head# 2024-06-27T21:23:15.561 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb7ca6b8:::benchmark_data_smithi192_88961_object87450:head# 2024-06-27T21:23:15.561 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb7cb7a2:::benchmark_data_smithi192_88961_object87010:head# 2024-06-27T21:23:15.562 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb8053cb:::benchmark_data_smithi192_88961_object69024:head# 2024-06-27T21:23:15.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb82c043:::benchmark_data_smithi192_88961_object59701:head# 2024-06-27T21:23:15.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb846b6b:::benchmark_data_smithi192_88961_object48542:head# 2024-06-27T21:23:15.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb84c792:::benchmark_data_smithi192_88961_object84471:head# 2024-06-27T21:23:15.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb84fbc5:::benchmark_data_smithi192_88961_object75253:head# 2024-06-27T21:23:15.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb8577f6:::benchmark_data_smithi192_88961_object89394:head# 2024-06-27T21:23:15.565 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb879358:::benchmark_data_smithi192_88961_object88184:head# 2024-06-27T21:23:15.565 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb8a9b94:::benchmark_data_smithi192_88961_object48641:head# 2024-06-27T21:23:15.566 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb8d7dfa:::benchmark_data_smithi192_88961_object55214:head# 2024-06-27T21:23:15.566 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb90f960:::benchmark_data_smithi192_88961_object61019:head# 2024-06-27T21:23:15.566 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb916f60:::benchmark_data_smithi192_88961_object62921:head# 2024-06-27T21:23:15.567 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb9194d6:::benchmark_data_smithi192_88961_object55588:head# 2024-06-27T21:23:15.567 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb950ec3:::benchmark_data_smithi192_88961_object88700:head# 2024-06-27T21:23:15.568 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb95f0ce:::benchmark_data_smithi192_88961_object52364:head# 2024-06-27T21:23:15.568 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb96b515:::benchmark_data_smithi192_88961_object44571:head# 2024-06-27T21:23:15.569 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb989f5a:::benchmark_data_smithi192_88961_object71283:head# 2024-06-27T21:23:15.569 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb9a5b9d:::benchmark_data_smithi192_88961_object88981:head# 2024-06-27T21:23:15.569 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb9a79db:::benchmark_data_smithi192_88961_object59028:head# 2024-06-27T21:23:15.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb9b0afe:::benchmark_data_smithi192_88961_object48541:head# 2024-06-27T21:23:15.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb9b53da:::benchmark_data_smithi192_88961_object72864:head# 2024-06-27T21:23:15.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bb9e00af:::benchmark_data_smithi192_88961_object78649:head# 2024-06-27T21:23:15.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bba0153c:::benchmark_data_smithi192_88961_object54758:head# 2024-06-27T21:23:15.572 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bba1af7a:::benchmark_data_smithi192_88961_object83811:head# 2024-06-27T21:23:15.572 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bba32bbe:::benchmark_data_smithi192_88961_object54313:head# 2024-06-27T21:23:15.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bba4cef5:::benchmark_data_smithi192_88961_object54050:head# 2024-06-27T21:23:15.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bba823f1:::benchmark_data_smithi192_88961_object62432:head# 2024-06-27T21:23:15.575 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbaab373:::benchmark_data_smithi192_88961_object72415:head# 2024-06-27T21:23:15.576 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbabc8f1:::benchmark_data_smithi192_88961_object53608:head# 2024-06-27T21:23:15.576 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbac4a6c:::benchmark_data_smithi192_88961_object60552:head# 2024-06-27T21:23:15.576 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbaca5af:::benchmark_data_smithi192_88961_object73188:head# 2024-06-27T21:23:15.577 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbacabeb:::benchmark_data_smithi192_88961_object82455:head# 2024-06-27T21:23:15.577 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbafd1c1:::benchmark_data_smithi192_88961_object43196:head# 2024-06-27T21:23:15.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbb164b9:::benchmark_data_smithi192_88961_object84050:head# 2024-06-27T21:23:15.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbb34f76:::benchmark_data_smithi192_88961_object58274:head# 2024-06-27T21:23:15.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbb4864b:::benchmark_data_smithi192_88961_object83075:head# 2024-06-27T21:23:15.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbb6160e:::benchmark_data_smithi192_88961_object90197:head# 2024-06-27T21:23:15.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbb931c2:::benchmark_data_smithi192_88961_object54856:head# 2024-06-27T21:23:15.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbbaabbb:::benchmark_data_smithi192_88961_object86972:head# 2024-06-27T21:23:15.580 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbc03cf8:::benchmark_data_smithi192_88961_object49336:head# 2024-06-27T21:23:15.580 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbc16cb0:::benchmark_data_smithi192_88961_object77291:head# 2024-06-27T21:23:15.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbc41287:::benchmark_data_smithi192_88961_object74668:head# 2024-06-27T21:23:15.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbc6ad12:::benchmark_data_smithi192_88961_object58025:head# 2024-06-27T21:23:15.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbc80c26:::benchmark_data_smithi192_88961_object56020:head# 2024-06-27T21:23:15.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbcced72:::benchmark_data_smithi192_88961_object85533:head# 2024-06-27T21:23:15.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbcdedf5:::benchmark_data_smithi192_88961_object76752:head# 2024-06-27T21:23:15.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbcee8a7:::benchmark_data_smithi192_88961_object74029:head# 2024-06-27T21:23:15.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbcfd7a5:::benchmark_data_smithi192_88961_object60634:head# 2024-06-27T21:23:15.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd019ab:::benchmark_data_smithi192_88961_object89850:head# 2024-06-27T21:23:15.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd08d2f:::benchmark_data_smithi192_88961_object56560:head# 2024-06-27T21:23:15.586 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd251f8:::benchmark_data_smithi192_88961_object46897:head# 2024-06-27T21:23:15.587 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd31999:::benchmark_data_smithi192_88961_object45414:head# 2024-06-27T21:23:15.587 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd43069:::benchmark_data_smithi192_88961_object74286:head# 2024-06-27T21:23:15.588 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd6b1bb:::benchmark_data_smithi192_88961_object59356:head# 2024-06-27T21:23:15.588 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd87749:::benchmark_data_smithi192_88961_object74402:head# 2024-06-27T21:23:15.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbd95f60:::benchmark_data_smithi192_88961_object76152:head# 2024-06-27T21:23:15.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbdb3c1f:::benchmark_data_smithi192_88961_object63763:head# 2024-06-27T21:23:15.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbdb5215:::benchmark_data_smithi192_88961_object56664:head# 2024-06-27T21:23:15.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbdf980d:::benchmark_data_smithi192_88961_object84428:head# 2024-06-27T21:23:15.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe0707e:::benchmark_data_smithi192_88961_object70386:head# 2024-06-27T21:23:15.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe15f2e:::benchmark_data_smithi192_88961_object81922:head# 2024-06-27T21:23:15.592 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe28aa0:::benchmark_data_smithi192_88961_object44374:head# 2024-06-27T21:23:15.592 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe53269:::benchmark_data_smithi192_88961_object43293:head# 2024-06-27T21:23:15.593 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe54ba1:::benchmark_data_smithi192_88961_object82824:head# 2024-06-27T21:23:15.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:15.594 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe812fd:::benchmark_data_smithi192_88961_object55393:head# 2024-06-27T21:23:15.594 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe96adc:::benchmark_data_smithi192_88961_object61145:head# 2024-06-27T21:23:15.594 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbe9880a:::benchmark_data_smithi192_88961_object80438:head# 2024-06-27T21:23:15.595 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbeba0a0:::benchmark_data_smithi192_88961_object71471:head# 2024-06-27T21:23:15.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbebfdcc:::benchmark_data_smithi192_88961_object54582:head# 2024-06-27T21:23:15.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbee8af4:::benchmark_data_smithi192_88961_object54182:head# 2024-06-27T21:23:15.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbeef3b2:::benchmark_data_smithi192_88961_object50095:head# 2024-06-27T21:23:15.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbef8f5a:::benchmark_data_smithi192_88961_object52149:head# 2024-06-27T21:23:15.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbefc223:::benchmark_data_smithi192_88961_object87807:head# 2024-06-27T21:23:15.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf166e3:::benchmark_data_smithi192_88961_object87767:head# 2024-06-27T21:23:15.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf1f5d5:::benchmark_data_smithi192_88961_object44642:head# 2024-06-27T21:23:15.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf1f730:::benchmark_data_smithi192_88961_object70758:head# 2024-06-27T21:23:15.599 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf22037:::benchmark_data_smithi192_88961_object71891:head# 2024-06-27T21:23:15.599 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf2f6a9:::benchmark_data_smithi192_88961_object80345:head# 2024-06-27T21:23:15.600 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf3ac4b:::benchmark_data_smithi192_88961_object89228:head# 2024-06-27T21:23:15.600 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf4ea0d:::benchmark_data_smithi192_88961_object43984:head# 2024-06-27T21:23:15.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf73367:::benchmark_data_smithi192_88961_object45609:head# 2024-06-27T21:23:15.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbf8ca18:::benchmark_data_smithi192_88961_object59156:head# 2024-06-27T21:23:15.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bbfc2259:::benchmark_data_smithi192_88961_object63703:head# 2024-06-27T21:23:15.602 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc01a3e1:::benchmark_data_smithi192_88961_object80773:head# 2024-06-27T21:23:15.603 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc02457a:::benchmark_data_smithi192_88961_object69284:head# 2024-06-27T21:23:15.604 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc026ef1:::benchmark_data_smithi192_88961_object65662:head# 2024-06-27T21:23:15.604 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc031e04:::benchmark_data_smithi192_88961_object85383:head# 2024-06-27T21:23:15.607 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc032d33:::benchmark_data_smithi192_88961_object84693:head# 2024-06-27T21:23:15.607 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc04065a:::benchmark_data_smithi192_88961_object62118:head# 2024-06-27T21:23:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc04a9ad:::benchmark_data_smithi192_88961_object75779:head# 2024-06-27T21:23:15.608 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc065ec3:::benchmark_data_smithi192_88961_object65754:head# 2024-06-27T21:23:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc0a3016:::benchmark_data_smithi192_88961_object52583:head# 2024-06-27T21:23:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc0e604c:::benchmark_data_smithi192_88961_object76946:head# 2024-06-27T21:23:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc10c60d:::benchmark_data_smithi192_88961_object69293:head# 2024-06-27T21:23:15.610 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc113308:::benchmark_data_smithi192_88961_object63240:head# 2024-06-27T21:23:15.611 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc11e4fd:::benchmark_data_smithi192_88961_object50185:head# 2024-06-27T21:23:15.613 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc133475:::benchmark_data_smithi192_88961_object66514:head# 2024-06-27T21:23:15.613 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc135a1f:::benchmark_data_smithi192_88961_object57237:head# 2024-06-27T21:23:15.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1574bc:::benchmark_data_smithi192_88961_object56923:head# 2024-06-27T21:23:15.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc15d1a1:::benchmark_data_smithi192_88961_object51305:head# 2024-06-27T21:23:15.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc15dd5c:::benchmark_data_smithi192_88961_object75646:head# 2024-06-27T21:23:15.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc15e8c2:::benchmark_data_smithi192_88961_object69502:head# 2024-06-27T21:23:15.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1630a3:::benchmark_data_smithi192_88961_object49766:head# 2024-06-27T21:23:15.616 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc172fb2:::benchmark_data_smithi192_88961_object78054:head# 2024-06-27T21:23:15.617 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc17b90a:::benchmark_data_smithi192_88961_object60045:head# 2024-06-27T21:23:15.617 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc18720d:::benchmark_data_smithi192_88961_object62336:head# 2024-06-27T21:23:15.617 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc18a627:::benchmark_data_smithi192_88961_object80204:head# 2024-06-27T21:23:15.618 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc18d231:::benchmark_data_smithi192_88961_object56812:head# 2024-06-27T21:23:15.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc194a4b:::benchmark_data_smithi192_88961_object73943:head# 2024-06-27T21:23:15.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1bd95a:::benchmark_data_smithi192_88961_object46110:head# 2024-06-27T21:23:15.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1c742a:::benchmark_data_smithi192_88961_object52179:head# 2024-06-27T21:23:15.620 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1c779d:::benchmark_data_smithi192_88961_object63658:head# 2024-06-27T21:23:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:15.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:15.621 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1d2b96:::benchmark_data_smithi192_88961_object57537:head# 2024-06-27T21:23:15.621 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1dd5a2:::benchmark_data_smithi192_88961_object71551:head# 2024-06-27T21:23:15.622 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc1fd54b:::benchmark_data_smithi192_88961_object83822:head# 2024-06-27T21:23:15.622 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2241b6:::benchmark_data_smithi192_88961_object52522:head# 2024-06-27T21:23:15.623 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2424a5:::benchmark_data_smithi192_88961_object78273:head# 2024-06-27T21:23:15.623 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2672fc:::benchmark_data_smithi192_88961_object53950:head# 2024-06-27T21:23:15.623 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2681bf:::benchmark_data_smithi192_88961_object76676:head# 2024-06-27T21:23:15.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc274de7:::benchmark_data_smithi192_88961_object59290:head# 2024-06-27T21:23:15.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc28adee:::benchmark_data_smithi192_88961_object84412:head# 2024-06-27T21:23:15.626 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2a3f0d:::benchmark_data_smithi192_88961_object56367:head# 2024-06-27T21:23:15.626 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2c4dcb:::benchmark_data_smithi192_88961_object58526:head# 2024-06-27T21:23:15.627 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2d5e6d:::benchmark_data_smithi192_88961_object66859:head# 2024-06-27T21:23:15.627 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc2e008b:::benchmark_data_smithi192_88961_object79149:head# 2024-06-27T21:23:15.628 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc305bf4:::benchmark_data_smithi192_88961_object80779:head# 2024-06-27T21:23:15.628 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc307bc4:::benchmark_data_smithi192_88961_object56265:head# 2024-06-27T21:23:15.629 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc30bdd2:::benchmark_data_smithi192_88961_object69501:head# 2024-06-27T21:23:15.629 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc32ffc5:::benchmark_data_smithi192_88961_object48315:head# 2024-06-27T21:23:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3371ae:::benchmark_data_smithi192_88961_object63409:head# 2024-06-27T21:23:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc348e8e:::benchmark_data_smithi192_88961_object81635:head# 2024-06-27T21:23:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc34eeba:::benchmark_data_smithi192_88961_object72300:head# 2024-06-27T21:23:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc34fe26:::benchmark_data_smithi192_88961_object52226:head# 2024-06-27T21:23:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc35410e:::benchmark_data_smithi192_88961_object77563:head# 2024-06-27T21:23:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc354292:::benchmark_data_smithi192_88961_object65231:head# 2024-06-27T21:23:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc356aaa:::benchmark_data_smithi192_88961_object88764:head# 2024-06-27T21:23:15.633 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc380fc1:::benchmark_data_smithi192_88961_object59631:head# 2024-06-27T21:23:15.633 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3936f0:::benchmark_data_smithi192_88961_object80656:head# 2024-06-27T21:23:15.634 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc39c658:::benchmark_data_smithi192_88961_object82272:head# 2024-06-27T21:23:15.634 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3a8716:::benchmark_data_smithi192_88961_object75388:head# 2024-06-27T21:23:15.636 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3cdf80:::benchmark_data_smithi192_88961_object70918:head# 2024-06-27T21:23:15.637 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3d9740:::benchmark_data_smithi192_88961_object65577:head# 2024-06-27T21:23:15.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3dde29:::benchmark_data_smithi192_88961_object46987:head# 2024-06-27T21:23:15.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3ebcae:::benchmark_data_smithi192_88961_object47719:head# 2024-06-27T21:23:15.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc3eebbb:::benchmark_data_smithi192_88961_object89288:head# 2024-06-27T21:23:15.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc41908a:::benchmark_data_smithi192_88961_object73543:head# 2024-06-27T21:23:15.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc4246c7:::benchmark_data_smithi192_88961_object46962:head# 2024-06-27T21:23:15.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc425158:::benchmark_data_smithi192_88961_object87406:head# 2024-06-27T21:23:15.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc459b96:::benchmark_data_smithi192_88961_object65585:head# 2024-06-27T21:23:15.641 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc495142:::benchmark_data_smithi192_88961_object85103:head# 2024-06-27T21:23:15.641 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc49ee45:::benchmark_data_smithi192_88961_object65286:head# 2024-06-27T21:23:15.641 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc4b13d6:::benchmark_data_smithi192_88961_object79647:head# 2024-06-27T21:23:15.642 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc4b991a:::benchmark_data_smithi192_88961_object61135:head# 2024-06-27T21:23:15.643 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc4cda97:::benchmark_data_smithi192_88961_object77917:head# 2024-06-27T21:23:15.643 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc4dcb3f:::benchmark_data_smithi192_88961_object48293:head# 2024-06-27T21:23:15.644 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc511d37:::benchmark_data_smithi192_88961_object87527:head# 2024-06-27T21:23:15.644 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc52fea3:::benchmark_data_smithi192_88961_object80044:head# 2024-06-27T21:23:15.644 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc544c70:::benchmark_data_smithi192_88961_object71086:head# 2024-06-27T21:23:15.645 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc577eee:::benchmark_data_smithi192_88961_object43095:head# 2024-06-27T21:23:15.645 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc5ca1e7:::benchmark_data_smithi192_88961_object62477:head# 2024-06-27T21:23:15.646 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc5d374a:::benchmark_data_smithi192_88961_object53819:head# 2024-06-27T21:23:15.646 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc61ad82:::benchmark_data_smithi192_88961_object71480:head# 2024-06-27T21:23:15.646 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc62e93b:::benchmark_data_smithi192_88961_object51584:head# 2024-06-27T21:23:15.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc632a80:::benchmark_data_smithi192_88961_object52935:head# 2024-06-27T21:23:15.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6337a0:::benchmark_data_smithi192_88961_object69197:head# 2024-06-27T21:23:15.648 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc63f81a:::benchmark_data_smithi192_88961_object82133:head# 2024-06-27T21:23:15.649 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc64897b:::benchmark_data_smithi192_88961_object64849:head# 2024-06-27T21:23:15.649 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc668cda:::benchmark_data_smithi192_88961_object76051:head# 2024-06-27T21:23:15.650 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc68145d:::benchmark_data_smithi192_88961_object69271:head# 2024-06-27T21:23:15.650 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6b2f40:::benchmark_data_smithi192_88961_object66546:head# 2024-06-27T21:23:15.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6b5c59:::benchmark_data_smithi192_88961_object45891:head# 2024-06-27T21:23:15.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6ce282:::benchmark_data_smithi192_88961_object58115:head# 2024-06-27T21:23:15.652 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6d225e:::benchmark_data_smithi192_88961_object80276:head# 2024-06-27T21:23:15.652 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6ee462:::benchmark_data_smithi192_88961_object74992:head# 2024-06-27T21:23:15.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc6ef33a:::benchmark_data_smithi192_88961_object44582:head# 2024-06-27T21:23:15.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc732e47:::benchmark_data_smithi192_88961_object54446:head# 2024-06-27T21:23:15.654 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc764b85:::benchmark_data_smithi192_88961_object89820:head# 2024-06-27T21:23:15.654 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc76f97c:::benchmark_data_smithi192_88961_object59407:head# 2024-06-27T21:23:15.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc790377:::benchmark_data_smithi192_88961_object79103:head# 2024-06-27T21:23:15.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc79b714:::benchmark_data_smithi192_88961_object43889:head# 2024-06-27T21:23:15.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc7a9227:::benchmark_data_smithi192_88961_object54353:head# 2024-06-27T21:23:15.656 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc7b7bfd:::benchmark_data_smithi192_88961_object74067:head# 2024-06-27T21:23:15.656 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc7d3153:::benchmark_data_smithi192_88961_object59102:head# 2024-06-27T21:23:15.656 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc7de333:::benchmark_data_smithi192_88961_object61196:head# 2024-06-27T21:23:15.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc7ed7e4:::benchmark_data_smithi192_88961_object53058:head# 2024-06-27T21:23:15.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc7ffdf1:::benchmark_data_smithi192_88961_object72913:head# 2024-06-27T21:23:15.658 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc800ca6:::benchmark_data_smithi192_88961_object79154:head# 2024-06-27T21:23:15.658 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc801a9e:::benchmark_data_smithi192_88961_object65409:head# 2024-06-27T21:23:15.658 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc803471:::benchmark_data_smithi192_88961_object77683:head# 2024-06-27T21:23:15.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc82753c:::benchmark_data_smithi192_88961_object79471:head# 2024-06-27T21:23:15.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc839d14:::benchmark_data_smithi192_88961_object88112:head# 2024-06-27T21:23:15.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc8706bf:::benchmark_data_smithi192_88961_object80227:head# 2024-06-27T21:23:15.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc87dfea:::benchmark_data_smithi192_88961_object57062:head# 2024-06-27T21:23:15.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc883b7c:::benchmark_data_smithi192_88961_object43755:head# 2024-06-27T21:23:15.661 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc88d220:::benchmark_data_smithi192_88961_object53818:head# 2024-06-27T21:23:15.661 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc89b262:::benchmark_data_smithi192_88961_object49554:head# 2024-06-27T21:23:15.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc8c66a4:::benchmark_data_smithi192_88961_object61095:head# 2024-06-27T21:23:15.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc8c8f5f:::benchmark_data_smithi192_88961_object52026:head# 2024-06-27T21:23:15.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc8cc995:::benchmark_data_smithi192_88961_object52301:head# 2024-06-27T21:23:15.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc905d69:::benchmark_data_smithi192_88961_object87179:head# 2024-06-27T21:23:15.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc909e69:::benchmark_data_smithi192_88961_object66957:head# 2024-06-27T21:23:15.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc928fee:::benchmark_data_smithi192_88961_object73404:head# 2024-06-27T21:23:15.664 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc92e2c2:::benchmark_data_smithi192_88961_object69236:head# 2024-06-27T21:23:15.664 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc961cea:::benchmark_data_smithi192_88961_object81364:head# 2024-06-27T21:23:15.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc975ddd:::benchmark_data_smithi192_88961_object70532:head# 2024-06-27T21:23:15.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc97949b:::benchmark_data_smithi192_88961_object90062:head# 2024-06-27T21:23:15.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc99a530:::benchmark_data_smithi192_88961_object74641:head# 2024-06-27T21:23:15.666 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc9a67fd:::benchmark_data_smithi192_88961_object48632:head# 2024-06-27T21:23:15.666 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc9e1a85:::benchmark_data_smithi192_88961_object68855:head# 2024-06-27T21:23:15.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bc9ed4a1:::benchmark_data_smithi192_88961_object60703:head# 2024-06-27T21:23:15.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca0e35d:::benchmark_data_smithi192_88961_object61463:head# 2024-06-27T21:23:15.668 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca2d420:::benchmark_data_smithi192_88961_object51446:head# 2024-06-27T21:23:15.668 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca3d1b9:::benchmark_data_smithi192_88961_object73112:head# 2024-06-27T21:23:15.669 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca45e85:::benchmark_data_smithi192_88961_object46681:head# 2024-06-27T21:23:15.669 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca4ee18:::benchmark_data_smithi192_88961_object73504:head# 2024-06-27T21:23:15.669 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca4ffb3:::benchmark_data_smithi192_88961_object69341:head# 2024-06-27T21:23:15.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca69adf:::benchmark_data_smithi192_88961_object86688:head# 2024-06-27T21:23:15.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca6f8a9:::benchmark_data_smithi192_88961_object48297:head# 2024-06-27T21:23:15.671 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bca99469:::benchmark_data_smithi192_88961_object88062:head# 2024-06-27T21:23:15.671 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcac5742:::benchmark_data_smithi192_88961_object88810:head# 2024-06-27T21:23:15.672 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcac97b5:::benchmark_data_smithi192_88961_object88920:head# 2024-06-27T21:23:15.672 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcae7def:::benchmark_data_smithi192_88961_object63271:head# 2024-06-27T21:23:15.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcaed639:::benchmark_data_smithi192_88961_object44404:head# 2024-06-27T21:23:15.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb11400:::benchmark_data_smithi192_88961_object87204:head# 2024-06-27T21:23:15.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb42807:::benchmark_data_smithi192_88961_object59903:head# 2024-06-27T21:23:15.674 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb4a5ee:::benchmark_data_smithi192_88961_object70811:head# 2024-06-27T21:23:15.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb5cb9c:::benchmark_data_smithi192_88961_object66910:head# 2024-06-27T21:23:15.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb65f22:::benchmark_data_smithi192_88961_object78058:head# 2024-06-27T21:23:15.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb684d1:::benchmark_data_smithi192_88961_object84651:head# 2024-06-27T21:23:15.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb6cdb5:::benchmark_data_smithi192_88961_object88526:head# 2024-06-27T21:23:15.676 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb7484a:::benchmark_data_smithi192_88961_object68384:head# 2024-06-27T21:23:15.676 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb7a306:::benchmark_data_smithi192_88961_object84466:head# 2024-06-27T21:23:15.676 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcb9e952:::benchmark_data_smithi192_88961_object49461:head# 2024-06-27T21:23:15.677 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcbb6a6d:::benchmark_data_smithi192_88961_object58926:head# 2024-06-27T21:23:15.677 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcbc201d:::benchmark_data_smithi192_88961_object63254:head# 2024-06-27T21:23:15.678 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcbd0ac4:::benchmark_data_smithi192_88961_object77523:head# 2024-06-27T21:23:15.680 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcbfd382:::benchmark_data_smithi192_88961_object43566:head# 2024-06-27T21:23:15.680 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc03dd9:::benchmark_data_smithi192_88961_object43074:head# 2024-06-27T21:23:15.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc16564:::benchmark_data_smithi192_88961_object43703:head# 2024-06-27T21:23:15.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc1b2fb:::benchmark_data_smithi192_88961_object69633:head# 2024-06-27T21:23:15.682 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc56588:::benchmark_data_smithi192_88961_object69779:head# 2024-06-27T21:23:15.682 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc61300:::benchmark_data_smithi192_88961_object70773:head# 2024-06-27T21:23:15.683 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc6a30c:::benchmark_data_smithi192_88961_object46904:head# 2024-06-27T21:23:15.683 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc8f074:::benchmark_data_smithi192_88961_object78890:head# 2024-06-27T21:23:15.683 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc97db3:::benchmark_data_smithi192_88961_object47420:head# 2024-06-27T21:23:15.684 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcc9ba69:::benchmark_data_smithi192_88961_object46883:head# 2024-06-27T21:23:15.687 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bccb108c:::benchmark_data_smithi192_88961_object43368:head# 2024-06-27T21:23:15.687 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bccb28b9:::benchmark_data_smithi192_88961_object87004:head# 2024-06-27T21:23:15.688 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bccc98ea:::benchmark_data_smithi192_88961_object49696:head# 2024-06-27T21:23:15.688 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bccca115:::benchmark_data_smithi192_88961_object89083:head# 2024-06-27T21:23:15.689 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bccd335d:::benchmark_data_smithi192_88961_object51347:head# 2024-06-27T21:23:15.690 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd009cf:::benchmark_data_smithi192_88961_object80481:head# 2024-06-27T21:23:15.690 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd20641:::benchmark_data_smithi192_88961_object63548:head# 2024-06-27T21:23:15.690 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd2da4b:::benchmark_data_smithi192_88961_object70306:head# 2024-06-27T21:23:15.691 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd3350d:::benchmark_data_smithi192_88961_object50913:head# 2024-06-27T21:23:15.691 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd36917:::benchmark_data_smithi192_88961_object71861:head# 2024-06-27T21:23:15.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd3a701:::benchmark_data_smithi192_88961_object52932:head# 2024-06-27T21:23:15.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd3df90:::benchmark_data_smithi192_88961_object77201:head# 2024-06-27T21:23:15.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd497e1:::benchmark_data_smithi192_88961_object63455:head# 2024-06-27T21:23:15.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd6c074:::benchmark_data_smithi192_88961_object49387:head# 2024-06-27T21:23:15.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcd7c138:::benchmark_data_smithi192_88961_object64947:head# 2024-06-27T21:23:15.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcda04d1:::benchmark_data_smithi192_88961_object67785:head# 2024-06-27T21:23:15.696 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcdb7739:::benchmark_data_smithi192_88961_object65975:head# 2024-06-27T21:23:15.697 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcdbf1e9:::benchmark_data_smithi192_88961_object74014:head# 2024-06-27T21:23:15.697 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcdc894d:::benchmark_data_smithi192_88961_object44837:head# 2024-06-27T21:23:15.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcddc403:::benchmark_data_smithi192_88961_object90133:head# 2024-06-27T21:23:15.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcde0071:::benchmark_data_smithi192_88961_object60164:head# 2024-06-27T21:23:15.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcdf4582:::benchmark_data_smithi192_88961_object46006:head# 2024-06-27T21:23:15.699 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcdf5c7f:::benchmark_data_smithi192_88961_object77411:head# 2024-06-27T21:23:15.699 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcdfb127:::benchmark_data_smithi192_88961_object60390:head# 2024-06-27T21:23:15.700 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bce05cae:::benchmark_data_smithi192_88961_object58913:head# 2024-06-27T21:23:15.700 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bce42f56:::benchmark_data_smithi192_88961_object43543:head# 2024-06-27T21:23:15.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bce536ef:::benchmark_data_smithi192_88961_object59201:head# 2024-06-27T21:23:15.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bce86fc4:::benchmark_data_smithi192_88961_object89833:head# 2024-06-27T21:23:15.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bceb7e45:::benchmark_data_smithi192_88961_object84057:head# 2024-06-27T21:23:15.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcebf5ff:::benchmark_data_smithi192_88961_object50879:head# 2024-06-27T21:23:15.705 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcef20dd:::benchmark_data_smithi192_88961_object73057:head# 2024-06-27T21:23:15.705 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcf18e7b:::benchmark_data_smithi192_88961_object55808:head# 2024-06-27T21:23:15.708 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcf1bba7:::benchmark_data_smithi192_88961_object67385:head# 2024-06-27T21:23:15.709 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcf2731f:::benchmark_data_smithi192_88961_object75047:head# 2024-06-27T21:23:15.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcf6e599:::benchmark_data_smithi192_88961_object86853:head# 2024-06-27T21:23:15.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcf6f305:::benchmark_data_smithi192_88961_object72019:head# 2024-06-27T21:23:15.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcf8af8d:::benchmark_data_smithi192_88961_object43596:head# 2024-06-27T21:23:15.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcfa7ecb:::benchmark_data_smithi192_88961_object88379:head# 2024-06-27T21:23:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcfb7804:::benchmark_data_smithi192_88961_object79892:head# 2024-06-27T21:23:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcfbd4b3:::benchmark_data_smithi192_88961_object54203:head# 2024-06-27T21:23:15.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:15 smithi190 ceph-mon[32247]: pgmap v925: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 10490/126051 objects degraded (8.322%); 7408/126051 objects misplaced (5.877%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:23:15.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:15 smithi190 ceph-mon[32247]: osdmap e352: 12 total, 11 up, 12 in 2024-06-27T21:23:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bcfc1dbf:::benchmark_data_smithi192_88961_object58945:head# 2024-06-27T21:23:15.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd000e1c:::benchmark_data_smithi192_88961_object58739:head# 2024-06-27T21:23:15.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd00ad2d:::benchmark_data_smithi192_88961_object77451:head# 2024-06-27T21:23:15.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd016012:::benchmark_data_smithi192_88961_object85918:head# 2024-06-27T21:23:15.718 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd02dcb2:::benchmark_data_smithi192_88961_object50615:head# 2024-06-27T21:23:15.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd053427:::benchmark_data_smithi192_88961_object88999:head# 2024-06-27T21:23:15.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd057622:::benchmark_data_smithi192_88961_object53628:head# 2024-06-27T21:23:15.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd06884c:::benchmark_data_smithi192_88961_object73566:head# 2024-06-27T21:23:15.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd06efd4:::benchmark_data_smithi192_88961_object62869:head# 2024-06-27T21:23:15.721 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd076da0:::benchmark_data_smithi192_88961_object57805:head# 2024-06-27T21:23:15.721 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd07807f:::benchmark_data_smithi192_88961_object89904:head# 2024-06-27T21:23:15.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:15.723 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0829ab:::benchmark_data_smithi192_88961_object86649:head# 2024-06-27T21:23:15.723 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0888ce:::benchmark_data_smithi192_88961_object63988:head# 2024-06-27T21:23:15.723 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd08b902:::benchmark_data_smithi192_88961_object81807:head# 2024-06-27T21:23:15.723 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0a1dcf:::benchmark_data_smithi192_88961_object88706:head# 2024-06-27T21:23:15.724 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0b748f:::benchmark_data_smithi192_88961_object57728:head# 2024-06-27T21:23:15.724 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0b8751:::benchmark_data_smithi192_88961_object74120:head# 2024-06-27T21:23:15.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0beb8c:::benchmark_data_smithi192_88961_object66332:head# 2024-06-27T21:23:15.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0c4731:::benchmark_data_smithi192_88961_object71533:head# 2024-06-27T21:23:15.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0c47e4:::benchmark_data_smithi192_88961_object52554:head# 2024-06-27T21:23:15.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0ce7da:::benchmark_data_smithi192_88961_object84154:head# 2024-06-27T21:23:15.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd0ead57:::benchmark_data_smithi192_88961_object74394:head# 2024-06-27T21:23:15.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd129eb8:::benchmark_data_smithi192_88961_object51686:head# 2024-06-27T21:23:15.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd16cb28:::benchmark_data_smithi192_88961_object78541:head# 2024-06-27T21:23:15.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1aeaf5:::benchmark_data_smithi192_88961_object68627:head# 2024-06-27T21:23:15.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1b4acb:::benchmark_data_smithi192_88961_object58821:head# 2024-06-27T21:23:15.728 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1cf3d8:::benchmark_data_smithi192_88961_object66006:head# 2024-06-27T21:23:15.728 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1d0dc6:::benchmark_data_smithi192_88961_object57479:head# 2024-06-27T21:23:15.728 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1df939:::benchmark_data_smithi192_88961_object54659:head# 2024-06-27T21:23:15.729 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1ebe2d:::benchmark_data_smithi192_88961_object56713:head# 2024-06-27T21:23:15.729 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1ed882:::benchmark_data_smithi192_88961_object76983:head# 2024-06-27T21:23:15.730 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1f140e:::benchmark_data_smithi192_88961_object90626:head# 2024-06-27T21:23:15.730 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1f87c0:::benchmark_data_smithi192_88961_object52136:head# 2024-06-27T21:23:15.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1ff8e4:::benchmark_data_smithi192_88961_object63610:head# 2024-06-27T21:23:15.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd1ffb9f:::benchmark_data_smithi192_88961_object62928:head# 2024-06-27T21:23:15.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd2098fc:::benchmark_data_smithi192_88961_object76599:head# 2024-06-27T21:23:15.732 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd20c570:::benchmark_data_smithi192_88961_object72649:head# 2024-06-27T21:23:15.732 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd20eced:::benchmark_data_smithi192_88961_object45347:head# 2024-06-27T21:23:15.733 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd213f1d:::benchmark_data_smithi192_88961_object64548:head# 2024-06-27T21:23:15.733 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd22b9fc:::benchmark_data_smithi192_88961_object69157:head# 2024-06-27T21:23:15.734 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd25366d:::benchmark_data_smithi192_88961_object55457:head# 2024-06-27T21:23:15.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd2661aa:::benchmark_data_smithi192_88961_object79656:head# 2024-06-27T21:23:15.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd28c773:::benchmark_data_smithi192_88961_object46144:head# 2024-06-27T21:23:15.736 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3202f5:::benchmark_data_smithi192_88961_object65711:head# 2024-06-27T21:23:15.736 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3261b3:::benchmark_data_smithi192_88961_object79293:head# 2024-06-27T21:23:15.737 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd32f5f5:::benchmark_data_smithi192_88961_object83001:head# 2024-06-27T21:23:15.737 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3a253e:::benchmark_data_smithi192_88961_object49314:head# 2024-06-27T21:23:15.738 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3afa4e:::benchmark_data_smithi192_88961_object60190:head# 2024-06-27T21:23:15.738 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3b57b3:::benchmark_data_smithi192_88961_object79283:head# 2024-06-27T21:23:15.738 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3ba930:::benchmark_data_smithi192_88961_object62829:head# 2024-06-27T21:23:15.739 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3c0b7e:::benchmark_data_smithi192_88961_object89715:head# 2024-06-27T21:23:15.739 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3c2bfb:::benchmark_data_smithi192_88961_object59692:head# 2024-06-27T21:23:15.739 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3cb9d2:::benchmark_data_smithi192_88961_object85222:head# 2024-06-27T21:23:15.740 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd3d1efa:::benchmark_data_smithi192_88961_object57003:head# 2024-06-27T21:23:15.740 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd4119e3:::benchmark_data_smithi192_88961_object89176:head# 2024-06-27T21:23:15.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd42775b:::benchmark_data_smithi192_88961_object53744:head# 2024-06-27T21:23:15.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd4460ba:::benchmark_data_smithi192_88961_object49410:head# 2024-06-27T21:23:15.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd45f219:::benchmark_data_smithi192_88961_object76836:head# 2024-06-27T21:23:15.742 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd47493b:::benchmark_data_smithi192_88961_object52572:head# 2024-06-27T21:23:15.742 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd4e141c:::benchmark_data_smithi192_88961_object81076:head# 2024-06-27T21:23:15.743 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd5088ca:::benchmark_data_smithi192_88961_object62465:head# 2024-06-27T21:23:15.743 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd518ddf:::benchmark_data_smithi192_88961_object70882:head# 2024-06-27T21:23:15.744 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd522750:::benchmark_data_smithi192_88961_object60055:head# 2024-06-27T21:23:15.744 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd5309fd:::benchmark_data_smithi192_88961_object71540:head# 2024-06-27T21:23:15.745 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd54754e:::benchmark_data_smithi192_88961_object54672:head# 2024-06-27T21:23:15.745 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd57daf9:::benchmark_data_smithi192_88961_object45804:head# 2024-06-27T21:23:15.746 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd59419c:::benchmark_data_smithi192_88961_object83801:head# 2024-06-27T21:23:15.746 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd5cc8b6:::benchmark_data_smithi192_88961_object77399:head# 2024-06-27T21:23:15.747 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd5d0f65:::benchmark_data_smithi192_88961_object84740:head# 2024-06-27T21:23:15.747 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd5d41ab:::benchmark_data_smithi192_88961_object62397:head# 2024-06-27T21:23:15.748 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd5f609e:::benchmark_data_smithi192_88961_object43748:head# 2024-06-27T21:23:15.748 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd61b1ad:::benchmark_data_smithi192_88961_object43896:head# 2024-06-27T21:23:15.749 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd629f49:::benchmark_data_smithi192_88961_object47350:head# 2024-06-27T21:23:15.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:15.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd62d0c1:::benchmark_data_smithi192_88961_object47792:head# 2024-06-27T21:23:15.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd63dfe3:::benchmark_data_smithi192_88961_object90335:head# 2024-06-27T21:23:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:15.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd659c82:::benchmark_data_smithi192_88961_object75333:head# 2024-06-27T21:23:15.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd659e6b:::benchmark_data_smithi192_88961_object87264:head# 2024-06-27T21:23:15.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd6633c3:::benchmark_data_smithi192_88961_object54053:head# 2024-06-27T21:23:15.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd66c072:::benchmark_data_smithi192_88961_object61438:head# 2024-06-27T21:23:15.752 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd6712b7:::benchmark_data_smithi192_88961_object66718:head# 2024-06-27T21:23:15.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd68d8ef:::benchmark_data_smithi192_88961_object76259:head# 2024-06-27T21:23:15.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd6b9a4b:::benchmark_data_smithi192_88961_object63257:head# 2024-06-27T21:23:15.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd6f05ea:::benchmark_data_smithi192_88961_object80855:head# 2024-06-27T21:23:15.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd6fd8b3:::benchmark_data_smithi192_88961_object64204:head# 2024-06-27T21:23:15.754 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd715666:::benchmark_data_smithi192_88961_object70865:head# 2024-06-27T21:23:15.754 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd721878:::benchmark_data_smithi192_88961_object52876:head# 2024-06-27T21:23:15.755 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd72d1a9:::benchmark_data_smithi192_88961_object63442:head# 2024-06-27T21:23:15.755 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd749a67:::benchmark_data_smithi192_88961_object57480:head# 2024-06-27T21:23:15.755 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd756053:::benchmark_data_smithi192_88961_object90292:head# 2024-06-27T21:23:15.756 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd783ba5:::benchmark_data_smithi192_88961_object77665:head# 2024-06-27T21:23:15.756 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd7b4d59:::benchmark_data_smithi192_88961_object82500:head# 2024-06-27T21:23:15.757 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd7bbea3:::benchmark_data_smithi192_88961_object85322:head# 2024-06-27T21:23:15.757 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd7ebce4:::benchmark_data_smithi192_88961_object89888:head# 2024-06-27T21:23:15.758 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd7fbfb1:::benchmark_data_smithi192_88961_object82046:head# 2024-06-27T21:23:15.758 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd7feb3c:::benchmark_data_smithi192_88961_object84927:head# 2024-06-27T21:23:15.758 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd803743:::benchmark_data_smithi192_88961_object58953:head# 2024-06-27T21:23:15.759 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd82d3fc:::benchmark_data_smithi192_88961_object61517:head# 2024-06-27T21:23:15.760 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd837f58:::benchmark_data_smithi192_88961_object67269:head# 2024-06-27T21:23:15.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd83dc05:::benchmark_data_smithi192_88961_object69853:head# 2024-06-27T21:23:15.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8612ea:::benchmark_data_smithi192_88961_object45904:head# 2024-06-27T21:23:15.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8631d4:::benchmark_data_smithi192_88961_object79289:head# 2024-06-27T21:23:15.762 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd866c9f:::benchmark_data_smithi192_88961_object81356:head# 2024-06-27T21:23:15.762 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd88c58c:::benchmark_data_smithi192_88961_object54614:head# 2024-06-27T21:23:15.763 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd88d558:::benchmark_data_smithi192_88961_object74189:head# 2024-06-27T21:23:15.763 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8b5aa5:::benchmark_data_smithi192_88961_object72752:head# 2024-06-27T21:23:15.764 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8c1f77:::benchmark_data_smithi192_88961_object66096:head# 2024-06-27T21:23:15.764 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8c49b3:::benchmark_data_smithi192_88961_object67239:head# 2024-06-27T21:23:15.764 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8ceb9d:::benchmark_data_smithi192_88961_object64223:head# 2024-06-27T21:23:15.765 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8cffb7:::benchmark_data_smithi192_88961_object75968:head# 2024-06-27T21:23:15.765 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8de900:::benchmark_data_smithi192_88961_object62763:head# 2024-06-27T21:23:15.766 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd8ebfcf:::benchmark_data_smithi192_88961_object68628:head# 2024-06-27T21:23:15.766 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd90dfb1:::benchmark_data_smithi192_88961_object84869:head# 2024-06-27T21:23:15.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd91306f:::benchmark_data_smithi192_88961_object77436:head# 2024-06-27T21:23:15.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd918f40:::benchmark_data_smithi192_88961_object45693:head# 2024-06-27T21:23:15.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd93e267:::benchmark_data_smithi192_88961_object82687:head# 2024-06-27T21:23:15.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd950f26:::benchmark_data_smithi192_88961_object74696:head# 2024-06-27T21:23:15.769 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd95909f:::benchmark_data_smithi192_88961_object51353:head# 2024-06-27T21:23:15.769 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd99c076:::benchmark_data_smithi192_88961_object83918:head# 2024-06-27T21:23:15.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd9b067f:::benchmark_data_smithi192_88961_object78468:head# 2024-06-27T21:23:15.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd9c0b81:::benchmark_data_smithi192_88961_object49063:head# 2024-06-27T21:23:15.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd9c2e2e:::benchmark_data_smithi192_88961_object83516:head# 2024-06-27T21:23:15.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd9c5fa1:::benchmark_data_smithi192_88961_object89198:head# 2024-06-27T21:23:15.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd9ce47b:::benchmark_data_smithi192_88961_object81605:head# 2024-06-27T21:23:15.774 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bd9d08d6:::benchmark_data_smithi192_88961_object67546:head# 2024-06-27T21:23:15.774 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda01172:::benchmark_data_smithi192_88961_object90669:head# 2024-06-27T21:23:15.775 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda11f69:::benchmark_data_smithi192_88961_object44814:head# 2024-06-27T21:23:15.775 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda169fb:::benchmark_data_smithi192_88961_object70189:head# 2024-06-27T21:23:15.776 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda338c1:::benchmark_data_smithi192_88961_object44919:head# 2024-06-27T21:23:15.776 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda38ca3:::benchmark_data_smithi192_88961_object67708:head# 2024-06-27T21:23:15.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda54a6a:::benchmark_data_smithi192_88961_object81157:head# 2024-06-27T21:23:15.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda84929:::benchmark_data_smithi192_88961_object60183:head# 2024-06-27T21:23:15.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda96488:::benchmark_data_smithi192_88961_object58355:head# 2024-06-27T21:23:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda9831e:::benchmark_data_smithi192_88961_object73611:head# 2024-06-27T21:23:15.778 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda9a4d2:::benchmark_data_smithi192_88961_object72035:head# 2024-06-27T21:23:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bda9d727:::benchmark_data_smithi192_88961_object71675:head# 2024-06-27T21:23:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdaa5256:::benchmark_data_smithi192_88961_object50917:head# 2024-06-27T21:23:15.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdaece18:::benchmark_data_smithi192_88961_object69479:head# 2024-06-27T21:23:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdaefaea:::benchmark_data_smithi192_88961_object80249:head# 2024-06-27T21:23:15.780 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdafb3d9:::benchmark_data_smithi192_88961_object44019:head# 2024-06-27T21:23:15.781 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb08f02:::benchmark_data_smithi192_88961_object83636:head# 2024-06-27T21:23:15.781 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb09165:::benchmark_data_smithi192_88961_object49192:head# 2024-06-27T21:23:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb0d327:::benchmark_data_smithi192_88961_object57007:head# 2024-06-27T21:23:15.782 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb309ff:::benchmark_data_smithi192_88961_object44468:head# 2024-06-27T21:23:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb486ca:::benchmark_data_smithi192_88961_object55984:head# 2024-06-27T21:23:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb93197:::benchmark_data_smithi192_88961_object76416:head# 2024-06-27T21:23:15.783 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdb98c3c:::benchmark_data_smithi192_88961_object58453:head# 2024-06-27T21:23:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdbad689:::benchmark_data_smithi192_88961_object48908:head# 2024-06-27T21:23:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdbed4ed:::benchmark_data_smithi192_88961_object68234:head# 2024-06-27T21:23:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc0a329:::benchmark_data_smithi192_88961_object78386:head# 2024-06-27T21:23:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc0a9dd:::benchmark_data_smithi192_88961_object46049:head# 2024-06-27T21:23:15.786 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc21dfd:::benchmark_data_smithi192_88961_object54508:head# 2024-06-27T21:23:15.786 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc30b42:::benchmark_data_smithi192_88961_object89267:head# 2024-06-27T21:23:15.787 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc33fd4:::benchmark_data_smithi192_88961_object72511:head# 2024-06-27T21:23:15.788 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc38e01:::benchmark_data_smithi192_88961_object62700:head# 2024-06-27T21:23:15.788 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc6146b:::benchmark_data_smithi192_88961_object47995:head# 2024-06-27T21:23:15.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdc9d625:::benchmark_data_smithi192_88961_object58239:head# 2024-06-27T21:23:15.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdca3b79:::benchmark_data_smithi192_88961_object50814:head# 2024-06-27T21:23:15.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdcace5a:::benchmark_data_smithi192_88961_object53854:head# 2024-06-27T21:23:15.790 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdcbcf63:::benchmark_data_smithi192_88961_object78621:head# 2024-06-27T21:23:15.791 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdcc19c4:::benchmark_data_smithi192_88961_object65484:head# 2024-06-27T21:23:15.791 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdcc7d9e:::benchmark_data_smithi192_88961_object73412:head# 2024-06-27T21:23:15.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdccde68:::benchmark_data_smithi192_88961_object67389:head# 2024-06-27T21:23:15.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdcfdef3:::benchmark_data_smithi192_88961_object50484:head# 2024-06-27T21:23:15.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdcffd4b:::benchmark_data_smithi192_88961_object88529:head# 2024-06-27T21:23:15.793 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd0d744:::benchmark_data_smithi192_88961_object68179:head# 2024-06-27T21:23:15.793 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd18b04:::benchmark_data_smithi192_88961_object68365:head# 2024-06-27T21:23:15.794 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd48426:::benchmark_data_smithi192_88961_object87035:head# 2024-06-27T21:23:15.794 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd4a2c1:::benchmark_data_smithi192_88961_object44902:head# 2024-06-27T21:23:15.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd57c29:::benchmark_data_smithi192_88961_object84128:head# 2024-06-27T21:23:15.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd61768:::benchmark_data_smithi192_88961_object85943:head# 2024-06-27T21:23:15.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdd9ca02:::benchmark_data_smithi192_88961_object65649:head# 2024-06-27T21:23:15.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdda06ee:::benchmark_data_smithi192_88961_object71535:head# 2024-06-27T21:23:15.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bddd5bbc:::benchmark_data_smithi192_88961_object75536:head# 2024-06-27T21:23:15.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bddf5bd0:::benchmark_data_smithi192_88961_object47361:head# 2024-06-27T21:23:15.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde0b4b9:::benchmark_data_smithi192_88961_object53424:head# 2024-06-27T21:23:15.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde0dd47:::benchmark_data_smithi192_88961_object68344:head# 2024-06-27T21:23:15.798 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde1049f:::benchmark_data_smithi192_88961_object75135:head# 2024-06-27T21:23:15.798 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde18d2f:::benchmark_data_smithi192_88961_object77497:head# 2024-06-27T21:23:15.799 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde39884:::benchmark_data_smithi192_88961_object46326:head# 2024-06-27T21:23:15.799 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde516f8:::benchmark_data_smithi192_88961_object74220:head# 2024-06-27T21:23:15.800 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde5d913:::benchmark_data_smithi192_88961_object44280:head# 2024-06-27T21:23:15.800 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde6cc4e:::benchmark_data_smithi192_88961_object74993:head# 2024-06-27T21:23:15.801 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde70775:::benchmark_data_smithi192_88961_object68939:head# 2024-06-27T21:23:15.802 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bde7cb4f:::benchmark_data_smithi192_88961_object86433:head# 2024-06-27T21:23:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdeaa515:::benchmark_data_smithi192_88961_object77051:head# 2024-06-27T21:23:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdeaebb6:::benchmark_data_smithi192_88961_object76410:head# 2024-06-27T21:23:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdeba915:::benchmark_data_smithi192_88961_object64803:head# 2024-06-27T21:23:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdebcd67:::benchmark_data_smithi192_88961_object51711:head# 2024-06-27T21:23:15.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf19856:::benchmark_data_smithi192_88961_object85971:head# 2024-06-27T21:23:15.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf206db:::benchmark_data_smithi192_88961_object49480:head# 2024-06-27T21:23:15.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf2530b:::benchmark_data_smithi192_88961_object57386:head# 2024-06-27T21:23:15.805 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf2be96:::benchmark_data_smithi192_88961_object58877:head# 2024-06-27T21:23:15.805 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf77a4b:::benchmark_data_smithi192_88961_object62587:head# 2024-06-27T21:23:15.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf788a1:::benchmark_data_smithi192_88961_object74195:head# 2024-06-27T21:23:15.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf81a60:::benchmark_data_smithi192_88961_object87686:head# 2024-06-27T21:23:15.807 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdf91b8e:::benchmark_data_smithi192_88961_object72087:head# 2024-06-27T21:23:15.807 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdfa555a:::benchmark_data_smithi192_88961_object65366:head# 2024-06-27T21:23:15.807 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdfc0aea:::benchmark_data_smithi192_88961_object85685:head# 2024-06-27T21:23:15.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdfc97cb:::benchmark_data_smithi192_88961_object72097:head# 2024-06-27T21:23:15.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdfddaea:::benchmark_data_smithi192_88961_object47649:head# 2024-06-27T21:23:15.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bdfe3ba8:::benchmark_data_smithi192_88961_object50329:head# 2024-06-27T21:23:15.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be00382f:::benchmark_data_smithi192_88961_object78222:head# 2024-06-27T21:23:15.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be028ad3:::benchmark_data_smithi192_88961_object90605:head# 2024-06-27T21:23:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be054f0a:::benchmark_data_smithi192_88961_object52566:head# 2024-06-27T21:23:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be06109f:::benchmark_data_smithi192_88961_object53735:head# 2024-06-27T21:23:15.811 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be09108a:::benchmark_data_smithi192_88961_object77945:head# 2024-06-27T21:23:15.811 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be0a7c22:::benchmark_data_smithi192_88961_object89257:head# 2024-06-27T21:23:15.811 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be0bdf8b:::benchmark_data_smithi192_88961_object64125:head# 2024-06-27T21:23:15.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be0ded6d:::benchmark_data_smithi192_88961_object54660:head# 2024-06-27T21:23:15.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be0f0ab8:::benchmark_data_smithi192_88961_object76265:head# 2024-06-27T21:23:15.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be0f7788:::benchmark_data_smithi192_88961_object83100:head# 2024-06-27T21:23:15.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be125485:::benchmark_data_smithi192_88961_object52397:head# 2024-06-27T21:23:15.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be13873a:::benchmark_data_smithi192_88961_object64213:head# 2024-06-27T21:23:15.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be1482cb:::benchmark_data_smithi192_88961_object80246:head# 2024-06-27T21:23:15.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be151c33:::benchmark_data_smithi192_88961_object66836:head# 2024-06-27T21:23:15.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be15fc6e:::benchmark_data_smithi192_88961_object64476:head# 2024-06-27T21:23:15.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be163aa3:::benchmark_data_smithi192_88961_object88484:head# 2024-06-27T21:23:15.816 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be19f6d6:::benchmark_data_smithi192_88961_object88897:head# 2024-06-27T21:23:15.816 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be1a88fb:::benchmark_data_smithi192_88961_object62707:head# 2024-06-27T21:23:15.817 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be1ba8a2:::benchmark_data_smithi192_88961_object57675:head# 2024-06-27T21:23:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be1ca07b:::benchmark_data_smithi192_88961_object55838:head# 2024-06-27T21:23:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be1dcd8a:::benchmark_data_smithi192_88961_object49481:head# 2024-06-27T21:23:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be1faeb0:::benchmark_data_smithi192_88961_object81453:head# 2024-06-27T21:23:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be207be9:::benchmark_data_smithi192_88961_object60368:head# 2024-06-27T21:23:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be22fb57:::benchmark_data_smithi192_88961_object73928:head# 2024-06-27T21:23:15.821 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be23eec0:::benchmark_data_smithi192_88961_object63804:head# 2024-06-27T21:23:15.821 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be24536d:::benchmark_data_smithi192_88961_object54415:head# 2024-06-27T21:23:15.821 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be246e5a:::benchmark_data_smithi192_88961_object60840:head# 2024-06-27T21:23:15.822 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be25f690:::benchmark_data_smithi192_88961_object58005:head# 2024-06-27T21:23:15.822 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be281769:::benchmark_data_smithi192_88961_object54835:head# 2024-06-27T21:23:15.822 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be2835c6:::benchmark_data_smithi192_88961_object67360:head# 2024-06-27T21:23:15.823 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be2b34f3:::benchmark_data_smithi192_88961_object88058:head# 2024-06-27T21:23:15.823 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be2f49f7:::benchmark_data_smithi192_88961_object45430:head# 2024-06-27T21:23:15.824 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be2f9a89:::benchmark_data_smithi192_88961_object49026:head# 2024-06-27T21:23:15.824 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be31be4d:::benchmark_data_smithi192_88961_object84445:head# 2024-06-27T21:23:15.825 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be3489ea:::benchmark_data_smithi192_88961_object61827:head# 2024-06-27T21:23:15.825 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be389784:::benchmark_data_smithi192_88961_object50632:head# 2024-06-27T21:23:15.825 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be38b41d:::benchmark_data_smithi192_88961_object58185:head# 2024-06-27T21:23:15.826 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be3c427b:::benchmark_data_smithi192_88961_object52488:head# 2024-06-27T21:23:15.826 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be3f923f:::benchmark_data_smithi192_88961_object88595:head# 2024-06-27T21:23:15.827 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be42d991:::benchmark_data_smithi192_88961_object53362:head# 2024-06-27T21:23:15.827 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be44e7da:::benchmark_data_smithi192_88961_object43533:head# 2024-06-27T21:23:15.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be4575fc:::benchmark_data_smithi192_88961_object52127:head# 2024-06-27T21:23:15.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be45edc1:::benchmark_data_smithi192_88961_object61317:head# 2024-06-27T21:23:15.828 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be46de66:::benchmark_data_smithi192_88961_object64261:head# 2024-06-27T21:23:15.829 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be49782e:::benchmark_data_smithi192_88961_object79943:head# 2024-06-27T21:23:15.829 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be49afd4:::benchmark_data_smithi192_88961_object49848:head# 2024-06-27T21:23:15.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be4d6dbc:::benchmark_data_smithi192_88961_object44812:head# 2024-06-27T21:23:15.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be4d9a82:::benchmark_data_smithi192_88961_object90491:head# 2024-06-27T21:23:15.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be50fdb0:::benchmark_data_smithi192_88961_object62754:head# 2024-06-27T21:23:15.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be5215c8:::benchmark_data_smithi192_88961_object59908:head# 2024-06-27T21:23:15.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be527016:::benchmark_data_smithi192_88961_object49264:head# 2024-06-27T21:23:15.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be531059:::benchmark_data_smithi192_88961_object55545:head# 2024-06-27T21:23:15.832 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be54b157:::benchmark_data_smithi192_88961_object43467:head# 2024-06-27T21:23:15.832 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be54ffa3:::benchmark_data_smithi192_88961_object49830:head# 2024-06-27T21:23:15.833 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be552f47:::benchmark_data_smithi192_88961_object58577:head# 2024-06-27T21:23:15.833 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be5553e4:::benchmark_data_smithi192_88961_object45785:head# 2024-06-27T21:23:15.834 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be5572c6:::benchmark_data_smithi192_88961_object74323:head# 2024-06-27T21:23:15.834 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be5a50bf:::benchmark_data_smithi192_88961_object70878:head# 2024-06-27T21:23:15.834 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be5a795d:::benchmark_data_smithi192_88961_object57928:head# 2024-06-27T21:23:15.835 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be5eadee:::benchmark_data_smithi192_88961_object80812:head# 2024-06-27T21:23:15.835 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be602c5b:::benchmark_data_smithi192_88961_object64008:head# 2024-06-27T21:23:15.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be627814:::benchmark_data_smithi192_88961_object56663:head# 2024-06-27T21:23:15.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be657f77:::benchmark_data_smithi192_88961_object66418:head# 2024-06-27T21:23:15.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be66567b:::benchmark_data_smithi192_88961_object88784:head# 2024-06-27T21:23:15.837 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be667344:::benchmark_data_smithi192_88961_object80279:head# 2024-06-27T21:23:15.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be6765da:::benchmark_data_smithi192_88961_object78961:head# 2024-06-27T21:23:15.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be681406:::benchmark_data_smithi192_88961_object85367:head# 2024-06-27T21:23:15.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be6c68ac:::benchmark_data_smithi192_88961_object63605:head# 2024-06-27T21:23:15.839 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be6e7c89:::benchmark_data_smithi192_88961_object86538:head# 2024-06-27T21:23:15.839 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be6f1b50:::benchmark_data_smithi192_88961_object61254:head# 2024-06-27T21:23:15.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be7189dc:::benchmark_data_smithi192_88961_object47120:head# 2024-06-27T21:23:15.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be73efba:::benchmark_data_smithi192_88961_object78588:head# 2024-06-27T21:23:15.841 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be744975:::benchmark_data_smithi192_88961_object72461:head# 2024-06-27T21:23:15.841 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be75576c:::benchmark_data_smithi192_88961_object65470:head# 2024-06-27T21:23:15.842 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be7611b9:::benchmark_data_smithi192_88961_object79068:head# 2024-06-27T21:23:15.842 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be79c277:::benchmark_data_smithi192_88961_object77356:head# 2024-06-27T21:23:15.843 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be7b6808:::benchmark_data_smithi192_88961_object73548:head# 2024-06-27T21:23:15.843 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be7dbbda:::benchmark_data_smithi192_88961_object73757:head# 2024-06-27T21:23:15.843 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be84b29f:::benchmark_data_smithi192_88961_object49735:head# 2024-06-27T21:23:15.844 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be85d775:::benchmark_data_smithi192_88961_object67758:head# 2024-06-27T21:23:15.844 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be876d70:::benchmark_data_smithi192_88961_object87924:head# 2024-06-27T21:23:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be889d70:::benchmark_data_smithi192_88961_object80394:head# 2024-06-27T21:23:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be88e3ff:::benchmark_data_smithi192_88961_object46023:head# 2024-06-27T21:23:15.846 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be8a41c2:::benchmark_data_smithi192_88961_object69843:head# 2024-06-27T21:23:15.846 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be8a4504:::benchmark_data_smithi192_88961_object62806:head# 2024-06-27T21:23:15.847 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be8e13b6:::benchmark_data_smithi192_88961_object64873:head# 2024-06-27T21:23:15.847 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be8f0da9:::benchmark_data_smithi192_88961_object53897:head# 2024-06-27T21:23:15.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be9290e7:::benchmark_data_smithi192_88961_object85589:head# 2024-06-27T21:23:15.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be9668a4:::benchmark_data_smithi192_88961_object69348:head# 2024-06-27T21:23:15.849 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be974a77:::benchmark_data_smithi192_88961_object80810:head# 2024-06-27T21:23:15.849 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be9b76cd:::benchmark_data_smithi192_88961_object68095:head# 2024-06-27T21:23:15.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:be9be462:::benchmark_data_smithi192_88961_object78453:head# 2024-06-27T21:23:15.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bea2ff65:::benchmark_data_smithi192_88961_object48430:head# 2024-06-27T21:23:15.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bea31183:::benchmark_data_smithi192_88961_object81800:head# 2024-06-27T21:23:15.851 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bea397cd:::benchmark_data_smithi192_88961_object72324:head# 2024-06-27T21:23:15.851 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bea6dc53:::benchmark_data_smithi192_88961_object67429:head# 2024-06-27T21:23:15.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:15.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beada10d:::benchmark_data_smithi192_88961_object67773:head# 2024-06-27T21:23:15.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beaf5005:::benchmark_data_smithi192_88961_object66586:head# 2024-06-27T21:23:15.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beaf7837:::benchmark_data_smithi192_88961_object79268:head# 2024-06-27T21:23:15.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beb03f15:::benchmark_data_smithi192_88961_object69384:head# 2024-06-27T21:23:15.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beb237d8:::benchmark_data_smithi192_88961_object43282:head# 2024-06-27T21:23:15.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beb318c0:::benchmark_data_smithi192_88961_object44822:head# 2024-06-27T21:23:15.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beb5595c:::benchmark_data_smithi192_88961_object55100:head# 2024-06-27T21:23:15.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beb9051b:::benchmark_data_smithi192_88961_object64799:head# 2024-06-27T21:23:15.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beb99d58:::benchmark_data_smithi192_88961_object62782:head# 2024-06-27T21:23:15.856 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bebc4e28:::benchmark_data_smithi192_88961_object53745:head# 2024-06-27T21:23:15.856 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bebe9b12:::benchmark_data_smithi192_88961_object75917:head# 2024-06-27T21:23:15.857 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bebefa0d:::benchmark_data_smithi192_88961_object89147:head# 2024-06-27T21:23:15.857 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bec0600e:::benchmark_data_smithi192_88961_object76211:head# 2024-06-27T21:23:15.858 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bec5ea64:::benchmark_data_smithi192_88961_object62546:head# 2024-06-27T21:23:15.858 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bec73978:::benchmark_data_smithi192_88961_object86197:head# 2024-06-27T21:23:15.858 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bec9d4f2:::benchmark_data_smithi192_88961_object63504:head# 2024-06-27T21:23:15.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beca0dea:::benchmark_data_smithi192_88961_object44936:head# 2024-06-27T21:23:15.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:becab14a:::benchmark_data_smithi192_88961_object59139:head# 2024-06-27T21:23:15.860 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:becacf3a:::benchmark_data_smithi192_88961_object58069:head# 2024-06-27T21:23:15.860 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:becb403f:::benchmark_data_smithi192_88961_object59998:head# 2024-06-27T21:23:15.861 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:becc290c:::benchmark_data_smithi192_88961_object87953:head# 2024-06-27T21:23:15.861 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:becdaacb:::benchmark_data_smithi192_88961_object71854:head# 2024-06-27T21:23:15.862 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:becffd12:::benchmark_data_smithi192_88961_object81561:head# 2024-06-27T21:23:15.862 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed01346:::benchmark_data_smithi192_88961_object71274:head# 2024-06-27T21:23:15.862 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed385e6:::benchmark_data_smithi192_88961_object52627:head# 2024-06-27T21:23:15.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed386f6:::benchmark_data_smithi192_88961_object84357:head# 2024-06-27T21:23:15.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed5d606:::benchmark_data_smithi192_88961_object81769:head# 2024-06-27T21:23:15.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed5edcf:::benchmark_data_smithi192_88961_object54755:head# 2024-06-27T21:23:15.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed690be:::benchmark_data_smithi192_88961_object72894:head# 2024-06-27T21:23:15.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bed87547:::benchmark_data_smithi192_88961_object47480:head# 2024-06-27T21:23:15.865 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bedb5f2d:::benchmark_data_smithi192_88961_object65807:head# 2024-06-27T21:23:15.865 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bedb6a3a:::benchmark_data_smithi192_88961_object87998:head# 2024-06-27T21:23:15.865 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bedd9225:::benchmark_data_smithi192_88961_object78372:head# 2024-06-27T21:23:15.866 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bedf30af:::benchmark_data_smithi192_88961_object62390:head# 2024-06-27T21:23:15.866 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee09a31:::benchmark_data_smithi192_88961_object66446:head# 2024-06-27T21:23:15.867 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee237b8:::benchmark_data_smithi192_88961_object78567:head# 2024-06-27T21:23:15.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee35840:::benchmark_data_smithi192_88961_object86343:head# 2024-06-27T21:23:15.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee3d44a:::benchmark_data_smithi192_88961_object71885:head# 2024-06-27T21:23:15.868 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee6a360:::benchmark_data_smithi192_88961_object79743:head# 2024-06-27T21:23:15.869 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee6b25a:::benchmark_data_smithi192_88961_object86328:head# 2024-06-27T21:23:15.869 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bee9c07c:::benchmark_data_smithi192_88961_object78817:head# 2024-06-27T21:23:15.869 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beec54bc:::benchmark_data_smithi192_88961_object44124:head# 2024-06-27T21:23:15.870 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beece4f3:::benchmark_data_smithi192_88961_object77068:head# 2024-06-27T21:23:15.870 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beed48b2:::benchmark_data_smithi192_88961_object52668:head# 2024-06-27T21:23:15.871 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bef21fea:::benchmark_data_smithi192_88961_object70253:head# 2024-06-27T21:23:15.871 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bef28c5f:::benchmark_data_smithi192_88961_object51973:head# 2024-06-27T21:23:15.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bef3f783:::benchmark_data_smithi192_88961_object50562:head# 2024-06-27T21:23:15.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bef6660f:::benchmark_data_smithi192_88961_object69589:head# 2024-06-27T21:23:15.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bef93df9:::benchmark_data_smithi192_88961_object69666:head# 2024-06-27T21:23:15.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:befe664a:::benchmark_data_smithi192_88961_object63613:head# 2024-06-27T21:23:15.873 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:beffc94a:::benchmark_data_smithi192_88961_object89502:head# 2024-06-27T21:23:15.873 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf001a0a:::benchmark_data_smithi192_88961_object45763:head# 2024-06-27T21:23:15.873 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf0075b4:::benchmark_data_smithi192_88961_object66479:head# 2024-06-27T21:23:15.874 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf021927:::benchmark_data_smithi192_88961_object59057:head# 2024-06-27T21:23:15.874 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf0365e2:::benchmark_data_smithi192_88961_object64520:head# 2024-06-27T21:23:15.875 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf0688ef:::benchmark_data_smithi192_88961_object55897:head# 2024-06-27T21:23:15.875 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf07b046:::benchmark_data_smithi192_88961_object84045:head# 2024-06-27T21:23:15.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:15.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf09746b:::benchmark_data_smithi192_88961_object87590:head# 2024-06-27T21:23:15.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf09b21c:::benchmark_data_smithi192_88961_object83234:head# 2024-06-27T21:23:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:15.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:15.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf0a89f9:::benchmark_data_smithi192_88961_object71370:head# 2024-06-27T21:23:15.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf0b901a:::benchmark_data_smithi192_88961_object54706:head# 2024-06-27T21:23:15.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf0c8d97:::benchmark_data_smithi192_88961_object58487:head# 2024-06-27T21:23:15.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf1128e3:::benchmark_data_smithi192_88961_object48112:head# 2024-06-27T21:23:15.878 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf14dc20:::benchmark_data_smithi192_88961_object78711:head# 2024-06-27T21:23:15.879 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf163f1d:::benchmark_data_smithi192_88961_object84491:head# 2024-06-27T21:23:15.879 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf17b2c0:::benchmark_data_smithi192_88961_object67530:head# 2024-06-27T21:23:15.879 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf18a7a7:::benchmark_data_smithi192_88961_object90001:head# 2024-06-27T21:23:15.880 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf1939a3:::benchmark_data_smithi192_88961_object67362:head# 2024-06-27T21:23:15.880 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf193e4b:::benchmark_data_smithi192_88961_object66795:head# 2024-06-27T21:23:15.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf198fff:::benchmark_data_smithi192_88961_object71614:head# 2024-06-27T21:23:15.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf1babcc:::benchmark_data_smithi192_88961_object82435:head# 2024-06-27T21:23:15.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf1bfaec:::benchmark_data_smithi192_88961_object83794:head# 2024-06-27T21:23:15.882 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf1db177:::benchmark_data_smithi192_88961_object75202:head# 2024-06-27T21:23:15.883 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2267ba:::benchmark_data_smithi192_88961_object83646:head# 2024-06-27T21:23:15.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2342ce:::benchmark_data_smithi192_88961_object79461:head# 2024-06-27T21:23:15.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf282fc3:::benchmark_data_smithi192_88961_object68721:head# 2024-06-27T21:23:15.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf283122:::benchmark_data_smithi192_88961_object63174:head# 2024-06-27T21:23:15.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf294222:::benchmark_data_smithi192_88961_object59964:head# 2024-06-27T21:23:15.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2bcd72:::benchmark_data_smithi192_88961_object48508:head# 2024-06-27T21:23:15.885 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2c7d6b:::benchmark_data_smithi192_88961_object50249:head# 2024-06-27T21:23:15.885 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2e0be3:::benchmark_data_smithi192_88961_object51285:head# 2024-06-27T21:23:15.885 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2efdf7:::benchmark_data_smithi192_88961_object89433:head# 2024-06-27T21:23:15.886 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf2fc6fe:::benchmark_data_smithi192_88961_object78227:head# 2024-06-27T21:23:15.886 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf30a18a:::benchmark_data_smithi192_88961_object45651:head# 2024-06-27T21:23:15.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf30fae8:::benchmark_data_smithi192_88961_object51629:head# 2024-06-27T21:23:15.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf32061c:::benchmark_data_smithi192_88961_object48738:head# 2024-06-27T21:23:15.888 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf32ab96:::benchmark_data_smithi192_88961_object78677:head# 2024-06-27T21:23:15.888 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf32bed9:::benchmark_data_smithi192_88961_object63596:head# 2024-06-27T21:23:15.889 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf32bf04:::benchmark_data_smithi192_88961_object83679:head# 2024-06-27T21:23:15.889 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf33c0b3:::benchmark_data_smithi192_88961_object87104:head# 2024-06-27T21:23:15.889 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3409da:::benchmark_data_smithi192_88961_object71982:head# 2024-06-27T21:23:15.890 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf34a4b4:::benchmark_data_smithi192_88961_object82296:head# 2024-06-27T21:23:15.890 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf351a07:::benchmark_data_smithi192_88961_object75088:head# 2024-06-27T21:23:15.890 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf354a04:::benchmark_data_smithi192_88961_object61627:head# 2024-06-27T21:23:15.891 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf35af65:::benchmark_data_smithi192_88961_object47073:head# 2024-06-27T21:23:15.891 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf35d849:::benchmark_data_smithi192_88961_object50192:head# 2024-06-27T21:23:15.891 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf36c574:::benchmark_data_smithi192_88961_object76022:head# 2024-06-27T21:23:15.892 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf37af81:::benchmark_data_smithi192_88961_object81516:head# 2024-06-27T21:23:15.892 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf39c626:::benchmark_data_smithi192_88961_object73755:head# 2024-06-27T21:23:15.893 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3a11ce:::benchmark_data_smithi192_88961_object67137:head# 2024-06-27T21:23:15.893 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3a207b:::benchmark_data_smithi192_88961_object86424:head# 2024-06-27T21:23:15.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3ad8cb:::benchmark_data_smithi192_88961_object71657:head# 2024-06-27T21:23:15.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3afa4e:::benchmark_data_smithi192_88961_object46363:head# 2024-06-27T21:23:15.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3cc4f8:::benchmark_data_smithi192_88961_object59410:head# 2024-06-27T21:23:15.894 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf3f007c:::benchmark_data_smithi192_88961_object47081:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf4132d7:::benchmark_data_smithi192_88961_object55685:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf42cf7b:::benchmark_data_smithi192_88961_object60466:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf43bc73:::benchmark_data_smithi192_88961_object89843:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf449ca1:::benchmark_data_smithi192_88961_object61659:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf455e3e:::benchmark_data_smithi192_88961_object50882:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf473d04:::benchmark_data_smithi192_88961_object76598:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf47bed4:::benchmark_data_smithi192_88961_object50315:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf492159:::benchmark_data_smithi192_88961_object66327:head# 2024-06-27T21:23:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf4b0943:::benchmark_data_smithi192_88961_object89256:head# 2024-06-27T21:23:15.900 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf4ed845:::benchmark_data_smithi192_88961_object53123:head# 2024-06-27T21:23:15.900 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf500259:::benchmark_data_smithi192_88961_object74652:head# 2024-06-27T21:23:15.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf520e67:::benchmark_data_smithi192_88961_object75978:head# 2024-06-27T21:23:15.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf52d0f4:::benchmark_data_smithi192_88961_object43283:head# 2024-06-27T21:23:15.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf556e98:::benchmark_data_smithi192_88961_object78971:head# 2024-06-27T21:23:15.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf599fe8:::benchmark_data_smithi192_88961_object48226:head# 2024-06-27T21:23:15.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf5b6b6d:::benchmark_data_smithi192_88961_object70048:head# 2024-06-27T21:23:15.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf5d49a9:::benchmark_data_smithi192_88961_object55626:head# 2024-06-27T21:23:15.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf5d933b:::benchmark_data_smithi192_88961_object45367:head# 2024-06-27T21:23:15.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf5e0fd5:::benchmark_data_smithi192_88961_object67902:head# 2024-06-27T21:23:15.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf5e62d8:::benchmark_data_smithi192_88961_object63925:head# 2024-06-27T21:23:15.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf5f4595:::benchmark_data_smithi192_88961_object80369:head# 2024-06-27T21:23:15.903 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf600fac:::benchmark_data_smithi192_88961_object42992:head# 2024-06-27T21:23:15.903 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf61b78d:::benchmark_data_smithi192_88961_object78142:head# 2024-06-27T21:23:15.904 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf629f9a:::benchmark_data_smithi192_88961_object48570:head# 2024-06-27T21:23:15.904 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf64ca28:::benchmark_data_smithi192_88961_object45944:head# 2024-06-27T21:23:15.904 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf663fb2:::benchmark_data_smithi192_88961_object76061:head# 2024-06-27T21:23:15.905 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf66541d:::benchmark_data_smithi192_88961_object64684:head# 2024-06-27T21:23:15.905 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf66f95b:::benchmark_data_smithi192_88961_object44550:head# 2024-06-27T21:23:15.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf6763f4:::benchmark_data_smithi192_88961_object68762:head# 2024-06-27T21:23:15.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf67c319:::benchmark_data_smithi192_88961_object63473:head# 2024-06-27T21:23:15.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf67cdca:::benchmark_data_smithi192_88961_object45464:head# 2024-06-27T21:23:15.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf6917a3:::benchmark_data_smithi192_88961_object66905:head# 2024-06-27T21:23:15.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf6c4f7c:::benchmark_data_smithi192_88961_object88865:head# 2024-06-27T21:23:15.908 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf70fb8d:::benchmark_data_smithi192_88961_object79252:head# 2024-06-27T21:23:15.908 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf72816d:::benchmark_data_smithi192_88961_object76625:head# 2024-06-27T21:23:15.911 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7282fb:::benchmark_data_smithi192_88961_object45550:head# 2024-06-27T21:23:15.912 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7397df:::benchmark_data_smithi192_88961_object75981:head# 2024-06-27T21:23:15.912 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf764343:::benchmark_data_smithi192_88961_object56692:head# 2024-06-27T21:23:15.912 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf770b16:::benchmark_data_smithi192_88961_object73101:head# 2024-06-27T21:23:15.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7803d7:::benchmark_data_smithi192_88961_object72439:head# 2024-06-27T21:23:15.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf78d7c0:::benchmark_data_smithi192_88961_object73854:head# 2024-06-27T21:23:15.914 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf78f1f1:::benchmark_data_smithi192_88961_object82914:head# 2024-06-27T21:23:15.919 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf79687b:::benchmark_data_smithi192_88961_object63784:head# 2024-06-27T21:23:15.919 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7bce01:::benchmark_data_smithi192_88961_object49825:head# 2024-06-27T21:23:15.919 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7d7600:::benchmark_data_smithi192_88961_object54128:head# 2024-06-27T21:23:15.919 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7dd979:::benchmark_data_smithi192_88961_object50026:head# 2024-06-27T21:23:15.919 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7ed3a7:::benchmark_data_smithi192_88961_object51548:head# 2024-06-27T21:23:15.920 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7ed76b:::benchmark_data_smithi192_88961_object88204:head# 2024-06-27T21:23:15.921 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7f0787:::benchmark_data_smithi192_88961_object73694:head# 2024-06-27T21:23:15.922 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf7f63f3:::benchmark_data_smithi192_88961_object76909:head# 2024-06-27T21:23:15.923 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf8195a2:::benchmark_data_smithi192_88961_object83671:head# 2024-06-27T21:23:15.925 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf83b46e:::benchmark_data_smithi192_88961_object76255:head# 2024-06-27T21:23:15.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf848d5a:::benchmark_data_smithi192_88961_object55888:head# 2024-06-27T21:23:15.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf84db13:::benchmark_data_smithi192_88961_object49800:head# 2024-06-27T21:23:15.927 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf868f37:::benchmark_data_smithi192_88961_object58157:head# 2024-06-27T21:23:15.927 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf897c5b:::benchmark_data_smithi192_88961_object72676:head# 2024-06-27T21:23:15.927 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf8c0523:::benchmark_data_smithi192_88961_object75349:head# 2024-06-27T21:23:15.928 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf91152f:::benchmark_data_smithi192_88961_object83667:head# 2024-06-27T21:23:15.928 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf91d9b5:::benchmark_data_smithi192_88961_object70399:head# 2024-06-27T21:23:15.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf92560d:::benchmark_data_smithi192_88961_object61077:head# 2024-06-27T21:23:15.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf978bfa:::benchmark_data_smithi192_88961_object51072:head# 2024-06-27T21:23:15.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf97bd56:::benchmark_data_smithi192_88961_object52493:head# 2024-06-27T21:23:15.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf98f704:::benchmark_data_smithi192_88961_object73118:head# 2024-06-27T21:23:15.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf98f716:::benchmark_data_smithi192_88961_object87526:head# 2024-06-27T21:23:15.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9992ee:::benchmark_data_smithi192_88961_object77492:head# 2024-06-27T21:23:15.932 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9a3c19:::benchmark_data_smithi192_88961_object65018:head# 2024-06-27T21:23:15.932 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9a5215:::benchmark_data_smithi192_88961_object78841:head# 2024-06-27T21:23:15.933 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9ce4a1:::benchmark_data_smithi192_88961_object68599:head# 2024-06-27T21:23:15.933 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9e1cca:::benchmark_data_smithi192_88961_object73912:head# 2024-06-27T21:23:15.934 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9e4ecf:::benchmark_data_smithi192_88961_object77966:head# 2024-06-27T21:23:15.934 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bf9e8a66:::benchmark_data_smithi192_88961_object53161:head# 2024-06-27T21:23:15.935 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa0e508:::benchmark_data_smithi192_88961_object79486:head# 2024-06-27T21:23:15.935 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa10314:::benchmark_data_smithi192_88961_object61301:head# 2024-06-27T21:23:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa1efb5:::benchmark_data_smithi192_88961_object52099:head# 2024-06-27T21:23:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa3301f:::benchmark_data_smithi192_88961_object84775:head# 2024-06-27T21:23:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa44896:::benchmark_data_smithi192_88961_object61667:head# 2024-06-27T21:23:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa5a531:::benchmark_data_smithi192_88961_object87224:head# 2024-06-27T21:23:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa6d2a7:::benchmark_data_smithi192_88961_object84716:head# 2024-06-27T21:23:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa74900:::benchmark_data_smithi192_88961_object72784:head# 2024-06-27T21:23:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa7cd1c:::benchmark_data_smithi192_88961_object62929:head# 2024-06-27T21:23:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfa91778:::benchmark_data_smithi192_88961_object62379:head# 2024-06-27T21:23:15.939 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfad2f6c:::benchmark_data_smithi192_88961_object44369:head# 2024-06-27T21:23:15.939 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfafe3c9:::benchmark_data_smithi192_88961_object52947:head# 2024-06-27T21:23:15.939 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb1c238:::benchmark_data_smithi192_88961_object83285:head# 2024-06-27T21:23:15.940 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb1d422:::benchmark_data_smithi192_88961_object81286:head# 2024-06-27T21:23:15.940 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb3c5dc:::benchmark_data_smithi192_88961_object43869:head# 2024-06-27T21:23:15.941 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb44561:::benchmark_data_smithi192_88961_object85373:head# 2024-06-27T21:23:15.941 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb5c328:::benchmark_data_smithi192_88961_object89076:head# 2024-06-27T21:23:15.941 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb79fef:::benchmark_data_smithi192_88961_object45102:head# 2024-06-27T21:23:15.942 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb850a5:::benchmark_data_smithi192_88961_object48970:head# 2024-06-27T21:23:15.942 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfb85676:::benchmark_data_smithi192_88961_object60334:head# 2024-06-27T21:23:15.942 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfba7bb5:::benchmark_data_smithi192_88961_object60004:head# 2024-06-27T21:23:15.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfbba593:::benchmark_data_smithi192_88961_object49598:head# 2024-06-27T21:23:15.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfbdb5ad:::benchmark_data_smithi192_88961_object84461:head# 2024-06-27T21:23:15.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfbe4107:::benchmark_data_smithi192_88961_object51688:head# 2024-06-27T21:23:15.944 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfbe7f12:::benchmark_data_smithi192_88961_object48801:head# 2024-06-27T21:23:15.944 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfc379bd:::benchmark_data_smithi192_88961_object77915:head# 2024-06-27T21:23:15.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfc54d6f:::benchmark_data_smithi192_88961_object61548:head# 2024-06-27T21:23:15.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfc632cc:::benchmark_data_smithi192_88961_object77198:head# 2024-06-27T21:23:15.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfc6b790:::benchmark_data_smithi192_88961_object66408:head# 2024-06-27T21:23:15.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:15 smithi151 ceph-mon[28965]: pgmap v925: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 10490/126051 objects degraded (8.322%); 7408/126051 objects misplaced (5.877%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:23:15.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:15 smithi151 ceph-mon[28965]: osdmap e352: 12 total, 11 up, 12 in 2024-06-27T21:23:15.946 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfc706bd:::benchmark_data_smithi192_88961_object70948:head# 2024-06-27T21:23:15.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfcc629b:::benchmark_data_smithi192_88961_object46738:head# 2024-06-27T21:23:15.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfcdc58d:::benchmark_data_smithi192_88961_object87192:head# 2024-06-27T21:23:15.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfcf436f:::benchmark_data_smithi192_88961_object46884:head# 2024-06-27T21:23:15.947 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd03084:::benchmark_data_smithi192_88961_object85768:head# 2024-06-27T21:23:15.948 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd0c1bc:::benchmark_data_smithi192_88961_object57688:head# 2024-06-27T21:23:15.948 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd0d07d:::benchmark_data_smithi192_88961_object65140:head# 2024-06-27T21:23:15.948 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd3bb69:::benchmark_data_smithi192_88961_object58759:head# 2024-06-27T21:23:15.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd5048b:::benchmark_data_smithi192_88961_object67059:head# 2024-06-27T21:23:15.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd522e3:::benchmark_data_smithi192_88961_object77942:head# 2024-06-27T21:23:15.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd53ec3:::benchmark_data_smithi192_88961_object79648:head# 2024-06-27T21:23:15.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd6f3d1:::benchmark_data_smithi192_88961_object62843:head# 2024-06-27T21:23:15.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd74fb9:::benchmark_data_smithi192_88961_object86087:head# 2024-06-27T21:23:15.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd7d198:::benchmark_data_smithi192_88961_object66176:head# 2024-06-27T21:23:15.951 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfd7ea39:::benchmark_data_smithi192_88961_object53165:head# 2024-06-27T21:23:15.951 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfddfc37:::benchmark_data_smithi192_88961_object86429:head# 2024-06-27T21:23:15.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfde4d62:::benchmark_data_smithi192_88961_object68146:head# 2024-06-27T21:23:15.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfdecec6:::benchmark_data_smithi192_88961_object71259:head# 2024-06-27T21:23:15.952 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfdf7049:::benchmark_data_smithi192_88961_object75302:head# 2024-06-27T21:23:15.953 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfe063fe:::benchmark_data_smithi192_88961_object55156:head# 2024-06-27T21:23:15.953 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfe1a62b:::benchmark_data_smithi192_88961_object79020:head# 2024-06-27T21:23:15.954 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfe2b9f7:::benchmark_data_smithi192_88961_object72341:head# 2024-06-27T21:23:15.954 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfe4abec:::benchmark_data_smithi192_88961_object51002:head# 2024-06-27T21:23:15.954 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfe61ee2:::benchmark_data_smithi192_88961_object61560:head# 2024-06-27T21:23:15.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfeb3752:::benchmark_data_smithi192_88961_object86848:head# 2024-06-27T21:23:15.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfeb3bec:::benchmark_data_smithi192_88961_object57230:head# 2024-06-27T21:23:15.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfeb682d:::benchmark_data_smithi192_88961_object49813:head# 2024-06-27T21:23:15.956 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfee6e86:::benchmark_data_smithi192_88961_object68201:head# 2024-06-27T21:23:15.956 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfef11f1:::benchmark_data_smithi192_88961_object63465:head# 2024-06-27T21:23:15.957 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff0142c:::benchmark_data_smithi192_88961_object78544:head# 2024-06-27T21:23:15.957 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff13f93:::benchmark_data_smithi192_88961_object50134:head# 2024-06-27T21:23:15.958 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff3bb72:::benchmark_data_smithi192_88961_object60573:head# 2024-06-27T21:23:15.958 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff5641b:::benchmark_data_smithi192_88961_object49049:head# 2024-06-27T21:23:15.958 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff58966:::benchmark_data_smithi192_88961_object89151:head# 2024-06-27T21:23:15.959 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff74081:::benchmark_data_smithi192_88961_object69754:head# 2024-06-27T21:23:15.959 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bff782bb:::benchmark_data_smithi192_88961_object59359:head# 2024-06-27T21:23:15.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bffa3c29:::benchmark_data_smithi192_88961_object76713:head# 2024-06-27T21:23:15.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:bfff9254:::benchmark_data_smithi192_88961_object87786:head# 2024-06-27T21:23:15.963 INFO:teuthology.orchestra.run.smithi190.stderr:Export successful 2024-06-27T21:23:15.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:16.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:15 smithi192 ceph-mon[32956]: pgmap v925: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 10490/126051 objects degraded (8.322%); 7408/126051 objects misplaced (5.877%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:23:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:15 smithi192 ceph-mon[32956]: osdmap e352: 12 total, 11 up, 12 in 2024-06-27T21:23:16.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:16.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:16.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:16.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:16.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:16.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:16.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:16.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:16.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:16.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:16.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:16.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:16.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:16.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:16.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:16.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -n osd.4 --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 4.d 2024-06-27T21:23:17.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:17.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:17.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:17.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:17.208 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/osd.4/config 2024-06-27T21:23:17.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:17.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:17.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:17.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:17.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:17.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:17.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:17.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:17.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:17.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:17.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:17.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:17.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:17.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:17 smithi151 ceph-mon[28965]: pgmap v927: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 9824/118803 objects degraded (8.269%); 7408/118803 objects misplaced (6.236%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:23:18.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:17 smithi190 ceph-mon[32247]: pgmap v927: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 9824/118803 objects degraded (8.269%); 7408/118803 objects misplaced (6.236%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:23:18.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:18.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:17 smithi192 ceph-mon[32956]: pgmap v927: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s; 9824/118803 objects degraded (8.269%); 7408/118803 objects misplaced (6.236%); 3.7 MiB/s, 58 objects/s recovering 2024-06-27T21:23:18.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-27T21:23:18.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-27T21:23:18.139 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:18.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:18.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:18.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:18.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:18.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:18.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:18.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:18.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:18.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:18.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:144549 4.c 4:305ef1ac:::benchmark_data_smithi192_88961_object53810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:18.704898+0000", 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.036199476000000001, 2024-06-27T21:23:18.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.036232318999999999, 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 144549 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:23:18.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.704898+0000", 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.704898+0000", 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.704900+0000", 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2680000000000001e-06 2024-06-27T21:23:18.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.704909+0000", 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8310000000000007e-06 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.704941+0000", 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2172000000000003e-05 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:23:18.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.704951+0000", 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0404000000000001e-05 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.705152+0000", 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020039799999999999 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.705286+0000", 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013423600000000001 2024-06-27T21:23:18.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6", 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.705401+0000", 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115522 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.709866+0000", 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044642409999999999 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:144571 4.0 4:08526cae:::benchmark_data_smithi192_88961_object53832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:18.732911+0000", 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0081861820000000002, 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0082915019999999992, 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:23:18.747 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 144571 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.732911+0000", 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.732911+0000", 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.732913+0000", 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9010000000000001e-06 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.732929+0000", 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5991999999999999e-05 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.732947+0000", 2024-06-27T21:23:18.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8612000000000001e-05 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.732957+0000", 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3999999999999998e-06 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.733024+0000", 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6903999999999995e-05 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.733161+0000", 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013743699999999999 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.733275+0000", 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113399 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738204+0000", 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049290560000000002 2024-06-27T21:23:18.751 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:144579 4.0 4:026dde8b:::benchmark_data_smithi192_88961_object53840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:18.738260+0000", 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0028372929999999998, 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030049, 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:23:18.752 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 144579 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738260+0000", 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738260+0000", 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738262+0000", 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8190000000000001e-06 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738270+0000", 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9449999999999993e-06 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:23:18.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738280+0000", 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0030000000000001e-05 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738285+0000", 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.728e-06 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738333+0000", 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7185999999999999e-05 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738456+0000", 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012365 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738562+0000", 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105993 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:144577 4.9 e353/282 4:961b22a2:::benchmark_data_smithi192_88961_object53838:head v 353'8998, mlcod=353'8996)", 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:18.738405+0000", 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00269178, 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029167450000000001, 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738405+0000", 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738405+0000", 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738406+0000", 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6299999999999993e-07 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738414+0000", 2024-06-27T21:23:18.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3200000000000002e-06 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738446+0000", 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2103000000000001e-05 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738454+0000", 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9489999999999996e-06 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738506+0000", 2024-06-27T21:23:18.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2484000000000001e-05 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:18.738525+0000", 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9205999999999999e-05 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-06-27T21:23:18.760 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:23:18.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:18 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 9636/117675 objects degraded (8.189%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:18.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:18 smithi190 ceph-mon[32247]: osdmap e353: 12 total, 11 up, 12 in 2024-06-27T21:23:18.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:18.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:18.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:18.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:18 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 9636/117675 objects degraded (8.189%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:18.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:18 smithi151 ceph-mon[28965]: osdmap e353: 12 total, 11 up, 12 in 2024-06-27T21:23:18.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:18.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:19.028 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-27T21:23:19.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:18 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 9636/117675 objects degraded (8.189%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:19.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:18 smithi192 ceph-mon[32956]: osdmap e353: 12 total, 11 up, 12 in 2024-06-27T21:23:19.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:19.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:19.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:19.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:19.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:19.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:19.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:19.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:19.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:19.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:19.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:23:19.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:19.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:19.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:19.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:19.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:19 smithi151 ceph-mon[28965]: pgmap v928: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 9636/117675 objects degraded (8.189%); 7408/117675 objects misplaced (6.295%); 3.1 MiB/s, 50 objects/s recovering 2024-06-27T21:23:19.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:19.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:19.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:19.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:19.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:19 smithi190 ceph-mon[32247]: pgmap v928: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 9636/117675 objects degraded (8.189%); 7408/117675 objects misplaced (6.295%); 3.1 MiB/s, 50 objects/s recovering 2024-06-27T21:23:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:20.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:19 smithi192 ceph-mon[32956]: pgmap v928: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 9636/117675 objects degraded (8.189%); 7408/117675 objects misplaced (6.295%); 3.1 MiB/s, 50 objects/s recovering 2024-06-27T21:23:20.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:20.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:20.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:20.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:23:20.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:20.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:20.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:20.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:20.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:20.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:20.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:20.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:20.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:20.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:20.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:20.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:20.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:20.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:20.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:21.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:21.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:20 smithi190 ceph-mon[32247]: 1.6 deep-scrub starts 2024-06-27T21:23:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:20 smithi190 ceph-mon[32247]: 1.6 deep-scrub ok 2024-06-27T21:23:21.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:20 smithi151 ceph-mon[28965]: 1.6 deep-scrub starts 2024-06-27T21:23:21.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:20 smithi151 ceph-mon[28965]: 1.6 deep-scrub ok 2024-06-27T21:23:21.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:20 smithi192 ceph-mon[32956]: 1.6 deep-scrub starts 2024-06-27T21:23:21.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:20 smithi192 ceph-mon[32956]: 1.6 deep-scrub ok 2024-06-27T21:23:21.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:21.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:21.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:21.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:21.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:21.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:21.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:21.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:21.619 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:21.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:21.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:21.669 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:21.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:21.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:21.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:21.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:21.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:21.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:21.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:21 smithi151 ceph-mon[28965]: pgmap v930: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 9436/112932 objects degraded (8.355%); 6759/112932 objects misplaced (5.985%); 3.6 MiB/s, 56 objects/s recovering 2024-06-27T21:23:21.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:21 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:23:21.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:21 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:23:22.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:21 smithi190 ceph-mon[32247]: pgmap v930: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 9436/112932 objects degraded (8.355%); 6759/112932 objects misplaced (5.985%); 3.6 MiB/s, 56 objects/s recovering 2024-06-27T21:23:22.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:21 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:23:22.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:21 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:23:22.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:22.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:21 smithi192 ceph-mon[32956]: pgmap v930: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 686 op/s; 9436/112932 objects degraded (8.355%); 6759/112932 objects misplaced (5.985%); 3.6 MiB/s, 56 objects/s recovering 2024-06-27T21:23:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:21 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:23:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:21 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:23:22.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:22.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:22.260 INFO:teuthology.orchestra.run.smithi190.stdout: marking collection for removal 2024-06-27T21:23:22.260 INFO:teuthology.orchestra.run.smithi190.stdout:setting '_remove' omap key 2024-06-27T21:23:22.269 INFO:teuthology.orchestra.run.smithi190.stdout:finish_remove_pgs 4.d_head removing 4.d 2024-06-27T21:23:22.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:22.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:22.335 INFO:teuthology.orchestra.run.smithi190.stdout:Remove successful 2024-06-27T21:23:22.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:22.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:22.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:22.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:22.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:22.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:22.718 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:23:22.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:22.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:22 smithi151 ceph-mon[28965]: 4.1 scrub starts 2024-06-27T21:23:22.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:22 smithi151 ceph-mon[28965]: 4.1 scrub ok 2024-06-27T21:23:22.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/521012576' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:23:22.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:22 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:23:22.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:22.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:22.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:22.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:23.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:22 smithi190 ceph-mon[32247]: 4.1 scrub starts 2024-06-27T21:23:23.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:22 smithi190 ceph-mon[32247]: 4.1 scrub ok 2024-06-27T21:23:23.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/521012576' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:23:23.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:22 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:23:23.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:22 smithi192 ceph-mon[32956]: 4.1 scrub starts 2024-06-27T21:23:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:22 smithi192 ceph-mon[32956]: 4.1 scrub ok 2024-06-27T21:23:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/521012576' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:23:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:22 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:23:23.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:23.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:23.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:23.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:23.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:23.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:23.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:23.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell -n osd.4 --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.d.4 2024-06-27T21:23:23.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:23.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:23.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:23.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:23.721 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/osd.4/config 2024-06-27T21:23:23.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:23.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:23.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:23.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:23 smithi151 ceph-mon[28965]: pgmap v931: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 710 op/s; 8990/108996 objects degraded (8.248%); 6759/108996 objects misplaced (6.201%); 1.7 MiB/s, 27 objects/s recovering 2024-06-27T21:23:23.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:23 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:23.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:23 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:23:23.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:23 smithi151 ceph-mon[28965]: osdmap e354: 12 total, 11 up, 12 in 2024-06-27T21:23:23.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:23.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:24.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:23 smithi190 ceph-mon[32247]: pgmap v931: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 710 op/s; 8990/108996 objects degraded (8.248%); 6759/108996 objects misplaced (6.201%); 1.7 MiB/s, 27 objects/s recovering 2024-06-27T21:23:24.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:23 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:24.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:23 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:23:24.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:23 smithi190 ceph-mon[32247]: osdmap e354: 12 total, 11 up, 12 in 2024-06-27T21:23:24.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:23 smithi192 ceph-mon[32956]: pgmap v931: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 710 op/s; 8990/108996 objects degraded (8.248%); 6759/108996 objects misplaced (6.201%); 1.7 MiB/s, 27 objects/s recovering 2024-06-27T21:23:24.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:23 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:24.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:23 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:23:24.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:23 smithi192 ceph-mon[32956]: osdmap e354: 12 total, 11 up, 12 in 2024-06-27T21:23:24.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:24.078 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:24.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:24.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:24.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:24.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:24.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:24.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:24.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:24.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:24.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:24.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:24.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:24.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:24.892 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:24.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:24.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 8990/108996 objects degraded (8.248%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:24.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:24.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:24.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:25.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:25.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:25.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 8990/108996 objects degraded (8.248%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:25.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:25.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:25.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 8990/108996 objects degraded (8.248%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:25.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:25.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:25.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:25.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:25.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:25.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:25.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:25.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:25.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:25.526 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:23:25.527 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:23:25.527 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:23:25.527 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:23:25.527 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:23:25.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:25.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:23:25.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:25.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:25.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:25.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:25.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:25.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:25.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:25.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:26.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:25 smithi190 ceph-mon[32247]: pgmap v933: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 8778/105567 objects degraded (8.315%); 6026/105567 objects misplaced (5.708%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:23:26.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:25 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:26.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:25 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:26.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:25 smithi190 ceph-mon[32247]: osdmap e355: 12 total, 11 up, 12 in 2024-06-27T21:23:26.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:25 smithi192 ceph-mon[32956]: pgmap v933: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 8778/105567 objects degraded (8.315%); 6026/105567 objects misplaced (5.708%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:23:26.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:25 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:26.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:25 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:26.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:25 smithi192 ceph-mon[32956]: osdmap e355: 12 total, 11 up, 12 in 2024-06-27T21:23:26.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:26.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:26.125 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-06-27T21:23:26.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:25 smithi151 ceph-mon[28965]: pgmap v933: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 8778/105567 objects degraded (8.315%); 6026/105567 objects misplaced (5.708%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:23:26.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:25 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:26.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:25 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:26.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:25 smithi151 ceph-mon[28965]: osdmap e355: 12 total, 11 up, 12 in 2024-06-27T21:23:26.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:26.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:26.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:26.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:26.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:23:26.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:26.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:26.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:26.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:26.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:26.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:26.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:23:26.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:23:26.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:23:26.812 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:23:26.833 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:26.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:23:26.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:23:26.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:26.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:23:26.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:23:26.894 INFO:teuthology.orchestra.run.smithi151.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-27T21:23:26.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:26.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:26.909 INFO:teuthology.orchestra.run.smithi190.stdout:get_pg_num_history pg_num_history pg_num_history(e347 pg_nums {1={161=11},4={282=16}} deleted_pools ) 2024-06-27T21:23:26.909 INFO:teuthology.orchestra.run.smithi190.stdout:Importing pgid 4.d 2024-06-27T21:23:26.921 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0004acc:::benchmark_data_smithi192_88961_object85358:head# 2024-06-27T21:23:26.921 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.922 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0005339:::benchmark_data_smithi192_88961_object45375:head# 2024-06-27T21:23:26.922 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.923 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b000b22e:::benchmark_data_smithi192_88961_object59051:head# 2024-06-27T21:23:26.923 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.923 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b000c10b:::benchmark_data_smithi192_88961_object59216:head# 2024-06-27T21:23:26.923 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.924 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b001fa71:::benchmark_data_smithi192_88961_object64829:head# 2024-06-27T21:23:26.924 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.924 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0031f54:::benchmark_data_smithi192_88961_object70895:head# 2024-06-27T21:23:26.924 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.924 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b004b9ce:::benchmark_data_smithi192_88961_object54652:head# 2024-06-27T21:23:26.924 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.925 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b006a44d:::benchmark_data_smithi192_88961_object72024:head# 2024-06-27T21:23:26.925 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.925 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00af01b:::benchmark_data_smithi192_88961_object69933:head# 2024-06-27T21:23:26.925 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.925 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00bdffe:::benchmark_data_smithi192_88961_object51483:head# 2024-06-27T21:23:26.925 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00d6665:::benchmark_data_smithi192_88961_object68732:head# 2024-06-27T21:23:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00e0b6c:::benchmark_data_smithi192_88961_object76654:head# 2024-06-27T21:23:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00f6de9:::benchmark_data_smithi192_88961_object50818:head# 2024-06-27T21:23:26.926 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.927 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00f7d2b:::benchmark_data_smithi192_88961_object83888:head# 2024-06-27T21:23:26.927 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.927 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b00f8b78:::benchmark_data_smithi192_88961_object48522:head# 2024-06-27T21:23:26.927 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.928 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b010175c:::benchmark_data_smithi192_88961_object60439:head# 2024-06-27T21:23:26.928 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.928 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0102d12:::benchmark_data_smithi192_88961_object62841:head# 2024-06-27T21:23:26.929 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.929 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b010f0df:::benchmark_data_smithi192_88961_object71054:head# 2024-06-27T21:23:26.929 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:23:26.930 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:23:26.930 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0110fe3:::benchmark_data_smithi192_88961_object67333:head# 2024-06-27T21:23:26.930 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.930 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b011cb76:::benchmark_data_smithi192_88961_object65053:head# 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0124027:::benchmark_data_smithi192_88961_object76317:head# 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b01518b6:::benchmark_data_smithi192_88961_object57985:head# 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b015a981:::benchmark_data_smithi192_88961_object61380:head# 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.931 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0162506:::benchmark_data_smithi192_88961_object79453:head# 2024-06-27T21:23:26.932 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.933 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0169fbf:::benchmark_data_smithi192_88961_object86519:head# 2024-06-27T21:23:26.933 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.933 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0193e0b:::benchmark_data_smithi192_88961_object85630:head# 2024-06-27T21:23:26.933 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.934 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b019697d:::benchmark_data_smithi192_88961_object90588:head# 2024-06-27T21:23:26.934 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.934 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b01dc879:::benchmark_data_smithi192_88961_object77247:head# 2024-06-27T21:23:26.934 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.935 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b020788d:::benchmark_data_smithi192_88961_object59514:head# 2024-06-27T21:23:26.935 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.935 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b020aa7a:::benchmark_data_smithi192_88961_object74975:head# 2024-06-27T21:23:26.936 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.936 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0222bc4:::benchmark_data_smithi192_88961_object75710:head# 2024-06-27T21:23:26.936 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.936 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0232d80:::benchmark_data_smithi192_88961_object72595:head# 2024-06-27T21:23:26.936 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.937 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0258b13:::benchmark_data_smithi192_88961_object53116:head# 2024-06-27T21:23:26.937 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.937 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b026131a:::benchmark_data_smithi192_88961_object53908:head# 2024-06-27T21:23:26.937 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.938 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b026a8aa:::benchmark_data_smithi192_88961_object90567:head# 2024-06-27T21:23:26.938 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.938 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b026f576:::benchmark_data_smithi192_88961_object49343:head# 2024-06-27T21:23:26.938 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0277be1:::benchmark_data_smithi192_88961_object69249:head# 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b027b239:::benchmark_data_smithi192_88961_object52351:head# 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b029ab65:::benchmark_data_smithi192_88961_object83322:head# 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02a3f49:::benchmark_data_smithi192_88961_object73312:head# 2024-06-27T21:23:26.939 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.940 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02bb6f5:::benchmark_data_smithi192_88961_object62462:head# 2024-06-27T21:23:26.940 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.940 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02bf647:::benchmark_data_smithi192_88961_object76587:head# 2024-06-27T21:23:26.941 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.941 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02c38eb:::benchmark_data_smithi192_88961_object55655:head# 2024-06-27T21:23:26.941 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.941 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02c69eb:::benchmark_data_smithi192_88961_object90462:head# 2024-06-27T21:23:26.942 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.942 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02d7de4:::benchmark_data_smithi192_88961_object66053:head# 2024-06-27T21:23:26.942 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.942 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02f2ae9:::benchmark_data_smithi192_88961_object74456:head# 2024-06-27T21:23:26.942 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.943 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02f3dbe:::benchmark_data_smithi192_88961_object47966:head# 2024-06-27T21:23:26.943 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.943 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02fb9b8:::benchmark_data_smithi192_88961_object81127:head# 2024-06-27T21:23:26.943 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.943 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b02fcdfa:::benchmark_data_smithi192_88961_object54092:head# 2024-06-27T21:23:26.943 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.944 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b030227a:::benchmark_data_smithi192_88961_object68770:head# 2024-06-27T21:23:26.944 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.944 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b031ca18:::benchmark_data_smithi192_88961_object76405:head# 2024-06-27T21:23:26.944 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0336e12:::benchmark_data_smithi192_88961_object63493:head# 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0368d87:::benchmark_data_smithi192_88961_object83041:head# 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b038fb94:::benchmark_data_smithi192_88961_object68018:head# 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b03e2c9e:::benchmark_data_smithi192_88961_object77409:head# 2024-06-27T21:23:26.945 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.946 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b03e3eb5:::benchmark_data_smithi192_88961_object82517:head# 2024-06-27T21:23:26.946 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.946 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b040096c:::benchmark_data_smithi192_88961_object65993:head# 2024-06-27T21:23:26.946 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.946 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b040d733:::benchmark_data_smithi192_88961_object46624:head# 2024-06-27T21:23:26.946 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.947 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0419461:::benchmark_data_smithi192_88961_object49242:head# 2024-06-27T21:23:26.947 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.947 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0447a44:::benchmark_data_smithi192_88961_object83262:head# 2024-06-27T21:23:26.947 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.947 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b04c0d46:::benchmark_data_smithi192_88961_object82385:head# 2024-06-27T21:23:26.947 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.948 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b04c948a:::benchmark_data_smithi192_88961_object76433:head# 2024-06-27T21:23:26.948 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.948 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b04d5e81:::benchmark_data_smithi192_88961_object85682:head# 2024-06-27T21:23:26.948 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.948 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b050c9f8:::benchmark_data_smithi192_88961_object57758:head# 2024-06-27T21:23:26.949 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.949 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b052809e:::benchmark_data_smithi192_88961_object80264:head# 2024-06-27T21:23:26.949 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.949 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0537b1f:::benchmark_data_smithi192_88961_object77353:head# 2024-06-27T21:23:26.949 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.950 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b053d5c3:::benchmark_data_smithi192_88961_object67875:head# 2024-06-27T21:23:26.950 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.950 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b054f09e:::benchmark_data_smithi192_88961_object53749:head# 2024-06-27T21:23:26.950 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.950 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0566aae:::benchmark_data_smithi192_88961_object54703:head# 2024-06-27T21:23:26.950 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b056e5de:::benchmark_data_smithi192_88961_object62602:head# 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b05870a0:::benchmark_data_smithi192_88961_object45374:head# 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b05ad45f:::benchmark_data_smithi192_88961_object49744:head# 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b05bb117:::benchmark_data_smithi192_88961_object62031:head# 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.951 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b05e04c3:::benchmark_data_smithi192_88961_object45313:head# 2024-06-27T21:23:26.952 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.952 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b05e050a:::benchmark_data_smithi192_88961_object79931:head# 2024-06-27T21:23:26.952 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b06212ef:::benchmark_data_smithi192_88961_object50685:head# 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0634a19:::benchmark_data_smithi192_88961_object77450:head# 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b063b9e6:::benchmark_data_smithi192_88961_object87881:head# 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0667678:::benchmark_data_smithi192_88961_object81132:head# 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b06a7adc:::benchmark_data_smithi192_88961_object55965:head# 2024-06-27T21:23:26.953 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b06aa327:::benchmark_data_smithi192_88961_object45848:head# 2024-06-27T21:23:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b06d102c:::benchmark_data_smithi192_88961_object52892:head# 2024-06-27T21:23:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b06d2376:::benchmark_data_smithi192_88961_object50294:head# 2024-06-27T21:23:26.954 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b070ba0f:::benchmark_data_smithi192_88961_object68354:head# 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b076b237:::benchmark_data_smithi192_88961_object58124:head# 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b079993a:::benchmark_data_smithi192_88961_object85058:head# 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b07a86fa:::benchmark_data_smithi192_88961_object44181:head# 2024-06-27T21:23:26.955 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.956 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b07cbc11:::benchmark_data_smithi192_88961_object72944:head# 2024-06-27T21:23:26.956 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b07d1cad:::benchmark_data_smithi192_88961_object69610:head# 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b07ec854:::benchmark_data_smithi192_88961_object58772:head# 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b080b09f:::benchmark_data_smithi192_88961_object66274:head# 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b080c6bc:::benchmark_data_smithi192_88961_object51998:head# 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b081f53d:::benchmark_data_smithi192_88961_object88188:head# 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b083c9c9:::benchmark_data_smithi192_88961_object83161:head# 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.957 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b084a43e:::benchmark_data_smithi192_88961_object59453:head# 2024-06-27T21:23:26.958 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0856df6:::benchmark_data_smithi192_88961_object88697:head# 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0859ec2:::benchmark_data_smithi192_88961_object57447:head# 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0866c9b:::benchmark_data_smithi192_88961_object47928:head# 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b086800c:::benchmark_data_smithi192_88961_object54678:head# 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b088718f:::benchmark_data_smithi192_88961_object55409:head# 2024-06-27T21:23:26.959 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.960 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b08a4988:::benchmark_data_smithi192_88961_object46705:head# 2024-06-27T21:23:26.960 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.960 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b08a4ea2:::benchmark_data_smithi192_88961_object76100:head# 2024-06-27T21:23:26.960 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.960 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b08dfa38:::benchmark_data_smithi192_88961_object58537:head# 2024-06-27T21:23:26.960 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.961 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:23:26.961 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:23:26.962 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b08fc276:::benchmark_data_smithi192_88961_object56803:head# 2024-06-27T21:23:26.962 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.962 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b08fe8a8:::benchmark_data_smithi192_88961_object82886:head# 2024-06-27T21:23:26.962 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0925c94:::benchmark_data_smithi192_88961_object89131:head# 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b092facb:::benchmark_data_smithi192_88961_object44040:head# 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0930b2e:::benchmark_data_smithi192_88961_object56964:head# 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0935913:::benchmark_data_smithi192_88961_object44000:head# 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b09478f7:::benchmark_data_smithi192_88961_object61956:head# 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b094cb19:::benchmark_data_smithi192_88961_object49398:head# 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.963 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b097c219:::benchmark_data_smithi192_88961_object48727:head# 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b097defc:::benchmark_data_smithi192_88961_object82111:head# 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b09bf574:::benchmark_data_smithi192_88961_object58933:head# 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b09c20bb:::benchmark_data_smithi192_88961_object70897:head# 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b09cc095:::benchmark_data_smithi192_88961_object45954:head# 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b09eb751:::benchmark_data_smithi192_88961_object54221:head# 2024-06-27T21:23:26.964 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.967 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b09eef1d:::benchmark_data_smithi192_88961_object60021:head# 2024-06-27T21:23:26.967 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0a20962:::benchmark_data_smithi192_88961_object71088:head# 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0a34672:::benchmark_data_smithi192_88961_object52125:head# 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0a3acaa:::benchmark_data_smithi192_88961_object50120:head# 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0a6481e:::benchmark_data_smithi192_88961_object59503:head# 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0a9ae0e:::benchmark_data_smithi192_88961_object71268:head# 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.968 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0aa66e6:::benchmark_data_smithi192_88961_object88933:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ab3a2b:::benchmark_data_smithi192_88961_object55273:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ab91ed:::benchmark_data_smithi192_88961_object54663:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ab9ce9:::benchmark_data_smithi192_88961_object45293:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0abbad3:::benchmark_data_smithi192_88961_object59220:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ac17c2:::benchmark_data_smithi192_88961_object61178:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0adc187:::benchmark_data_smithi192_88961_object84046:head# 2024-06-27T21:23:26.969 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0adce32:::benchmark_data_smithi192_88961_object76217:head# 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0af0a68:::benchmark_data_smithi192_88961_object64779:head# 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0af599d:::benchmark_data_smithi192_88961_object81791:head# 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0b0f442:::benchmark_data_smithi192_88961_object78979:head# 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0b3f864:::benchmark_data_smithi192_88961_object77297:head# 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0b545e2:::benchmark_data_smithi192_88961_object70879:head# 2024-06-27T21:23:26.970 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.971 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0bfa327:::benchmark_data_smithi192_88961_object60146:head# 2024-06-27T21:23:26.971 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.971 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0bfb53a:::benchmark_data_smithi192_88961_object71444:head# 2024-06-27T21:23:26.971 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c02f76:::benchmark_data_smithi192_88961_object59769:head# 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c25a8e:::benchmark_data_smithi192_88961_object63645:head# 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c32434:::benchmark_data_smithi192_88961_object66634:head# 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.973 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c35834:::benchmark_data_smithi192_88961_object60336:head# 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c4538a:::benchmark_data_smithi192_88961_object82054:head# 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c52f0d:::benchmark_data_smithi192_88961_object54066:head# 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c5b7a3:::benchmark_data_smithi192_88961_object54218:head# 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c6264c:::benchmark_data_smithi192_88961_object57775:head# 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0c7ddfc:::benchmark_data_smithi192_88961_object79801:head# 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.974 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0caecff:::benchmark_data_smithi192_88961_object86282:head# 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0cb59f8:::benchmark_data_smithi192_88961_object56186:head# 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0cc8206:::benchmark_data_smithi192_88961_object68155:head# 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ce4e75:::benchmark_data_smithi192_88961_object72133:head# 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0cea05d:::benchmark_data_smithi192_88961_object60007:head# 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0cf854f:::benchmark_data_smithi192_88961_object48471:head# 2024-06-27T21:23:26.975 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d0d652:::benchmark_data_smithi192_88961_object48811:head# 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d17ec5:::benchmark_data_smithi192_88961_object46795:head# 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d21b06:::benchmark_data_smithi192_88961_object44191:head# 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d25f94:::benchmark_data_smithi192_88961_object45583:head# 2024-06-27T21:23:26.977 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d304fb:::benchmark_data_smithi192_88961_object55653:head# 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d3f6fa:::benchmark_data_smithi192_88961_object61376:head# 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d42e03:::benchmark_data_smithi192_88961_object68596:head# 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d42f29:::benchmark_data_smithi192_88961_object76017:head# 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d5b7f9:::benchmark_data_smithi192_88961_object52881:head# 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d627b4:::benchmark_data_smithi192_88961_object86940:head# 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.978 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d8f0c9:::benchmark_data_smithi192_88961_object78776:head# 2024-06-27T21:23:26.979 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.979 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0d8f802:::benchmark_data_smithi192_88961_object50200:head# 2024-06-27T21:23:26.979 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.979 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0da9f94:::benchmark_data_smithi192_88961_object50623:head# 2024-06-27T21:23:26.979 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.980 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0dc75c1:::benchmark_data_smithi192_88961_object60856:head# 2024-06-27T21:23:26.980 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0dcd16f:::benchmark_data_smithi192_88961_object60727:head# 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0dce0af:::benchmark_data_smithi192_88961_object45562:head# 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0dda33e:::benchmark_data_smithi192_88961_object60395:head# 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ddf097:::benchmark_data_smithi192_88961_object72344:head# 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0e10463:::benchmark_data_smithi192_88961_object59671:head# 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0e30646:::benchmark_data_smithi192_88961_object57366:head# 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.981 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0e62f7e:::benchmark_data_smithi192_88961_object64801:head# 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0e86cc7:::benchmark_data_smithi192_88961_object44363:head# 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0e9d62e:::benchmark_data_smithi192_88961_object50543:head# 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ec0a6b:::benchmark_data_smithi192_88961_object48343:head# 2024-06-27T21:23:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.983 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ec1822:::benchmark_data_smithi192_88961_object71179:head# 2024-06-27T21:23:26.983 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.983 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0ec81b6:::benchmark_data_smithi192_88961_object70335:head# 2024-06-27T21:23:26.983 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0edb2ff:::benchmark_data_smithi192_88961_object52981:head# 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f19f78:::benchmark_data_smithi192_88961_object75435:head# 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f34c09:::benchmark_data_smithi192_88961_object73778:head# 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f428c9:::benchmark_data_smithi192_88961_object58437:head# 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f42a3b:::benchmark_data_smithi192_88961_object54722:head# 2024-06-27T21:23:26.984 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.985 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f63a02:::benchmark_data_smithi192_88961_object87933:head# 2024-06-27T21:23:26.985 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.985 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f6ae8e:::benchmark_data_smithi192_88961_object55092:head# 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f8d9eb:::benchmark_data_smithi192_88961_object68925:head# 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0f91a50:::benchmark_data_smithi192_88961_object89697:head# 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0fb4012:::benchmark_data_smithi192_88961_object80167:head# 2024-06-27T21:23:26.986 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.987 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0fc4d00:::benchmark_data_smithi192_88961_object55376:head# 2024-06-27T21:23:26.987 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.987 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0fdeb08:::benchmark_data_smithi192_88961_object56757:head# 2024-06-27T21:23:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b0feda33:::benchmark_data_smithi192_88961_object90458:head# 2024-06-27T21:23:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b10102ea:::benchmark_data_smithi192_88961_object67630:head# 2024-06-27T21:23:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b10105e9:::benchmark_data_smithi192_88961_object46282:head# 2024-06-27T21:23:26.991 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.991 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b102c04e:::benchmark_data_smithi192_88961_object44841:head# 2024-06-27T21:23:26.991 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.992 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1053508:::benchmark_data_smithi192_88961_object80730:head# 2024-06-27T21:23:26.992 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.992 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b10703f8:::benchmark_data_smithi192_88961_object67617:head# 2024-06-27T21:23:26.993 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.993 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b107957d:::benchmark_data_smithi192_88961_object59645:head# 2024-06-27T21:23:26.993 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.993 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b10949ab:::benchmark_data_smithi192_88961_object83314:head# 2024-06-27T21:23:26.993 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:23:26.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:23:26.995 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b10b9dcd:::benchmark_data_smithi192_88961_object85851:head# 2024-06-27T21:23:26.995 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b10fd701:::benchmark_data_smithi192_88961_object47261:head# 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b110023d:::benchmark_data_smithi192_88961_object88703:head# 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1101901:::benchmark_data_smithi192_88961_object53439:head# 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1110da9:::benchmark_data_smithi192_88961_object65131:head# 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1124a80:::benchmark_data_smithi192_88961_object86292:head# 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1127213:::benchmark_data_smithi192_88961_object56107:head# 2024-06-27T21:23:26.996 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.997 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b11293ed:::benchmark_data_smithi192_88961_object58119:head# 2024-06-27T21:23:26.997 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.997 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b114053f:::benchmark_data_smithi192_88961_object60697:head# 2024-06-27T21:23:26.997 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.998 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1140856:::benchmark_data_smithi192_88961_object70896:head# 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b114df6f:::benchmark_data_smithi192_88961_object69723:head# 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b11711f8:::benchmark_data_smithi192_88961_object69486:head# 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b118a5a9:::benchmark_data_smithi192_88961_object85492:head# 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b11b0316:::benchmark_data_smithi192_88961_object77252:head# 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b11bfb72:::benchmark_data_smithi192_88961_object73345:head# 2024-06-27T21:23:26.999 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b11d7125:::benchmark_data_smithi192_88961_object63921:head# 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b11ebc93:::benchmark_data_smithi192_88961_object71323:head# 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1208458:::benchmark_data_smithi192_88961_object60846:head# 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b123f6c5:::benchmark_data_smithi192_88961_object45947:head# 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1241cd1:::benchmark_data_smithi192_88961_object88955:head# 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1250798:::benchmark_data_smithi192_88961_object89154:head# 2024-06-27T21:23:27.000 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.004 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1257812:::benchmark_data_smithi192_88961_object73949:head# 2024-06-27T21:23:27.004 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1262110:::benchmark_data_smithi192_88961_object82044:head# 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1278ce2:::benchmark_data_smithi192_88961_object45611:head# 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b127fc4a:::benchmark_data_smithi192_88961_object52284:head# 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1283f7f:::benchmark_data_smithi192_88961_object66524:head# 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1297e25:::benchmark_data_smithi192_88961_object61193:head# 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b12b254a:::benchmark_data_smithi192_88961_object46656:head# 2024-06-27T21:23:27.005 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b12c2ae5:::benchmark_data_smithi192_88961_object58887:head# 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b12d6204:::benchmark_data_smithi192_88961_object85598:head# 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b12e7237:::benchmark_data_smithi192_88961_object85261:head# 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b131ab89:::benchmark_data_smithi192_88961_object83884:head# 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1327fc7:::benchmark_data_smithi192_88961_object46753:head# 2024-06-27T21:23:27.006 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b133181a:::benchmark_data_smithi192_88961_object54763:head# 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1347dd4:::benchmark_data_smithi192_88961_object65598:head# 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b13602ea:::benchmark_data_smithi192_88961_object57342:head# 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b13603d2:::benchmark_data_smithi192_88961_object74141:head# 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1393989:::benchmark_data_smithi192_88961_object86017:head# 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b13a6d53:::benchmark_data_smithi192_88961_object85427:head# 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.007 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b13e1ef0:::benchmark_data_smithi192_88961_object59487:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b13f6c8f:::benchmark_data_smithi192_88961_object47727:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b140028f:::benchmark_data_smithi192_88961_object45132:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1400380:::benchmark_data_smithi192_88961_object72672:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b140cc59:::benchmark_data_smithi192_88961_object78387:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b141e507:::benchmark_data_smithi192_88961_object44523:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b142ebc2:::benchmark_data_smithi192_88961_object68153:head# 2024-06-27T21:23:27.008 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:27.012 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b142f2b1:::benchmark_data_smithi192_88961_object50022:head# 2024-06-27T21:23:27.012 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.012 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1450851:::benchmark_data_smithi192_88961_object68086:head# 2024-06-27T21:23:27.012 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.012 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b146bcda:::benchmark_data_smithi192_88961_object57789:head# 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b14795a7:::benchmark_data_smithi192_88961_object64889:head# 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b148892a:::benchmark_data_smithi192_88961_object62164:head# 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b14932ea:::benchmark_data_smithi192_88961_object50202:head# 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b14af43c:::benchmark_data_smithi192_88961_object64321:head# 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b14d8d8e:::benchmark_data_smithi192_88961_object84379:head# 2024-06-27T21:23:27.013 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b14f5769:::benchmark_data_smithi192_88961_object70004:head# 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b15062e5:::benchmark_data_smithi192_88961_object71122:head# 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b150b88b:::benchmark_data_smithi192_88961_object51254:head# 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b150c84a:::benchmark_data_smithi192_88961_object44315:head# 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b154acca:::benchmark_data_smithi192_88961_object81084:head# 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.014 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1550b02:::benchmark_data_smithi192_88961_object84747:head# 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1579e4d:::benchmark_data_smithi192_88961_object75574:head# 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1588f92:::benchmark_data_smithi192_88961_object65244:head# 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1593688:::benchmark_data_smithi192_88961_object62004:head# 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b159a64c:::benchmark_data_smithi192_88961_object81318:head# 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b15cacd9:::benchmark_data_smithi192_88961_object75091:head# 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.015 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b15dabfa:::benchmark_data_smithi192_88961_object77207:head# 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b15db2d0:::benchmark_data_smithi192_88961_object47663:head# 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b15f2bdb:::benchmark_data_smithi192_88961_object73089:head# 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b161c131:::benchmark_data_smithi192_88961_object65433:head# 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b162e7d9:::benchmark_data_smithi192_88961_object59389:head# 2024-06-27T21:23:27.016 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.018 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1639b55:::benchmark_data_smithi192_88961_object57254:head# 2024-06-27T21:23:27.018 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.018 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1639c07:::benchmark_data_smithi192_88961_object47444:head# 2024-06-27T21:23:27.018 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b16529a3:::benchmark_data_smithi192_88961_object70043:head# 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b16be8a2:::benchmark_data_smithi192_88961_object63657:head# 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b16c7d93:::benchmark_data_smithi192_88961_object90537:head# 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b16f63a0:::benchmark_data_smithi192_88961_object71545:head# 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b16faf6c:::benchmark_data_smithi192_88961_object81832:head# 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b16fd924:::benchmark_data_smithi192_88961_object45617:head# 2024-06-27T21:23:27.019 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.021 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17100db:::benchmark_data_smithi192_88961_object49399:head# 2024-06-27T21:23:27.021 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.022 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17161fa:::benchmark_data_smithi192_88961_object72587:head# 2024-06-27T21:23:27.022 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.022 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b171e503:::benchmark_data_smithi192_88961_object60175:head# 2024-06-27T21:23:27.022 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.023 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1723ea9:::benchmark_data_smithi192_88961_object52290:head# 2024-06-27T21:23:27.023 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.023 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b173a61f:::benchmark_data_smithi192_88961_object60937:head# 2024-06-27T21:23:27.023 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.023 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b174c0c2:::benchmark_data_smithi192_88961_object67847:head# 2024-06-27T21:23:27.023 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.025 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1791e02:::benchmark_data_smithi192_88961_object84252:head# 2024-06-27T21:23:27.025 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:23:27.025 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17b2aaf:::benchmark_data_smithi192_88961_object71532:head# 2024-06-27T21:23:27.026 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:23:27.026 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17bd424:::benchmark_data_smithi192_88961_object71106:head# 2024-06-27T21:23:27.026 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.027 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17d853f:::benchmark_data_smithi192_88961_object51701:head# 2024-06-27T21:23:27.027 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.027 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17da211:::benchmark_data_smithi192_88961_object88394:head# 2024-06-27T21:23:27.027 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.027 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17ee8ed:::benchmark_data_smithi192_88961_object82080:head# 2024-06-27T21:23:27.027 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.028 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b17f53a3:::benchmark_data_smithi192_88961_object46228:head# 2024-06-27T21:23:27.028 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.028 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1804ab5:::benchmark_data_smithi192_88961_object60998:head# 2024-06-27T21:23:27.028 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.028 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b180f6af:::benchmark_data_smithi192_88961_object64930:head# 2024-06-27T21:23:27.029 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.029 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b182a9ad:::benchmark_data_smithi192_88961_object45864:head# 2024-06-27T21:23:27.029 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.029 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b184552b:::benchmark_data_smithi192_88961_object70191:head# 2024-06-27T21:23:27.029 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.030 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b184d560:::benchmark_data_smithi192_88961_object52408:head# 2024-06-27T21:23:27.030 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.030 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b184f88a:::benchmark_data_smithi192_88961_object84253:head# 2024-06-27T21:23:27.031 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.031 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1865f89:::benchmark_data_smithi192_88961_object87107:head# 2024-06-27T21:23:27.031 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.031 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1875258:::benchmark_data_smithi192_88961_object49353:head# 2024-06-27T21:23:27.031 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b18853ed:::benchmark_data_smithi192_88961_object46815:head# 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b188a51a:::benchmark_data_smithi192_88961_object63374:head# 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b188eff1:::benchmark_data_smithi192_88961_object75602:head# 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b18a8838:::benchmark_data_smithi192_88961_object87673:head# 2024-06-27T21:23:27.032 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b18a8ba2:::benchmark_data_smithi192_88961_object61109:head# 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b18c6683:::benchmark_data_smithi192_88961_object45393:head# 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b191dccf:::benchmark_data_smithi192_88961_object57009:head# 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.033 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b193c308:::benchmark_data_smithi192_88961_object50664:head# 2024-06-27T21:23:27.034 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.034 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b19480e2:::benchmark_data_smithi192_88961_object82696:head# 2024-06-27T21:23:27.034 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.034 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1959a16:::benchmark_data_smithi192_88961_object57098:head# 2024-06-27T21:23:27.034 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.035 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b19af94f:::benchmark_data_smithi192_88961_object53724:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b19b1d1b:::benchmark_data_smithi192_88961_object79526:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1a1bc11:::benchmark_data_smithi192_88961_object80557:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1a4818b:::benchmark_data_smithi192_88961_object68256:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1a496ae:::benchmark_data_smithi192_88961_object63847:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1a56fe8:::benchmark_data_smithi192_88961_object62884:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1a6c31c:::benchmark_data_smithi192_88961_object75921:head# 2024-06-27T21:23:27.036 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.037 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1a73d17:::benchmark_data_smithi192_88961_object66911:head# 2024-06-27T21:23:27.037 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.037 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1ac09c8:::benchmark_data_smithi192_88961_object71265:head# 2024-06-27T21:23:27.037 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1adeb5e:::benchmark_data_smithi192_88961_object72697:head# 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1ae09cb:::benchmark_data_smithi192_88961_object79214:head# 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1af04c4:::benchmark_data_smithi192_88961_object47540:head# 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b3b3f6:::benchmark_data_smithi192_88961_object82038:head# 2024-06-27T21:23:27.038 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.039 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b3b67a:::benchmark_data_smithi192_88961_object82243:head# 2024-06-27T21:23:27.039 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.039 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b3da8a:::benchmark_data_smithi192_88961_object53398:head# 2024-06-27T21:23:27.039 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.039 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b40f9c:::benchmark_data_smithi192_88961_object86523:head# 2024-06-27T21:23:27.039 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:27.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:27.042 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b5f472:::benchmark_data_smithi192_88961_object84607:head# 2024-06-27T21:23:27.042 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.042 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b755a2:::benchmark_data_smithi192_88961_object86874:head# 2024-06-27T21:23:27.042 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.042 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1b94c90:::benchmark_data_smithi192_88961_object85191:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1bb5374:::benchmark_data_smithi192_88961_object46134:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1be3b35:::benchmark_data_smithi192_88961_object69651:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1be5161:::benchmark_data_smithi192_88961_object55051:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1bf8281:::benchmark_data_smithi192_88961_object83684:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c09795:::benchmark_data_smithi192_88961_object88040:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c0acc5:::benchmark_data_smithi192_88961_object46359:head# 2024-06-27T21:23:27.043 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c135a9:::benchmark_data_smithi192_88961_object49974:head# 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c36eb8:::benchmark_data_smithi192_88961_object80536:head# 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c3f089:::benchmark_data_smithi192_88961_object55977:head# 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c4b9f6:::benchmark_data_smithi192_88961_object76116:head# 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c6eacf:::benchmark_data_smithi192_88961_object89409:head# 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1c8bae1:::benchmark_data_smithi192_88961_object86648:head# 2024-06-27T21:23:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.045 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1cac732:::benchmark_data_smithi192_88961_object87641:head# 2024-06-27T21:23:27.045 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.045 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1cb5414:::benchmark_data_smithi192_88961_object59212:head# 2024-06-27T21:23:27.045 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.045 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1cca20c:::benchmark_data_smithi192_88961_object47591:head# 2024-06-27T21:23:27.045 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.046 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1cca438:::benchmark_data_smithi192_88961_object70100:head# 2024-06-27T21:23:27.046 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.046 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d02577:::benchmark_data_smithi192_88961_object66944:head# 2024-06-27T21:23:27.046 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.046 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d17869:::benchmark_data_smithi192_88961_object67906:head# 2024-06-27T21:23:27.047 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.049 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d1b02f:::benchmark_data_smithi192_88961_object49638:head# 2024-06-27T21:23:27.049 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.049 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d1bf47:::benchmark_data_smithi192_88961_object83727:head# 2024-06-27T21:23:27.050 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.050 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d2b16b:::benchmark_data_smithi192_88961_object45113:head# 2024-06-27T21:23:27.050 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.050 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d44e8d:::benchmark_data_smithi192_88961_object78645:head# 2024-06-27T21:23:27.050 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.051 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d52687:::benchmark_data_smithi192_88961_object81188:head# 2024-06-27T21:23:27.051 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.051 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d6a1e9:::benchmark_data_smithi192_88961_object49235:head# 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d89932:::benchmark_data_smithi192_88961_object82281:head# 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1d97bf6:::benchmark_data_smithi192_88961_object48783:head# 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1db28ce:::benchmark_data_smithi192_88961_object69691:head# 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1dc05f8:::benchmark_data_smithi192_88961_object51189:head# 2024-06-27T21:23:27.052 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.053 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1df91bd:::benchmark_data_smithi192_88961_object89970:head# 2024-06-27T21:23:27.053 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.053 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1dff40e:::benchmark_data_smithi192_88961_object57933:head# 2024-06-27T21:23:27.053 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e0defa:::benchmark_data_smithi192_88961_object74527:head# 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e20806:::benchmark_data_smithi192_88961_object57323:head# 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e29bd5:::benchmark_data_smithi192_88961_object46439:head# 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e2e5d7:::benchmark_data_smithi192_88961_object43767:head# 2024-06-27T21:23:27.054 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e59f12:::benchmark_data_smithi192_88961_object64224:head# 2024-06-27T21:23:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e60e3a:::benchmark_data_smithi192_88961_object86751:head# 2024-06-27T21:23:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e67aaa:::benchmark_data_smithi192_88961_object47192:head# 2024-06-27T21:23:27.055 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e6c131:::benchmark_data_smithi192_88961_object69255:head# 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e77c5a:::benchmark_data_smithi192_88961_object55894:head# 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e7b61c:::benchmark_data_smithi192_88961_object71257:head# 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1e94af3:::benchmark_data_smithi192_88961_object71333:head# 2024-06-27T21:23:27.056 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.057 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1ed21e0:::benchmark_data_smithi192_88961_object62580:head# 2024-06-27T21:23:27.057 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.057 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1ef06f7:::benchmark_data_smithi192_88961_object49525:head# 2024-06-27T21:23:27.057 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.057 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1ef2d00:::benchmark_data_smithi192_88961_object62055:head# 2024-06-27T21:23:27.057 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f222c8:::benchmark_data_smithi192_88961_object51775:head# 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f2937e:::benchmark_data_smithi192_88961_object86993:head# 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f43bcc:::benchmark_data_smithi192_88961_object46426:head# 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f5f36e:::benchmark_data_smithi192_88961_object79637:head# 2024-06-27T21:23:27.058 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f6fcbd:::benchmark_data_smithi192_88961_object87112:head# 2024-06-27T21:23:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f87799:::benchmark_data_smithi192_88961_object48197:head# 2024-06-27T21:23:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1f8e5ac:::benchmark_data_smithi192_88961_object86622:head# 2024-06-27T21:23:27.061 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1fa7fd3:::benchmark_data_smithi192_88961_object85023:head# 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1faa763:::benchmark_data_smithi192_88961_object87556:head# 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1fccb97:::benchmark_data_smithi192_88961_object59273:head# 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1fea38a:::benchmark_data_smithi192_88961_object89158:head# 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b1ff3419:::benchmark_data_smithi192_88961_object84058:head# 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b204e3ca:::benchmark_data_smithi192_88961_object52405:head# 2024-06-27T21:23:27.062 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2050aa5:::benchmark_data_smithi192_88961_object50541:head# 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2053c55:::benchmark_data_smithi192_88961_object75286:head# 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b20735b6:::benchmark_data_smithi192_88961_object54932:head# 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b208205b:::benchmark_data_smithi192_88961_object45868:head# 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b208c067:::benchmark_data_smithi192_88961_object79331:head# 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b20b8a48:::benchmark_data_smithi192_88961_object55976:head# 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.063 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b20d84e4:::benchmark_data_smithi192_88961_object73335:head# 2024-06-27T21:23:27.064 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.067 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b20ef4b7:::benchmark_data_smithi192_88961_object48215:head# 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b20f8503:::benchmark_data_smithi192_88961_object54933:head# 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b20fbcf4:::benchmark_data_smithi192_88961_object60230:head# 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2102350:::benchmark_data_smithi192_88961_object72478:head# 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b210cd6d:::benchmark_data_smithi192_88961_object64714:head# 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b21165cd:::benchmark_data_smithi192_88961_object87739:head# 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.068 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2147782:::benchmark_data_smithi192_88961_object46924:head# 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b214a0ac:::benchmark_data_smithi192_88961_object52684:head# 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b216f034:::benchmark_data_smithi192_88961_object83318:head# 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2173c38:::benchmark_data_smithi192_88961_object76154:head# 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b217f364:::benchmark_data_smithi192_88961_object49396:head# 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2182a99:::benchmark_data_smithi192_88961_object54911:head# 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.069 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b218b2e8:::benchmark_data_smithi192_88961_object69419:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b21bb050:::benchmark_data_smithi192_88961_object49400:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b21bf403:::benchmark_data_smithi192_88961_object70689:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b21bf621:::benchmark_data_smithi192_88961_object48777:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b21e973b:::benchmark_data_smithi192_88961_object75492:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b21fce42:::benchmark_data_smithi192_88961_object45017:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2212da1:::benchmark_data_smithi192_88961_object80348:head# 2024-06-27T21:23:27.070 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b225e205:::benchmark_data_smithi192_88961_object46876:head# 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b226284f:::benchmark_data_smithi192_88961_object49352:head# 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22714db:::benchmark_data_smithi192_88961_object46372:head# 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2272dc5:::benchmark_data_smithi192_88961_object74137:head# 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b227ffc3:::benchmark_data_smithi192_88961_object70071:head# 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22933f5:::benchmark_data_smithi192_88961_object82723:head# 2024-06-27T21:23:27.071 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:23:27.073 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22b9f68:::benchmark_data_smithi192_88961_object46869:head# 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22bc78d:::benchmark_data_smithi192_88961_object53874:head# 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22c94a5:::benchmark_data_smithi192_88961_object82857:head# 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22c9b2d:::benchmark_data_smithi192_88961_object50235:head# 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22cde6d:::benchmark_data_smithi192_88961_object68329:head# 2024-06-27T21:23:27.080 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22e8b0e:::benchmark_data_smithi192_88961_object73250:head# 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22f196a:::benchmark_data_smithi192_88961_object58765:head# 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22f3d57:::benchmark_data_smithi192_88961_object73403:head# 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b22ff9ca:::benchmark_data_smithi192_88961_object56622:head# 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b233b1ac:::benchmark_data_smithi192_88961_object62793:head# 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2379098:::benchmark_data_smithi192_88961_object46755:head# 2024-06-27T21:23:27.081 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b237f75b:::benchmark_data_smithi192_88961_object61765:head# 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b239e80b:::benchmark_data_smithi192_88961_object90654:head# 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b23b4c82:::benchmark_data_smithi192_88961_object69635:head# 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b23b6960:::benchmark_data_smithi192_88961_object82855:head# 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b23f6561:::benchmark_data_smithi192_88961_object82286:head# 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b23f94ab:::benchmark_data_smithi192_88961_object60382:head# 2024-06-27T21:23:27.082 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b241670d:::benchmark_data_smithi192_88961_object75063:head# 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b24168b5:::benchmark_data_smithi192_88961_object47523:head# 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2432052:::benchmark_data_smithi192_88961_object54786:head# 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b244fa0b:::benchmark_data_smithi192_88961_object62451:head# 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2455125:::benchmark_data_smithi192_88961_object85022:head# 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b245d121:::benchmark_data_smithi192_88961_object72713:head# 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.083 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2467c08:::benchmark_data_smithi192_88961_object66572:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2491cbf:::benchmark_data_smithi192_88961_object79356:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b24c2db4:::benchmark_data_smithi192_88961_object50434:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b24fc09c:::benchmark_data_smithi192_88961_object89498:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b250ab9a:::benchmark_data_smithi192_88961_object81699:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2513f20:::benchmark_data_smithi192_88961_object55355:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2518b7e:::benchmark_data_smithi192_88961_object90194:head# 2024-06-27T21:23:27.084 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b252d58c:::benchmark_data_smithi192_88961_object64615:head# 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2534dc3:::benchmark_data_smithi192_88961_object57440:head# 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b253b730:::benchmark_data_smithi192_88961_object77471:head# 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b25511bb:::benchmark_data_smithi192_88961_object44749:head# 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2558ca3:::benchmark_data_smithi192_88961_object69145:head# 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b25809a2:::benchmark_data_smithi192_88961_object88246:head# 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.085 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2597126:::benchmark_data_smithi192_88961_object86910:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b25c5e1d:::benchmark_data_smithi192_88961_object62082:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b25ca0e7:::benchmark_data_smithi192_88961_object75728:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b25dbd0c:::benchmark_data_smithi192_88961_object79773:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b25ed63b:::benchmark_data_smithi192_88961_object76407:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2615d8e:::benchmark_data_smithi192_88961_object87121:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b263bf64:::benchmark_data_smithi192_88961_object48648:head# 2024-06-27T21:23:27.086 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2650a1e:::benchmark_data_smithi192_88961_object68652:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b265bc2c:::benchmark_data_smithi192_88961_object61568:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b26648e2:::benchmark_data_smithi192_88961_object79224:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b266503f:::benchmark_data_smithi192_88961_object69390:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b26b8cd1:::benchmark_data_smithi192_88961_object86783:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b26bc08b:::benchmark_data_smithi192_88961_object47741:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2707448:::benchmark_data_smithi192_88961_object64015:head# 2024-06-27T21:23:27.087 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.088 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b272a984:::benchmark_data_smithi192_88961_object43135:head# 2024-06-27T21:23:27.088 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.088 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b274314c:::benchmark_data_smithi192_88961_object71577:head# 2024-06-27T21:23:27.088 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.088 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2744e12:::benchmark_data_smithi192_88961_object53168:head# 2024-06-27T21:23:27.088 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2749880:::benchmark_data_smithi192_88961_object61907:head# 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b27dfca2:::benchmark_data_smithi192_88961_object62568:head# 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b27e1a61:::benchmark_data_smithi192_88961_object56489:head# 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b28021b8:::benchmark_data_smithi192_88961_object83906:head# 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.095 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b281766a:::benchmark_data_smithi192_88961_object63621:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2839428:::benchmark_data_smithi192_88961_object43744:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2864010:::benchmark_data_smithi192_88961_object46703:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b286989c:::benchmark_data_smithi192_88961_object81179:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b286b77d:::benchmark_data_smithi192_88961_object59014:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b28a34da:::benchmark_data_smithi192_88961_object60233:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b28a5b20:::benchmark_data_smithi192_88961_object72971:head# 2024-06-27T21:23:27.096 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b28c0edf:::benchmark_data_smithi192_88961_object48263:head# 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29181e3:::benchmark_data_smithi192_88961_object88374:head# 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2927309:::benchmark_data_smithi192_88961_object72878:head# 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2927aa7:::benchmark_data_smithi192_88961_object87879:head# 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b293ef59:::benchmark_data_smithi192_88961_object62746:head# 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b295152a:::benchmark_data_smithi192_88961_object73968:head# 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.097 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2962895:::benchmark_data_smithi192_88961_object61733:head# 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2970791:::benchmark_data_smithi192_88961_object77518:head# 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2982e27:::benchmark_data_smithi192_88961_object88003:head# 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29924f4:::benchmark_data_smithi192_88961_object66336:head# 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2993711:::benchmark_data_smithi192_88961_object80013:head# 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29b53a1:::benchmark_data_smithi192_88961_object48199:head# 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.098 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29cd88b:::benchmark_data_smithi192_88961_object78596:head# 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29e40d2:::benchmark_data_smithi192_88961_object83720:head# 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29f8d23:::benchmark_data_smithi192_88961_object86106:head# 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b29f9a2d:::benchmark_data_smithi192_88961_object59896:head# 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2a0995f:::benchmark_data_smithi192_88961_object49523:head# 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2a331c5:::benchmark_data_smithi192_88961_object48747:head# 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.099 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2a5691a:::benchmark_data_smithi192_88961_object46996:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2a6232b:::benchmark_data_smithi192_88961_object83462:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2a76049:::benchmark_data_smithi192_88961_object44051:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2a7aefe:::benchmark_data_smithi192_88961_object84088:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ab6997:::benchmark_data_smithi192_88961_object68205:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ab7c8c:::benchmark_data_smithi192_88961_object49536:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ac8e6a:::benchmark_data_smithi192_88961_object84800:head# 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.100 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ae3bbc:::benchmark_data_smithi192_88961_object76425:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2b3de70:::benchmark_data_smithi192_88961_object52647:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2b64a5e:::benchmark_data_smithi192_88961_object43530:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2b70eee:::benchmark_data_smithi192_88961_object65599:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2b7c1d4:::benchmark_data_smithi192_88961_object60620:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2b905de:::benchmark_data_smithi192_88961_object47772:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2b99b45:::benchmark_data_smithi192_88961_object49767:head# 2024-06-27T21:23:27.101 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2baa057:::benchmark_data_smithi192_88961_object53482:head# 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2bc75f1:::benchmark_data_smithi192_88961_object56952:head# 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2be7819:::benchmark_data_smithi192_88961_object57041:head# 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2c797c2:::benchmark_data_smithi192_88961_object86365:head# 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2c7c958:::benchmark_data_smithi192_88961_object87921:head# 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2c839a2:::benchmark_data_smithi192_88961_object64528:head# 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.102 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2c989c2:::benchmark_data_smithi192_88961_object78191:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2c9b611:::benchmark_data_smithi192_88961_object58272:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ca01a1:::benchmark_data_smithi192_88961_object53931:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cb9ac7:::benchmark_data_smithi192_88961_object55724:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cba18c:::benchmark_data_smithi192_88961_object74058:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cbcd0a:::benchmark_data_smithi192_88961_object53893:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cbe581:::benchmark_data_smithi192_88961_object69204:head# 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.103 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cc873a:::benchmark_data_smithi192_88961_object49847:head# 2024-06-27T21:23:27.104 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.104 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cd1d5f:::benchmark_data_smithi192_88961_object61690:head# 2024-06-27T21:23:27.104 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:23:27.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:23:27.113 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cd2943:::benchmark_data_smithi192_88961_object88245:head# 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cd8bf5:::benchmark_data_smithi192_88961_object54688:head# 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ce5968:::benchmark_data_smithi192_88961_object49563:head# 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2cff831:::benchmark_data_smithi192_88961_object78960:head# 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d1c5fb:::benchmark_data_smithi192_88961_object60982:head# 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d20e69:::benchmark_data_smithi192_88961_object68043:head# 2024-06-27T21:23:27.114 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d232eb:::benchmark_data_smithi192_88961_object60513:head# 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d2fd58:::benchmark_data_smithi192_88961_object72255:head# 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d2ff8d:::benchmark_data_smithi192_88961_object55543:head# 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d68bf9:::benchmark_data_smithi192_88961_object90397:head# 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d69075:::benchmark_data_smithi192_88961_object55342:head# 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d6fd8f:::benchmark_data_smithi192_88961_object81530:head# 2024-06-27T21:23:27.115 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d7ef7b:::benchmark_data_smithi192_88961_object80633:head# 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d88271:::benchmark_data_smithi192_88961_object81281:head# 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d8cc71:::benchmark_data_smithi192_88961_object75843:head# 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2d9e047:::benchmark_data_smithi192_88961_object57481:head# 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2daf348:::benchmark_data_smithi192_88961_object43622:head# 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2db0687:::benchmark_data_smithi192_88961_object88828:head# 2024-06-27T21:23:27.116 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2dc7b32:::benchmark_data_smithi192_88961_object51559:head# 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2dcdfa9:::benchmark_data_smithi192_88961_object62872:head# 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2dd7492:::benchmark_data_smithi192_88961_object62535:head# 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ddd99f:::benchmark_data_smithi192_88961_object53906:head# 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e0183a:::benchmark_data_smithi192_88961_object56993:head# 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.117 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e2af94:::benchmark_data_smithi192_88961_object60166:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e38b47:::benchmark_data_smithi192_88961_object57796:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e3db39:::benchmark_data_smithi192_88961_object66245:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e5f9a8:::benchmark_data_smithi192_88961_object54318:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e656e6:::benchmark_data_smithi192_88961_object69757:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e6817c:::benchmark_data_smithi192_88961_object74126:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e6a9ed:::benchmark_data_smithi192_88961_object70838:head# 2024-06-27T21:23:27.118 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e85037:::benchmark_data_smithi192_88961_object57883:head# 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e9021b:::benchmark_data_smithi192_88961_object71622:head# 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2e94b1d:::benchmark_data_smithi192_88961_object58118:head# 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2eb76f4:::benchmark_data_smithi192_88961_object57053:head# 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ecbfe2:::benchmark_data_smithi192_88961_object63319:head# 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ecce78:::benchmark_data_smithi192_88961_object88586:head# 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.119 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ecfae0:::benchmark_data_smithi192_88961_object71819:head# 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ed7391:::benchmark_data_smithi192_88961_object75770:head# 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2eda202:::benchmark_data_smithi192_88961_object77896:head# 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2ee8b01:::benchmark_data_smithi192_88961_object57408:head# 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2eec7f2:::benchmark_data_smithi192_88961_object53087:head# 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2f26658:::benchmark_data_smithi192_88961_object58479:head# 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.120 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2f37ce4:::benchmark_data_smithi192_88961_object65715:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2f70a43:::benchmark_data_smithi192_88961_object73486:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2f7905c:::benchmark_data_smithi192_88961_object61076:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2f7d265:::benchmark_data_smithi192_88961_object56988:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2f8b907:::benchmark_data_smithi192_88961_object55254:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2fa2065:::benchmark_data_smithi192_88961_object63373:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2fb5c35:::benchmark_data_smithi192_88961_object45368:head# 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.121 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2fc9d71:::benchmark_data_smithi192_88961_object84204:head# 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2fea75f:::benchmark_data_smithi192_88961_object83399:head# 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b2fee9ec:::benchmark_data_smithi192_88961_object75665:head# 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3022042:::benchmark_data_smithi192_88961_object80152:head# 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3031cce:::benchmark_data_smithi192_88961_object78607:head# 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3035353:::benchmark_data_smithi192_88961_object52114:head# 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.122 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30431dc:::benchmark_data_smithi192_88961_object87942:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b306e790:::benchmark_data_smithi192_88961_object87895:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b306ec99:::benchmark_data_smithi192_88961_object85016:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3078ce6:::benchmark_data_smithi192_88961_object48219:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b307a785:::benchmark_data_smithi192_88961_object63364:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3083d49:::benchmark_data_smithi192_88961_object74296:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b308f927:::benchmark_data_smithi192_88961_object43097:head# 2024-06-27T21:23:27.123 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30adaec:::benchmark_data_smithi192_88961_object52487:head# 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30b414a:::benchmark_data_smithi192_88961_object71166:head# 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30c17c6:::benchmark_data_smithi192_88961_object68827:head# 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30c1c22:::benchmark_data_smithi192_88961_object67221:head# 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30c516a:::benchmark_data_smithi192_88961_object86830:head# 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30ccc07:::benchmark_data_smithi192_88961_object46470:head# 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30cd534:::benchmark_data_smithi192_88961_object54030:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b30e0447:::benchmark_data_smithi192_88961_object51005:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b311459f:::benchmark_data_smithi192_88961_object73334:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b313fad7:::benchmark_data_smithi192_88961_object81309:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b315f8f9:::benchmark_data_smithi192_88961_object47633:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3174ae4:::benchmark_data_smithi192_88961_object82552:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b31768ac:::benchmark_data_smithi192_88961_object71661:head# 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.125 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b318095d:::benchmark_data_smithi192_88961_object74594:head# 2024-06-27T21:23:27.126 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.126 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3192686:::benchmark_data_smithi192_88961_object47703:head# 2024-06-27T21:23:27.126 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:23:27.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:23:27.138 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b31c2219:::benchmark_data_smithi192_88961_object80691:head# 2024-06-27T21:23:27.138 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b31c9520:::benchmark_data_smithi192_88961_object82413:head# 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b320471e:::benchmark_data_smithi192_88961_object85658:head# 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b322270f:::benchmark_data_smithi192_88961_object47758:head# 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b324bd28:::benchmark_data_smithi192_88961_object87849:head# 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b324f3e1:::benchmark_data_smithi192_88961_object71585:head# 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.139 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3255ccf:::benchmark_data_smithi192_88961_object58050:head# 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b325d033:::benchmark_data_smithi192_88961_object80457:head# 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3270e05:::benchmark_data_smithi192_88961_object89070:head# 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b327542d:::benchmark_data_smithi192_88961_object75353:head# 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b328a87c:::benchmark_data_smithi192_88961_object60322:head# 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b32a8d4b:::benchmark_data_smithi192_88961_object69413:head# 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.140 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b32c9708:::benchmark_data_smithi192_88961_object46605:head# 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b32f6a2d:::benchmark_data_smithi192_88961_object53042:head# 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b331603d:::benchmark_data_smithi192_88961_object65283:head# 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b334341f:::benchmark_data_smithi192_88961_object65248:head# 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b334f4c5:::benchmark_data_smithi192_88961_object77289:head# 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3359929:::benchmark_data_smithi192_88961_object79952:head# 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.141 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b336f1c6:::benchmark_data_smithi192_88961_object59071:head# 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b33b96f8:::benchmark_data_smithi192_88961_object65424:head# 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b33c27f3:::benchmark_data_smithi192_88961_object45913:head# 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b33d30cc:::benchmark_data_smithi192_88961_object68414:head# 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b34053c5:::benchmark_data_smithi192_88961_object67206:head# 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3412708:::benchmark_data_smithi192_88961_object80614:head# 2024-06-27T21:23:27.142 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b341a50d:::benchmark_data_smithi192_88961_object89848:head# 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b341e998:::benchmark_data_smithi192_88961_object52145:head# 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b343a44f:::benchmark_data_smithi192_88961_object55478:head# 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b344c758:::benchmark_data_smithi192_88961_object58754:head# 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b344f335:::benchmark_data_smithi192_88961_object87578:head# 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3453cec:::benchmark_data_smithi192_88961_object83729:head# 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.143 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3455d3c:::benchmark_data_smithi192_88961_object65510:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b348fe1b:::benchmark_data_smithi192_88961_object51578:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b349f9e6:::benchmark_data_smithi192_88961_object71688:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b34fad76:::benchmark_data_smithi192_88961_object74804:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b350c18a:::benchmark_data_smithi192_88961_object64287:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3510b8b:::benchmark_data_smithi192_88961_object73030:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3515545:::benchmark_data_smithi192_88961_object87041:head# 2024-06-27T21:23:27.144 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3523925:::benchmark_data_smithi192_88961_object58894:head# 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b353cb70:::benchmark_data_smithi192_88961_object45066:head# 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b35710c8:::benchmark_data_smithi192_88961_object50121:head# 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b357d4c6:::benchmark_data_smithi192_88961_object61276:head# 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b35a6654:::benchmark_data_smithi192_88961_object69467:head# 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b35b2428:::benchmark_data_smithi192_88961_object45781:head# 2024-06-27T21:23:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b35b2862:::benchmark_data_smithi192_88961_object67115:head# 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b35e2734:::benchmark_data_smithi192_88961_object60309:head# 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b35eb9e7:::benchmark_data_smithi192_88961_object77431:head# 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b36057e0:::benchmark_data_smithi192_88961_object87134:head# 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b361b6f8:::benchmark_data_smithi192_88961_object80558:head# 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3620934:::benchmark_data_smithi192_88961_object66267:head# 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.146 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b362d6a3:::benchmark_data_smithi192_88961_object56297:head# 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b365bf07:::benchmark_data_smithi192_88961_object82980:head# 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3662597:::benchmark_data_smithi192_88961_object75989:head# 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b366d503:::benchmark_data_smithi192_88961_object81631:head# 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b366e749:::benchmark_data_smithi192_88961_object86894:head# 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b36bea62:::benchmark_data_smithi192_88961_object83081:head# 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.147 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b36bf392:::benchmark_data_smithi192_88961_object65931:head# 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b36ffb9f:::benchmark_data_smithi192_88961_object72005:head# 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3700771:::benchmark_data_smithi192_88961_object58803:head# 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3707ada:::benchmark_data_smithi192_88961_object43941:head# 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b370d613:::benchmark_data_smithi192_88961_object85735:head# 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b37160a7:::benchmark_data_smithi192_88961_object66440:head# 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.148 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b371699c:::benchmark_data_smithi192_88961_object67000:head# 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3737abc:::benchmark_data_smithi192_88961_object54122:head# 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b374b9c0:::benchmark_data_smithi192_88961_object72388:head# 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3774a79:::benchmark_data_smithi192_88961_object62891:head# 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b37dcb5f:::benchmark_data_smithi192_88961_object47357:head# 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b38230d1:::benchmark_data_smithi192_88961_object45098:head# 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.149 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b382e0c5:::benchmark_data_smithi192_88961_object84019:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3862083:::benchmark_data_smithi192_88961_object44034:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3879b3c:::benchmark_data_smithi192_88961_object76314:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b389c65c:::benchmark_data_smithi192_88961_object56182:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b38b90af:::benchmark_data_smithi192_88961_object75105:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b38bcbf1:::benchmark_data_smithi192_88961_object83836:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b38df052:::benchmark_data_smithi192_88961_object75296:head# 2024-06-27T21:23:27.150 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b38fa24f:::benchmark_data_smithi192_88961_object50136:head# 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b392757d:::benchmark_data_smithi192_88961_object81904:head# 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b393ac1e:::benchmark_data_smithi192_88961_object73984:head# 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3997731:::benchmark_data_smithi192_88961_object49765:head# 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b39c370e:::benchmark_data_smithi192_88961_object74028:head# 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b39d24ef:::benchmark_data_smithi192_88961_object83863:head# 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.151 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b39d9a49:::benchmark_data_smithi192_88961_object68091:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b39df4ae:::benchmark_data_smithi192_88961_object68597:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b39e38e2:::benchmark_data_smithi192_88961_object65461:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a42665:::benchmark_data_smithi192_88961_object89606:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a52f0f:::benchmark_data_smithi192_88961_object85508:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a55ac1:::benchmark_data_smithi192_88961_object84933:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a67ffc:::benchmark_data_smithi192_88961_object48949:head# 2024-06-27T21:23:27.152 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:27.173 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a7b4b1:::benchmark_data_smithi192_88961_object69071:head# 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a7ba39:::benchmark_data_smithi192_88961_object85126:head# 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a7d460:::benchmark_data_smithi192_88961_object65658:head# 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3a83dbd:::benchmark_data_smithi192_88961_object69606:head# 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3aa3d11:::benchmark_data_smithi192_88961_object52497:head# 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3ab573f:::benchmark_data_smithi192_88961_object81998:head# 2024-06-27T21:23:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3ad9a56:::benchmark_data_smithi192_88961_object68004:head# 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3ae39d7:::benchmark_data_smithi192_88961_object57406:head# 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3ae435b:::benchmark_data_smithi192_88961_object70847:head# 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b0078c:::benchmark_data_smithi192_88961_object55293:head# 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b070a3:::benchmark_data_smithi192_88961_object87734:head# 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b34e61:::benchmark_data_smithi192_88961_object50287:head# 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.175 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b38b70:::benchmark_data_smithi192_88961_object47940:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b41a5f:::benchmark_data_smithi192_88961_object56248:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b5a5f3:::benchmark_data_smithi192_88961_object68537:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3b81e18:::benchmark_data_smithi192_88961_object61478:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3bad26a:::benchmark_data_smithi192_88961_object62553:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3bbff02:::benchmark_data_smithi192_88961_object49528:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3bc8cd3:::benchmark_data_smithi192_88961_object48627:head# 2024-06-27T21:23:27.176 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c2bb89:::benchmark_data_smithi192_88961_object58484:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c340ca:::benchmark_data_smithi192_88961_object77655:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c3a58b:::benchmark_data_smithi192_88961_object51970:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c4885e:::benchmark_data_smithi192_88961_object86762:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c51bea:::benchmark_data_smithi192_88961_object82468:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c6fe84:::benchmark_data_smithi192_88961_object53677:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c7abae:::benchmark_data_smithi192_88961_object81008:head# 2024-06-27T21:23:27.177 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c91911:::benchmark_data_smithi192_88961_object48251:head# 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c93f64:::benchmark_data_smithi192_88961_object55371:head# 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3c9433a:::benchmark_data_smithi192_88961_object55827:head# 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3ca2c6c:::benchmark_data_smithi192_88961_object75437:head# 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cb19ad:::benchmark_data_smithi192_88961_object66835:head# 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cb40fa:::benchmark_data_smithi192_88961_object75303:head# 2024-06-27T21:23:27.178 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cb58e3:::benchmark_data_smithi192_88961_object63264:head# 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cb6852:::benchmark_data_smithi192_88961_object58026:head# 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cc9903:::benchmark_data_smithi192_88961_object51135:head# 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cd1c6c:::benchmark_data_smithi192_88961_object85418:head# 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cf7603:::benchmark_data_smithi192_88961_object70752:head# 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3cf7cc4:::benchmark_data_smithi192_88961_object90201:head# 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.179 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d188b8:::benchmark_data_smithi192_88961_object76883:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d24217:::benchmark_data_smithi192_88961_object50188:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d33cbc:::benchmark_data_smithi192_88961_object53462:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d3c629:::benchmark_data_smithi192_88961_object53326:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d4795f:::benchmark_data_smithi192_88961_object76322:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d59c97:::benchmark_data_smithi192_88961_object84254:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d797c0:::benchmark_data_smithi192_88961_object56937:head# 2024-06-27T21:23:27.180 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d8ec1f:::benchmark_data_smithi192_88961_object73784:head# 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3d9fd12:::benchmark_data_smithi192_88961_object52527:head# 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3db6310:::benchmark_data_smithi192_88961_object78774:head# 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3dcf2fb:::benchmark_data_smithi192_88961_object59612:head# 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3deacee:::benchmark_data_smithi192_88961_object65282:head# 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3df086e:::benchmark_data_smithi192_88961_object88252:head# 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.181 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e0b1ff:::benchmark_data_smithi192_88961_object72808:head# 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e19950:::benchmark_data_smithi192_88961_object46845:head# 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e29c27:::benchmark_data_smithi192_88961_object53072:head# 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e3e7a3:::benchmark_data_smithi192_88961_object62132:head# 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e40842:::benchmark_data_smithi192_88961_object56790:head# 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e4d4bc:::benchmark_data_smithi192_88961_object78508:head# 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.182 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e54b67:::benchmark_data_smithi192_88961_object45725:head# 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e66641:::benchmark_data_smithi192_88961_object57659:head# 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e78543:::benchmark_data_smithi192_88961_object80639:head# 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3e8333d:::benchmark_data_smithi192_88961_object70947:head# 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3ea60ae:::benchmark_data_smithi192_88961_object87476:head# 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3eaaf67:::benchmark_data_smithi192_88961_object78042:head# 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3eb5a00:::benchmark_data_smithi192_88961_object53673:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3edf535:::benchmark_data_smithi192_88961_object51569:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3f2cc56:::benchmark_data_smithi192_88961_object52389:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3f4ae8e:::benchmark_data_smithi192_88961_object88973:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3f63454:::benchmark_data_smithi192_88961_object73387:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3f902c5:::benchmark_data_smithi192_88961_object59991:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3f9de23:::benchmark_data_smithi192_88961_object65672:head# 2024-06-27T21:23:27.184 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3fbb0ca:::benchmark_data_smithi192_88961_object47445:head# 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b3fdb89f:::benchmark_data_smithi192_88961_object90070:head# 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4008d43:::benchmark_data_smithi192_88961_object46709:head# 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b402d5fc:::benchmark_data_smithi192_88961_object60211:head# 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40642f7:::benchmark_data_smithi192_88961_object59924:head# 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b406b6b1:::benchmark_data_smithi192_88961_object58505:head# 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.185 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b406eb18:::benchmark_data_smithi192_88961_object57281:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40741e8:::benchmark_data_smithi192_88961_object66280:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40a924f:::benchmark_data_smithi192_88961_object82097:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40b65bc:::benchmark_data_smithi192_88961_object68575:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40cf3b6:::benchmark_data_smithi192_88961_object87572:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40ef503:::benchmark_data_smithi192_88961_object59123:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40f78eb:::benchmark_data_smithi192_88961_object45674:head# 2024-06-27T21:23:27.186 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b40fc315:::benchmark_data_smithi192_88961_object88867:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b41109aa:::benchmark_data_smithi192_88961_object71971:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b412e9e4:::benchmark_data_smithi192_88961_object77625:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b413ee25:::benchmark_data_smithi192_88961_object48357:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4144113:::benchmark_data_smithi192_88961_object68514:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b415ed41:::benchmark_data_smithi192_88961_object65116:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4167466:::benchmark_data_smithi192_88961_object50579:head# 2024-06-27T21:23:27.187 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b416e755:::benchmark_data_smithi192_88961_object54595:head# 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b41806a1:::benchmark_data_smithi192_88961_object43722:head# 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b41caae1:::benchmark_data_smithi192_88961_object58726:head# 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b41d5824:::benchmark_data_smithi192_88961_object50464:head# 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b41e5784:::benchmark_data_smithi192_88961_object68238:head# 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b41eeb5d:::benchmark_data_smithi192_88961_object61526:head# 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.188 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b420089c:::benchmark_data_smithi192_88961_object64096:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4206309:::benchmark_data_smithi192_88961_object53205:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42088cb:::benchmark_data_smithi192_88961_object70020:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b424582c:::benchmark_data_smithi192_88961_object85241:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4260513:::benchmark_data_smithi192_88961_object84878:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42695d8:::benchmark_data_smithi192_88961_object73252:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4277970:::benchmark_data_smithi192_88961_object57699:head# 2024-06-27T21:23:27.189 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4298ffd:::benchmark_data_smithi192_88961_object61819:head# 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b429df2d:::benchmark_data_smithi192_88961_object73675:head# 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42a3cbe:::benchmark_data_smithi192_88961_object88210:head# 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42b752b:::benchmark_data_smithi192_88961_object64939:head# 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42be6e8:::benchmark_data_smithi192_88961_object76261:head# 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42d1784:::benchmark_data_smithi192_88961_object53138:head# 2024-06-27T21:23:27.190 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b42d1fd8:::benchmark_data_smithi192_88961_object73425:head# 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43078a0:::benchmark_data_smithi192_88961_object57452:head# 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b432715c:::benchmark_data_smithi192_88961_object72758:head# 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b432ffd8:::benchmark_data_smithi192_88961_object62103:head# 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4371055:::benchmark_data_smithi192_88961_object74874:head# 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b439210b:::benchmark_data_smithi192_88961_object45272:head# 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.191 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b439c5b4:::benchmark_data_smithi192_88961_object82828:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43a0c36:::benchmark_data_smithi192_88961_object49323:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43a5b01:::benchmark_data_smithi192_88961_object64827:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43a62f1:::benchmark_data_smithi192_88961_object47326:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43c3c01:::benchmark_data_smithi192_88961_object87394:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43d6903:::benchmark_data_smithi192_88961_object58212:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43db4d7:::benchmark_data_smithi192_88961_object58345:head# 2024-06-27T21:23:27.192 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b43e6e7f:::benchmark_data_smithi192_88961_object77233:head# 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4409837:::benchmark_data_smithi192_88961_object77327:head# 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b44203da:::benchmark_data_smithi192_88961_object80582:head# 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4424054:::benchmark_data_smithi192_88961_object88721:head# 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b442451f:::benchmark_data_smithi192_88961_object82522:head# 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b444dbe3:::benchmark_data_smithi192_88961_object80310:head# 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.193 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b44646e4:::benchmark_data_smithi192_88961_object75027:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4482e30:::benchmark_data_smithi192_88961_object80744:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b449bf9b:::benchmark_data_smithi192_88961_object73926:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b449eca6:::benchmark_data_smithi192_88961_object51508:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b44a5955:::benchmark_data_smithi192_88961_object66871:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b44ac2e8:::benchmark_data_smithi192_88961_object85802:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b44c699a:::benchmark_data_smithi192_88961_object59314:head# 2024-06-27T21:23:27.194 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b44eb691:::benchmark_data_smithi192_88961_object81229:head# 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4506366:::benchmark_data_smithi192_88961_object57597:head# 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b451c4b9:::benchmark_data_smithi192_88961_object78346:head# 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4534d5d:::benchmark_data_smithi192_88961_object81138:head# 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b453f62b:::benchmark_data_smithi192_88961_object84198:head# 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b456ac9f:::benchmark_data_smithi192_88961_object79418:head# 2024-06-27T21:23:27.195 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:27.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b45855ad:::benchmark_data_smithi192_88961_object80455:head# 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b459818f:::benchmark_data_smithi192_88961_object42965:head# 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b459d171:::benchmark_data_smithi192_88961_object58773:head# 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.211 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b45a1da2:::benchmark_data_smithi192_88961_object79159:head# 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b45b7f43:::benchmark_data_smithi192_88961_object45333:head# 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b45fc96b:::benchmark_data_smithi192_88961_object71495:head# 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b462adc5:::benchmark_data_smithi192_88961_object55469:head# 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b46543d5:::benchmark_data_smithi192_88961_object61922:head# 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b466198e:::benchmark_data_smithi192_88961_object82503:head# 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.212 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4665010:::benchmark_data_smithi192_88961_object53860:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b46936b6:::benchmark_data_smithi192_88961_object59735:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b46bb098:::benchmark_data_smithi192_88961_object59131:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b46cc33e:::benchmark_data_smithi192_88961_object62213:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b46d7a9a:::benchmark_data_smithi192_88961_object86397:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b46ebc11:::benchmark_data_smithi192_88961_object77521:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4724f19:::benchmark_data_smithi192_88961_object73406:head# 2024-06-27T21:23:27.213 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b473d382:::benchmark_data_smithi192_88961_object58876:head# 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4747ad7:::benchmark_data_smithi192_88961_object52157:head# 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b47599f7:::benchmark_data_smithi192_88961_object59272:head# 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b475ab65:::benchmark_data_smithi192_88961_object74574:head# 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4774652:::benchmark_data_smithi192_88961_object76961:head# 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4775315:::benchmark_data_smithi192_88961_object67723:head# 2024-06-27T21:23:27.214 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b47915f4:::benchmark_data_smithi192_88961_object50673:head# 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b47a1f58:::benchmark_data_smithi192_88961_object56603:head# 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b47c8498:::benchmark_data_smithi192_88961_object89703:head# 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b47dbf7c:::benchmark_data_smithi192_88961_object86898:head# 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b47e2462:::benchmark_data_smithi192_88961_object60804:head# 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b48077ab:::benchmark_data_smithi192_88961_object78077:head# 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.215 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b481d0d8:::benchmark_data_smithi192_88961_object72530:head# 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b484eeb8:::benchmark_data_smithi192_88961_object90132:head# 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4853d93:::benchmark_data_smithi192_88961_object86615:head# 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4861d03:::benchmark_data_smithi192_88961_object46652:head# 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b486c844:::benchmark_data_smithi192_88961_object80214:head# 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b489d27d:::benchmark_data_smithi192_88961_object62764:head# 2024-06-27T21:23:27.216 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b48aff28:::benchmark_data_smithi192_88961_object63788:head# 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4900402:::benchmark_data_smithi192_88961_object61030:head# 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b491d455:::benchmark_data_smithi192_88961_object58107:head# 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b49b50d6:::benchmark_data_smithi192_88961_object49791:head# 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b49c8983:::benchmark_data_smithi192_88961_object89701:head# 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b49d69a7:::benchmark_data_smithi192_88961_object82310:head# 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.217 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b49d9a22:::benchmark_data_smithi192_88961_object51291:head# 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b49e314b:::benchmark_data_smithi192_88961_object44884:head# 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b49fb69f:::benchmark_data_smithi192_88961_object70827:head# 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a0e141:::benchmark_data_smithi192_88961_object57869:head# 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a2085d:::benchmark_data_smithi192_88961_object62761:head# 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a2c0ca:::benchmark_data_smithi192_88961_object81888:head# 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.218 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a40664:::benchmark_data_smithi192_88961_object49944:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a4d660:::benchmark_data_smithi192_88961_object78558:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a5eb19:::benchmark_data_smithi192_88961_object76678:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a69c40:::benchmark_data_smithi192_88961_object43876:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a6d17a:::benchmark_data_smithi192_88961_object75338:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a88bbb:::benchmark_data_smithi192_88961_object55544:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a901d1:::benchmark_data_smithi192_88961_object79169:head# 2024-06-27T21:23:27.219 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a9666e:::benchmark_data_smithi192_88961_object81755:head# 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4a9facb:::benchmark_data_smithi192_88961_object54041:head# 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4ab193d:::benchmark_data_smithi192_88961_object50259:head# 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4acbd78:::benchmark_data_smithi192_88961_object68093:head# 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b3b11b:::benchmark_data_smithi192_88961_object67287:head# 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b4876c:::benchmark_data_smithi192_88961_object75605:head# 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.220 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b5641e:::benchmark_data_smithi192_88961_object88701:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b66c66:::benchmark_data_smithi192_88961_object83207:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b6b63e:::benchmark_data_smithi192_88961_object60383:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b6c3e3:::benchmark_data_smithi192_88961_object68772:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b91468:::benchmark_data_smithi192_88961_object90656:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b91f14:::benchmark_data_smithi192_88961_object81303:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4b9a481:::benchmark_data_smithi192_88961_object75229:head# 2024-06-27T21:23:27.221 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4ba2dbd:::benchmark_data_smithi192_88961_object62606:head# 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4bca496:::benchmark_data_smithi192_88961_object64391:head# 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4bdc4f5:::benchmark_data_smithi192_88961_object49615:head# 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4be0da0:::benchmark_data_smithi192_88961_object50531:head# 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c0a587:::benchmark_data_smithi192_88961_object66477:head# 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c2a74c:::benchmark_data_smithi192_88961_object61266:head# 2024-06-27T21:23:27.222 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c2ed22:::benchmark_data_smithi192_88961_object67458:head# 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c57c3b:::benchmark_data_smithi192_88961_object46472:head# 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c5eba5:::benchmark_data_smithi192_88961_object63720:head# 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c66219:::benchmark_data_smithi192_88961_object77452:head# 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c6d71e:::benchmark_data_smithi192_88961_object85678:head# 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c7250b:::benchmark_data_smithi192_88961_object84446:head# 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.223 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c72c3f:::benchmark_data_smithi192_88961_object47959:head# 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4c8356d:::benchmark_data_smithi192_88961_object48332:head# 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4ca79f5:::benchmark_data_smithi192_88961_object63700:head# 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4cb91f3:::benchmark_data_smithi192_88961_object49451:head# 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4cc46ac:::benchmark_data_smithi192_88961_object76216:head# 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4cd7a52:::benchmark_data_smithi192_88961_object58832:head# 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.224 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4cdc222:::benchmark_data_smithi192_88961_object69323:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4cf9ec6:::benchmark_data_smithi192_88961_object44794:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d16aea:::benchmark_data_smithi192_88961_object58949:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d2010f:::benchmark_data_smithi192_88961_object60297:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d23477:::benchmark_data_smithi192_88961_object82067:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d447cb:::benchmark_data_smithi192_88961_object60737:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d4b2c5:::benchmark_data_smithi192_88961_object45675:head# 2024-06-27T21:23:27.225 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d55366:::benchmark_data_smithi192_88961_object69209:head# 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d5ee01:::benchmark_data_smithi192_88961_object45471:head# 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d61901:::benchmark_data_smithi192_88961_object43438:head# 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d84fd4:::benchmark_data_smithi192_88961_object53951:head# 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d93111:::benchmark_data_smithi192_88961_object68854:head# 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4d99741:::benchmark_data_smithi192_88961_object57033:head# 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.226 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4dabb23:::benchmark_data_smithi192_88961_object83353:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4df41e1:::benchmark_data_smithi192_88961_object62762:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4dfd639:::benchmark_data_smithi192_88961_object72503:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4e3622f:::benchmark_data_smithi192_88961_object75564:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4e498fd:::benchmark_data_smithi192_88961_object81854:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4e6c440:::benchmark_data_smithi192_88961_object53739:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4e88633:::benchmark_data_smithi192_88961_object83460:head# 2024-06-27T21:23:27.227 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4e8c9c4:::benchmark_data_smithi192_88961_object47441:head# 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4eb98c2:::benchmark_data_smithi192_88961_object52427:head# 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4ecd47a:::benchmark_data_smithi192_88961_object53440:head# 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4ee71e6:::benchmark_data_smithi192_88961_object52492:head# 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4ee7ee2:::benchmark_data_smithi192_88961_object63272:head# 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4eedf72:::benchmark_data_smithi192_88961_object60515:head# 2024-06-27T21:23:27.228 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.229 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f0ef23:::benchmark_data_smithi192_88961_object78761:head# 2024-06-27T21:23:27.229 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.229 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f172d5:::benchmark_data_smithi192_88961_object48280:head# 2024-06-27T21:23:27.229 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.229 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f3e482:::benchmark_data_smithi192_88961_object65022:head# 2024-06-27T21:23:27.229 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.239 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f410a3:::benchmark_data_smithi192_88961_object68780:head# 2024-06-27T21:23:27.239 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.239 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f49582:::benchmark_data_smithi192_88961_object78616:head# 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f6f180:::benchmark_data_smithi192_88961_object75908:head# 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f80a15:::benchmark_data_smithi192_88961_object75872:head# 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f87f3d:::benchmark_data_smithi192_88961_object49094:head# 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f8fdfd:::benchmark_data_smithi192_88961_object66162:head# 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4f99e57:::benchmark_data_smithi192_88961_object85724:head# 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.240 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4fb3648:::benchmark_data_smithi192_88961_object80271:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4fb3c4b:::benchmark_data_smithi192_88961_object75443:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4fc62c0:::benchmark_data_smithi192_88961_object45217:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4fd84cd:::benchmark_data_smithi192_88961_object80638:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b4fe956e:::benchmark_data_smithi192_88961_object69042:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5001821:::benchmark_data_smithi192_88961_object49630:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b505f941:::benchmark_data_smithi192_88961_object60848:head# 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.241 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b507ef2f:::benchmark_data_smithi192_88961_object83677:head# 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b50a44e8:::benchmark_data_smithi192_88961_object74308:head# 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b50a9886:::benchmark_data_smithi192_88961_object79465:head# 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b50b05ee:::benchmark_data_smithi192_88961_object78917:head# 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b50f3f46:::benchmark_data_smithi192_88961_object55889:head# 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b510f298:::benchmark_data_smithi192_88961_object61826:head# 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.242 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b51e1e33:::benchmark_data_smithi192_88961_object83192:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b51ea504:::benchmark_data_smithi192_88961_object71729:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b51ef08a:::benchmark_data_smithi192_88961_object48151:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b520b680:::benchmark_data_smithi192_88961_object87119:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b521f6d3:::benchmark_data_smithi192_88961_object52020:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5226c56:::benchmark_data_smithi192_88961_object83281:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b522fae0:::benchmark_data_smithi192_88961_object52044:head# 2024-06-27T21:23:27.243 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5255039:::benchmark_data_smithi192_88961_object60253:head# 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52658e2:::benchmark_data_smithi192_88961_object85399:head# 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b526eddf:::benchmark_data_smithi192_88961_object55251:head# 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b526efa0:::benchmark_data_smithi192_88961_object84754:head# 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5296894:::benchmark_data_smithi192_88961_object49493:head# 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b529cf3a:::benchmark_data_smithi192_88961_object54728:head# 2024-06-27T21:23:27.244 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52a4d34:::benchmark_data_smithi192_88961_object78704:head# 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52b0359:::benchmark_data_smithi192_88961_object69088:head# 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52b852b:::benchmark_data_smithi192_88961_object90041:head# 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52c8849:::benchmark_data_smithi192_88961_object65601:head# 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52d72c1:::benchmark_data_smithi192_88961_object58634:head# 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52d808a:::benchmark_data_smithi192_88961_object58172:head# 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.245 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b52daeac:::benchmark_data_smithi192_88961_object80646:head# 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5331f4a:::benchmark_data_smithi192_88961_object88138:head# 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5334d37:::benchmark_data_smithi192_88961_object63942:head# 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b534dab2:::benchmark_data_smithi192_88961_object84300:head# 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5352355:::benchmark_data_smithi192_88961_object62751:head# 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b535c14a:::benchmark_data_smithi192_88961_object52970:head# 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.246 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b536873f:::benchmark_data_smithi192_88961_object88980:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5376920:::benchmark_data_smithi192_88961_object68785:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b538924e:::benchmark_data_smithi192_88961_object76581:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b538b5b1:::benchmark_data_smithi192_88961_object73271:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53977de:::benchmark_data_smithi192_88961_object82840:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53b47ae:::benchmark_data_smithi192_88961_object80980:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53b93c6:::benchmark_data_smithi192_88961_object49043:head# 2024-06-27T21:23:27.247 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53bbf25:::benchmark_data_smithi192_88961_object49182:head# 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53c791f:::benchmark_data_smithi192_88961_object75243:head# 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53c7c44:::benchmark_data_smithi192_88961_object66819:head# 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53cb51c:::benchmark_data_smithi192_88961_object75844:head# 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53cd96a:::benchmark_data_smithi192_88961_object75853:head# 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53cf9cd:::benchmark_data_smithi192_88961_object69436:head# 2024-06-27T21:23:27.248 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b53d9210:::benchmark_data_smithi192_88961_object66253:head# 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b541c326:::benchmark_data_smithi192_88961_object50401:head# 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5426e9e:::benchmark_data_smithi192_88961_object57768:head# 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b542f04f:::benchmark_data_smithi192_88961_object45735:head# 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5438b9c:::benchmark_data_smithi192_88961_object47854:head# 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54415b2:::benchmark_data_smithi192_88961_object90200:head# 2024-06-27T21:23:27.249 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5448993:::benchmark_data_smithi192_88961_object68744:head# 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5454619:::benchmark_data_smithi192_88961_object60106:head# 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b546b723:::benchmark_data_smithi192_88961_object82802:head# 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5471c4a:::benchmark_data_smithi192_88961_object66699:head# 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5480026:::benchmark_data_smithi192_88961_object47964:head# 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54aa8fb:::benchmark_data_smithi192_88961_object84448:head# 2024-06-27T21:23:27.250 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54b65d0:::benchmark_data_smithi192_88961_object72958:head# 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54c6af7:::benchmark_data_smithi192_88961_object80651:head# 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54ee090:::benchmark_data_smithi192_88961_object84560:head# 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54f364e:::benchmark_data_smithi192_88961_object44785:head# 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b54fa8e6:::benchmark_data_smithi192_88961_object64452:head# 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b550250e:::benchmark_data_smithi192_88961_object45295:head# 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.251 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b551241f:::benchmark_data_smithi192_88961_object60681:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b553ad6b:::benchmark_data_smithi192_88961_object61285:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b553b381:::benchmark_data_smithi192_88961_object54572:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b553f24c:::benchmark_data_smithi192_88961_object46483:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b556bb22:::benchmark_data_smithi192_88961_object76927:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b55927e2:::benchmark_data_smithi192_88961_object75912:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b559b8c3:::benchmark_data_smithi192_88961_object77442:head# 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.252 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b55ca7f2:::benchmark_data_smithi192_88961_object48745:head# 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b55f4011:::benchmark_data_smithi192_88961_object78522:head# 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5605008:::benchmark_data_smithi192_88961_object56666:head# 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56319be:::benchmark_data_smithi192_88961_object89814:head# 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5634849:::benchmark_data_smithi192_88961_object73331:head# 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5681d12:::benchmark_data_smithi192_88961_object81420:head# 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.253 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b568d519:::benchmark_data_smithi192_88961_object47029:head# 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5692c93:::benchmark_data_smithi192_88961_object63564:head# 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b569333c:::benchmark_data_smithi192_88961_object86913:head# 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56a705c:::benchmark_data_smithi192_88961_object55693:head# 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56ad5af:::benchmark_data_smithi192_88961_object55208:head# 2024-06-27T21:23:27.254 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.272 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56b4a83:::benchmark_data_smithi192_88961_object75777:head# 2024-06-27T21:23:27.272 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.272 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56c176c:::benchmark_data_smithi192_88961_object86801:head# 2024-06-27T21:23:27.272 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.272 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56eadcb:::benchmark_data_smithi192_88961_object88146:head# 2024-06-27T21:23:27.272 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56ed0ad:::benchmark_data_smithi192_88961_object52315:head# 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56f294c:::benchmark_data_smithi192_88961_object44618:head# 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b56fbd64:::benchmark_data_smithi192_88961_object58649:head# 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5703701:::benchmark_data_smithi192_88961_object71676:head# 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b57317b9:::benchmark_data_smithi192_88961_object49689:head# 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5755838:::benchmark_data_smithi192_88961_object75298:head# 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.273 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b57583b4:::benchmark_data_smithi192_88961_object80955:head# 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b577ac68:::benchmark_data_smithi192_88961_object75592:head# 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5782654:::benchmark_data_smithi192_88961_object87520:head# 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b57ba9fa:::benchmark_data_smithi192_88961_object85590:head# 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b580236c:::benchmark_data_smithi192_88961_object45399:head# 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b581ce91:::benchmark_data_smithi192_88961_object44461:head# 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.274 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b581e5c2:::benchmark_data_smithi192_88961_object73264:head# 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b582e97b:::benchmark_data_smithi192_88961_object87201:head# 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58555f2:::benchmark_data_smithi192_88961_object60505:head# 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5867969:::benchmark_data_smithi192_88961_object73679:head# 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5870363:::benchmark_data_smithi192_88961_object48391:head# 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b587856b:::benchmark_data_smithi192_88961_object70680:head# 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.275 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b588108f:::benchmark_data_smithi192_88961_object83788:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b588f7bc:::benchmark_data_smithi192_88961_object75319:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58943bc:::benchmark_data_smithi192_88961_object74869:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5896717:::benchmark_data_smithi192_88961_object70091:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58d0241:::benchmark_data_smithi192_88961_object62588:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58e033c:::benchmark_data_smithi192_88961_object87517:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58e57f4:::benchmark_data_smithi192_88961_object52608:head# 2024-06-27T21:23:27.276 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58e588a:::benchmark_data_smithi192_88961_object74272:head# 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b58fe230:::benchmark_data_smithi192_88961_object77127:head# 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59016ef:::benchmark_data_smithi192_88961_object76253:head# 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5919ece:::benchmark_data_smithi192_88961_object47601:head# 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b591a6df:::benchmark_data_smithi192_88961_object71712:head# 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b596964a:::benchmark_data_smithi192_88961_object64338:head# 2024-06-27T21:23:27.277 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5980e9f:::benchmark_data_smithi192_88961_object49144:head# 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59a661e:::benchmark_data_smithi192_88961_object71475:head# 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59a7bfc:::benchmark_data_smithi192_88961_object66016:head# 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59bdff2:::benchmark_data_smithi192_88961_object83494:head# 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59e8ea9:::benchmark_data_smithi192_88961_object42973:head# 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59f1b8b:::benchmark_data_smithi192_88961_object81847:head# 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.278 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b59f1f2c:::benchmark_data_smithi192_88961_object82122:head# 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a168fa:::benchmark_data_smithi192_88961_object66686:head# 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a38beb:::benchmark_data_smithi192_88961_object85398:head# 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a3bad2:::benchmark_data_smithi192_88961_object86721:head# 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a44a28:::benchmark_data_smithi192_88961_object54929:head# 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a7c817:::benchmark_data_smithi192_88961_object55442:head# 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.279 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a8a52a:::benchmark_data_smithi192_88961_object53373:head# 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5a93226:::benchmark_data_smithi192_88961_object66855:head# 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ab11ad:::benchmark_data_smithi192_88961_object53600:head# 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ac5594:::benchmark_data_smithi192_88961_object63718:head# 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ad15e1:::benchmark_data_smithi192_88961_object69900:head# 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5b35e8e:::benchmark_data_smithi192_88961_object87793:head# 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.280 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5b4cd0a:::benchmark_data_smithi192_88961_object63894:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5b61ee6:::benchmark_data_smithi192_88961_object89281:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5b6b5d7:::benchmark_data_smithi192_88961_object77505:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5b9041d:::benchmark_data_smithi192_88961_object81781:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5bb912d:::benchmark_data_smithi192_88961_object63880:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5bc0f10:::benchmark_data_smithi192_88961_object50181:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5bc47cb:::benchmark_data_smithi192_88961_object59023:head# 2024-06-27T21:23:27.281 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5bd8ac8:::benchmark_data_smithi192_88961_object51460:head# 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5c4697c:::benchmark_data_smithi192_88961_object66450:head# 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5c9553f:::benchmark_data_smithi192_88961_object75729:head# 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5cd205a:::benchmark_data_smithi192_88961_object64904:head# 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5cedc37:::benchmark_data_smithi192_88961_object58567:head# 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5cf5059:::benchmark_data_smithi192_88961_object51503:head# 2024-06-27T21:23:27.282 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5d041ca:::benchmark_data_smithi192_88961_object89846:head# 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5d26909:::benchmark_data_smithi192_88961_object63206:head# 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5d79a0d:::benchmark_data_smithi192_88961_object57687:head# 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5d827ab:::benchmark_data_smithi192_88961_object73172:head# 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5d8d5c1:::benchmark_data_smithi192_88961_object82699:head# 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.283 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5d915ec:::benchmark_data_smithi192_88961_object79785:head# 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5da80d0:::benchmark_data_smithi192_88961_object50527:head# 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5db26c9:::benchmark_data_smithi192_88961_object81524:head# 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5db4afe:::benchmark_data_smithi192_88961_object71353:head# 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5db7f9b:::benchmark_data_smithi192_88961_object62632:head# 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5dd90f3:::benchmark_data_smithi192_88961_object73249:head# 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.284 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ddaeee:::benchmark_data_smithi192_88961_object61779:head# 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e00c0d:::benchmark_data_smithi192_88961_object72149:head# 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e1925c:::benchmark_data_smithi192_88961_object50408:head# 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e3e83f:::benchmark_data_smithi192_88961_object68353:head# 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e51496:::benchmark_data_smithi192_88961_object71443:head# 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e5422e:::benchmark_data_smithi192_88961_object43887:head# 2024-06-27T21:23:27.285 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e66b07:::benchmark_data_smithi192_88961_object59723:head# 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e68d89:::benchmark_data_smithi192_88961_object54969:head# 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5e87dc4:::benchmark_data_smithi192_88961_object53387:head# 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5eb4b4f:::benchmark_data_smithi192_88961_object84701:head# 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ec06f8:::benchmark_data_smithi192_88961_object55607:head# 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ee09e3:::benchmark_data_smithi192_88961_object85937:head# 2024-06-27T21:23:27.286 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5ef4d51:::benchmark_data_smithi192_88961_object69225:head# 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5efe235:::benchmark_data_smithi192_88961_object72638:head# 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5f10f48:::benchmark_data_smithi192_88961_object78669:head# 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5f18564:::benchmark_data_smithi192_88961_object63334:head# 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5f2173d:::benchmark_data_smithi192_88961_object49614:head# 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5f576e7:::benchmark_data_smithi192_88961_object67082:head# 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5f68908:::benchmark_data_smithi192_88961_object74178:head# 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5f9fe34:::benchmark_data_smithi192_88961_object84476:head# 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5fc1dad:::benchmark_data_smithi192_88961_object49518:head# 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5fdeba1:::benchmark_data_smithi192_88961_object83354:head# 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b5fe2dff:::benchmark_data_smithi192_88961_object89143:head# 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b60026c9:::benchmark_data_smithi192_88961_object81204:head# 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6004745:::benchmark_data_smithi192_88961_object62456:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6036b13:::benchmark_data_smithi192_88961_object85849:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b603bf4d:::benchmark_data_smithi192_88961_object50676:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6060bfa:::benchmark_data_smithi192_88961_object86712:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6068a35:::benchmark_data_smithi192_88961_object55302:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b607404c:::benchmark_data_smithi192_88961_object47511:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b60cd7b5:::benchmark_data_smithi192_88961_object80690:head# 2024-06-27T21:23:27.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b60e97aa:::benchmark_data_smithi192_88961_object88520:head# 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b60f21e0:::benchmark_data_smithi192_88961_object65425:head# 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b610a48c:::benchmark_data_smithi192_88961_object56766:head# 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61390c8:::benchmark_data_smithi192_88961_object83206:head# 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b613fcf7:::benchmark_data_smithi192_88961_object43323:head# 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61a093e:::benchmark_data_smithi192_88961_object63009:head# 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.290 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61c14d4:::benchmark_data_smithi192_88961_object79498:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61cb3a5:::benchmark_data_smithi192_88961_object51309:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61e97a9:::benchmark_data_smithi192_88961_object83067:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61ebb12:::benchmark_data_smithi192_88961_object54123:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b61f8c8c:::benchmark_data_smithi192_88961_object45886:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b622cb6c:::benchmark_data_smithi192_88961_object79475:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b622d273:::benchmark_data_smithi192_88961_object83168:head# 2024-06-27T21:23:27.291 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b622f55e:::benchmark_data_smithi192_88961_object78730:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6250b67:::benchmark_data_smithi192_88961_object47686:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6251526:::benchmark_data_smithi192_88961_object65543:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6258c3f:::benchmark_data_smithi192_88961_object67911:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b625a9f0:::benchmark_data_smithi192_88961_object88336:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6270e37:::benchmark_data_smithi192_88961_object58675:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b627460a:::benchmark_data_smithi192_88961_object87174:head# 2024-06-27T21:23:27.292 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.293 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b627636e:::benchmark_data_smithi192_88961_object70307:head# 2024-06-27T21:23:27.293 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6289a0b:::benchmark_data_smithi192_88961_object76743:head# 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b62abfae:::benchmark_data_smithi192_88961_object56789:head# 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b62ae722:::benchmark_data_smithi192_88961_object58776:head# 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b62e4487:::benchmark_data_smithi192_88961_object43565:head# 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b62f74a5:::benchmark_data_smithi192_88961_object63066:head# 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.308 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b631bdb9:::benchmark_data_smithi192_88961_object65962:head# 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b631c34c:::benchmark_data_smithi192_88961_object90254:head# 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b631f04d:::benchmark_data_smithi192_88961_object90679:head# 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6325f24:::benchmark_data_smithi192_88961_object55860:head# 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6329650:::benchmark_data_smithi192_88961_object55062:head# 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6364a25:::benchmark_data_smithi192_88961_object80506:head# 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.309 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6370243:::benchmark_data_smithi192_88961_object57327:head# 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63aa41e:::benchmark_data_smithi192_88961_object84688:head# 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63bc387:::benchmark_data_smithi192_88961_object86594:head# 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63d2ed3:::benchmark_data_smithi192_88961_object54624:head# 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63d4713:::benchmark_data_smithi192_88961_object87693:head# 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63ea525:::benchmark_data_smithi192_88961_object70597:head# 2024-06-27T21:23:27.310 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63ebc91:::benchmark_data_smithi192_88961_object51081:head# 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63ee422:::benchmark_data_smithi192_88961_object65086:head# 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b63f8046:::benchmark_data_smithi192_88961_object87081:head# 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6406790:::benchmark_data_smithi192_88961_object49683:head# 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6458e76:::benchmark_data_smithi192_88961_object81775:head# 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b646d2f0:::benchmark_data_smithi192_88961_object64985:head# 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b64829fe:::benchmark_data_smithi192_88961_object43037:head# 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b64aa763:::benchmark_data_smithi192_88961_object57553:head# 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b64ca395:::benchmark_data_smithi192_88961_object54836:head# 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b64d094a:::benchmark_data_smithi192_88961_object43674:head# 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b64e7b5d:::benchmark_data_smithi192_88961_object62758:head# 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.312 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b64fdfbc:::benchmark_data_smithi192_88961_object68784:head# 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6508dbe:::benchmark_data_smithi192_88961_object66279:head# 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b650fe33:::benchmark_data_smithi192_88961_object64369:head# 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65634a9:::benchmark_data_smithi192_88961_object47242:head# 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65651ef:::benchmark_data_smithi192_88961_object58518:head# 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65a8fea:::benchmark_data_smithi192_88961_object76651:head# 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.313 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65b5bb3:::benchmark_data_smithi192_88961_object75701:head# 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65ca8fe:::benchmark_data_smithi192_88961_object45666:head# 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65d18d1:::benchmark_data_smithi192_88961_object43877:head# 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65d44d3:::benchmark_data_smithi192_88961_object64502:head# 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65d5258:::benchmark_data_smithi192_88961_object59354:head# 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65dca54:::benchmark_data_smithi192_88961_object75197:head# 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.314 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65e8ca3:::benchmark_data_smithi192_88961_object59144:head# 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b65f2081:::benchmark_data_smithi192_88961_object78973:head# 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6601ce4:::benchmark_data_smithi192_88961_object56048:head# 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6605830:::benchmark_data_smithi192_88961_object59788:head# 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b66101ee:::benchmark_data_smithi192_88961_object61456:head# 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b663ca78:::benchmark_data_smithi192_88961_object70763:head# 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.315 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6658c84:::benchmark_data_smithi192_88961_object89513:head# 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b665a1af:::benchmark_data_smithi192_88961_object63346:head# 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b665ccc8:::benchmark_data_smithi192_88961_object84846:head# 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b667441a:::benchmark_data_smithi192_88961_object81469:head# 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b668cd5c:::benchmark_data_smithi192_88961_object85976:head# 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b668f1aa:::benchmark_data_smithi192_88961_object79207:head# 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.316 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6698348:::benchmark_data_smithi192_88961_object59970:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b66f8093:::benchmark_data_smithi192_88961_object45300:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b672bfdf:::benchmark_data_smithi192_88961_object46147:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6730b89:::benchmark_data_smithi192_88961_object73328:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67439a2:::benchmark_data_smithi192_88961_object50745:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67473aa:::benchmark_data_smithi192_88961_object61137:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b676288b:::benchmark_data_smithi192_88961_object84017:head# 2024-06-27T21:23:27.317 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67678d9:::benchmark_data_smithi192_88961_object86808:head# 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b676b1d4:::benchmark_data_smithi192_88961_object57140:head# 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67797f4:::benchmark_data_smithi192_88961_object48540:head# 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67a38f5:::benchmark_data_smithi192_88961_object89263:head# 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67dd177:::benchmark_data_smithi192_88961_object43385:head# 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67e6423:::benchmark_data_smithi192_88961_object44096:head# 2024-06-27T21:23:27.318 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b67f9365:::benchmark_data_smithi192_88961_object71184:head# 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6805fe6:::benchmark_data_smithi192_88961_object88403:head# 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6819963:::benchmark_data_smithi192_88961_object60926:head# 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6825fe1:::benchmark_data_smithi192_88961_object68437:head# 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b686c95e:::benchmark_data_smithi192_88961_object78814:head# 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b687deac:::benchmark_data_smithi192_88961_object86716:head# 2024-06-27T21:23:27.319 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b689a5dc:::benchmark_data_smithi192_88961_object57538:head# 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b68a9c3c:::benchmark_data_smithi192_88961_object59189:head# 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b68cf7c7:::benchmark_data_smithi192_88961_object90039:head# 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b68fdc7b:::benchmark_data_smithi192_88961_object79004:head# 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b690a1f1:::benchmark_data_smithi192_88961_object48802:head# 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b690aa47:::benchmark_data_smithi192_88961_object51943:head# 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.320 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b691a89c:::benchmark_data_smithi192_88961_object56582:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6929b65:::benchmark_data_smithi192_88961_object82869:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b693dfb9:::benchmark_data_smithi192_88961_object76115:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b694b6a6:::benchmark_data_smithi192_88961_object49171:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b694d16a:::benchmark_data_smithi192_88961_object89615:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69752ad:::benchmark_data_smithi192_88961_object46644:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b697c178:::benchmark_data_smithi192_88961_object83896:head# 2024-06-27T21:23:27.321 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69803d9:::benchmark_data_smithi192_88961_object63979:head# 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69a1faa:::benchmark_data_smithi192_88961_object76395:head# 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69c67b6:::benchmark_data_smithi192_88961_object70378:head# 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69cd768:::benchmark_data_smithi192_88961_object84726:head# 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69f3135:::benchmark_data_smithi192_88961_object63871:head# 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69f5ce3:::benchmark_data_smithi192_88961_object58780:head# 2024-06-27T21:23:27.322 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b69fcd21:::benchmark_data_smithi192_88961_object58150:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a46598:::benchmark_data_smithi192_88961_object43464:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a6015d:::benchmark_data_smithi192_88961_object72706:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a606e5:::benchmark_data_smithi192_88961_object83339:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a61ae0:::benchmark_data_smithi192_88961_object82762:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a6f2cf:::benchmark_data_smithi192_88961_object52295:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a723f0:::benchmark_data_smithi192_88961_object56427:head# 2024-06-27T21:23:27.323 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a7f0fd:::benchmark_data_smithi192_88961_object87851:head# 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6a90d63:::benchmark_data_smithi192_88961_object49095:head# 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6aada4d:::benchmark_data_smithi192_88961_object59522:head# 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6ac7dfc:::benchmark_data_smithi192_88961_object61655:head# 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b04c41:::benchmark_data_smithi192_88961_object70420:head# 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b23c66:::benchmark_data_smithi192_88961_object54553:head# 2024-06-27T21:23:27.324 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b23ff6:::benchmark_data_smithi192_88961_object73722:head# 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b4a2e4:::benchmark_data_smithi192_88961_object87955:head# 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b5122e:::benchmark_data_smithi192_88961_object44195:head# 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b5b0ac:::benchmark_data_smithi192_88961_object83621:head# 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b7c014:::benchmark_data_smithi192_88961_object75808:head# 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6b8ede2:::benchmark_data_smithi192_88961_object65481:head# 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.325 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6ba644f:::benchmark_data_smithi192_88961_object78202:head# 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6bb2b43:::benchmark_data_smithi192_88961_object80057:head# 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6bd0268:::benchmark_data_smithi192_88961_object69961:head# 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6beb63e:::benchmark_data_smithi192_88961_object88627:head# 2024-06-27T21:23:27.326 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:27.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:27.344 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6bf1fab:::benchmark_data_smithi192_88961_object77527:head# 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6bfb12b:::benchmark_data_smithi192_88961_object60019:head# 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c1944c:::benchmark_data_smithi192_88961_object50261:head# 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c2de8f:::benchmark_data_smithi192_88961_object79949:head# 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c49a64:::benchmark_data_smithi192_88961_object90625:head# 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c4d8c8:::benchmark_data_smithi192_88961_object71589:head# 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.345 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c4da84:::benchmark_data_smithi192_88961_object71237:head# 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c59d6d:::benchmark_data_smithi192_88961_object69877:head# 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c77981:::benchmark_data_smithi192_88961_object83957:head# 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c7a5b2:::benchmark_data_smithi192_88961_object60675:head# 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c8028d:::benchmark_data_smithi192_88961_object77368:head# 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6c82f17:::benchmark_data_smithi192_88961_object83363:head# 2024-06-27T21:23:27.346 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6cb82bd:::benchmark_data_smithi192_88961_object70429:head# 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6cbe570:::benchmark_data_smithi192_88961_object60209:head# 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6cbf49a:::benchmark_data_smithi192_88961_object75051:head# 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6cd03d2:::benchmark_data_smithi192_88961_object52794:head# 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6cd2308:::benchmark_data_smithi192_88961_object70613:head# 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6ce0b75:::benchmark_data_smithi192_88961_object59282:head# 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.347 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6d01f0c:::benchmark_data_smithi192_88961_object68629:head# 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6d0e177:::benchmark_data_smithi192_88961_object46855:head# 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6d2353e:::benchmark_data_smithi192_88961_object69316:head# 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6d60c12:::benchmark_data_smithi192_88961_object78543:head# 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6d74f52:::benchmark_data_smithi192_88961_object80694:head# 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6d8795a:::benchmark_data_smithi192_88961_object84516:head# 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.348 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6dbf552:::benchmark_data_smithi192_88961_object80079:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6dd9400:::benchmark_data_smithi192_88961_object50473:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6dfb172:::benchmark_data_smithi192_88961_object85980:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6e185f5:::benchmark_data_smithi192_88961_object73017:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6e267c9:::benchmark_data_smithi192_88961_object44183:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6e2d0f5:::benchmark_data_smithi192_88961_object63379:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6e63f1f:::benchmark_data_smithi192_88961_object83240:head# 2024-06-27T21:23:27.349 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6e84745:::benchmark_data_smithi192_88961_object63029:head# 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6e8bc7d:::benchmark_data_smithi192_88961_object77623:head# 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6eb0589:::benchmark_data_smithi192_88961_object62074:head# 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6eb990e:::benchmark_data_smithi192_88961_object78769:head# 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6eebd34:::benchmark_data_smithi192_88961_object62294:head# 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f23c00:::benchmark_data_smithi192_88961_object73465:head# 2024-06-27T21:23:27.350 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f2c81d:::benchmark_data_smithi192_88961_object85035:head# 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f2f67e:::benchmark_data_smithi192_88961_object87464:head# 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f322c0:::benchmark_data_smithi192_88961_object46403:head# 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f43dd5:::benchmark_data_smithi192_88961_object67316:head# 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f76b52:::benchmark_data_smithi192_88961_object87152:head# 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6f8bab9:::benchmark_data_smithi192_88961_object60951:head# 2024-06-27T21:23:27.351 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6fbf4b8:::benchmark_data_smithi192_88961_object66159:head# 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6fc6747:::benchmark_data_smithi192_88961_object80992:head# 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6fc7398:::benchmark_data_smithi192_88961_object89339:head# 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6fd748b:::benchmark_data_smithi192_88961_object81968:head# 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6ff7a5b:::benchmark_data_smithi192_88961_object65540:head# 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b6ff8b6a:::benchmark_data_smithi192_88961_object54204:head# 2024-06-27T21:23:27.352 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b700ad0f:::benchmark_data_smithi192_88961_object46218:head# 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b700c975:::benchmark_data_smithi192_88961_object72113:head# 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7014924:::benchmark_data_smithi192_88961_object84422:head# 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7020058:::benchmark_data_smithi192_88961_object75179:head# 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b70235c5:::benchmark_data_smithi192_88961_object52502:head# 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b702eedc:::benchmark_data_smithi192_88961_object63005:head# 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.353 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b703ca8d:::benchmark_data_smithi192_88961_object72832:head# 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7045576:::benchmark_data_smithi192_88961_object50033:head# 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b70d2bf5:::benchmark_data_smithi192_88961_object77238:head# 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b70e0c2a:::benchmark_data_smithi192_88961_object48513:head# 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b70ee090:::benchmark_data_smithi192_88961_object79926:head# 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7130a85:::benchmark_data_smithi192_88961_object48853:head# 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.354 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b71389b7:::benchmark_data_smithi192_88961_object55557:head# 2024-06-27T21:23:27.355 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.355 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7177aa6:::benchmark_data_smithi192_88961_object62560:head# 2024-06-27T21:23:27.357 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b718a494:::benchmark_data_smithi192_88961_object48067:head# 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b71910ae:::benchmark_data_smithi192_88961_object51662:head# 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b719427a:::benchmark_data_smithi192_88961_object65205:head# 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b719ec01:::benchmark_data_smithi192_88961_object85061:head# 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b71c664e:::benchmark_data_smithi192_88961_object72157:head# 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.358 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b71cbe85:::benchmark_data_smithi192_88961_object50804:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b71dc7f2:::benchmark_data_smithi192_88961_object66473:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7201894:::benchmark_data_smithi192_88961_object82666:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7207bba:::benchmark_data_smithi192_88961_object73645:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7211fa7:::benchmark_data_smithi192_88961_object62337:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b724573a:::benchmark_data_smithi192_88961_object82575:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b725fd72:::benchmark_data_smithi192_88961_object83346:head# 2024-06-27T21:23:27.359 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7274ac5:::benchmark_data_smithi192_88961_object84462:head# 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b728824c:::benchmark_data_smithi192_88961_object73656:head# 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7295ba5:::benchmark_data_smithi192_88961_object49686:head# 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b72c0141:::benchmark_data_smithi192_88961_object61338:head# 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b72c2c4a:::benchmark_data_smithi192_88961_object45212:head# 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b72c2ecc:::benchmark_data_smithi192_88961_object82457:head# 2024-06-27T21:23:27.360 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b72c754e:::benchmark_data_smithi192_88961_object77650:head# 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b72d0060:::benchmark_data_smithi192_88961_object67787:head# 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b72f3374:::benchmark_data_smithi192_88961_object68369:head# 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b733331c:::benchmark_data_smithi192_88961_object55670:head# 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b734dd69:::benchmark_data_smithi192_88961_object65024:head# 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b736635f:::benchmark_data_smithi192_88961_object65364:head# 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.361 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b738a558:::benchmark_data_smithi192_88961_object88418:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b738b4b1:::benchmark_data_smithi192_88961_object58864:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b73a6ec8:::benchmark_data_smithi192_88961_object43741:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b73c56b2:::benchmark_data_smithi192_88961_object60636:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b73c9484:::benchmark_data_smithi192_88961_object77030:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b73cfdde:::benchmark_data_smithi192_88961_object63419:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b73e131f:::benchmark_data_smithi192_88961_object83158:head# 2024-06-27T21:23:27.362 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b741ece0:::benchmark_data_smithi192_88961_object87583:head# 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b742be32:::benchmark_data_smithi192_88961_object45040:head# 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7440a26:::benchmark_data_smithi192_88961_object79907:head# 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b744435a:::benchmark_data_smithi192_88961_object50482:head# 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b74606a8:::benchmark_data_smithi192_88961_object89092:head# 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b746a648:::benchmark_data_smithi192_88961_object49662:head# 2024-06-27T21:23:27.363 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b746be58:::benchmark_data_smithi192_88961_object54338:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7481316:::benchmark_data_smithi192_88961_object56000:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b748c9ad:::benchmark_data_smithi192_88961_object70474:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7498daf:::benchmark_data_smithi192_88961_object79274:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b749c4d0:::benchmark_data_smithi192_88961_object62210:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b74a4155:::benchmark_data_smithi192_88961_object65413:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b74c6a13:::benchmark_data_smithi192_88961_object65340:head# 2024-06-27T21:23:27.364 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b74cbb64:::benchmark_data_smithi192_88961_object82835:head# 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7505aaf:::benchmark_data_smithi192_88961_object68539:head# 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b750afd1:::benchmark_data_smithi192_88961_object70780:head# 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b754a57f:::benchmark_data_smithi192_88961_object70788:head# 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b754abdc:::benchmark_data_smithi192_88961_object52737:head# 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b75598bc:::benchmark_data_smithi192_88961_object88456:head# 2024-06-27T21:23:27.365 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7561b63:::benchmark_data_smithi192_88961_object50924:head# 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b75ab834:::benchmark_data_smithi192_88961_object59849:head# 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b75bc7bc:::benchmark_data_smithi192_88961_object56362:head# 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b75bd1f3:::benchmark_data_smithi192_88961_object66101:head# 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b75c694d:::benchmark_data_smithi192_88961_object66040:head# 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b75dbbbe:::benchmark_data_smithi192_88961_object44504:head# 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.366 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7615173:::benchmark_data_smithi192_88961_object80937:head# 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b76376c2:::benchmark_data_smithi192_88961_object64030:head# 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7699694:::benchmark_data_smithi192_88961_object85888:head# 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b76a8921:::benchmark_data_smithi192_88961_object60798:head# 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b76b5d73:::benchmark_data_smithi192_88961_object87161:head# 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b771086e:::benchmark_data_smithi192_88961_object50852:head# 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.367 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7742d3d:::benchmark_data_smithi192_88961_object77926:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77683eb:::benchmark_data_smithi192_88961_object81643:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7773363:::benchmark_data_smithi192_88961_object46070:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b777df05:::benchmark_data_smithi192_88961_object52731:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77804c5:::benchmark_data_smithi192_88961_object62102:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b778a992:::benchmark_data_smithi192_88961_object89584:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7790487:::benchmark_data_smithi192_88961_object46714:head# 2024-06-27T21:23:27.368 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.369 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77947a6:::benchmark_data_smithi192_88961_object43023:head# 2024-06-27T21:23:27.369 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.369 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77abcce:::benchmark_data_smithi192_88961_object74241:head# 2024-06-27T21:23:27.369 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.369 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77b30fa:::benchmark_data_smithi192_88961_object79143:head# 2024-06-27T21:23:27.369 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.384 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77c9fcc:::benchmark_data_smithi192_88961_object81895:head# 2024-06-27T21:23:27.384 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77cf000:::benchmark_data_smithi192_88961_object57132:head# 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77cf7bc:::benchmark_data_smithi192_88961_object86385:head# 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77fa32d:::benchmark_data_smithi192_88961_object87396:head# 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77feaf1:::benchmark_data_smithi192_88961_object71154:head# 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b77fee99:::benchmark_data_smithi192_88961_object44426:head# 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b78236a0:::benchmark_data_smithi192_88961_object67596:head# 2024-06-27T21:23:27.385 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b782c960:::benchmark_data_smithi192_88961_object45436:head# 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7835fac:::benchmark_data_smithi192_88961_object62228:head# 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7855dc0:::benchmark_data_smithi192_88961_object70459:head# 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b785bc74:::benchmark_data_smithi192_88961_object47282:head# 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b786b2de:::benchmark_data_smithi192_88961_object47459:head# 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b788e52d:::benchmark_data_smithi192_88961_object68133:head# 2024-06-27T21:23:27.386 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b78b93f9:::benchmark_data_smithi192_88961_object78575:head# 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b78bae93:::benchmark_data_smithi192_88961_object89748:head# 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b78c721c:::benchmark_data_smithi192_88961_object72418:head# 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b78f3e81:::benchmark_data_smithi192_88961_object90400:head# 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b78ffc63:::benchmark_data_smithi192_88961_object73318:head# 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b790372e:::benchmark_data_smithi192_88961_object43499:head# 2024-06-27T21:23:27.387 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b79102c1:::benchmark_data_smithi192_88961_object52785:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b79197a5:::benchmark_data_smithi192_88961_object76790:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b792998d:::benchmark_data_smithi192_88961_object86611:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7935703:::benchmark_data_smithi192_88961_object73479:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b793c11d:::benchmark_data_smithi192_88961_object81746:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7972306:::benchmark_data_smithi192_88961_object78840:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b798d2b3:::benchmark_data_smithi192_88961_object59398:head# 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.388 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7998dde:::benchmark_data_smithi192_88961_object74357:head# 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b79ad291:::benchmark_data_smithi192_88961_object54220:head# 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b79cc2f6:::benchmark_data_smithi192_88961_object81654:head# 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b79d829c:::benchmark_data_smithi192_88961_object88406:head# 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7a07a8e:::benchmark_data_smithi192_88961_object87291:head# 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7a0d0b3:::benchmark_data_smithi192_88961_object70900:head# 2024-06-27T21:23:27.389 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7a3a5b2:::benchmark_data_smithi192_88961_object87612:head# 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7a90f22:::benchmark_data_smithi192_88961_object48044:head# 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7aa01f3:::benchmark_data_smithi192_88961_object79104:head# 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7aa1f67:::benchmark_data_smithi192_88961_object72506:head# 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7acd029:::benchmark_data_smithi192_88961_object46480:head# 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ae1df6:::benchmark_data_smithi192_88961_object51683:head# 2024-06-27T21:23:27.390 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ae39a3:::benchmark_data_smithi192_88961_object51490:head# 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ae3d62:::benchmark_data_smithi192_88961_object81440:head# 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b07f0d:::benchmark_data_smithi192_88961_object48455:head# 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b15578:::benchmark_data_smithi192_88961_object60480:head# 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b18a72:::benchmark_data_smithi192_88961_object49651:head# 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b3a7d1:::benchmark_data_smithi192_88961_object73430:head# 2024-06-27T21:23:27.391 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b4812b:::benchmark_data_smithi192_88961_object76655:head# 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b4fcb3:::benchmark_data_smithi192_88961_object52216:head# 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b51bbe:::benchmark_data_smithi192_88961_object46498:head# 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b5623f:::benchmark_data_smithi192_88961_object88967:head# 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b56e49:::benchmark_data_smithi192_88961_object75450:head# 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b61ceb:::benchmark_data_smithi192_88961_object57617:head# 2024-06-27T21:23:27.392 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b78367:::benchmark_data_smithi192_88961_object54637:head# 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b82df0:::benchmark_data_smithi192_88961_object68610:head# 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b89e88:::benchmark_data_smithi192_88961_object47875:head# 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7b95743:::benchmark_data_smithi192_88961_object67193:head# 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ba131b:::benchmark_data_smithi192_88961_object88731:head# 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.393 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7bc3e8a:::benchmark_data_smithi192_88961_object79177:head# 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7bdb28f:::benchmark_data_smithi192_88961_object84510:head# 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7bdc26f:::benchmark_data_smithi192_88961_object90214:head# 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7bdf096:::benchmark_data_smithi192_88961_object71777:head# 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7be808f:::benchmark_data_smithi192_88961_object73767:head# 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c01fd2:::benchmark_data_smithi192_88961_object76466:head# 2024-06-27T21:23:27.394 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c166e8:::benchmark_data_smithi192_88961_object54424:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c172d7:::benchmark_data_smithi192_88961_object48100:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c19b9c:::benchmark_data_smithi192_88961_object56707:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c3a7e4:::benchmark_data_smithi192_88961_object87012:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c4687e:::benchmark_data_smithi192_88961_object83358:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7c70c8c:::benchmark_data_smithi192_88961_object60018:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7cc7230:::benchmark_data_smithi192_88961_object58983:head# 2024-06-27T21:23:27.395 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7cfade6:::benchmark_data_smithi192_88961_object86675:head# 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7cfc5c4:::benchmark_data_smithi192_88961_object72488:head# 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d0a812:::benchmark_data_smithi192_88961_object68803:head# 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d1da73:::benchmark_data_smithi192_88961_object73167:head# 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d21ab8:::benchmark_data_smithi192_88961_object66284:head# 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d24867:::benchmark_data_smithi192_88961_object45405:head# 2024-06-27T21:23:27.396 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d42812:::benchmark_data_smithi192_88961_object77584:head# 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d49771:::benchmark_data_smithi192_88961_object46281:head# 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d6fef4:::benchmark_data_smithi192_88961_object64451:head# 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d71efa:::benchmark_data_smithi192_88961_object51355:head# 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d74eb3:::benchmark_data_smithi192_88961_object47993:head# 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d7f2f9:::benchmark_data_smithi192_88961_object44676:head# 2024-06-27T21:23:27.397 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7d9c0cd:::benchmark_data_smithi192_88961_object78290:head# 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7da4a91:::benchmark_data_smithi192_88961_object83312:head# 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7daafe8:::benchmark_data_smithi192_88961_object52883:head# 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7de3424:::benchmark_data_smithi192_88961_object60603:head# 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e12cda:::benchmark_data_smithi192_88961_object50627:head# 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.398 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e2377f:::benchmark_data_smithi192_88961_object67102:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e3d6e6:::benchmark_data_smithi192_88961_object53459:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e4301a:::benchmark_data_smithi192_88961_object53164:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e57b16:::benchmark_data_smithi192_88961_object67134:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e913fc:::benchmark_data_smithi192_88961_object83288:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7e9f80e:::benchmark_data_smithi192_88961_object53830:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ea40c1:::benchmark_data_smithi192_88961_object77600:head# 2024-06-27T21:23:27.399 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ec65a0:::benchmark_data_smithi192_88961_object76585:head# 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ed2188:::benchmark_data_smithi192_88961_object84722:head# 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ee4842:::benchmark_data_smithi192_88961_object81644:head# 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7efdd13:::benchmark_data_smithi192_88961_object45711:head# 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f0a630:::benchmark_data_smithi192_88961_object88966:head# 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f28009:::benchmark_data_smithi192_88961_object45343:head# 2024-06-27T21:23:27.400 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f4a1e5:::benchmark_data_smithi192_88961_object62087:head# 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f608bb:::benchmark_data_smithi192_88961_object66419:head# 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f6d1c3:::benchmark_data_smithi192_88961_object71119:head# 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f7ced2:::benchmark_data_smithi192_88961_object65198:head# 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7f9b5b4:::benchmark_data_smithi192_88961_object74131:head# 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7fd06b5:::benchmark_data_smithi192_88961_object86877:head# 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.401 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7fd7729:::benchmark_data_smithi192_88961_object47904:head# 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7fe7862:::benchmark_data_smithi192_88961_object85577:head# 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b7ffe51f:::benchmark_data_smithi192_88961_object52159:head# 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b801687b:::benchmark_data_smithi192_88961_object53783:head# 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b802cad6:::benchmark_data_smithi192_88961_object51393:head# 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8036d78:::benchmark_data_smithi192_88961_object63685:head# 2024-06-27T21:23:27.402 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8071999:::benchmark_data_smithi192_88961_object65416:head# 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b807e1dc:::benchmark_data_smithi192_88961_object61205:head# 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b809a232:::benchmark_data_smithi192_88961_object48979:head# 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b809ae59:::benchmark_data_smithi192_88961_object83415:head# 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b809d2b0:::benchmark_data_smithi192_88961_object55830:head# 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b80a43c2:::benchmark_data_smithi192_88961_object59448:head# 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.403 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b80eba65:::benchmark_data_smithi192_88961_object58731:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b80ee5c8:::benchmark_data_smithi192_88961_object60196:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b80f0ebb:::benchmark_data_smithi192_88961_object60403:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b81038b7:::benchmark_data_smithi192_88961_object78823:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8123154:::benchmark_data_smithi192_88961_object77321:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8124233:::benchmark_data_smithi192_88961_object50968:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b81250c2:::benchmark_data_smithi192_88961_object78824:head# 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.404 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b813dbee:::benchmark_data_smithi192_88961_object45987:head# 2024-06-27T21:23:27.405 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.419 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b815aea2:::benchmark_data_smithi192_88961_object67676:head# 2024-06-27T21:23:27.419 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.419 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b817260e:::benchmark_data_smithi192_88961_object66210:head# 2024-06-27T21:23:27.419 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b81a4002:::benchmark_data_smithi192_88961_object43764:head# 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b81c4550:::benchmark_data_smithi192_88961_object62747:head# 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b81d41a9:::benchmark_data_smithi192_88961_object87534:head# 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b81e3a6e:::benchmark_data_smithi192_88961_object89068:head# 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8200055:::benchmark_data_smithi192_88961_object44871:head# 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8214ffc:::benchmark_data_smithi192_88961_object68130:head# 2024-06-27T21:23:27.420 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82197f3:::benchmark_data_smithi192_88961_object45933:head# 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b822f8fb:::benchmark_data_smithi192_88961_object83152:head# 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8230c94:::benchmark_data_smithi192_88961_object73935:head# 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82510ed:::benchmark_data_smithi192_88961_object44959:head# 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82663b5:::benchmark_data_smithi192_88961_object67626:head# 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b828b72c:::benchmark_data_smithi192_88961_object70427:head# 2024-06-27T21:23:27.421 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b828ea40:::benchmark_data_smithi192_88961_object62993:head# 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8295785:::benchmark_data_smithi192_88961_object90380:head# 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82a36cc:::benchmark_data_smithi192_88961_object67197:head# 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82a6d49:::benchmark_data_smithi192_88961_object65819:head# 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82e65da:::benchmark_data_smithi192_88961_object61749:head# 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.422 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b82f2ae4:::benchmark_data_smithi192_88961_object46356:head# 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8318ab2:::benchmark_data_smithi192_88961_object79853:head# 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b836f15f:::benchmark_data_smithi192_88961_object46170:head# 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b83855a9:::benchmark_data_smithi192_88961_object67177:head# 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b83a4f8d:::benchmark_data_smithi192_88961_object78916:head# 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b83ac178:::benchmark_data_smithi192_88961_object85586:head# 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.423 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b83c6581:::benchmark_data_smithi192_88961_object78201:head# 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b83d74b2:::benchmark_data_smithi192_88961_object53277:head# 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b83d9582:::benchmark_data_smithi192_88961_object46575:head# 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8402895:::benchmark_data_smithi192_88961_object83792:head# 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8414a0b:::benchmark_data_smithi192_88961_object72688:head# 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8423c29:::benchmark_data_smithi192_88961_object86714:head# 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.424 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b84357eb:::benchmark_data_smithi192_88961_object45193:head# 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8439cc3:::benchmark_data_smithi192_88961_object68137:head# 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8465c9c:::benchmark_data_smithi192_88961_object43769:head# 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b84693b2:::benchmark_data_smithi192_88961_object69743:head# 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b847d589:::benchmark_data_smithi192_88961_object48894:head# 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8480ea6:::benchmark_data_smithi192_88961_object76574:head# 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.425 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b84d3a40:::benchmark_data_smithi192_88961_object50169:head# 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b84da017:::benchmark_data_smithi192_88961_object48213:head# 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b84e9206:::benchmark_data_smithi192_88961_object61033:head# 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8504a9f:::benchmark_data_smithi192_88961_object81321:head# 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b851be47:::benchmark_data_smithi192_88961_object60198:head# 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b852237f:::benchmark_data_smithi192_88961_object76352:head# 2024-06-27T21:23:27.426 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b852ccb1:::benchmark_data_smithi192_88961_object47488:head# 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8531f66:::benchmark_data_smithi192_88961_object51785:head# 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b853b060:::benchmark_data_smithi192_88961_object55674:head# 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8540741:::benchmark_data_smithi192_88961_object81901:head# 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8541367:::benchmark_data_smithi192_88961_object65763:head# 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8553135:::benchmark_data_smithi192_88961_object68701:head# 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.427 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8564ca9:::benchmark_data_smithi192_88961_object65627:head# 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b856ad88:::benchmark_data_smithi192_88961_object72302:head# 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8594544:::benchmark_data_smithi192_88961_object73773:head# 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8599bbc:::benchmark_data_smithi192_88961_object52907:head# 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b85a06a5:::benchmark_data_smithi192_88961_object44178:head# 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b85b3fa8:::benchmark_data_smithi192_88961_object50485:head# 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.428 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b85ce39f:::benchmark_data_smithi192_88961_object82006:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8604fa6:::benchmark_data_smithi192_88961_object44886:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b860ea61:::benchmark_data_smithi192_88961_object69752:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8615e71:::benchmark_last_metadata:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b862894a:::benchmark_data_smithi192_88961_object81224:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8647723:::benchmark_data_smithi192_88961_object84816:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8673606:::benchmark_data_smithi192_88961_object68240:head# 2024-06-27T21:23:27.429 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b86876ea:::benchmark_data_smithi192_88961_object66315:head# 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b868efae:::benchmark_data_smithi192_88961_object84248:head# 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b86988db:::benchmark_data_smithi192_88961_object55076:head# 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b86c5546:::benchmark_data_smithi192_88961_object54946:head# 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b86cc07c:::benchmark_data_smithi192_88961_object50288:head# 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.430 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b86cca07:::benchmark_data_smithi192_88961_object70981:head# 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87282f4:::benchmark_data_smithi192_88961_object87727:head# 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b872b512:::benchmark_data_smithi192_88961_object79694:head# 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b873531b:::benchmark_data_smithi192_88961_object79299:head# 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8735c60:::benchmark_data_smithi192_88961_object83196:head# 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b873c342:::benchmark_data_smithi192_88961_object72614:head# 2024-06-27T21:23:27.431 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8740295:::benchmark_data_smithi192_88961_object86159:head# 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b877b681:::benchmark_data_smithi192_88961_object52228:head# 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b878c1f8:::benchmark_data_smithi192_88961_object64247:head# 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b879ccee:::benchmark_data_smithi192_88961_object77403:head# 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87a44f4:::benchmark_data_smithi192_88961_object68036:head# 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87b02a6:::benchmark_data_smithi192_88961_object66060:head# 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.432 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87b3942:::benchmark_data_smithi192_88961_object88800:head# 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87d968f:::benchmark_data_smithi192_88961_object72858:head# 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87df77e:::benchmark_data_smithi192_88961_object51950:head# 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87ec49a:::benchmark_data_smithi192_88961_object64771:head# 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b87fda5a:::benchmark_data_smithi192_88961_object84205:head# 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b880c406:::benchmark_data_smithi192_88961_object65780:head# 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.433 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b880f5fb:::benchmark_data_smithi192_88961_object67547:head# 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b882ecd6:::benchmark_data_smithi192_88961_object88068:head# 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b883a906:::benchmark_data_smithi192_88961_object73760:head# 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b885086c:::benchmark_data_smithi192_88961_object78340:head# 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8893b45:::benchmark_data_smithi192_88961_object80918:head# 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b88aafea:::benchmark_data_smithi192_88961_object44254:head# 2024-06-27T21:23:27.434 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b88ac095:::benchmark_data_smithi192_88961_object44811:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b88c4b2b:::benchmark_data_smithi192_88961_object76794:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b88c955e:::benchmark_data_smithi192_88961_object83810:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b88d38c1:::benchmark_data_smithi192_88961_object48439:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b890b240:::benchmark_data_smithi192_88961_object56198:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b890d11f:::benchmark_data_smithi192_88961_object51757:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b892d813:::benchmark_data_smithi192_88961_object45488:head# 2024-06-27T21:23:27.435 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8980f92:::benchmark_data_smithi192_88961_object51388:head# 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8989eef:::benchmark_data_smithi192_88961_object65390:head# 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8997640:::benchmark_data_smithi192_88961_object50686:head# 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b89a1498:::benchmark_data_smithi192_88961_object69177:head# 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b89a1d58:::benchmark_data_smithi192_88961_object78437:head# 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b89b1c02:::benchmark_data_smithi192_88961_object51590:head# 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.436 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b89d8d20:::benchmark_data_smithi192_88961_object55684:head# 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b89f09cf:::benchmark_data_smithi192_88961_object66483:head# 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b89fcb58:::benchmark_data_smithi192_88961_object86936:head# 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8a14b8a:::benchmark_data_smithi192_88961_object75900:head# 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8a2b568:::benchmark_data_smithi192_88961_object68373:head# 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8a5622b:::benchmark_data_smithi192_88961_object82563:head# 2024-06-27T21:23:27.437 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8a7e9b7:::benchmark_data_smithi192_88961_object86268:head# 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8a8b36f:::benchmark_data_smithi192_88961_object87843:head# 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8a9326e:::benchmark_data_smithi192_88961_object50752:head# 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8ab3c7c:::benchmark_data_smithi192_88961_object57111:head# 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.449 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8ac78f0:::benchmark_data_smithi192_88961_object75898:head# 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8ae0eef:::benchmark_data_smithi192_88961_object90343:head# 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8aeef9e:::benchmark_data_smithi192_88961_object69168:head# 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b0b9e6:::benchmark_data_smithi192_88961_object56218:head# 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b0d489:::benchmark_data_smithi192_88961_object58909:head# 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b15aa5:::benchmark_data_smithi192_88961_object85135:head# 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.450 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b16752:::benchmark_data_smithi192_88961_object43762:head# 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b27f19:::benchmark_data_smithi192_88961_object64671:head# 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b423c7:::benchmark_data_smithi192_88961_object79688:head# 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b52f07:::benchmark_data_smithi192_88961_object81540:head# 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8b750a8:::benchmark_data_smithi192_88961_object76588:head# 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8bbb4fc:::benchmark_data_smithi192_88961_object47547:head# 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.451 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8bbdc99:::benchmark_data_smithi192_88961_object77119:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8bd52b0:::benchmark_data_smithi192_88961_object48302:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8bd866f:::benchmark_data_smithi192_88961_object53480:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8c23d66:::benchmark_data_smithi192_88961_object67521:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8c2bc8f:::benchmark_data_smithi192_88961_object82090:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8c4def5:::benchmark_data_smithi192_88961_object51494:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8c550c0:::benchmark_data_smithi192_88961_object75009:head# 2024-06-27T21:23:27.452 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8c7b6fc:::benchmark_data_smithi192_88961_object49278:head# 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8c7fc7a:::benchmark_data_smithi192_88961_object57037:head# 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8caf73b:::benchmark_data_smithi192_88961_object74450:head# 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8cd75c9:::benchmark_data_smithi192_88961_object51962:head# 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d0b594:::benchmark_data_smithi192_88961_object50165:head# 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d13655:::benchmark_data_smithi192_88961_object52750:head# 2024-06-27T21:23:27.453 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d1b8bb:::benchmark_data_smithi192_88961_object71383:head# 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d1c01e:::benchmark_data_smithi192_88961_object54751:head# 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d1c5c2:::benchmark_data_smithi192_88961_object71481:head# 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d27b67:::benchmark_data_smithi192_88961_object80452:head# 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d7a494:::benchmark_data_smithi192_88961_object58311:head# 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.454 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8d870c6:::benchmark_data_smithi192_88961_object46183:head# 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8db140c:::benchmark_data_smithi192_88961_object64561:head# 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8db4e06:::benchmark_data_smithi192_88961_object84486:head# 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8dedb84:::benchmark_data_smithi192_88961_object55303:head# 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8e009b6:::benchmark_data_smithi192_88961_object83316:head# 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8e2753e:::benchmark_data_smithi192_88961_object56347:head# 2024-06-27T21:23:27.455 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8e43200:::benchmark_data_smithi192_88961_object80664:head# 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8e651ba:::benchmark_data_smithi192_88961_object52862:head# 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8e6f7a2:::benchmark_data_smithi192_88961_object86112:head# 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8eb263d:::benchmark_data_smithi192_88961_object83381:head# 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8eb5c4b:::benchmark_data_smithi192_88961_object52898:head# 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8ed7c11:::benchmark_data_smithi192_88961_object55338:head# 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8edbd52:::benchmark_data_smithi192_88961_object58946:head# 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8ee1170:::benchmark_data_smithi192_88961_object45311:head# 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8eead08:::benchmark_data_smithi192_88961_object70138:head# 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8f15296:::benchmark_data_smithi192_88961_object56504:head# 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8f6d086:::benchmark_data_smithi192_88961_object85057:head# 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.457 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8f8157e:::benchmark_data_smithi192_88961_object82297:head# 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8f89b70:::benchmark_data_smithi192_88961_object80859:head# 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8f9f457:::benchmark_data_smithi192_88961_object80288:head# 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8fa6151:::benchmark_data_smithi192_88961_object88514:head# 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8fa8bf5:::benchmark_data_smithi192_88961_object76399:head# 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b8fb19ec:::benchmark_data_smithi192_88961_object72315:head# 2024-06-27T21:23:27.458 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b902c49b:::benchmark_data_smithi192_88961_object62396:head# 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b903a531:::benchmark_data_smithi192_88961_object61024:head# 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b903c2b7:::benchmark_data_smithi192_88961_object55799:head# 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b906a862:::benchmark_data_smithi192_88961_object59575:head# 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9073805:::benchmark_data_smithi192_88961_object45326:head# 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9074bc9:::benchmark_data_smithi192_88961_object90186:head# 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.459 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9085d1a:::benchmark_data_smithi192_88961_object79780:head# 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b908cf32:::benchmark_data_smithi192_88961_object72869:head# 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9093b30:::benchmark_data_smithi192_88961_object65867:head# 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b909493f:::benchmark_data_smithi192_88961_object64629:head# 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b90b1947:::benchmark_data_smithi192_88961_object69895:head# 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b90b5802:::benchmark_data_smithi192_88961_object77425:head# 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.460 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b90e1dbf:::benchmark_data_smithi192_88961_object76538:head# 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b90ffb18:::benchmark_data_smithi192_88961_object82177:head# 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9105f78:::benchmark_data_smithi192_88961_object49076:head# 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9108362:::benchmark_data_smithi192_88961_object87676:head# 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91142d2:::benchmark_data_smithi192_88961_object57297:head# 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b914eb55:::benchmark_data_smithi192_88961_object69270:head# 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.461 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9186d49:::benchmark_data_smithi192_88961_object61639:head# 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91ba582:::benchmark_data_smithi192_88961_object57570:head# 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91ba7e5:::benchmark_data_smithi192_88961_object65280:head# 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91d0053:::benchmark_data_smithi192_88961_object90613:head# 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91d34b0:::benchmark_data_smithi192_88961_object77078:head# 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91d9696:::benchmark_data_smithi192_88961_object82361:head# 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.462 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b91f2b5d:::benchmark_data_smithi192_88961_object84427:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9202abe:::benchmark_data_smithi192_88961_object45969:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b92258b7:::benchmark_data_smithi192_88961_object82736:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9234bb0:::benchmark_data_smithi192_88961_object90029:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b923b033:::benchmark_data_smithi192_88961_object59268:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b924ba6c:::benchmark_data_smithi192_88961_object52032:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9266534:::benchmark_data_smithi192_88961_object49428:head# 2024-06-27T21:23:27.463 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9267067:::benchmark_data_smithi192_88961_object74928:head# 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b928c0ee:::benchmark_data_smithi192_88961_object89206:head# 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b92d936a:::benchmark_data_smithi192_88961_object64205:head# 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b92f01a2:::benchmark_data_smithi192_88961_object84172:head# 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.472 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b92ffade:::benchmark_data_smithi192_88961_object76855:head# 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9337603:::benchmark_data_smithi192_88961_object60155:head# 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9339f3e:::benchmark_data_smithi192_88961_object79713:head# 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b934e7e2:::benchmark_data_smithi192_88961_object65362:head# 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9372e72:::benchmark_data_smithi192_88961_object87299:head# 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9380d51:::benchmark_data_smithi192_88961_object67466:head# 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.473 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9381383:::benchmark_data_smithi192_88961_object48437:head# 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b938936a:::benchmark_data_smithi192_88961_object61079:head# 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b93b634c:::benchmark_data_smithi192_88961_object85476:head# 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b93bf00e:::benchmark_data_smithi192_88961_object90646:head# 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b943c417:::benchmark_data_smithi192_88961_object78697:head# 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b943e2be:::benchmark_data_smithi192_88961_object82697:head# 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.474 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b943f118:::benchmark_data_smithi192_88961_object85788:head# 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9456b82:::benchmark_data_smithi192_88961_object77565:head# 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b945c97f:::benchmark_data_smithi192_88961_object80475:head# 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b94ab563:::benchmark_data_smithi192_88961_object54503:head# 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b94bc138:::benchmark_data_smithi192_88961_object49289:head# 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b94dff16:::benchmark_data_smithi192_88961_object59552:head# 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.475 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b94e2e02:::benchmark_data_smithi192_88961_object87659:head# 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b94e4686:::benchmark_data_smithi192_88961_object54165:head# 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b94e6d62:::benchmark_data_smithi192_88961_object52504:head# 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9505dd5:::benchmark_data_smithi192_88961_object72060:head# 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b950be96:::benchmark_data_smithi192_88961_object44779:head# 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9510b79:::benchmark_data_smithi192_88961_object61781:head# 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.476 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9527cf7:::benchmark_data_smithi192_88961_object86030:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b954e5d0:::benchmark_data_smithi192_88961_object82022:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9553f29:::benchmark_data_smithi192_88961_object56701:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b95be180:::benchmark_data_smithi192_88961_object87940:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b95cb431:::benchmark_data_smithi192_88961_object47399:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b95d4bc3:::benchmark_data_smithi192_88961_object49280:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b95df2e0:::benchmark_data_smithi192_88961_object50806:head# 2024-06-27T21:23:27.477 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b95f0fec:::benchmark_data_smithi192_88961_object69795:head# 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b961b071:::benchmark_data_smithi192_88961_object74341:head# 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9635748:::benchmark_data_smithi192_88961_object44244:head# 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96432d8:::benchmark_data_smithi192_88961_object60102:head# 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b965c1bc:::benchmark_data_smithi192_88961_object68450:head# 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.478 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9663e19:::benchmark_data_smithi192_88961_object73419:head# 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9667bdf:::benchmark_data_smithi192_88961_object60640:head# 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b969b363:::benchmark_data_smithi192_88961_object55802:head# 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96a4daf:::benchmark_data_smithi192_88961_object54070:head# 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96bffc7:::benchmark_data_smithi192_88961_object73418:head# 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.479 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96d45b6:::benchmark_data_smithi192_88961_object45890:head# 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96d5744:::benchmark_data_smithi192_88961_object86717:head# 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96e54af:::benchmark_data_smithi192_88961_object50542:head# 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b96f67bf:::benchmark_data_smithi192_88961_object85965:head# 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b971128e:::benchmark_data_smithi192_88961_object61252:head# 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9726128:::benchmark_data_smithi192_88961_object70438:head# 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b973621b:::benchmark_data_smithi192_88961_object72196:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9762d04:::benchmark_data_smithi192_88961_object52481:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9772026:::benchmark_data_smithi192_88961_object71697:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9775f8b:::benchmark_data_smithi192_88961_object85632:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b977d018:::benchmark_data_smithi192_88961_object65936:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b97a4706:::benchmark_data_smithi192_88961_object57032:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b97c72d8:::benchmark_data_smithi192_88961_object50276:head# 2024-06-27T21:23:27.481 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b97cfc67:::benchmark_data_smithi192_88961_object76225:head# 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b97daf75:::benchmark_data_smithi192_88961_object66300:head# 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b97f0c3f:::benchmark_data_smithi192_88961_object43133:head# 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9821bba:::benchmark_data_smithi192_88961_object49548:head# 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9841a29:::benchmark_data_smithi192_88961_object46807:head# 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b985a263:::benchmark_data_smithi192_88961_object67790:head# 2024-06-27T21:23:27.482 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9884e88:::benchmark_data_smithi192_88961_object89424:head# 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b989d7b9:::benchmark_data_smithi192_88961_object67155:head# 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b98bc02c:::benchmark_data_smithi192_88961_object46389:head# 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b98c9bc0:::benchmark_data_smithi192_88961_object47159:head# 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b98d0a49:::benchmark_data_smithi192_88961_object71919:head# 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99190c0:::benchmark_data_smithi192_88961_object55014:head# 2024-06-27T21:23:27.483 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9926549:::benchmark_data_smithi192_88961_object52144:head# 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b994eebf:::benchmark_data_smithi192_88961_object55828:head# 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9961ba7:::benchmark_data_smithi192_88961_object50646:head# 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99a1f72:::benchmark_data_smithi192_88961_object47309:head# 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99c1e1b:::benchmark_data_smithi192_88961_object82193:head# 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99d860c:::benchmark_data_smithi192_88961_object62880:head# 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.484 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99deef8:::benchmark_data_smithi192_88961_object89717:head# 2024-06-27T21:23:27.485 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.486 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99ec9a7:::benchmark_data_smithi192_88961_object77882:head# 2024-06-27T21:23:27.486 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.486 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b99f43f1:::benchmark_data_smithi192_88961_object56370:head# 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9a34df5:::benchmark_data_smithi192_88961_object62931:head# 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9a46cc1:::benchmark_data_smithi192_88961_object83631:head# 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9a50050:::benchmark_data_smithi192_88961_object52238:head# 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9a51590:::benchmark_data_smithi192_88961_object62025:head# 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9a67338:::benchmark_data_smithi192_88961_object72778:head# 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.487 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9a780d8:::benchmark_data_smithi192_88961_object56557:head# 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9ab3fc5:::benchmark_data_smithi192_88961_object69247:head# 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9adb02f:::benchmark_data_smithi192_88961_object76099:head# 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9adc995:::benchmark_data_smithi192_88961_object54087:head# 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9ae0aee:::benchmark_data_smithi192_88961_object66379:head# 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b1aa52:::benchmark_data_smithi192_88961_object63756:head# 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.488 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b31747:::benchmark_data_smithi192_88961_object50708:head# 2024-06-27T21:23:27.489 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.489 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b3817c:::benchmark_data_smithi192_88961_object45546:head# 2024-06-27T21:23:27.489 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:27.490 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b4d075:::benchmark_data_smithi192_88961_object50476:head# 2024-06-27T21:23:27.490 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.490 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b65b49:::benchmark_data_smithi192_88961_object75714:head# 2024-06-27T21:23:27.490 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:27.500 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b6b9a6:::benchmark_data_smithi192_88961_object76180:head# 2024-06-27T21:23:27.500 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.500 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b7e8e0:::benchmark_data_smithi192_88961_object65445:head# 2024-06-27T21:23:27.500 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b91b42:::benchmark_data_smithi192_88961_object48336:head# 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b946c6:::benchmark_data_smithi192_88961_object47443:head# 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9b969e3:::benchmark_data_smithi192_88961_object79779:head# 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9bb1213:::benchmark_data_smithi192_88961_object66642:head# 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9bb140a:::benchmark_data_smithi192_88961_object79112:head# 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.501 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9bc8491:::benchmark_data_smithi192_88961_object59372:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9be3217:::benchmark_data_smithi192_88961_object56260:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9be4eb1:::benchmark_data_smithi192_88961_object72368:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c0f86c:::benchmark_data_smithi192_88961_object81779:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c2dc6f:::benchmark_data_smithi192_88961_object65414:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c3b0ef:::benchmark_data_smithi192_88961_object86634:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c6a591:::benchmark_data_smithi192_88961_object57456:head# 2024-06-27T21:23:27.502 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c6fd17:::benchmark_data_smithi192_88961_object65264:head# 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c7ae75:::benchmark_data_smithi192_88961_object90037:head# 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c859f7:::benchmark_data_smithi192_88961_object59286:head# 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9c8f2c8:::benchmark_data_smithi192_88961_object72367:head# 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9cac8c5:::benchmark_data_smithi192_88961_object79786:head# 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9cb708a:::benchmark_data_smithi192_88961_object50576:head# 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.503 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9cbf62c:::benchmark_data_smithi192_88961_object45327:head# 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9ccae14:::benchmark_data_smithi192_88961_object76335:head# 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9cf86e6:::benchmark_data_smithi192_88961_object67434:head# 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9cfd8ad:::benchmark_data_smithi192_88961_object75123:head# 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9d01e54:::benchmark_data_smithi192_88961_object60983:head# 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9d13948:::benchmark_data_smithi192_88961_object66655:head# 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.504 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9d60f57:::benchmark_data_smithi192_88961_object80930:head# 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9d7321e:::benchmark_data_smithi192_88961_object60282:head# 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9ddd625:::benchmark_data_smithi192_88961_object48238:head# 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9df9674:::benchmark_data_smithi192_88961_object62812:head# 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e044f2:::benchmark_data_smithi192_88961_object81047:head# 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e0f6bb:::benchmark_data_smithi192_88961_object44367:head# 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.505 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e2445c:::benchmark_data_smithi192_88961_object44384:head# 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e2c5e6:::benchmark_data_smithi192_88961_object70229:head# 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e5b57b:::benchmark_data_smithi192_88961_object90182:head# 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e6d5b9:::benchmark_data_smithi192_88961_object51990:head# 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9e9866f:::benchmark_data_smithi192_88961_object67375:head# 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9ee81a0:::benchmark_data_smithi192_88961_object54375:head# 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.506 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f03454:::benchmark_data_smithi192_88961_object89013:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f209a9:::benchmark_data_smithi192_88961_object60705:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f27d60:::benchmark_data_smithi192_88961_object89098:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f2e4fa:::benchmark_data_smithi192_88961_object82576:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f41272:::benchmark_data_smithi192_88961_object86771:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f46f18:::benchmark_data_smithi192_88961_object50768:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f549aa:::benchmark_data_smithi192_88961_object72039:head# 2024-06-27T21:23:27.507 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f623b1:::benchmark_data_smithi192_88961_object52314:head# 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f708e4:::benchmark_data_smithi192_88961_object86015:head# 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9f74513:::benchmark_data_smithi192_88961_object81005:head# 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9faa516:::benchmark_data_smithi192_88961_object48955:head# 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9faa955:::benchmark_data_smithi192_88961_object82807:head# 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9fde356:::benchmark_data_smithi192_88961_object71794:head# 2024-06-27T21:23:27.508 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:b9ffc5e5:::benchmark_data_smithi192_88961_object43025:head# 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba005285:::benchmark_data_smithi192_88961_object55899:head# 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba02aadc:::benchmark_data_smithi192_88961_object88426:head# 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba0437eb:::benchmark_data_smithi192_88961_object88217:head# 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba04e076:::benchmark_data_smithi192_88961_object46534:head# 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba05da2b:::benchmark_data_smithi192_88961_object69683:head# 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.509 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba06e31b:::benchmark_data_smithi192_88961_object51527:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba073a95:::benchmark_data_smithi192_88961_object46202:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba08107f:::benchmark_data_smithi192_88961_object73633:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba0ae05b:::benchmark_data_smithi192_88961_object77542:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba0c2deb:::benchmark_data_smithi192_88961_object60973:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba0d1950:::benchmark_data_smithi192_88961_object74979:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba13d211:::benchmark_data_smithi192_88961_object47963:head# 2024-06-27T21:23:27.510 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba14b746:::benchmark_data_smithi192_88961_object58385:head# 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba14cee1:::benchmark_data_smithi192_88961_object61693:head# 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba15faa4:::benchmark_data_smithi192_88961_object53289:head# 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba161d1f:::benchmark_data_smithi192_88961_object59145:head# 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba17ed8e:::benchmark_data_smithi192_88961_object74694:head# 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba1868b4:::benchmark_data_smithi192_88961_object68198:head# 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.511 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba1a0a84:::benchmark_data_smithi192_88961_object67201:head# 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba1a5e61:::benchmark_data_smithi192_88961_object57964:head# 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba1b48e3:::benchmark_data_smithi192_88961_object46810:head# 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba1eaab7:::benchmark_data_smithi192_88961_object84188:head# 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba216337:::benchmark_data_smithi192_88961_object89101:head# 2024-06-27T21:23:27.512 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.519 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba22f2ee:::benchmark_data_smithi192_88961_object66380:head# 2024-06-27T21:23:27.519 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.519 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba24cbcf:::benchmark_data_smithi192_88961_object71751:head# 2024-06-27T21:23:27.519 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.519 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2502b6:::benchmark_data_smithi192_88961_object66744:head# 2024-06-27T21:23:27.519 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba283a13:::benchmark_data_smithi192_88961_object45477:head# 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2846ee:::benchmark_data_smithi192_88961_object76361:head# 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba28ccbc:::benchmark_data_smithi192_88961_object84147:head# 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba298094:::benchmark_data_smithi192_88961_object70673:head# 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2a31f1:::benchmark_data_smithi192_88961_object81088:head# 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2b00e7:::benchmark_data_smithi192_88961_object49281:head# 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.520 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2c0982:::benchmark_data_smithi192_88961_object48446:head# 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2dd4d5:::benchmark_data_smithi192_88961_object87376:head# 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba2ebd84:::benchmark_data_smithi192_88961_object45127:head# 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3072d1:::benchmark_data_smithi192_88961_object82664:head# 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3301e0:::benchmark_data_smithi192_88961_object59944:head# 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3358dd:::benchmark_data_smithi192_88961_object43241:head# 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.521 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba34ee05:::benchmark_data_smithi192_88961_object60379:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba35c462:::benchmark_data_smithi192_88961_object58742:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba36c98f:::benchmark_data_smithi192_88961_object56460:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba37157f:::benchmark_data_smithi192_88961_object69037:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3774e2:::benchmark_data_smithi192_88961_object81410:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba38b1f6:::benchmark_data_smithi192_88961_object84614:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3add6d:::benchmark_data_smithi192_88961_object50509:head# 2024-06-27T21:23:27.522 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3d50bc:::benchmark_data_smithi192_88961_object60321:head# 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba3f7abc:::benchmark_data_smithi192_88961_object76298:head# 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4068de:::benchmark_data_smithi192_88961_object58429:head# 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba438835:::benchmark_data_smithi192_88961_object54372:head# 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba45e3b5:::benchmark_data_smithi192_88961_object47735:head# 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba46521e:::benchmark_data_smithi192_88961_object76994:head# 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.523 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4697f7:::benchmark_data_smithi192_88961_object64134:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba46eabf:::benchmark_data_smithi192_88961_object84161:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba48407b:::benchmark_data_smithi192_88961_object84298:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba48aaa7:::benchmark_data_smithi192_88961_object74866:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba492b09:::benchmark_data_smithi192_88961_object62463:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4942ab:::benchmark_data_smithi192_88961_object68870:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4a016d:::benchmark_data_smithi192_88961_object71089:head# 2024-06-27T21:23:27.524 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4b3bf1:::benchmark_data_smithi192_88961_object45527:head# 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4b59bf:::benchmark_data_smithi192_88961_object81757:head# 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4b6dd0:::benchmark_data_smithi192_88961_object76155:head# 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4e6cca:::benchmark_data_smithi192_88961_object81956:head# 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba4f698d:::benchmark_data_smithi192_88961_object62972:head# 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba501b34:::benchmark_data_smithi192_88961_object90404:head# 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.525 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba50c038:::benchmark_data_smithi192_88961_object45203:head# 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba50ef17:::benchmark_data_smithi192_88961_object72820:head# 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba52d16a:::benchmark_data_smithi192_88961_object47031:head# 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba54ab4e:::benchmark_data_smithi192_88961_object50426:head# 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba554c31:::benchmark_data_smithi192_88961_object76378:head# 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba576dfc:::benchmark_data_smithi192_88961_object89387:head# 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.526 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba595558:::benchmark_data_smithi192_88961_object76838:head# 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba5bdaec:::benchmark_data_smithi192_88961_object48816:head# 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba5ca36e:::benchmark_data_smithi192_88961_object69418:head# 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba5e4b4d:::benchmark_data_smithi192_88961_object43116:head# 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba61906f:::benchmark_data_smithi192_88961_object84053:head# 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba637918:::benchmark_data_smithi192_88961_object85721:head# 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.527 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba66e577:::benchmark_data_smithi192_88961_object88213:head# 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba67fda4:::benchmark_data_smithi192_88961_object76028:head# 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba6ac071:::benchmark_data_smithi192_88961_object63710:head# 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba6bb220:::benchmark_data_smithi192_88961_object65933:head# 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba6f9861:::benchmark_data_smithi192_88961_object62958:head# 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba74c177:::benchmark_data_smithi192_88961_object47778:head# 2024-06-27T21:23:27.528 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba74ee8f:::benchmark_data_smithi192_88961_object89977:head# 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba76db83:::benchmark_data_smithi192_88961_object61387:head# 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba77c716:::benchmark_data_smithi192_88961_object86083:head# 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba7b08c1:::benchmark_data_smithi192_88961_object53419:head# 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.537 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba7c7b96:::benchmark_data_smithi192_88961_object69428:head# 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba7e5dd6:::benchmark_data_smithi192_88961_object46087:head# 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba7f5b41:::benchmark_data_smithi192_88961_object79667:head# 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba80e1e9:::benchmark_data_smithi192_88961_object85997:head# 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba8157cf:::benchmark_data_smithi192_88961_object72281:head# 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba818d28:::benchmark_data_smithi192_88961_object71799:head# 2024-06-27T21:23:27.538 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba81d4ef:::benchmark_data_smithi192_88961_object62520:head# 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba81ddf0:::benchmark_data_smithi192_88961_object79922:head# 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba81eddb:::benchmark_data_smithi192_88961_object84501:head# 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba8256db:::benchmark_data_smithi192_88961_object59492:head# 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba845bd5:::benchmark_data_smithi192_88961_object46108:head# 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba85dca5:::benchmark_data_smithi192_88961_object79582:head# 2024-06-27T21:23:27.539 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba866a64:::benchmark_data_smithi192_88961_object90269:head# 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba8985c9:::benchmark_data_smithi192_88961_object58561:head# 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba89961a:::benchmark_data_smithi192_88961_object83478:head# 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba89ce7d:::benchmark_data_smithi192_88961_object87384:head# 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba8a3a83:::benchmark_data_smithi192_88961_object54035:head# 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba8ddc1e:::benchmark_data_smithi192_88961_object65869:head# 2024-06-27T21:23:27.540 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba9147dd:::benchmark_data_smithi192_88961_object69942:head# 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba9172ab:::benchmark_data_smithi192_88961_object51725:head# 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba929103:::benchmark_data_smithi192_88961_object65908:head# 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba932543:::benchmark_data_smithi192_88961_object44829:head# 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba94f94c:::benchmark_data_smithi192_88961_object60241:head# 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba970cd6:::benchmark_data_smithi192_88961_object74678:head# 2024-06-27T21:23:27.541 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba99742a:::benchmark_data_smithi192_88961_object47139:head# 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba9ac536:::benchmark_data_smithi192_88961_object44730:head# 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba9bec95:::benchmark_data_smithi192_88961_object73728:head# 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:ba9ea348:::benchmark_data_smithi192_88961_object85013:head# 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa0ce56:::benchmark_data_smithi192_88961_object46604:head# 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa1dca6:::benchmark_data_smithi192_88961_object59582:head# 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.542 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa265cc:::benchmark_data_smithi192_88961_object57249:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa3219b:::benchmark_data_smithi192_88961_object54571:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa3a69e:::benchmark_data_smithi192_88961_object77951:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa5962d:::benchmark_data_smithi192_88961_object66633:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa6a6d9:::benchmark_data_smithi192_88961_object78738:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa748d2:::benchmark_data_smithi192_88961_object70275:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa7e1ce:::benchmark_data_smithi192_88961_object84779:head# 2024-06-27T21:23:27.543 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baa97ff8:::benchmark_data_smithi192_88961_object85596:head# 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baaa5229:::benchmark_data_smithi192_88961_object48155:head# 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baac74b0:::benchmark_data_smithi192_88961_object43157:head# 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baaca298:::benchmark_data_smithi192_88961_object79355:head# 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baadf722:::benchmark_data_smithi192_88961_object51176:head# 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab058bd:::benchmark_data_smithi192_88961_object47404:head# 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.544 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab333a0:::benchmark_data_smithi192_88961_object48174:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab33e71:::benchmark_data_smithi192_88961_object73690:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab37bec:::benchmark_data_smithi192_88961_object79913:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab3ebe7:::benchmark_data_smithi192_88961_object74221:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab4489a:::benchmark_data_smithi192_88961_object85123:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab7f872:::benchmark_data_smithi192_88961_object78910:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bab8877d:::benchmark_data_smithi192_88961_object49644:head# 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.545 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:babaf580:::benchmark_data_smithi192_88961_object56277:head# 2024-06-27T21:23:27.546 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.546 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:babb97e6:::benchmark_data_smithi192_88961_object79987:head# 2024-06-27T21:23:27.546 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.552 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:babd5288:::benchmark_data_smithi192_88961_object71559:head# 2024-06-27T21:23:27.552 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.552 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac03983:::benchmark_data_smithi192_88961_object69694:head# 2024-06-27T21:23:27.552 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac12523:::benchmark_data_smithi192_88961_object50229:head# 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac1df33:::benchmark_data_smithi192_88961_object85480:head# 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac4298b:::benchmark_data_smithi192_88961_object87577:head# 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac4ea1b:::benchmark_data_smithi192_88961_object59129:head# 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac55a30:::benchmark_data_smithi192_88961_object76700:head# 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac5e37a:::benchmark_data_smithi192_88961_object74756:head# 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.553 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac612cb:::benchmark_data_smithi192_88961_object71874:head# 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac69683:::benchmark_data_smithi192_88961_object54126:head# 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac72758:::benchmark_data_smithi192_88961_object66167:head# 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac76958:::benchmark_data_smithi192_88961_object52253:head# 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bac81b22:::benchmark_data_smithi192_88961_object88902:head# 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bacbc825:::benchmark_data_smithi192_88961_object48159:head# 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bacd1aa3:::benchmark_data_smithi192_88961_object55007:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bace35fa:::benchmark_data_smithi192_88961_object54523:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad2e891:::benchmark_data_smithi192_88961_object75251:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad44c60:::benchmark_data_smithi192_88961_object63844:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad4bc81:::benchmark_data_smithi192_88961_object44964:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad55f7f:::benchmark_data_smithi192_88961_object50452:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad5b567:::benchmark_data_smithi192_88961_object56531:head# 2024-06-27T21:23:27.555 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad6621f:::benchmark_data_smithi192_88961_object64184:head# 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad6df76:::benchmark_data_smithi192_88961_object72004:head# 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad72c9c:::benchmark_data_smithi192_88961_object44199:head# 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bad83b10:::benchmark_data_smithi192_88961_object75352:head# 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:badc9e67:::benchmark_data_smithi192_88961_object69865:head# 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baddd0a8:::benchmark_data_smithi192_88961_object89171:head# 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.556 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:badeb8e2:::benchmark_data_smithi192_88961_object69574:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bae02167:::benchmark_data_smithi192_88961_object69885:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bae2e1fd:::benchmark_data_smithi192_88961_object86846:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bae2f055:::benchmark_data_smithi192_88961_object87246:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bae5002b:::benchmark_data_smithi192_88961_object50951:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bae80500:::benchmark_data_smithi192_88961_object85099:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bae82141:::benchmark_data_smithi192_88961_object63477:head# 2024-06-27T21:23:27.557 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baeb9f4e:::benchmark_data_smithi192_88961_object50851:head# 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baed29a6:::benchmark_data_smithi192_88961_object49561:head# 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baefaa26:::benchmark_data_smithi192_88961_object49286:head# 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baf2760e:::benchmark_data_smithi192_88961_object61455:head# 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baf31a24:::benchmark_data_smithi192_88961_object61304:head# 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baf58dd9:::benchmark_data_smithi192_88961_object87358:head# 2024-06-27T21:23:27.558 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.561 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baf5ee7f:::benchmark_data_smithi192_88961_object45515:head# 2024-06-27T21:23:27.561 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.561 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baf905c2:::benchmark_data_smithi192_88961_object61282:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:baf92bda:::benchmark_data_smithi192_88961_object52897:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bafc8ae8:::benchmark_data_smithi192_88961_object55524:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bafea1c5:::benchmark_data_smithi192_88961_object67619:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb02010b:::benchmark_data_smithi192_88961_object85887:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0265b0:::benchmark_data_smithi192_88961_object60407:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb02a715:::benchmark_data_smithi192_88961_object64438:head# 2024-06-27T21:23:27.562 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb038d16:::benchmark_data_smithi192_88961_object44717:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb071985:::benchmark_data_smithi192_88961_object80123:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0bbd55:::benchmark_data_smithi192_88961_object46513:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0c2101:::benchmark_data_smithi192_88961_object81379:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0e4c45:::benchmark_data_smithi192_88961_object47652:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0e7ff7:::benchmark_data_smithi192_88961_object56450:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0fca03:::benchmark_data_smithi192_88961_object52201:head# 2024-06-27T21:23:27.563 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.580 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0fcf35:::benchmark_data_smithi192_88961_object86826:head# 2024-06-27T21:23:27.580 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.580 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb0fe035:::benchmark_data_smithi192_88961_object75108:head# 2024-06-27T21:23:27.580 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb119eb9:::benchmark_data_smithi192_88961_object54199:head# 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb14241f:::benchmark_data_smithi192_88961_object59122:head# 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb143a31:::benchmark_data_smithi192_88961_object82941:head# 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb14ac6e:::benchmark_data_smithi192_88961_object71617:head# 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1531fb:::benchmark_data_smithi192_88961_object81394:head# 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb155faf:::benchmark_data_smithi192_88961_object64210:head# 2024-06-27T21:23:27.581 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1636b0:::benchmark_data_smithi192_88961_object89857:head# 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1692eb:::benchmark_data_smithi192_88961_object45235:head# 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1801e0:::benchmark_data_smithi192_88961_object59541:head# 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1b3615:::benchmark_data_smithi192_88961_object87961:head# 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1d5314:::benchmark_data_smithi192_88961_object76119:head# 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1e4a0a:::benchmark_data_smithi192_88961_object79354:head# 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.582 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb1f53a7:::benchmark_data_smithi192_88961_object60303:head# 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb200cfc:::benchmark_data_smithi192_88961_object73287:head# 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb20a07e:::benchmark_data_smithi192_88961_object75643:head# 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb23ef5a:::benchmark_data_smithi192_88961_object68936:head# 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb249f1c:::benchmark_data_smithi192_88961_object45381:head# 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb26b129:::benchmark_data_smithi192_88961_object81996:head# 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.583 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb26d66c:::benchmark_data_smithi192_88961_object66730:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb2802b4:::benchmark_data_smithi192_88961_object54160:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb2858c3:::benchmark_data_smithi192_88961_object56935:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb28a690:::benchmark_data_smithi192_88961_object59685:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb2b6fc6:::benchmark_data_smithi192_88961_object43803:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb2f72b7:::benchmark_data_smithi192_88961_object60293:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb32eb2b:::benchmark_data_smithi192_88961_object71411:head# 2024-06-27T21:23:27.584 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb332cdb:::benchmark_data_smithi192_88961_object73662:head# 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb33863f:::benchmark_data_smithi192_88961_object59711:head# 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb348869:::benchmark_data_smithi192_88961_object85154:head# 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb34c3a5:::benchmark_data_smithi192_88961_object87183:head# 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3520cb:::benchmark_data_smithi192_88961_object55739:head# 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb359cd5:::benchmark_data_smithi192_88961_object61261:head# 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.585 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb36c058:::benchmark_data_smithi192_88961_object47193:head# 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3832c9:::benchmark_data_smithi192_88961_object46222:head# 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3b3068:::benchmark_data_smithi192_88961_object56691:head# 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3c274e:::benchmark_data_smithi192_88961_object64693:head# 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3d1c36:::benchmark_data_smithi192_88961_object59324:head# 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3f1666:::benchmark_data_smithi192_88961_object49194:head# 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.586 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb3f92bf:::benchmark_data_smithi192_88961_object47446:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb45befc:::benchmark_data_smithi192_88961_object59279:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb4771c6:::benchmark_data_smithi192_88961_object70483:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb48ecbb:::benchmark_data_smithi192_88961_object58253:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb4946e6:::benchmark_data_smithi192_88961_object87873:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb4ac95c:::benchmark_data_smithi192_88961_object83210:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb4b3ff3:::benchmark_data_smithi192_88961_object61443:head# 2024-06-27T21:23:27.587 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb4e1b88:::benchmark_data_smithi192_88961_object69760:head# 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb4eed1a:::benchmark_data_smithi192_88961_object50015:head# 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb50376f:::benchmark_data_smithi192_88961_object58616:head# 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb51887f:::benchmark_data_smithi192_88961_object76518:head# 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb53a3eb:::benchmark_data_smithi192_88961_object47934:head# 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb548bb2:::benchmark_data_smithi192_88961_object66822:head# 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.588 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb5579f3:::benchmark_data_smithi192_88961_object87290:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb56d518:::benchmark_data_smithi192_88961_object53846:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb5946f8:::benchmark_data_smithi192_88961_object53790:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb5f6188:::benchmark_data_smithi192_88961_object72291:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb62fae8:::benchmark_data_smithi192_88961_object47280:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb6731bf:::benchmark_data_smithi192_88961_object68882:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb682011:::benchmark_data_smithi192_88961_object90558:head# 2024-06-27T21:23:27.589 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb68f7a5:::benchmark_data_smithi192_88961_object65351:head# 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb690318:::benchmark_data_smithi192_88961_object79430:head# 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb697b08:::benchmark_data_smithi192_88961_object73510:head# 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb6b799b:::benchmark_data_smithi192_88961_object59710:head# 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb6d35ed:::benchmark_data_smithi192_88961_object71462:head# 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb6f1091:::benchmark_data_smithi192_88961_object55318:head# 2024-06-27T21:23:27.590 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb72e5b6:::benchmark_data_smithi192_88961_object65691:head# 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb72fd11:::benchmark_data_smithi192_88961_object47286:head# 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb74f951:::benchmark_data_smithi192_88961_object57021:head# 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb75ef69:::benchmark_data_smithi192_88961_object70366:head# 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb771999:::benchmark_data_smithi192_88961_object86077:head# 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb7c1cd7:::benchmark_data_smithi192_88961_object72354:head# 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.591 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb7ca6b8:::benchmark_data_smithi192_88961_object87450:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb7cb7a2:::benchmark_data_smithi192_88961_object87010:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb8053cb:::benchmark_data_smithi192_88961_object69024:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb82c043:::benchmark_data_smithi192_88961_object59701:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb846b6b:::benchmark_data_smithi192_88961_object48542:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb84c792:::benchmark_data_smithi192_88961_object84471:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb84fbc5:::benchmark_data_smithi192_88961_object75253:head# 2024-06-27T21:23:27.592 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb8577f6:::benchmark_data_smithi192_88961_object89394:head# 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb879358:::benchmark_data_smithi192_88961_object88184:head# 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb8a9b94:::benchmark_data_smithi192_88961_object48641:head# 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb8d7dfa:::benchmark_data_smithi192_88961_object55214:head# 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb90f960:::benchmark_data_smithi192_88961_object61019:head# 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb916f60:::benchmark_data_smithi192_88961_object62921:head# 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.593 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb9194d6:::benchmark_data_smithi192_88961_object55588:head# 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb950ec3:::benchmark_data_smithi192_88961_object88700:head# 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb95f0ce:::benchmark_data_smithi192_88961_object52364:head# 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb96b515:::benchmark_data_smithi192_88961_object44571:head# 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb989f5a:::benchmark_data_smithi192_88961_object71283:head# 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb9a5b9d:::benchmark_data_smithi192_88961_object88981:head# 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.594 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb9a79db:::benchmark_data_smithi192_88961_object59028:head# 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb9b0afe:::benchmark_data_smithi192_88961_object48541:head# 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb9b53da:::benchmark_data_smithi192_88961_object72864:head# 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bb9e00af:::benchmark_data_smithi192_88961_object78649:head# 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bba0153c:::benchmark_data_smithi192_88961_object54758:head# 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bba1af7a:::benchmark_data_smithi192_88961_object83811:head# 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.595 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bba32bbe:::benchmark_data_smithi192_88961_object54313:head# 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bba4cef5:::benchmark_data_smithi192_88961_object54050:head# 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bba823f1:::benchmark_data_smithi192_88961_object62432:head# 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbaab373:::benchmark_data_smithi192_88961_object72415:head# 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbabc8f1:::benchmark_data_smithi192_88961_object53608:head# 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbac4a6c:::benchmark_data_smithi192_88961_object60552:head# 2024-06-27T21:23:27.596 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbaca5af:::benchmark_data_smithi192_88961_object73188:head# 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbacabeb:::benchmark_data_smithi192_88961_object82455:head# 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbafd1c1:::benchmark_data_smithi192_88961_object43196:head# 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbb164b9:::benchmark_data_smithi192_88961_object84050:head# 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbb34f76:::benchmark_data_smithi192_88961_object58274:head# 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbb4864b:::benchmark_data_smithi192_88961_object83075:head# 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.597 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbb6160e:::benchmark_data_smithi192_88961_object90197:head# 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbb931c2:::benchmark_data_smithi192_88961_object54856:head# 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbbaabbb:::benchmark_data_smithi192_88961_object86972:head# 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbc03cf8:::benchmark_data_smithi192_88961_object49336:head# 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbc16cb0:::benchmark_data_smithi192_88961_object77291:head# 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbc41287:::benchmark_data_smithi192_88961_object74668:head# 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.598 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbc6ad12:::benchmark_data_smithi192_88961_object58025:head# 2024-06-27T21:23:27.599 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.599 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbc80c26:::benchmark_data_smithi192_88961_object56020:head# 2024-06-27T21:23:27.599 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:27.619 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbcced72:::benchmark_data_smithi192_88961_object85533:head# 2024-06-27T21:23:27.619 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.619 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbcdedf5:::benchmark_data_smithi192_88961_object76752:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbcee8a7:::benchmark_data_smithi192_88961_object74029:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbcfd7a5:::benchmark_data_smithi192_88961_object60634:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd019ab:::benchmark_data_smithi192_88961_object89850:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd08d2f:::benchmark_data_smithi192_88961_object56560:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd251f8:::benchmark_data_smithi192_88961_object46897:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd31999:::benchmark_data_smithi192_88961_object45414:head# 2024-06-27T21:23:27.620 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd43069:::benchmark_data_smithi192_88961_object74286:head# 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd6b1bb:::benchmark_data_smithi192_88961_object59356:head# 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd87749:::benchmark_data_smithi192_88961_object74402:head# 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbd95f60:::benchmark_data_smithi192_88961_object76152:head# 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbdb3c1f:::benchmark_data_smithi192_88961_object63763:head# 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbdb5215:::benchmark_data_smithi192_88961_object56664:head# 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.621 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbdf980d:::benchmark_data_smithi192_88961_object84428:head# 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe0707e:::benchmark_data_smithi192_88961_object70386:head# 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe15f2e:::benchmark_data_smithi192_88961_object81922:head# 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe28aa0:::benchmark_data_smithi192_88961_object44374:head# 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe53269:::benchmark_data_smithi192_88961_object43293:head# 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe54ba1:::benchmark_data_smithi192_88961_object82824:head# 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.622 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe812fd:::benchmark_data_smithi192_88961_object55393:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe96adc:::benchmark_data_smithi192_88961_object61145:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbe9880a:::benchmark_data_smithi192_88961_object80438:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbeba0a0:::benchmark_data_smithi192_88961_object71471:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbebfdcc:::benchmark_data_smithi192_88961_object54582:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbee8af4:::benchmark_data_smithi192_88961_object54182:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbeef3b2:::benchmark_data_smithi192_88961_object50095:head# 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.623 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbef8f5a:::benchmark_data_smithi192_88961_object52149:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbefc223:::benchmark_data_smithi192_88961_object87807:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf166e3:::benchmark_data_smithi192_88961_object87767:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf1f5d5:::benchmark_data_smithi192_88961_object44642:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf1f730:::benchmark_data_smithi192_88961_object70758:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf22037:::benchmark_data_smithi192_88961_object71891:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf2f6a9:::benchmark_data_smithi192_88961_object80345:head# 2024-06-27T21:23:27.624 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf3ac4b:::benchmark_data_smithi192_88961_object89228:head# 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf4ea0d:::benchmark_data_smithi192_88961_object43984:head# 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf73367:::benchmark_data_smithi192_88961_object45609:head# 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbf8ca18:::benchmark_data_smithi192_88961_object59156:head# 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bbfc2259:::benchmark_data_smithi192_88961_object63703:head# 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc01a3e1:::benchmark_data_smithi192_88961_object80773:head# 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc02457a:::benchmark_data_smithi192_88961_object69284:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc026ef1:::benchmark_data_smithi192_88961_object65662:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc031e04:::benchmark_data_smithi192_88961_object85383:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc032d33:::benchmark_data_smithi192_88961_object84693:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc04065a:::benchmark_data_smithi192_88961_object62118:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc04a9ad:::benchmark_data_smithi192_88961_object75779:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc065ec3:::benchmark_data_smithi192_88961_object65754:head# 2024-06-27T21:23:27.626 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc0a3016:::benchmark_data_smithi192_88961_object52583:head# 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc0e604c:::benchmark_data_smithi192_88961_object76946:head# 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc10c60d:::benchmark_data_smithi192_88961_object69293:head# 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc113308:::benchmark_data_smithi192_88961_object63240:head# 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc11e4fd:::benchmark_data_smithi192_88961_object50185:head# 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc133475:::benchmark_data_smithi192_88961_object66514:head# 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.627 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc135a1f:::benchmark_data_smithi192_88961_object57237:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1574bc:::benchmark_data_smithi192_88961_object56923:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc15d1a1:::benchmark_data_smithi192_88961_object51305:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc15dd5c:::benchmark_data_smithi192_88961_object75646:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc15e8c2:::benchmark_data_smithi192_88961_object69502:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1630a3:::benchmark_data_smithi192_88961_object49766:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc172fb2:::benchmark_data_smithi192_88961_object78054:head# 2024-06-27T21:23:27.628 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc17b90a:::benchmark_data_smithi192_88961_object60045:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc18720d:::benchmark_data_smithi192_88961_object62336:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc18a627:::benchmark_data_smithi192_88961_object80204:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc18d231:::benchmark_data_smithi192_88961_object56812:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc194a4b:::benchmark_data_smithi192_88961_object73943:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1bd95a:::benchmark_data_smithi192_88961_object46110:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1c742a:::benchmark_data_smithi192_88961_object52179:head# 2024-06-27T21:23:27.629 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1c779d:::benchmark_data_smithi192_88961_object63658:head# 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1d2b96:::benchmark_data_smithi192_88961_object57537:head# 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1dd5a2:::benchmark_data_smithi192_88961_object71551:head# 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc1fd54b:::benchmark_data_smithi192_88961_object83822:head# 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2241b6:::benchmark_data_smithi192_88961_object52522:head# 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2424a5:::benchmark_data_smithi192_88961_object78273:head# 2024-06-27T21:23:27.630 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2672fc:::benchmark_data_smithi192_88961_object53950:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2681bf:::benchmark_data_smithi192_88961_object76676:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc274de7:::benchmark_data_smithi192_88961_object59290:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc28adee:::benchmark_data_smithi192_88961_object84412:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2a3f0d:::benchmark_data_smithi192_88961_object56367:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2c4dcb:::benchmark_data_smithi192_88961_object58526:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2d5e6d:::benchmark_data_smithi192_88961_object66859:head# 2024-06-27T21:23:27.631 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc2e008b:::benchmark_data_smithi192_88961_object79149:head# 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc305bf4:::benchmark_data_smithi192_88961_object80779:head# 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc307bc4:::benchmark_data_smithi192_88961_object56265:head# 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc30bdd2:::benchmark_data_smithi192_88961_object69501:head# 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc32ffc5:::benchmark_data_smithi192_88961_object48315:head# 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3371ae:::benchmark_data_smithi192_88961_object63409:head# 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.632 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc348e8e:::benchmark_data_smithi192_88961_object81635:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc34eeba:::benchmark_data_smithi192_88961_object72300:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc34fe26:::benchmark_data_smithi192_88961_object52226:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc35410e:::benchmark_data_smithi192_88961_object77563:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc354292:::benchmark_data_smithi192_88961_object65231:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc356aaa:::benchmark_data_smithi192_88961_object88764:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc380fc1:::benchmark_data_smithi192_88961_object59631:head# 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.633 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3936f0:::benchmark_data_smithi192_88961_object80656:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc39c658:::benchmark_data_smithi192_88961_object82272:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3a8716:::benchmark_data_smithi192_88961_object75388:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3cdf80:::benchmark_data_smithi192_88961_object70918:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3d9740:::benchmark_data_smithi192_88961_object65577:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3dde29:::benchmark_data_smithi192_88961_object46987:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3ebcae:::benchmark_data_smithi192_88961_object47719:head# 2024-06-27T21:23:27.634 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc3eebbb:::benchmark_data_smithi192_88961_object89288:head# 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc41908a:::benchmark_data_smithi192_88961_object73543:head# 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc4246c7:::benchmark_data_smithi192_88961_object46962:head# 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc425158:::benchmark_data_smithi192_88961_object87406:head# 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc459b96:::benchmark_data_smithi192_88961_object65585:head# 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc495142:::benchmark_data_smithi192_88961_object85103:head# 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.635 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc49ee45:::benchmark_data_smithi192_88961_object65286:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc4b13d6:::benchmark_data_smithi192_88961_object79647:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc4b991a:::benchmark_data_smithi192_88961_object61135:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc4cda97:::benchmark_data_smithi192_88961_object77917:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc4dcb3f:::benchmark_data_smithi192_88961_object48293:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc511d37:::benchmark_data_smithi192_88961_object87527:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc52fea3:::benchmark_data_smithi192_88961_object80044:head# 2024-06-27T21:23:27.636 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc544c70:::benchmark_data_smithi192_88961_object71086:head# 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc577eee:::benchmark_data_smithi192_88961_object43095:head# 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc5ca1e7:::benchmark_data_smithi192_88961_object62477:head# 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc5d374a:::benchmark_data_smithi192_88961_object53819:head# 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc61ad82:::benchmark_data_smithi192_88961_object71480:head# 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc62e93b:::benchmark_data_smithi192_88961_object51584:head# 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.637 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc632a80:::benchmark_data_smithi192_88961_object52935:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6337a0:::benchmark_data_smithi192_88961_object69197:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc63f81a:::benchmark_data_smithi192_88961_object82133:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc64897b:::benchmark_data_smithi192_88961_object64849:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc668cda:::benchmark_data_smithi192_88961_object76051:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc68145d:::benchmark_data_smithi192_88961_object69271:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6b2f40:::benchmark_data_smithi192_88961_object66546:head# 2024-06-27T21:23:27.638 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6b5c59:::benchmark_data_smithi192_88961_object45891:head# 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6ce282:::benchmark_data_smithi192_88961_object58115:head# 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6d225e:::benchmark_data_smithi192_88961_object80276:head# 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6ee462:::benchmark_data_smithi192_88961_object74992:head# 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc6ef33a:::benchmark_data_smithi192_88961_object44582:head# 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc732e47:::benchmark_data_smithi192_88961_object54446:head# 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.639 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc764b85:::benchmark_data_smithi192_88961_object89820:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc76f97c:::benchmark_data_smithi192_88961_object59407:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc790377:::benchmark_data_smithi192_88961_object79103:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc79b714:::benchmark_data_smithi192_88961_object43889:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc7a9227:::benchmark_data_smithi192_88961_object54353:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc7b7bfd:::benchmark_data_smithi192_88961_object74067:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc7d3153:::benchmark_data_smithi192_88961_object59102:head# 2024-06-27T21:23:27.640 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc7de333:::benchmark_data_smithi192_88961_object61196:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc7ed7e4:::benchmark_data_smithi192_88961_object53058:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc7ffdf1:::benchmark_data_smithi192_88961_object72913:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc800ca6:::benchmark_data_smithi192_88961_object79154:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc801a9e:::benchmark_data_smithi192_88961_object65409:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc803471:::benchmark_data_smithi192_88961_object77683:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc82753c:::benchmark_data_smithi192_88961_object79471:head# 2024-06-27T21:23:27.641 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.642 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc839d14:::benchmark_data_smithi192_88961_object88112:head# 2024-06-27T21:23:27.642 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.642 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc8706bf:::benchmark_data_smithi192_88961_object80227:head# 2024-06-27T21:23:27.642 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:27.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:27.670 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc87dfea:::benchmark_data_smithi192_88961_object57062:head# 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc883b7c:::benchmark_data_smithi192_88961_object43755:head# 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc88d220:::benchmark_data_smithi192_88961_object53818:head# 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc89b262:::benchmark_data_smithi192_88961_object49554:head# 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc8c66a4:::benchmark_data_smithi192_88961_object61095:head# 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.671 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc8c8f5f:::benchmark_data_smithi192_88961_object52026:head# 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc8cc995:::benchmark_data_smithi192_88961_object52301:head# 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc905d69:::benchmark_data_smithi192_88961_object87179:head# 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc909e69:::benchmark_data_smithi192_88961_object66957:head# 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc928fee:::benchmark_data_smithi192_88961_object73404:head# 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc92e2c2:::benchmark_data_smithi192_88961_object69236:head# 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.672 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc961cea:::benchmark_data_smithi192_88961_object81364:head# 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc975ddd:::benchmark_data_smithi192_88961_object70532:head# 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc97949b:::benchmark_data_smithi192_88961_object90062:head# 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc99a530:::benchmark_data_smithi192_88961_object74641:head# 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc9a67fd:::benchmark_data_smithi192_88961_object48632:head# 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc9e1a85:::benchmark_data_smithi192_88961_object68855:head# 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.673 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bc9ed4a1:::benchmark_data_smithi192_88961_object60703:head# 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca0e35d:::benchmark_data_smithi192_88961_object61463:head# 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca2d420:::benchmark_data_smithi192_88961_object51446:head# 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca3d1b9:::benchmark_data_smithi192_88961_object73112:head# 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca45e85:::benchmark_data_smithi192_88961_object46681:head# 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca4ee18:::benchmark_data_smithi192_88961_object73504:head# 2024-06-27T21:23:27.674 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca4ffb3:::benchmark_data_smithi192_88961_object69341:head# 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca69adf:::benchmark_data_smithi192_88961_object86688:head# 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca6f8a9:::benchmark_data_smithi192_88961_object48297:head# 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bca99469:::benchmark_data_smithi192_88961_object88062:head# 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcac5742:::benchmark_data_smithi192_88961_object88810:head# 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcac97b5:::benchmark_data_smithi192_88961_object88920:head# 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.675 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcae7def:::benchmark_data_smithi192_88961_object63271:head# 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcaed639:::benchmark_data_smithi192_88961_object44404:head# 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb11400:::benchmark_data_smithi192_88961_object87204:head# 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb42807:::benchmark_data_smithi192_88961_object59903:head# 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb4a5ee:::benchmark_data_smithi192_88961_object70811:head# 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb5cb9c:::benchmark_data_smithi192_88961_object66910:head# 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.676 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb65f22:::benchmark_data_smithi192_88961_object78058:head# 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb684d1:::benchmark_data_smithi192_88961_object84651:head# 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb6cdb5:::benchmark_data_smithi192_88961_object88526:head# 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb7484a:::benchmark_data_smithi192_88961_object68384:head# 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb7a306:::benchmark_data_smithi192_88961_object84466:head# 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcb9e952:::benchmark_data_smithi192_88961_object49461:head# 2024-06-27T21:23:27.677 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcbb6a6d:::benchmark_data_smithi192_88961_object58926:head# 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcbc201d:::benchmark_data_smithi192_88961_object63254:head# 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcbd0ac4:::benchmark_data_smithi192_88961_object77523:head# 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcbfd382:::benchmark_data_smithi192_88961_object43566:head# 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc03dd9:::benchmark_data_smithi192_88961_object43074:head# 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc16564:::benchmark_data_smithi192_88961_object43703:head# 2024-06-27T21:23:27.678 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc1b2fb:::benchmark_data_smithi192_88961_object69633:head# 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc56588:::benchmark_data_smithi192_88961_object69779:head# 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc61300:::benchmark_data_smithi192_88961_object70773:head# 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc6a30c:::benchmark_data_smithi192_88961_object46904:head# 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc8f074:::benchmark_data_smithi192_88961_object78890:head# 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.679 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc97db3:::benchmark_data_smithi192_88961_object47420:head# 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcc9ba69:::benchmark_data_smithi192_88961_object46883:head# 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bccb108c:::benchmark_data_smithi192_88961_object43368:head# 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bccb28b9:::benchmark_data_smithi192_88961_object87004:head# 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bccc98ea:::benchmark_data_smithi192_88961_object49696:head# 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bccca115:::benchmark_data_smithi192_88961_object89083:head# 2024-06-27T21:23:27.680 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bccd335d:::benchmark_data_smithi192_88961_object51347:head# 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd009cf:::benchmark_data_smithi192_88961_object80481:head# 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd20641:::benchmark_data_smithi192_88961_object63548:head# 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd2da4b:::benchmark_data_smithi192_88961_object70306:head# 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd3350d:::benchmark_data_smithi192_88961_object50913:head# 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.681 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd36917:::benchmark_data_smithi192_88961_object71861:head# 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd3a701:::benchmark_data_smithi192_88961_object52932:head# 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd3df90:::benchmark_data_smithi192_88961_object77201:head# 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd497e1:::benchmark_data_smithi192_88961_object63455:head# 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd6c074:::benchmark_data_smithi192_88961_object49387:head# 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcd7c138:::benchmark_data_smithi192_88961_object64947:head# 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.682 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcda04d1:::benchmark_data_smithi192_88961_object67785:head# 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcdb7739:::benchmark_data_smithi192_88961_object65975:head# 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcdbf1e9:::benchmark_data_smithi192_88961_object74014:head# 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcdc894d:::benchmark_data_smithi192_88961_object44837:head# 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcddc403:::benchmark_data_smithi192_88961_object90133:head# 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcde0071:::benchmark_data_smithi192_88961_object60164:head# 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.683 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcdf4582:::benchmark_data_smithi192_88961_object46006:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcdf5c7f:::benchmark_data_smithi192_88961_object77411:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcdfb127:::benchmark_data_smithi192_88961_object60390:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bce05cae:::benchmark_data_smithi192_88961_object58913:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bce42f56:::benchmark_data_smithi192_88961_object43543:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bce536ef:::benchmark_data_smithi192_88961_object59201:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bce86fc4:::benchmark_data_smithi192_88961_object89833:head# 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.684 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bceb7e45:::benchmark_data_smithi192_88961_object84057:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcebf5ff:::benchmark_data_smithi192_88961_object50879:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcef20dd:::benchmark_data_smithi192_88961_object73057:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcf18e7b:::benchmark_data_smithi192_88961_object55808:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcf1bba7:::benchmark_data_smithi192_88961_object67385:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcf2731f:::benchmark_data_smithi192_88961_object75047:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcf6e599:::benchmark_data_smithi192_88961_object86853:head# 2024-06-27T21:23:27.685 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcf6f305:::benchmark_data_smithi192_88961_object72019:head# 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcf8af8d:::benchmark_data_smithi192_88961_object43596:head# 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcfa7ecb:::benchmark_data_smithi192_88961_object88379:head# 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcfb7804:::benchmark_data_smithi192_88961_object79892:head# 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcfbd4b3:::benchmark_data_smithi192_88961_object54203:head# 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bcfc1dbf:::benchmark_data_smithi192_88961_object58945:head# 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.686 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd000e1c:::benchmark_data_smithi192_88961_object58739:head# 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd00ad2d:::benchmark_data_smithi192_88961_object77451:head# 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd016012:::benchmark_data_smithi192_88961_object85918:head# 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd02dcb2:::benchmark_data_smithi192_88961_object50615:head# 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd053427:::benchmark_data_smithi192_88961_object88999:head# 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd057622:::benchmark_data_smithi192_88961_object53628:head# 2024-06-27T21:23:27.687 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd06884c:::benchmark_data_smithi192_88961_object73566:head# 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd06efd4:::benchmark_data_smithi192_88961_object62869:head# 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd076da0:::benchmark_data_smithi192_88961_object57805:head# 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd07807f:::benchmark_data_smithi192_88961_object89904:head# 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0829ab:::benchmark_data_smithi192_88961_object86649:head# 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0888ce:::benchmark_data_smithi192_88961_object63988:head# 2024-06-27T21:23:27.688 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd08b902:::benchmark_data_smithi192_88961_object81807:head# 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0a1dcf:::benchmark_data_smithi192_88961_object88706:head# 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0b748f:::benchmark_data_smithi192_88961_object57728:head# 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0b8751:::benchmark_data_smithi192_88961_object74120:head# 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0beb8c:::benchmark_data_smithi192_88961_object66332:head# 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0c4731:::benchmark_data_smithi192_88961_object71533:head# 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.689 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0c47e4:::benchmark_data_smithi192_88961_object52554:head# 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0ce7da:::benchmark_data_smithi192_88961_object84154:head# 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd0ead57:::benchmark_data_smithi192_88961_object74394:head# 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd129eb8:::benchmark_data_smithi192_88961_object51686:head# 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd16cb28:::benchmark_data_smithi192_88961_object78541:head# 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1aeaf5:::benchmark_data_smithi192_88961_object68627:head# 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.690 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1b4acb:::benchmark_data_smithi192_88961_object58821:head# 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1cf3d8:::benchmark_data_smithi192_88961_object66006:head# 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1d0dc6:::benchmark_data_smithi192_88961_object57479:head# 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1df939:::benchmark_data_smithi192_88961_object54659:head# 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1ebe2d:::benchmark_data_smithi192_88961_object56713:head# 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1ed882:::benchmark_data_smithi192_88961_object76983:head# 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1f140e:::benchmark_data_smithi192_88961_object90626:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1f87c0:::benchmark_data_smithi192_88961_object52136:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1ff8e4:::benchmark_data_smithi192_88961_object63610:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd1ffb9f:::benchmark_data_smithi192_88961_object62928:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd2098fc:::benchmark_data_smithi192_88961_object76599:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd20c570:::benchmark_data_smithi192_88961_object72649:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd20eced:::benchmark_data_smithi192_88961_object45347:head# 2024-06-27T21:23:27.692 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd213f1d:::benchmark_data_smithi192_88961_object64548:head# 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd22b9fc:::benchmark_data_smithi192_88961_object69157:head# 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd25366d:::benchmark_data_smithi192_88961_object55457:head# 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd2661aa:::benchmark_data_smithi192_88961_object79656:head# 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd28c773:::benchmark_data_smithi192_88961_object46144:head# 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3202f5:::benchmark_data_smithi192_88961_object65711:head# 2024-06-27T21:23:27.693 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3261b3:::benchmark_data_smithi192_88961_object79293:head# 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd32f5f5:::benchmark_data_smithi192_88961_object83001:head# 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3a253e:::benchmark_data_smithi192_88961_object49314:head# 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3afa4e:::benchmark_data_smithi192_88961_object60190:head# 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3b57b3:::benchmark_data_smithi192_88961_object79283:head# 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3ba930:::benchmark_data_smithi192_88961_object62829:head# 2024-06-27T21:23:27.694 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3c0b7e:::benchmark_data_smithi192_88961_object89715:head# 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3c2bfb:::benchmark_data_smithi192_88961_object59692:head# 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3cb9d2:::benchmark_data_smithi192_88961_object85222:head# 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd3d1efa:::benchmark_data_smithi192_88961_object57003:head# 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd4119e3:::benchmark_data_smithi192_88961_object89176:head# 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd42775b:::benchmark_data_smithi192_88961_object53744:head# 2024-06-27T21:23:27.695 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd4460ba:::benchmark_data_smithi192_88961_object49410:head# 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd45f219:::benchmark_data_smithi192_88961_object76836:head# 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd47493b:::benchmark_data_smithi192_88961_object52572:head# 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd4e141c:::benchmark_data_smithi192_88961_object81076:head# 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd5088ca:::benchmark_data_smithi192_88961_object62465:head# 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd518ddf:::benchmark_data_smithi192_88961_object70882:head# 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.696 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd522750:::benchmark_data_smithi192_88961_object60055:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd5309fd:::benchmark_data_smithi192_88961_object71540:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd54754e:::benchmark_data_smithi192_88961_object54672:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd57daf9:::benchmark_data_smithi192_88961_object45804:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd59419c:::benchmark_data_smithi192_88961_object83801:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd5cc8b6:::benchmark_data_smithi192_88961_object77399:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd5d0f65:::benchmark_data_smithi192_88961_object84740:head# 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.697 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd5d41ab:::benchmark_data_smithi192_88961_object62397:head# 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd5f609e:::benchmark_data_smithi192_88961_object43748:head# 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd61b1ad:::benchmark_data_smithi192_88961_object43896:head# 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd629f49:::benchmark_data_smithi192_88961_object47350:head# 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd62d0c1:::benchmark_data_smithi192_88961_object47792:head# 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd63dfe3:::benchmark_data_smithi192_88961_object90335:head# 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.698 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd659c82:::benchmark_data_smithi192_88961_object75333:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd659e6b:::benchmark_data_smithi192_88961_object87264:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd6633c3:::benchmark_data_smithi192_88961_object54053:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd66c072:::benchmark_data_smithi192_88961_object61438:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd6712b7:::benchmark_data_smithi192_88961_object66718:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd68d8ef:::benchmark_data_smithi192_88961_object76259:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd6b9a4b:::benchmark_data_smithi192_88961_object63257:head# 2024-06-27T21:23:27.699 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd6f05ea:::benchmark_data_smithi192_88961_object80855:head# 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd6fd8b3:::benchmark_data_smithi192_88961_object64204:head# 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd715666:::benchmark_data_smithi192_88961_object70865:head# 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd721878:::benchmark_data_smithi192_88961_object52876:head# 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd72d1a9:::benchmark_data_smithi192_88961_object63442:head# 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd749a67:::benchmark_data_smithi192_88961_object57480:head# 2024-06-27T21:23:27.700 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd756053:::benchmark_data_smithi192_88961_object90292:head# 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd783ba5:::benchmark_data_smithi192_88961_object77665:head# 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd7b4d59:::benchmark_data_smithi192_88961_object82500:head# 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd7bbea3:::benchmark_data_smithi192_88961_object85322:head# 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd7ebce4:::benchmark_data_smithi192_88961_object89888:head# 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd7fbfb1:::benchmark_data_smithi192_88961_object82046:head# 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.701 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd7feb3c:::benchmark_data_smithi192_88961_object84927:head# 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd803743:::benchmark_data_smithi192_88961_object58953:head# 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd82d3fc:::benchmark_data_smithi192_88961_object61517:head# 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd837f58:::benchmark_data_smithi192_88961_object67269:head# 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd83dc05:::benchmark_data_smithi192_88961_object69853:head# 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8612ea:::benchmark_data_smithi192_88961_object45904:head# 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.702 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8631d4:::benchmark_data_smithi192_88961_object79289:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd866c9f:::benchmark_data_smithi192_88961_object81356:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd88c58c:::benchmark_data_smithi192_88961_object54614:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd88d558:::benchmark_data_smithi192_88961_object74189:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8b5aa5:::benchmark_data_smithi192_88961_object72752:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8c1f77:::benchmark_data_smithi192_88961_object66096:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8c49b3:::benchmark_data_smithi192_88961_object67239:head# 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.703 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8ceb9d:::benchmark_data_smithi192_88961_object64223:head# 2024-06-27T21:23:27.704 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.704 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8cffb7:::benchmark_data_smithi192_88961_object75968:head# 2024-06-27T21:23:27.704 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8de900:::benchmark_data_smithi192_88961_object62763:head# 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd8ebfcf:::benchmark_data_smithi192_88961_object68628:head# 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd90dfb1:::benchmark_data_smithi192_88961_object84869:head# 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd91306f:::benchmark_data_smithi192_88961_object77436:head# 2024-06-27T21:23:27.724 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd918f40:::benchmark_data_smithi192_88961_object45693:head# 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd93e267:::benchmark_data_smithi192_88961_object82687:head# 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd950f26:::benchmark_data_smithi192_88961_object74696:head# 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd95909f:::benchmark_data_smithi192_88961_object51353:head# 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd99c076:::benchmark_data_smithi192_88961_object83918:head# 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd9b067f:::benchmark_data_smithi192_88961_object78468:head# 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.725 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd9c0b81:::benchmark_data_smithi192_88961_object49063:head# 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd9c2e2e:::benchmark_data_smithi192_88961_object83516:head# 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd9c5fa1:::benchmark_data_smithi192_88961_object89198:head# 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd9ce47b:::benchmark_data_smithi192_88961_object81605:head# 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bd9d08d6:::benchmark_data_smithi192_88961_object67546:head# 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda01172:::benchmark_data_smithi192_88961_object90669:head# 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.726 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda11f69:::benchmark_data_smithi192_88961_object44814:head# 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda169fb:::benchmark_data_smithi192_88961_object70189:head# 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda338c1:::benchmark_data_smithi192_88961_object44919:head# 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda38ca3:::benchmark_data_smithi192_88961_object67708:head# 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda54a6a:::benchmark_data_smithi192_88961_object81157:head# 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda84929:::benchmark_data_smithi192_88961_object60183:head# 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.727 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda96488:::benchmark_data_smithi192_88961_object58355:head# 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda9831e:::benchmark_data_smithi192_88961_object73611:head# 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda9a4d2:::benchmark_data_smithi192_88961_object72035:head# 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bda9d727:::benchmark_data_smithi192_88961_object71675:head# 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdaa5256:::benchmark_data_smithi192_88961_object50917:head# 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdaece18:::benchmark_data_smithi192_88961_object69479:head# 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.728 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdaefaea:::benchmark_data_smithi192_88961_object80249:head# 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdafb3d9:::benchmark_data_smithi192_88961_object44019:head# 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb08f02:::benchmark_data_smithi192_88961_object83636:head# 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb09165:::benchmark_data_smithi192_88961_object49192:head# 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb0d327:::benchmark_data_smithi192_88961_object57007:head# 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb309ff:::benchmark_data_smithi192_88961_object44468:head# 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.729 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb486ca:::benchmark_data_smithi192_88961_object55984:head# 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb93197:::benchmark_data_smithi192_88961_object76416:head# 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdb98c3c:::benchmark_data_smithi192_88961_object58453:head# 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdbad689:::benchmark_data_smithi192_88961_object48908:head# 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdbed4ed:::benchmark_data_smithi192_88961_object68234:head# 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc0a329:::benchmark_data_smithi192_88961_object78386:head# 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.730 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc0a9dd:::benchmark_data_smithi192_88961_object46049:head# 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc21dfd:::benchmark_data_smithi192_88961_object54508:head# 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc30b42:::benchmark_data_smithi192_88961_object89267:head# 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc33fd4:::benchmark_data_smithi192_88961_object72511:head# 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc38e01:::benchmark_data_smithi192_88961_object62700:head# 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc6146b:::benchmark_data_smithi192_88961_object47995:head# 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.731 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdc9d625:::benchmark_data_smithi192_88961_object58239:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdca3b79:::benchmark_data_smithi192_88961_object50814:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdcace5a:::benchmark_data_smithi192_88961_object53854:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdcbcf63:::benchmark_data_smithi192_88961_object78621:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdcc19c4:::benchmark_data_smithi192_88961_object65484:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdcc7d9e:::benchmark_data_smithi192_88961_object73412:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdccde68:::benchmark_data_smithi192_88961_object67389:head# 2024-06-27T21:23:27.732 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdcfdef3:::benchmark_data_smithi192_88961_object50484:head# 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdcffd4b:::benchmark_data_smithi192_88961_object88529:head# 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd0d744:::benchmark_data_smithi192_88961_object68179:head# 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd18b04:::benchmark_data_smithi192_88961_object68365:head# 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd48426:::benchmark_data_smithi192_88961_object87035:head# 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd4a2c1:::benchmark_data_smithi192_88961_object44902:head# 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.733 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd57c29:::benchmark_data_smithi192_88961_object84128:head# 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd61768:::benchmark_data_smithi192_88961_object85943:head# 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdd9ca02:::benchmark_data_smithi192_88961_object65649:head# 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdda06ee:::benchmark_data_smithi192_88961_object71535:head# 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bddd5bbc:::benchmark_data_smithi192_88961_object75536:head# 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bddf5bd0:::benchmark_data_smithi192_88961_object47361:head# 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.734 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde0b4b9:::benchmark_data_smithi192_88961_object53424:head# 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde0dd47:::benchmark_data_smithi192_88961_object68344:head# 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde1049f:::benchmark_data_smithi192_88961_object75135:head# 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde18d2f:::benchmark_data_smithi192_88961_object77497:head# 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde39884:::benchmark_data_smithi192_88961_object46326:head# 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde516f8:::benchmark_data_smithi192_88961_object74220:head# 2024-06-27T21:23:27.735 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde5d913:::benchmark_data_smithi192_88961_object44280:head# 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde6cc4e:::benchmark_data_smithi192_88961_object74993:head# 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde70775:::benchmark_data_smithi192_88961_object68939:head# 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bde7cb4f:::benchmark_data_smithi192_88961_object86433:head# 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdeaa515:::benchmark_data_smithi192_88961_object77051:head# 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdeaebb6:::benchmark_data_smithi192_88961_object76410:head# 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.736 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdeba915:::benchmark_data_smithi192_88961_object64803:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdebcd67:::benchmark_data_smithi192_88961_object51711:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf19856:::benchmark_data_smithi192_88961_object85971:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf206db:::benchmark_data_smithi192_88961_object49480:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf2530b:::benchmark_data_smithi192_88961_object57386:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf2be96:::benchmark_data_smithi192_88961_object58877:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf77a4b:::benchmark_data_smithi192_88961_object62587:head# 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.737 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf788a1:::benchmark_data_smithi192_88961_object74195:head# 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf81a60:::benchmark_data_smithi192_88961_object87686:head# 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdf91b8e:::benchmark_data_smithi192_88961_object72087:head# 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdfa555a:::benchmark_data_smithi192_88961_object65366:head# 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdfc0aea:::benchmark_data_smithi192_88961_object85685:head# 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdfc97cb:::benchmark_data_smithi192_88961_object72097:head# 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.738 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdfddaea:::benchmark_data_smithi192_88961_object47649:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bdfe3ba8:::benchmark_data_smithi192_88961_object50329:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be00382f:::benchmark_data_smithi192_88961_object78222:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be028ad3:::benchmark_data_smithi192_88961_object90605:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be054f0a:::benchmark_data_smithi192_88961_object52566:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be06109f:::benchmark_data_smithi192_88961_object53735:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be09108a:::benchmark_data_smithi192_88961_object77945:head# 2024-06-27T21:23:27.739 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be0a7c22:::benchmark_data_smithi192_88961_object89257:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be0bdf8b:::benchmark_data_smithi192_88961_object64125:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be0ded6d:::benchmark_data_smithi192_88961_object54660:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be0f0ab8:::benchmark_data_smithi192_88961_object76265:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be0f7788:::benchmark_data_smithi192_88961_object83100:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be125485:::benchmark_data_smithi192_88961_object52397:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be13873a:::benchmark_data_smithi192_88961_object64213:head# 2024-06-27T21:23:27.740 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be1482cb:::benchmark_data_smithi192_88961_object80246:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be151c33:::benchmark_data_smithi192_88961_object66836:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be15fc6e:::benchmark_data_smithi192_88961_object64476:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be163aa3:::benchmark_data_smithi192_88961_object88484:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be19f6d6:::benchmark_data_smithi192_88961_object88897:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be1a88fb:::benchmark_data_smithi192_88961_object62707:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be1ba8a2:::benchmark_data_smithi192_88961_object57675:head# 2024-06-27T21:23:27.741 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be1ca07b:::benchmark_data_smithi192_88961_object55838:head# 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be1dcd8a:::benchmark_data_smithi192_88961_object49481:head# 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be1faeb0:::benchmark_data_smithi192_88961_object81453:head# 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be207be9:::benchmark_data_smithi192_88961_object60368:head# 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be22fb57:::benchmark_data_smithi192_88961_object73928:head# 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be23eec0:::benchmark_data_smithi192_88961_object63804:head# 2024-06-27T21:23:27.742 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be24536d:::benchmark_data_smithi192_88961_object54415:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be246e5a:::benchmark_data_smithi192_88961_object60840:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be25f690:::benchmark_data_smithi192_88961_object58005:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be281769:::benchmark_data_smithi192_88961_object54835:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be2835c6:::benchmark_data_smithi192_88961_object67360:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be2b34f3:::benchmark_data_smithi192_88961_object88058:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be2f49f7:::benchmark_data_smithi192_88961_object45430:head# 2024-06-27T21:23:27.743 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be2f9a89:::benchmark_data_smithi192_88961_object49026:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be31be4d:::benchmark_data_smithi192_88961_object84445:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be3489ea:::benchmark_data_smithi192_88961_object61827:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be389784:::benchmark_data_smithi192_88961_object50632:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be38b41d:::benchmark_data_smithi192_88961_object58185:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be3c427b:::benchmark_data_smithi192_88961_object52488:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be3f923f:::benchmark_data_smithi192_88961_object88595:head# 2024-06-27T21:23:27.744 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be42d991:::benchmark_data_smithi192_88961_object53362:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be44e7da:::benchmark_data_smithi192_88961_object43533:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be4575fc:::benchmark_data_smithi192_88961_object52127:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be45edc1:::benchmark_data_smithi192_88961_object61317:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be46de66:::benchmark_data_smithi192_88961_object64261:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be49782e:::benchmark_data_smithi192_88961_object79943:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be49afd4:::benchmark_data_smithi192_88961_object49848:head# 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.745 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be4d6dbc:::benchmark_data_smithi192_88961_object44812:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be4d9a82:::benchmark_data_smithi192_88961_object90491:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be50fdb0:::benchmark_data_smithi192_88961_object62754:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be5215c8:::benchmark_data_smithi192_88961_object59908:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be527016:::benchmark_data_smithi192_88961_object49264:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be531059:::benchmark_data_smithi192_88961_object55545:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be54b157:::benchmark_data_smithi192_88961_object43467:head# 2024-06-27T21:23:27.746 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be54ffa3:::benchmark_data_smithi192_88961_object49830:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be552f47:::benchmark_data_smithi192_88961_object58577:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be5553e4:::benchmark_data_smithi192_88961_object45785:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be5572c6:::benchmark_data_smithi192_88961_object74323:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be5a50bf:::benchmark_data_smithi192_88961_object70878:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be5a795d:::benchmark_data_smithi192_88961_object57928:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be5eadee:::benchmark_data_smithi192_88961_object80812:head# 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.747 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be602c5b:::benchmark_data_smithi192_88961_object64008:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be627814:::benchmark_data_smithi192_88961_object56663:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be657f77:::benchmark_data_smithi192_88961_object66418:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be66567b:::benchmark_data_smithi192_88961_object88784:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be667344:::benchmark_data_smithi192_88961_object80279:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be6765da:::benchmark_data_smithi192_88961_object78961:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be681406:::benchmark_data_smithi192_88961_object85367:head# 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.748 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be6c68ac:::benchmark_data_smithi192_88961_object63605:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be6e7c89:::benchmark_data_smithi192_88961_object86538:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be6f1b50:::benchmark_data_smithi192_88961_object61254:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be7189dc:::benchmark_data_smithi192_88961_object47120:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be73efba:::benchmark_data_smithi192_88961_object78588:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be744975:::benchmark_data_smithi192_88961_object72461:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be75576c:::benchmark_data_smithi192_88961_object65470:head# 2024-06-27T21:23:27.749 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be7611b9:::benchmark_data_smithi192_88961_object79068:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be79c277:::benchmark_data_smithi192_88961_object77356:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be7b6808:::benchmark_data_smithi192_88961_object73548:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be7dbbda:::benchmark_data_smithi192_88961_object73757:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be84b29f:::benchmark_data_smithi192_88961_object49735:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be85d775:::benchmark_data_smithi192_88961_object67758:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be876d70:::benchmark_data_smithi192_88961_object87924:head# 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.750 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be889d70:::benchmark_data_smithi192_88961_object80394:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be88e3ff:::benchmark_data_smithi192_88961_object46023:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be8a41c2:::benchmark_data_smithi192_88961_object69843:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be8a4504:::benchmark_data_smithi192_88961_object62806:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be8e13b6:::benchmark_data_smithi192_88961_object64873:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be8f0da9:::benchmark_data_smithi192_88961_object53897:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be9290e7:::benchmark_data_smithi192_88961_object85589:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be9668a4:::benchmark_data_smithi192_88961_object69348:head# 2024-06-27T21:23:27.751 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.752 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be974a77:::benchmark_data_smithi192_88961_object80810:head# 2024-06-27T21:23:27.752 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be9b76cd:::benchmark_data_smithi192_88961_object68095:head# 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:be9be462:::benchmark_data_smithi192_88961_object78453:head# 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bea2ff65:::benchmark_data_smithi192_88961_object48430:head# 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.768 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bea31183:::benchmark_data_smithi192_88961_object81800:head# 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bea397cd:::benchmark_data_smithi192_88961_object72324:head# 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bea6dc53:::benchmark_data_smithi192_88961_object67429:head# 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beada10d:::benchmark_data_smithi192_88961_object67773:head# 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beaf5005:::benchmark_data_smithi192_88961_object66586:head# 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beaf7837:::benchmark_data_smithi192_88961_object79268:head# 2024-06-27T21:23:27.769 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beb03f15:::benchmark_data_smithi192_88961_object69384:head# 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beb237d8:::benchmark_data_smithi192_88961_object43282:head# 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beb318c0:::benchmark_data_smithi192_88961_object44822:head# 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beb5595c:::benchmark_data_smithi192_88961_object55100:head# 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beb9051b:::benchmark_data_smithi192_88961_object64799:head# 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.770 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beb99d58:::benchmark_data_smithi192_88961_object62782:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bebc4e28:::benchmark_data_smithi192_88961_object53745:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bebe9b12:::benchmark_data_smithi192_88961_object75917:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bebefa0d:::benchmark_data_smithi192_88961_object89147:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bec0600e:::benchmark_data_smithi192_88961_object76211:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bec5ea64:::benchmark_data_smithi192_88961_object62546:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bec73978:::benchmark_data_smithi192_88961_object86197:head# 2024-06-27T21:23:27.771 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bec9d4f2:::benchmark_data_smithi192_88961_object63504:head# 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beca0dea:::benchmark_data_smithi192_88961_object44936:head# 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:becab14a:::benchmark_data_smithi192_88961_object59139:head# 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:becacf3a:::benchmark_data_smithi192_88961_object58069:head# 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:becb403f:::benchmark_data_smithi192_88961_object59998:head# 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:becc290c:::benchmark_data_smithi192_88961_object87953:head# 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:becdaacb:::benchmark_data_smithi192_88961_object71854:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:becffd12:::benchmark_data_smithi192_88961_object81561:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed01346:::benchmark_data_smithi192_88961_object71274:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed385e6:::benchmark_data_smithi192_88961_object52627:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed386f6:::benchmark_data_smithi192_88961_object84357:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed5d606:::benchmark_data_smithi192_88961_object81769:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed5edcf:::benchmark_data_smithi192_88961_object54755:head# 2024-06-27T21:23:27.773 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed690be:::benchmark_data_smithi192_88961_object72894:head# 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bed87547:::benchmark_data_smithi192_88961_object47480:head# 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bedb5f2d:::benchmark_data_smithi192_88961_object65807:head# 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bedb6a3a:::benchmark_data_smithi192_88961_object87998:head# 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bedd9225:::benchmark_data_smithi192_88961_object78372:head# 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bedf30af:::benchmark_data_smithi192_88961_object62390:head# 2024-06-27T21:23:27.774 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee09a31:::benchmark_data_smithi192_88961_object66446:head# 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee237b8:::benchmark_data_smithi192_88961_object78567:head# 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee35840:::benchmark_data_smithi192_88961_object86343:head# 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee3d44a:::benchmark_data_smithi192_88961_object71885:head# 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee6a360:::benchmark_data_smithi192_88961_object79743:head# 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee6b25a:::benchmark_data_smithi192_88961_object86328:head# 2024-06-27T21:23:27.775 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bee9c07c:::benchmark_data_smithi192_88961_object78817:head# 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beec54bc:::benchmark_data_smithi192_88961_object44124:head# 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beece4f3:::benchmark_data_smithi192_88961_object77068:head# 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beed48b2:::benchmark_data_smithi192_88961_object52668:head# 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bef21fea:::benchmark_data_smithi192_88961_object70253:head# 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bef28c5f:::benchmark_data_smithi192_88961_object51973:head# 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.776 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bef3f783:::benchmark_data_smithi192_88961_object50562:head# 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bef6660f:::benchmark_data_smithi192_88961_object69589:head# 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bef93df9:::benchmark_data_smithi192_88961_object69666:head# 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:befe664a:::benchmark_data_smithi192_88961_object63613:head# 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:beffc94a:::benchmark_data_smithi192_88961_object89502:head# 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf001a0a:::benchmark_data_smithi192_88961_object45763:head# 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.777 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf0075b4:::benchmark_data_smithi192_88961_object66479:head# 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf021927:::benchmark_data_smithi192_88961_object59057:head# 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf0365e2:::benchmark_data_smithi192_88961_object64520:head# 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf0688ef:::benchmark_data_smithi192_88961_object55897:head# 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf07b046:::benchmark_data_smithi192_88961_object84045:head# 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf09746b:::benchmark_data_smithi192_88961_object87590:head# 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.778 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf09b21c:::benchmark_data_smithi192_88961_object83234:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf0a89f9:::benchmark_data_smithi192_88961_object71370:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf0b901a:::benchmark_data_smithi192_88961_object54706:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf0c8d97:::benchmark_data_smithi192_88961_object58487:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf1128e3:::benchmark_data_smithi192_88961_object48112:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf14dc20:::benchmark_data_smithi192_88961_object78711:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf163f1d:::benchmark_data_smithi192_88961_object84491:head# 2024-06-27T21:23:27.779 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf17b2c0:::benchmark_data_smithi192_88961_object67530:head# 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf18a7a7:::benchmark_data_smithi192_88961_object90001:head# 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf1939a3:::benchmark_data_smithi192_88961_object67362:head# 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf193e4b:::benchmark_data_smithi192_88961_object66795:head# 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf198fff:::benchmark_data_smithi192_88961_object71614:head# 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf1babcc:::benchmark_data_smithi192_88961_object82435:head# 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.780 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf1bfaec:::benchmark_data_smithi192_88961_object83794:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf1db177:::benchmark_data_smithi192_88961_object75202:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2267ba:::benchmark_data_smithi192_88961_object83646:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2342ce:::benchmark_data_smithi192_88961_object79461:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf282fc3:::benchmark_data_smithi192_88961_object68721:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf283122:::benchmark_data_smithi192_88961_object63174:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf294222:::benchmark_data_smithi192_88961_object59964:head# 2024-06-27T21:23:27.781 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2bcd72:::benchmark_data_smithi192_88961_object48508:head# 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2c7d6b:::benchmark_data_smithi192_88961_object50249:head# 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2e0be3:::benchmark_data_smithi192_88961_object51285:head# 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2efdf7:::benchmark_data_smithi192_88961_object89433:head# 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf2fc6fe:::benchmark_data_smithi192_88961_object78227:head# 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf30a18a:::benchmark_data_smithi192_88961_object45651:head# 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.782 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf30fae8:::benchmark_data_smithi192_88961_object51629:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf32061c:::benchmark_data_smithi192_88961_object48738:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf32ab96:::benchmark_data_smithi192_88961_object78677:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf32bed9:::benchmark_data_smithi192_88961_object63596:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf32bf04:::benchmark_data_smithi192_88961_object83679:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf33c0b3:::benchmark_data_smithi192_88961_object87104:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3409da:::benchmark_data_smithi192_88961_object71982:head# 2024-06-27T21:23:27.783 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf34a4b4:::benchmark_data_smithi192_88961_object82296:head# 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf351a07:::benchmark_data_smithi192_88961_object75088:head# 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf354a04:::benchmark_data_smithi192_88961_object61627:head# 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf35af65:::benchmark_data_smithi192_88961_object47073:head# 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf35d849:::benchmark_data_smithi192_88961_object50192:head# 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf36c574:::benchmark_data_smithi192_88961_object76022:head# 2024-06-27T21:23:27.784 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf37af81:::benchmark_data_smithi192_88961_object81516:head# 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf39c626:::benchmark_data_smithi192_88961_object73755:head# 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3a11ce:::benchmark_data_smithi192_88961_object67137:head# 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3a207b:::benchmark_data_smithi192_88961_object86424:head# 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3ad8cb:::benchmark_data_smithi192_88961_object71657:head# 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3afa4e:::benchmark_data_smithi192_88961_object46363:head# 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.785 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3cc4f8:::benchmark_data_smithi192_88961_object59410:head# 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf3f007c:::benchmark_data_smithi192_88961_object47081:head# 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf4132d7:::benchmark_data_smithi192_88961_object55685:head# 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf42cf7b:::benchmark_data_smithi192_88961_object60466:head# 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf43bc73:::benchmark_data_smithi192_88961_object89843:head# 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf449ca1:::benchmark_data_smithi192_88961_object61659:head# 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.786 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf455e3e:::benchmark_data_smithi192_88961_object50882:head# 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf473d04:::benchmark_data_smithi192_88961_object76598:head# 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf47bed4:::benchmark_data_smithi192_88961_object50315:head# 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf492159:::benchmark_data_smithi192_88961_object66327:head# 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf4b0943:::benchmark_data_smithi192_88961_object89256:head# 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf4ed845:::benchmark_data_smithi192_88961_object53123:head# 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.787 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf500259:::benchmark_data_smithi192_88961_object74652:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf520e67:::benchmark_data_smithi192_88961_object75978:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf52d0f4:::benchmark_data_smithi192_88961_object43283:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf556e98:::benchmark_data_smithi192_88961_object78971:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf599fe8:::benchmark_data_smithi192_88961_object48226:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf5b6b6d:::benchmark_data_smithi192_88961_object70048:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf5d49a9:::benchmark_data_smithi192_88961_object55626:head# 2024-06-27T21:23:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf5d933b:::benchmark_data_smithi192_88961_object45367:head# 2024-06-27T21:23:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf5e0fd5:::benchmark_data_smithi192_88961_object67902:head# 2024-06-27T21:23:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf5e62d8:::benchmark_data_smithi192_88961_object63925:head# 2024-06-27T21:23:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:27.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:27.804 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf5f4595:::benchmark_data_smithi192_88961_object80369:head# 2024-06-27T21:23:27.804 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf600fac:::benchmark_data_smithi192_88961_object42992:head# 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf61b78d:::benchmark_data_smithi192_88961_object78142:head# 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf629f9a:::benchmark_data_smithi192_88961_object48570:head# 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf64ca28:::benchmark_data_smithi192_88961_object45944:head# 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf663fb2:::benchmark_data_smithi192_88961_object76061:head# 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.805 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf66541d:::benchmark_data_smithi192_88961_object64684:head# 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf66f95b:::benchmark_data_smithi192_88961_object44550:head# 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf6763f4:::benchmark_data_smithi192_88961_object68762:head# 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf67c319:::benchmark_data_smithi192_88961_object63473:head# 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf67cdca:::benchmark_data_smithi192_88961_object45464:head# 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf6917a3:::benchmark_data_smithi192_88961_object66905:head# 2024-06-27T21:23:27.806 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf6c4f7c:::benchmark_data_smithi192_88961_object88865:head# 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf70fb8d:::benchmark_data_smithi192_88961_object79252:head# 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf72816d:::benchmark_data_smithi192_88961_object76625:head# 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7282fb:::benchmark_data_smithi192_88961_object45550:head# 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7397df:::benchmark_data_smithi192_88961_object75981:head# 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf764343:::benchmark_data_smithi192_88961_object56692:head# 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.807 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf770b16:::benchmark_data_smithi192_88961_object73101:head# 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7803d7:::benchmark_data_smithi192_88961_object72439:head# 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf78d7c0:::benchmark_data_smithi192_88961_object73854:head# 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf78f1f1:::benchmark_data_smithi192_88961_object82914:head# 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf79687b:::benchmark_data_smithi192_88961_object63784:head# 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7bce01:::benchmark_data_smithi192_88961_object49825:head# 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.808 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7d7600:::benchmark_data_smithi192_88961_object54128:head# 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7dd979:::benchmark_data_smithi192_88961_object50026:head# 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7ed3a7:::benchmark_data_smithi192_88961_object51548:head# 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7ed76b:::benchmark_data_smithi192_88961_object88204:head# 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7f0787:::benchmark_data_smithi192_88961_object73694:head# 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf7f63f3:::benchmark_data_smithi192_88961_object76909:head# 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.809 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf8195a2:::benchmark_data_smithi192_88961_object83671:head# 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf83b46e:::benchmark_data_smithi192_88961_object76255:head# 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf848d5a:::benchmark_data_smithi192_88961_object55888:head# 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf84db13:::benchmark_data_smithi192_88961_object49800:head# 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf868f37:::benchmark_data_smithi192_88961_object58157:head# 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf897c5b:::benchmark_data_smithi192_88961_object72676:head# 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.810 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf8c0523:::benchmark_data_smithi192_88961_object75349:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf91152f:::benchmark_data_smithi192_88961_object83667:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf91d9b5:::benchmark_data_smithi192_88961_object70399:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf92560d:::benchmark_data_smithi192_88961_object61077:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf978bfa:::benchmark_data_smithi192_88961_object51072:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf97bd56:::benchmark_data_smithi192_88961_object52493:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf98f704:::benchmark_data_smithi192_88961_object73118:head# 2024-06-27T21:23:27.811 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf98f716:::benchmark_data_smithi192_88961_object87526:head# 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9992ee:::benchmark_data_smithi192_88961_object77492:head# 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9a3c19:::benchmark_data_smithi192_88961_object65018:head# 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9a5215:::benchmark_data_smithi192_88961_object78841:head# 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9ce4a1:::benchmark_data_smithi192_88961_object68599:head# 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9e1cca:::benchmark_data_smithi192_88961_object73912:head# 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.812 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9e4ecf:::benchmark_data_smithi192_88961_object77966:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bf9e8a66:::benchmark_data_smithi192_88961_object53161:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa0e508:::benchmark_data_smithi192_88961_object79486:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa10314:::benchmark_data_smithi192_88961_object61301:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa1efb5:::benchmark_data_smithi192_88961_object52099:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa3301f:::benchmark_data_smithi192_88961_object84775:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa44896:::benchmark_data_smithi192_88961_object61667:head# 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.813 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa5a531:::benchmark_data_smithi192_88961_object87224:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa6d2a7:::benchmark_data_smithi192_88961_object84716:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa74900:::benchmark_data_smithi192_88961_object72784:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa7cd1c:::benchmark_data_smithi192_88961_object62929:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfa91778:::benchmark_data_smithi192_88961_object62379:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfad2f6c:::benchmark_data_smithi192_88961_object44369:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfafe3c9:::benchmark_data_smithi192_88961_object52947:head# 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.814 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb1c238:::benchmark_data_smithi192_88961_object83285:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb1d422:::benchmark_data_smithi192_88961_object81286:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb3c5dc:::benchmark_data_smithi192_88961_object43869:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb44561:::benchmark_data_smithi192_88961_object85373:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb5c328:::benchmark_data_smithi192_88961_object89076:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb79fef:::benchmark_data_smithi192_88961_object45102:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb850a5:::benchmark_data_smithi192_88961_object48970:head# 2024-06-27T21:23:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfb85676:::benchmark_data_smithi192_88961_object60334:head# 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfba7bb5:::benchmark_data_smithi192_88961_object60004:head# 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfbba593:::benchmark_data_smithi192_88961_object49598:head# 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfbdb5ad:::benchmark_data_smithi192_88961_object84461:head# 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfbe4107:::benchmark_data_smithi192_88961_object51688:head# 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfbe7f12:::benchmark_data_smithi192_88961_object48801:head# 2024-06-27T21:23:27.816 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfc379bd:::benchmark_data_smithi192_88961_object77915:head# 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfc54d6f:::benchmark_data_smithi192_88961_object61548:head# 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfc632cc:::benchmark_data_smithi192_88961_object77198:head# 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfc6b790:::benchmark_data_smithi192_88961_object66408:head# 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfc706bd:::benchmark_data_smithi192_88961_object70948:head# 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfcc629b:::benchmark_data_smithi192_88961_object46738:head# 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.817 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfcdc58d:::benchmark_data_smithi192_88961_object87192:head# 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfcf436f:::benchmark_data_smithi192_88961_object46884:head# 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd03084:::benchmark_data_smithi192_88961_object85768:head# 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd0c1bc:::benchmark_data_smithi192_88961_object57688:head# 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd0d07d:::benchmark_data_smithi192_88961_object65140:head# 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd3bb69:::benchmark_data_smithi192_88961_object58759:head# 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.818 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd5048b:::benchmark_data_smithi192_88961_object67059:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd522e3:::benchmark_data_smithi192_88961_object77942:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd53ec3:::benchmark_data_smithi192_88961_object79648:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd6f3d1:::benchmark_data_smithi192_88961_object62843:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd74fb9:::benchmark_data_smithi192_88961_object86087:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd7d198:::benchmark_data_smithi192_88961_object66176:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfd7ea39:::benchmark_data_smithi192_88961_object53165:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfddfc37:::benchmark_data_smithi192_88961_object86429:head# 2024-06-27T21:23:27.819 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfde4d62:::benchmark_data_smithi192_88961_object68146:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfdecec6:::benchmark_data_smithi192_88961_object71259:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfdf7049:::benchmark_data_smithi192_88961_object75302:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfe063fe:::benchmark_data_smithi192_88961_object55156:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfe1a62b:::benchmark_data_smithi192_88961_object79020:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfe2b9f7:::benchmark_data_smithi192_88961_object72341:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfe4abec:::benchmark_data_smithi192_88961_object51002:head# 2024-06-27T21:23:27.820 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfe61ee2:::benchmark_data_smithi192_88961_object61560:head# 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfeb3752:::benchmark_data_smithi192_88961_object86848:head# 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfeb3bec:::benchmark_data_smithi192_88961_object57230:head# 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfeb682d:::benchmark_data_smithi192_88961_object49813:head# 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfee6e86:::benchmark_data_smithi192_88961_object68201:head# 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfef11f1:::benchmark_data_smithi192_88961_object63465:head# 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.821 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff0142c:::benchmark_data_smithi192_88961_object78544:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff13f93:::benchmark_data_smithi192_88961_object50134:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff3bb72:::benchmark_data_smithi192_88961_object60573:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff5641b:::benchmark_data_smithi192_88961_object49049:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff58966:::benchmark_data_smithi192_88961_object89151:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff74081:::benchmark_data_smithi192_88961_object69754:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bff782bb:::benchmark_data_smithi192_88961_object59359:head# 2024-06-27T21:23:27.822 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.823 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bffa3c29:::benchmark_data_smithi192_88961_object76713:head# 2024-06-27T21:23:27.823 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.823 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:bfff9254:::benchmark_data_smithi192_88961_object87786:head# 2024-06-27T21:23:27.823 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-06-27T21:23:27.823 INFO:teuthology.orchestra.run.smithi190.stdout:write_pg epoch 347 info 4.d( v 347'8364 (0'0,347'8364] local-lis/les=301/302 n=2974 ec=282/282 lis/c=301/301 les/c/f=302/302/0 sis=301) 2024-06-27T21:23:27.875 INFO:teuthology.orchestra.run.smithi190.stdout:Import successful 2024-06-27T21:23:27.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:27.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:27.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-27T21:23:27.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-27T21:23:28.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:27 smithi190 ceph-mon[32247]: pgmap v935: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 8129/98178 objects degraded (8.280%); 6026/98178 objects misplaced (6.138%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:23:28.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2451268511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:28.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:28.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:27 smithi192 ceph-mon[32956]: pgmap v935: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 8129/98178 objects degraded (8.280%); 6026/98178 objects misplaced (6.138%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:23:28.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2451268511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:28.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:28.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:27 smithi151 ceph-mon[28965]: pgmap v935: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.0 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 8129/98178 objects degraded (8.280%); 6026/98178 objects misplaced (6.138%); 3.9 MiB/s, 62 objects/s recovering 2024-06-27T21:23:28.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2451268511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:28.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:28.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:28.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:28.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:28.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:28.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:28.467 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:28.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:28.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:28.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:28.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:28.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:28.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:28.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:28.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:28.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:28 smithi151 ceph-mon[28965]: osdmap e356: 12 total, 11 up, 12 in 2024-06-27T21:23:28.961 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:28.961 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:28.961 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:28 smithi190 ceph-mon[32247]: osdmap e356: 12 total, 11 up, 12 in 2024-06-27T21:23:28.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:28.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:29.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/exp.4.d.4 2024-06-27T21:23:29.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:29.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:29.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:28 smithi192 ceph-mon[32956]: osdmap e356: 12 total, 11 up, 12 in 2024-06-27T21:23:29.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:29.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:29.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:29.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:29.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:29.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:29.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:29.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:29.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:29.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:29.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:29.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:29.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:29.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:29.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:23:29.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:30.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:29 smithi190 ceph-mon[32247]: pgmap v937: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 7890/96774 objects degraded (8.153%); 6026/96774 objects misplaced (6.227%); 2.5 MiB/s, 40 objects/s recovering 2024-06-27T21:23:30.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7890/96774 objects degraded (8.153%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:30.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:30.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:29 smithi192 ceph-mon[32956]: pgmap v937: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 7890/96774 objects degraded (8.153%); 6026/96774 objects misplaced (6.227%); 2.5 MiB/s, 40 objects/s recovering 2024-06-27T21:23:30.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7890/96774 objects degraded (8.153%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:30.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:29 smithi151 ceph-mon[28965]: pgmap v937: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 2.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 7890/96774 objects degraded (8.153%); 6026/96774 objects misplaced (6.227%); 2.5 MiB/s, 40 objects/s recovering 2024-06-27T21:23:30.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7890/96774 objects degraded (8.153%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:30.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:30.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:30.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:30.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:30.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:30.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:30.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:30.536 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:30.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:30.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:30.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:30.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:30.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:30.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:30.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:31.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:31.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:31.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:31.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:31.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:31.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:31.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:31.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:31.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:31.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:31.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:31.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:31.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:31.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:31 smithi151 ceph-mon[28965]: pgmap v938: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 7634/90807 objects degraded (8.407%); 5313/90807 objects misplaced (5.851%); 4.1 MiB/s, 66 objects/s recovering 2024-06-27T21:23:31.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:31.924 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:23:31.924 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:23:31.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:31.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:32.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:31 smithi190 ceph-mon[32247]: pgmap v938: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 7634/90807 objects degraded (8.407%); 5313/90807 objects misplaced (5.851%); 4.1 MiB/s, 66 objects/s recovering 2024-06-27T21:23:32.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:31 smithi192 ceph-mon[32956]: pgmap v938: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.8 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 7634/90807 objects degraded (8.407%); 5313/90807 objects misplaced (5.851%); 4.1 MiB/s, 66 objects/s recovering 2024-06-27T21:23:32.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:32.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:32.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:32.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:32.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:32.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:32.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:32.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:32.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:32.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:32.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:32.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:23:32.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:32.750 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:23:32.766 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:32.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:32.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:32.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:32.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:32.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/319099297' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:32.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:32 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:33.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:33.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/319099297' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:33.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:32 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:33.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:33.033 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:33.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/319099297' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:33.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:32 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:23:33.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:33.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:33.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:33.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:33.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:33.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:33.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:33.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:33.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:33.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:33.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:33.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:33.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:33.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:33.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:33.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:33.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:33 smithi151 ceph-mon[28965]: pgmap v939: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 7135/86505 objects degraded (8.248%); 5313/86505 objects misplaced (6.142%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:23:33.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:33 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:33.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:33 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:23:33.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:33 smithi151 ceph-mon[28965]: osdmap e357: 12 total, 11 up, 12 in 2024-06-27T21:23:34.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:33 smithi190 ceph-mon[32247]: pgmap v939: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 7135/86505 objects degraded (8.248%); 5313/86505 objects misplaced (6.142%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:23:34.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:33 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:34.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:33 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:23:34.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:33 smithi190 ceph-mon[32247]: osdmap e357: 12 total, 11 up, 12 in 2024-06-27T21:23:34.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:34.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:33 smithi192 ceph-mon[32956]: pgmap v939: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 7135/86505 objects degraded (8.248%); 5313/86505 objects misplaced (6.142%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:23:34.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:33 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:34.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:33 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:23:34.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:33 smithi192 ceph-mon[32956]: osdmap e357: 12 total, 11 up, 12 in 2024-06-27T21:23:34.059 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:34.066 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11] 2024-06-27T21:23:34.066 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-06-27T21:23:34.066 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-06-27T21:23:34.066 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4.service 2024-06-27T21:23:34.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl start ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4 2024-06-27T21:23:34.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:34.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:34.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:34.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:34.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:34.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:34.447 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:34 smithi190 systemd[1]: Starting Ceph osd.4 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:23:34.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:34.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:34.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:34.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:34.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:34.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:34.889 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7135/86505 objects degraded (8.248%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:34.902 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:23:34.902 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:23:34.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:34.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:34.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:35.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7135/86505 objects degraded (8.248%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:35.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7135/86505 objects degraded (8.248%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:35.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:35.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:35.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:35.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:35.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:35.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:35.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:35.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:35.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:35.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:35.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:35.722 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:35 smithi190 podman[99881]: 2024-06-27 21:23:35.467660844 +0000 UTC m=+0.556943534 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:23:35.723 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:35 smithi190 podman[99881]: 2024-06-27 21:23:35.591340243 +0000 UTC m=+0.680622943 container create 3927bc298adc217aa616bc81ba8296bd9980423a032e263b334cd225b963cb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:23:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:35.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:35.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:35.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:35.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:35.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:35 smithi151 ceph-mon[28965]: pgmap v941: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 6885/82770 objects degraded (8.318%); 4551/82770 objects misplaced (5.498%); 3.8 MiB/s, 61 objects/s recovering 2024-06-27T21:23:35.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:35.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:36.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:35 smithi190 ceph-mon[32247]: pgmap v941: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 6885/82770 objects degraded (8.318%); 4551/82770 objects misplaced (5.498%); 3.8 MiB/s, 61 objects/s recovering 2024-06-27T21:23:36.012 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:35 smithi190 podman[99881]: 2024-06-27 21:23:35.917250081 +0000 UTC m=+1.006532792 container init 3927bc298adc217aa616bc81ba8296bd9980423a032e263b334cd225b963cb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:23:36.013 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:35 smithi190 podman[99881]: 2024-06-27 21:23:35.923501773 +0000 UTC m=+1.012784470 container start 3927bc298adc217aa616bc81ba8296bd9980423a032e263b334cd225b963cb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True) 2024-06-27T21:23:36.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:35 smithi192 ceph-mon[32956]: pgmap v941: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.7 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 641 op/s; 6885/82770 objects degraded (8.318%); 4551/82770 objects misplaced (5.498%); 3.8 MiB/s, 61 objects/s recovering 2024-06-27T21:23:36.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:36.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:36.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:23:36.214 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:23:36.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:23:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:23:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-06-27T21:23:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:23:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:23:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:23:36.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:36.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:23:36.238 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:23:36.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:36.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:23:36.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:23:36.309 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:23:36.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:23:36.345 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:23:36.345 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:23:36.358 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 podman[99881]: 2024-06-27 21:23:36.036181353 +0000 UTC m=+1.125464050 container attach 3927bc298adc217aa616bc81ba8296bd9980423a032e263b334cd225b963cb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:23:36.358 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-27T21:23:36.358 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-27T21:23:36.358 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-27T21:23:36.359 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-06-27T21:23:36.359 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-27T21:23:36.359 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-27T21:23:36.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:36.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:36.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:36.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:23:36.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:36.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:36.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-27T21:23:36.680 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-06-27T21:23:36.681 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-06-27T21:23:36.681 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-27T21:23:36.681 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-27T21:23:36.681 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate[99967]: --> ceph-volume raw activate successful for osd ID: 4 2024-06-27T21:23:36.681 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 bash[99881]: --> ceph-volume raw activate successful for osd ID: 4 2024-06-27T21:23:36.681 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:36 smithi190 podman[99881]: 2024-06-27 21:23:36.464730734 +0000 UTC m=+1.554013428 container died 3927bc298adc217aa616bc81ba8296bd9980423a032e263b334cd225b963cb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-27T21:23:36.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:36.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:36.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:36.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3265292609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:36.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:36.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:37.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3265292609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:37.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3265292609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:37.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:37.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-06-27T21:23:37.176 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-06-27T21:23:37.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:37.261 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:37.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:37.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:37.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:37.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:37.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:37.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:37.661 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:37 smithi190 podman[100100]: 2024-06-27 21:23:37.400188008 +0000 UTC m=+0.962249486 container remove 3927bc298adc217aa616bc81ba8296bd9980423a032e263b334cd225b963cb14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-27T21:23:37.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:37.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:37.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:37.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:37.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:37.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:37 smithi151 ceph-mon[28965]: pgmap v942: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 6132/74604 objects degraded (8.219%); 4551/74604 objects misplaced (6.100%); 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:23:37.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:37.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:37 smithi190 ceph-mon[32247]: pgmap v942: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 6132/74604 objects degraded (8.219%); 4551/74604 objects misplaced (6.100%); 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:23:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:37.953 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:37.953 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:37 smithi190 podman[100212]: 2024-06-27 21:23:37.567191357 +0000 UTC m=+0.033932871 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:23:37.953 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:37 smithi190 podman[100212]: 2024-06-27 21:23:37.684656604 +0000 UTC m=+0.151398115 container create 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-06-27T21:23:37.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:38.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:37 smithi192 ceph-mon[32956]: pgmap v942: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 6132/74604 objects degraded (8.219%); 4551/74604 objects misplaced (6.100%); 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:23:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:38.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:38.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:23:38.068 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:38.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:38.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:38.230 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:37 smithi190 podman[100212]: 2024-06-27 21:23:37.954639996 +0000 UTC m=+0.421381518 container init 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:23:38.230 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:37 smithi190 podman[100212]: 2024-06-27 21:23:37.959347486 +0000 UTC m=+0.426088991 container start 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:23:38.230 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:37 smithi190 ceph-osd[100258]: -- 172.21.15.190:0/4173642013 <== mon.2 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555857987a40 con 0x555858750400 2024-06-27T21:23:38.230 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:38 smithi190 bash[100212]: 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a 2024-06-27T21:23:38.231 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:38 smithi190 systemd[1]: Started Ceph osd.4 for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:23:38.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:38.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:38.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:38.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:38.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:38.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:38.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:38.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:38.762 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:38 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:23:38.502+0000 7f20b2917640 -1 Falling back to public interface 2024-06-27T21:23:38.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:38.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:38.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:38.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:38 smithi151 ceph-mon[28965]: osdmap e358: 12 total, 11 up, 12 in 2024-06-27T21:23:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:38.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:38.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:39.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:38 smithi192 ceph-mon[32956]: osdmap e358: 12 total, 11 up, 12 in 2024-06-27T21:23:39.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:39.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:39.110 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:38 smithi190 ceph-mon[32247]: osdmap e358: 12 total, 11 up, 12 in 2024-06-27T21:23:39.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:23:39.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:39.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:39.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:39.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:39.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:39.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:39.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:39.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:39.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:39.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:39.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:39.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:39.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:40.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:39 smithi190 ceph-mon[32247]: pgmap v944: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 5898/73101 objects degraded (8.068%); 4551/73101 objects misplaced (6.226%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:23:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5898/73101 objects degraded (8.068%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:40.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:40.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:39 smithi192 ceph-mon[32956]: pgmap v944: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 5898/73101 objects degraded (8.068%); 4551/73101 objects misplaced (6.226%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:23:40.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:40.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:40.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5898/73101 objects degraded (8.068%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:40.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:39 smithi151 ceph-mon[28965]: pgmap v944: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 5898/73101 objects degraded (8.068%); 4551/73101 objects misplaced (6.226%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:23:40.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:40.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:40.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5898/73101 objects degraded (8.068%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:40.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:40.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:40.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:40.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:40.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:40.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:40.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:40.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:40.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:40.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:40.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:40.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:40.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:40.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:40.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:41.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:40 smithi190 ceph-mon[32247]: osdmap e359: 12 total, 11 up, 12 in 2024-06-27T21:23:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/358355867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:23:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:40 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:23:41.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:41.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:41.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:41.163 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:23:41.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:40 smithi151 ceph-mon[28965]: osdmap e359: 12 total, 11 up, 12 in 2024-06-27T21:23:41.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/358355867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:23:41.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:40 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:23:41.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:41.195 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:23:41.195 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:23:41.195 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:23:41.195 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:23:41.195 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:23:41.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:41.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:40 smithi192 ceph-mon[32956]: osdmap e359: 12 total, 11 up, 12 in 2024-06-27T21:23:41.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/358355867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:23:41.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:40 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:23:41.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:41.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:41.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:41.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:41.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:41.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:41.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:41.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:41.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:41.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:41.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:41.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:42.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:42.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:41 smithi190 ceph-mon[32247]: pgmap v946: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 5630/66906 objects degraded (8.415%); 3903/66906 objects misplaced (5.834%); 4.1 MiB/s, 64 objects/s recovering 2024-06-27T21:23:42.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:41 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:42.084 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:41 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:23:42.084 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:41 smithi190 ceph-mon[32247]: osdmap e360: 12 total, 11 up, 12 in 2024-06-27T21:23:42.084 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:41 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:23:41.814+0000 7f20b2917640 -1 osd.4 347 log_to_monitors true 2024-06-27T21:23:42.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:42.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:23:42.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:42.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:41 smithi151 ceph-mon[28965]: pgmap v946: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 5630/66906 objects degraded (8.415%); 3903/66906 objects misplaced (5.834%); 4.1 MiB/s, 64 objects/s recovering 2024-06-27T21:23:42.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:41 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:42.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:41 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:23:42.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:41 smithi151 ceph-mon[28965]: osdmap e360: 12 total, 11 up, 12 in 2024-06-27T21:23:42.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:42.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:41 smithi192 ceph-mon[32956]: pgmap v946: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 5630/66906 objects degraded (8.415%); 3903/66906 objects misplaced (5.834%); 4.1 MiB/s, 64 objects/s recovering 2024-06-27T21:23:42.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:41 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:23:42.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:41 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:23:42.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:41 smithi192 ceph-mon[32956]: osdmap e360: 12 total, 11 up, 12 in 2024-06-27T21:23:42.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:42.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:42.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:42.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:no valid command found; 10 closest matches: 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:0 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:1 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:2 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:abort 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:assert 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:bluefs files list 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:bluefs stats 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:bluestore allocator dump block 2024-06-27T21:23:42.493 INFO:teuthology.orchestra.run.smithi190.stderr:bluestore allocator fragmentation block 2024-06-27T21:23:42.494 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: invalid command 2024-06-27T21:23:42.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:42.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:42.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:42.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:42.770 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:42.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:42.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:42.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:42.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:42.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:43.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:42 smithi151 ceph-mon[28965]: from='osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:23:43.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:42 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:23:43.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:43.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:43.207 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:42 smithi190 ceph-mon[32247]: from='osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:23:43.207 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:42 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:23:43.207 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:23:42 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:23:42.834+0000 7f20a66ba640 -1 osd.4 361 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-27T21:23:43.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:43.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:23:43.252 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-06-27T21:23:43.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:42 smithi192 ceph-mon[32956]: from='osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:23:43.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:42 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-27T21:23:43.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:43.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:43.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:43.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:43.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:43.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:43.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:43.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:43.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:43.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:43.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:43.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:43.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:43.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:44.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:44.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:44.173 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:43 smithi190 ceph-mon[32247]: pgmap v948: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 5126/62367 objects degraded (8.219%); 3903/62367 objects misplaced (6.258%); 2.2 MiB/s, 35 objects/s recovering 2024-06-27T21:23:44.173 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:43 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-27T21:23:44.173 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:43 smithi190 ceph-mon[32247]: osdmap e361: 12 total, 11 up, 12 in 2024-06-27T21:23:44.173 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:43 smithi190 ceph-mon[32247]: from='osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:23:44.173 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:43 smithi190 ceph-mon[32247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:23:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:43 smithi151 ceph-mon[28965]: pgmap v948: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 5126/62367 objects degraded (8.219%); 3903/62367 objects misplaced (6.258%); 2.2 MiB/s, 35 objects/s recovering 2024-06-27T21:23:44.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:43 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-27T21:23:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:43 smithi151 ceph-mon[28965]: osdmap e361: 12 total, 11 up, 12 in 2024-06-27T21:23:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:43 smithi151 ceph-mon[28965]: from='osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:23:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:43 smithi151 ceph-mon[28965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:23:44.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:44.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:43 smithi192 ceph-mon[32956]: pgmap v948: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.3 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 5126/62367 objects degraded (8.219%); 3903/62367 objects misplaced (6.258%); 2.2 MiB/s, 35 objects/s recovering 2024-06-27T21:23:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:43 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-27T21:23:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:43 smithi192 ceph-mon[32956]: osdmap e361: 12 total, 11 up, 12 in 2024-06-27T21:23:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:43 smithi192 ceph-mon[32956]: from='osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:23:44.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:43 smithi192 ceph-mon[32956]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-06-27T21:23:44.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:44.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:44.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:44.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:44.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:44.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:44.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:44.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:44.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:44.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:44.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:44.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:44.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:45.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-27T21:23:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633] boot 2024-06-27T21:23:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: osdmap e362: 12 total, 12 up, 12 in 2024-06-27T21:23:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:23:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:23:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5126/62367 objects degraded (8.219%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:44 smithi190 ceph-mon[32247]: osdmap e363: 12 total, 12 up, 12 in 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121442 2.2 2:478d92b6:::benchmark_data_smithi192_59818_object29367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:46.100577+0000", 2024-06-27T21:23:45.083 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 598.98208028399995, 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.223586971, 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121442 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100577+0000", 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100577+0000", 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100578+0000", 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0869999999999999e-06 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100582+0000", 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3869999999999997e-06 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100594+0000", 2024-06-27T21:23:45.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2021000000000001e-05 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100600+0000", 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3430000000000003e-06 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100657+0000", 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7036000000000001e-05 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100783+0000", 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125826 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.100911+0000", 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128477 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.104303+0000", 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033924519999999998 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.104493+0000", 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000189614 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.324088+0000", 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21959461799999999 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.324130+0000", 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2110999999999997e-05 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.324164+0000", 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3998999999999998e-05 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:121449 2.3 2:cd1bd32e:::benchmark_data_smithi192_59818_object29374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:13:46.119375+0000", 2024-06-27T21:23:45.091 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 598.96328255399999, 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.342841958, 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 121449 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119375+0000", 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119375+0000", 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119376+0000", 2024-06-27T21:23:45.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6750000000000001e-06 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119383+0000", 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8410000000000002e-06 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119401+0000", 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7441000000000001e-05 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119410+0000", 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5370000000000003e-06 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119470+0000", 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0226999999999998e-05 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119554+0000", 2024-06-27T21:23:45.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3463999999999999e-05 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.119622+0000", 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8194999999999993e-05 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.122203+0000", 2024-06-27T21:23:45.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025806240000000001 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.122441+0000", 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023860299999999999 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.462142+0000", 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33970106700000002 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.462182+0000", 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9298999999999998e-05 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:13:46.462217+0000", 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4984999999999998e-05 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.098 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24508.0:159656 2.3 2:c721332e:::benchmark_data_smithi192_59818_object67581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:14:43.397534+0000", 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 541.68512352300002, 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22404874499999999, 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24508", 2024-06-27T21:23:45.099 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/333143582", 2024-06-27T21:23:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.101 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 159656 2024-06-27T21:23:45.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.101 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397534+0000", 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397534+0000", 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397535+0000", 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.623e-06 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397542+0000", 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7870000000000004e-06 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397561+0000", 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9289000000000001e-05 2024-06-27T21:23:45.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397570+0000", 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9500000000000007e-06 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397635+0000", 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4441000000000002e-05 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397770+0000", 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013576300000000001 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.397902+0000", 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013167300000000001 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.401685+0000", 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037830910000000001 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.401731+0000", 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5924999999999998e-05 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621473+0000", 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.219741833 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621537+0000", 2024-06-27T21:23:45.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3675000000000003e-05 2024-06-27T21:23:45.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621582+0000", 2024-06-27T21:23:45.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5695000000000002e-05 2024-06-27T21:23:45.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.109 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24508.0:159656 2.3 e158/104 ondisk, result = 0)", 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:14:43.403092+0000", 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 541.67956548400002, 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.218495682, 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.110 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.403092+0000", 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.403092+0000", 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.403093+0000", 2024-06-27T21:23:45.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8899999999999998e-07 2024-06-27T21:23:45.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.403098+0000", 2024-06-27T21:23:45.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.502e-06 2024-06-27T21:23:45.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621399+0000", 2024-06-27T21:23:45.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21830091300000001 2024-06-27T21:23:45.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621406+0000", 2024-06-27T21:23:45.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5570000000000001e-06 2024-06-27T21:23:45.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621458+0000", 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2543000000000002e-05 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621467+0000", 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9740000000000008e-06 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:14:43.621587+0000", 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012030400000000001 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.117 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:34085 3.c 3:31ab7127:::benchmark_data_smithi192_74314_object34084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:40.792879+0000", 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 424.289778087, 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24000338399999999, 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 34085 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.118 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.792879+0000", 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.792879+0000", 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.792883+0000", 2024-06-27T21:23:45.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2010000000000001e-06 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.792932+0000", 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8562e-05 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.792948+0000", 2024-06-27T21:23:45.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5829999999999999e-05 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.792955+0000", 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8290000000000008e-06 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.793011+0000", 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5065999999999998e-05 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.793124+0000", 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113947 2024-06-27T21:23:45.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.793301+0000", 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000176787 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.794901+0000", 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001600092 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:40.795196+0000", 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029422699999999999 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032774+0000", 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.237578862 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032838+0000", 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3967999999999995e-05 2024-06-27T21:23:45.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:41.032882+0000", 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4013e-05 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.126 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:41324 3.c 3:35d56568:::benchmark_data_smithi192_74314_object41323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:16:46.496883+0000", 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 418.58577445700001, 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21720515000000001, 2024-06-27T21:23:45.127 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 41324 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.128 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496883+0000", 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496883+0000", 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496884+0000", 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4980000000000001e-06 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496925+0000", 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1217999999999997e-05 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496936+0000", 2024-06-27T21:23:45.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0869e-05 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496943+0000", 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8510000000000001e-06 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.496986+0000", 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2861999999999997e-05 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.497072+0000", 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5972999999999998e-05 2024-06-27T21:23:45.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.497162+0000", 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9581999999999997e-05 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.498533+0000", 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001371262 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.498897+0000", 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036452499999999998 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.713983+0000", 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21508580999999999 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.714052+0000", 2024-06-27T21:23:45.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9010999999999994e-05 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:16:46.714088+0000", 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5689000000000003e-05 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:91006 3.c 3:3f210be7:::benchmark_data_smithi192_74314_object91005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:27.762909+0000", 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 377.31974834300001, 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.220284902, 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 91006 2024-06-27T21:23:45.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762909+0000", 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762909+0000", 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762911+0000", 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.731e-06 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762970+0000", 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9595000000000001e-05 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762984+0000", 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3728e-05 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.762991+0000", 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8759999999999999e-06 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763046+0000", 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4910000000000001e-05 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763162+0000", 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115986 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.763295+0000", 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013328600000000001 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.765973+0000", 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026781690000000002 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.766213+0000", 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000239616 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.983105+0000", 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21689233699999999 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.983156+0000", 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0858e-05 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:27.983194+0000", 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7809999999999999e-05 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.144 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:110815 3.5 3:acba5e69:::benchmark_data_smithi192_74314_object4177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:52.426752+0000", 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 352.65590469699998, 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26183841299999999, 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.145 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 110815 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.426752+0000", 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.426752+0000", 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.426753+0000", 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1299999999999998e-07 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.426758+0000", 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1899999999999997e-06 2024-06-27T21:23:45.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.426768+0000", 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.048e-05 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.426773+0000", 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3820000000000003e-06 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.450503+0000", 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023729689000000002 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.450619+0000", 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011547799999999999 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.466944+0000", 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016325276 2024-06-27T21:23:45.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.467057+0000", 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113482 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.467191+0000", 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013367300000000001 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.473483+0000", 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0062921660000000001 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.484283+0000", 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010799806 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.684069+0000", 2024-06-27T21:23:45.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19978574099999999 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.684136+0000", 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7308e-05 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.688591+0000", 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044548290000000004 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.152 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:110836 3.c 3:324d7afa:::benchmark_data_smithi192_74314_object4198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:52.454753+0000", 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 352.627903747, 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.217954271, 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.153 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 110836 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454753+0000", 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454753+0000", 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454755+0000", 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3969999999999999e-06 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454760+0000", 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6779999999999997e-06 2024-06-27T21:23:45.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454777+0000", 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.664e-05 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454785+0000", 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9300000000000003e-06 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454856+0000", 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0627e-05 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.454967+0000", 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000111224 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.455034+0000", 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7482000000000002e-05 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.459060+0000", 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040253320000000004 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.459898+0000", 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00083869499999999998 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.672646+0000", 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21274715699999999 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.672680+0000", 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4076999999999999e-05 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.672708+0000", 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8031999999999999e-05 2024-06-27T21:23:45.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:110842 3.5 3:a3de5652:::benchmark_data_smithi192_74314_object4204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:17:52.463420+0000", 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 352.619237548, 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22955552500000001, 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.160 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 110842 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.463420+0000", 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.463420+0000", 2024-06-27T21:23:45.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.463421+0000", 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.221e-06 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.463426+0000", 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9869999999999999e-06 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.463436+0000", 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9000000000000001e-06 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.463447+0000", 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1588e-05 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.470612+0000", 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071646710000000001 2024-06-27T21:23:45.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.470724+0000", 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112353 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.470819+0000", 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4679000000000006e-05 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.484244+0000", 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013425095999999999 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.484328+0000", 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3641999999999996e-05 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.688728+0000", 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20440070399999999 2024-06-27T21:23:45.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.688802+0000", 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3477000000000001e-05 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:17:52.692975+0000", 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004173207 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:123874 3.d e235/186 3:bf6c1432:::benchmark_data_smithi192_74314_object17236:head v 235'7855, mlcod=235'7851)", 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:09.119702+0000", 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 335.96295546699997, 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.281865015, 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.119702+0000", 2024-06-27T21:23:45.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.119702+0000", 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.119702+0000", 2024-06-27T21:23:45.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5600000000000005e-07 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.119710+0000", 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2910000000000002e-06 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.371086+0000", 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25137608099999997 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.371092+0000", 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7930000000000003e-06 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.373874+0000", 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027826330000000001 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.373892+0000", 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7788999999999999e-05 2024-06-27T21:23:45.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.401528+0000", 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.027635501999999999 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.401567+0000", 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9169999999999999e-05 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:156158 3.5 3:af64f47b:::benchmark_data_smithi192_74314_object49520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:48.210878+0000", 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 296.87177937299998, 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.235521916, 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 156158 2024-06-27T21:23:45.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.210878+0000", 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.210878+0000", 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.210879+0000", 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5510000000000001e-06 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.210884+0000", 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.391e-06 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.210894+0000", 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0726e-05 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.210902+0000", 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8839999999999994e-06 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.222028+0000", 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011125407 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.222102+0000", 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3898000000000003e-05 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.317287+0000", 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.095185510000000001 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.317356+0000", 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9065000000000004e-05 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.317447+0000", 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0797000000000005e-05 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.325309+0000", 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0078622669999999992 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.335087+0000", 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0097779830000000005 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.439702+0000", 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104614261 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.439801+0000", 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9331000000000002e-05 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:48.446400+0000", 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065988449999999999 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.15132.0:213264 3.4 e272/230 3:20d1c801:::benchmark_data_smithi192_74314_object106626:head v 272'13548, mlcod=272'13545)", 2024-06-27T21:23:45.180 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:07.087490+0000", 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 217.99516709700001, 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37944926600000001, 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.087490+0000", 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.087490+0000", 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.087491+0000", 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7199999999999997e-07 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.087496+0000", 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1449999999999999e-06 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.463032+0000", 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.375535599 2024-06-27T21:23:45.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.463055+0000", 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3329999999999999e-05 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.463099+0000", 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3818e-05 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.463115+0000", 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6143000000000001e-05 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.466905+0000", 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037898350000000001 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:07.466939+0000", 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4523999999999997e-05 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.185 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(3.5 e275,227 3:adf1ee1e:::benchmark_data_smithi192_74314_object52843:head,249'10165)", 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:17.132578+0000", 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 207.95007870800001, 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41151987099999998, 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:23:45.186 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.132578+0000", 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.132578+0000", 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.132579+0000", 2024-06-27T21:23:45.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.132583+0000", 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8060000000000001e-06 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.543669+0000", 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41108639899999999 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.543676+0000", 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.584e-06 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.543746+0000", 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0544999999999996e-05 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:17.544098+0000", 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035194300000000001 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(3.5 e276,227 3:a51a6756:::benchmark_data_smithi192_74314_object79728:head,262'11909)", 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:19.213064+0000", 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 205.86959339500001, 2024-06-27T21:23:45.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25357206399999999, 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.213064+0000", 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.213064+0000", 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.213064+0000", 2024-06-27T21:23:45.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:45.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:45.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0999999999999998e-07 2024-06-27T21:23:45.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.213068+0000", 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8489999999999999e-06 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.466113+0000", 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25304505599999999 2024-06-27T21:23:45.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.466122+0000", 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.456e-06 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.466229+0000", 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107149 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:19.466636+0000", 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040694400000000001 2024-06-27T21:23:45.197 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.198 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.198 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.198 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:693 4.1 e288/282 4:851bfe3a:::benchmark_data_smithi192_88961_object692:head v 288'41, mlcod=288'38)", 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:43.821644+0000", 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 181.26101297599999, 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22977736500000001, 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821644+0000", 2024-06-27T21:23:45.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821644+0000", 2024-06-27T21:23:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821645+0000", 2024-06-27T21:23:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.155e-06 2024-06-27T21:23:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821746+0000", 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010046400000000001 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.050812+0000", 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22906600199999999 2024-06-27T21:23:45.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.050827+0000", 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5605999999999999e-05 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.050956+0000", 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012841999999999999 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.051002+0000", 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6372000000000003e-05 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.051385+0000", 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038305300000000001 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.051422+0000", 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6293e-05 2024-06-27T21:23:45.207 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:121157 4.2 e342/282 4:4e90fc28:::benchmark_data_smithi192_88961_object30418:head v 342'7610, mlcod=342'7607)", 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:45.407230+0000", 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 59.675427345999999, 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22298259000000001, 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407230+0000", 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407230+0000", 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407231+0000", 2024-06-27T21:23:45.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9899999999999999e-07 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407237+0000", 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0820000000000004e-06 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627012+0000", 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21977543199999999 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627018+0000", 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4079999999999997e-06 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627070+0000", 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2119e-05 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627084+0000", 2024-06-27T21:23:45.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4353e-05 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.630177+0000", 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030928629999999999 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.630212+0000", 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5434e-05 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:122582 4.1 e343/282 4:800d8a34:::benchmark_data_smithi192_88961_object31843:head v 343'7629, mlcod=343'7624)", 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:47.131786+0000", 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 57.950871126999999, 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.233553917, 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131786+0000", 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131786+0000", 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131787+0000", 2024-06-27T21:23:45.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131792+0000", 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0919999999999997e-06 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361794+0000", 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23000235199999999 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361800+0000", 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1489999999999999e-06 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361863+0000", 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2535999999999999e-05 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361887+0000", 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3864999999999999e-05 2024-06-27T21:23:45.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.365300+0000", 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00341274 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.365340+0000", 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0342000000000001e-05 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.217 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:130521 4.1 e346/282 4:8d4b026c:::benchmark_data_smithi192_88961_object39782:head v 346'8090, mlcod=345'8088)", 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:56.565130+0000", 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.517527108000003, 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41281596300000001, 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565130+0000", 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565130+0000", 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565131+0000", 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.82e-07 2024-06-27T21:23:45.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565137+0000", 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2809999999999997e-06 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974674+0000", 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40953738000000001 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974679+0000", 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2530000000000004e-06 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974759+0000", 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9676000000000001e-05 2024-06-27T21:23:45.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974778+0000", 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8512000000000002e-05 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.977918+0000", 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031405159999999999 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.977946+0000", 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7863000000000001e-05 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:142791 4.2 e352/282 4:4c977d5e:::benchmark_data_smithi192_88961_object52052:head v 352'8989, mlcod=352'8984)", 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:23:15.826305+0000", 2024-06-27T21:23:45.223 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.256351713000001, 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16482132099999999, 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.826305+0000", 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:45.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.826305+0000", 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.826307+0000", 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4839999999999999e-06 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.826313+0000", 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2550000000000003e-06 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.987732+0000", 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16141841700000001 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.987737+0000", 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6540000000000004e-06 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:45.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.987794+0000", 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6796999999999998e-05 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.987817+0000", 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3218e-05 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.991089+0000", 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032717639999999999 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:45.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:15.991127+0000", 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7731999999999997e-05 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:45.228 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633] boot 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: osdmap e362: 12 total, 12 up, 12 in 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5126/62367 objects degraded (8.219%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:45.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:44 smithi151 ceph-mon[28965]: osdmap e363: 12 total, 12 up, 12 in 2024-06-27T21:23:45.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-27T21:23:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: osd.4 [v2:172.21.15.190:6800/3587059633,v1:172.21.15.190:6801/3587059633] boot 2024-06-27T21:23:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: osdmap e362: 12 total, 12 up, 12 in 2024-06-27T21:23:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-27T21:23:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:23:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5126/62367 objects degraded (8.219%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:23:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:44 smithi192 ceph-mon[32956]: osdmap e363: 12 total, 12 up, 12 in 2024-06-27T21:23:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:45.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:45.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:45.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:45.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:45.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:45.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:45.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:45.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:45.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:45.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:45.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:45.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:23:46.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:46.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:46.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:46.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:46.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:46.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:45 smithi151 ceph-mon[28965]: pgmap v951: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 4897/59007 objects degraded (8.299%); 3237/59007 objects misplaced (5.486%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:23:46.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:45 smithi151 ceph-mon[28965]: osdmap e364: 12 total, 12 up, 12 in 2024-06-27T21:23:46.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:46.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:46.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:46.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:45 smithi190 ceph-mon[32247]: pgmap v951: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 4897/59007 objects degraded (8.299%); 3237/59007 objects misplaced (5.486%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:23:46.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:45 smithi190 ceph-mon[32247]: osdmap e364: 12 total, 12 up, 12 in 2024-06-27T21:23:46.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:46.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:46.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:45 smithi192 ceph-mon[32956]: pgmap v951: 27 pgs: 1 active+recovering+undersized+remapped, 3 active+undersized, 4 active+undersized+degraded, 19 active+clean; 1.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 4897/59007 objects degraded (8.299%); 3237/59007 objects misplaced (5.486%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:23:46.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:45 smithi192 ceph-mon[32956]: osdmap e364: 12 total, 12 up, 12 in 2024-06-27T21:23:46.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:46.376 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:46.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:46.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:46.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:46.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:46.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:46.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:46.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:46.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:46.896 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:46.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:46.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:47.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:47.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:47.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:47.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:47.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:47.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:47.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:47.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:47.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:47.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:47.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:47.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:47.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:47.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:47.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:47.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:48.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:48.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:48.086 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:47 smithi190 ceph-mon[32247]: pgmap v954: 27 pgs: 1 activating+undersized+degraded+remapped, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded, 21 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 3057/53655 objects degraded (5.698%); 3237/53655 objects misplaced (6.033%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:23:48.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:48.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:47 smithi151 ceph-mon[28965]: pgmap v954: 27 pgs: 1 activating+undersized+degraded+remapped, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded, 21 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 3057/53655 objects degraded (5.698%); 3237/53655 objects misplaced (6.033%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:23:48.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:48.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:23:48.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:47 smithi192 ceph-mon[32956]: pgmap v954: 27 pgs: 1 activating+undersized+degraded+remapped, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded, 21 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 3057/53655 objects degraded (5.698%); 3237/53655 objects misplaced (6.033%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:23:48.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:48.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:48.438 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:23:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:165421 4.a 4:57cddfad:::benchmark_data_smithi192_88961_object74682:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e365)", 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:23:48.431629+0000", 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0074628899999999998, 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074903410000000002, 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for ondisk", 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 165421 2024-06-27T21:23:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431629+0000", 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431629+0000", 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431630+0000", 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.06e-07 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431633+0000", 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2930000000000001e-06 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431641+0000", 2024-06-27T21:23:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4370000000000001e-06 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431647+0000", 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1639999999999998e-06 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431811+0000", 2024-06-27T21:23:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016473099999999999 2024-06-27T21:23:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for ondisk", 2024-06-27T21:23:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.431855+0000", 2024-06-27T21:23:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3291000000000003e-05 2024-06-27T21:23:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:165434 4.5 4:aae5c56d:::benchmark_data_smithi192_88961_object74695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-27T21:23:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:23:48.433940+0000", 2024-06-27T21:23:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0051522859999999998, 2024-06-27T21:23:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052196320000000001, 2024-06-27T21:23:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:23:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:23:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:23:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 165434 2024-06-27T21:23:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.433940+0000", 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.433940+0000", 2024-06-27T21:23:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.433941+0000", 2024-06-27T21:23:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.204e-06 2024-06-27T21:23:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.433946+0000", 2024-06-27T21:23:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8199999999999996e-06 2024-06-27T21:23:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.433960+0000", 2024-06-27T21:23:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3872e-05 2024-06-27T21:23:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.433967+0000", 2024-06-27T21:23:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5299999999999999e-06 2024-06-27T21:23:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.434079+0000", 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000111304 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.434209+0000", 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013046100000000001 2024-06-27T21:23:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 10,11", 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.434329+0000", 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120259 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25312.0:165434 4.5 e365/320)", 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:23:48.438882+0000", 2024-06-27T21:23:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.000209753, 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030874500000000003, 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.438882+0000", 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.438882+0000", 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.438883+0000", 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529999999999999e-06 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.438893+0000", 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3249999999999997e-06 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.438917+0000", 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3971000000000001e-05 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.438925+0000", 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3019999999999995e-06 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:165421 4.a 4:57cddfad:::benchmark_data_smithi192_88961_object74682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:23:48.407749+0000", 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.031342885000000001, 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.031466635999999999, 2024-06-27T21:23:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 165421 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.407749+0000", 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.407749+0000", 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.407751+0000", 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.471e-06 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.407757+0000", 2024-06-27T21:23:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3409999999999999e-06 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.407773+0000", 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.573e-05 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.407781+0000", 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5779999999999999e-06 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.413539+0000", 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005757229 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.413671+0000", 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013254299999999999 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,10", 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.413803+0000", 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013215199999999999 2024-06-27T21:23:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.418532+0000", 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047290600000000002 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:23:48.418641+0000", 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000108998 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:23:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-06-27T21:23:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 4 2024-06-27T21:23:48.464 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:23:48.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:48.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:48.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:48.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:48.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:48.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:48.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:48.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:48.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:48.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:49.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:49.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:49.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:49.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: osdmap e365: 12 total, 12 up, 12 in 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:23:49.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:23:49.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: osdmap e365: 12 total, 12 up, 12 in 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:23:49.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:23:49.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:49.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:23:49.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:49.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:49.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: osdmap e365: 12 total, 12 up, 12 in 2024-06-27T21:23:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:23:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:23:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:23:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:23:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:23:49.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:49.345 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:49.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:49.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:49.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:49.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:49.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:49.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:49.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:49.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:49.880 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:49.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:49.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:49.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:50.012 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:49 smithi151 ceph-mon[28965]: pgmap v956: 27 pgs: 1 activating+undersized+degraded+remapped, 2 remapped+peering, 1 peering, 2 active+recovering+undersized+remapped, 21 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 1908/52914 objects degraded (3.606%); 3237/52914 objects misplaced (6.117%); 3.3 MiB/s, 65 objects/s recovering 2024-06-27T21:23:50.012 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:49 smithi151 ceph-mon[28965]: osdmap e366: 12 total, 12 up, 12 in 2024-06-27T21:23:50.012 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1908/52914 objects degraded (3.606%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:23:50.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:49 smithi190 ceph-mon[32247]: pgmap v956: 27 pgs: 1 activating+undersized+degraded+remapped, 2 remapped+peering, 1 peering, 2 active+recovering+undersized+remapped, 21 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 1908/52914 objects degraded (3.606%); 3237/52914 objects misplaced (6.117%); 3.3 MiB/s, 65 objects/s recovering 2024-06-27T21:23:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:49 smithi190 ceph-mon[32247]: osdmap e366: 12 total, 12 up, 12 in 2024-06-27T21:23:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1908/52914 objects degraded (3.606%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:23:50.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:50.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:50.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:50.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:50.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:49 smithi192 ceph-mon[32956]: pgmap v956: 27 pgs: 1 activating+undersized+degraded+remapped, 2 remapped+peering, 1 peering, 2 active+recovering+undersized+remapped, 21 active+clean; 1.1 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 1908/52914 objects degraded (3.606%); 3237/52914 objects misplaced (6.117%); 3.3 MiB/s, 65 objects/s recovering 2024-06-27T21:23:50.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:49 smithi192 ceph-mon[32956]: osdmap e366: 12 total, 12 up, 12 in 2024-06-27T21:23:50.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1908/52914 objects degraded (3.606%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:23:50.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:50.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:50.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:50.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:50.569 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:50.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:50.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:50.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:50.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:50.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:50.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:50.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:50.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:51.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:51.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:51.128 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:50 smithi190 ceph-mon[32247]: pgmap v958: 27 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 22 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 1908/49437 objects degraded (3.859%); 2664/49437 objects misplaced (5.389%); 2.2 MiB/s, 116 objects/s recovering 2024-06-27T21:23:51.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:51.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:50 smithi151 ceph-mon[28965]: pgmap v958: 27 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 22 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 1908/49437 objects degraded (3.859%); 2664/49437 objects misplaced (5.389%); 2.2 MiB/s, 116 objects/s recovering 2024-06-27T21:23:51.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.278 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:51.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:51.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:50 smithi192 ceph-mon[32956]: pgmap v958: 27 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 22 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 1908/49437 objects degraded (3.859%); 2664/49437 objects misplaced (5.389%); 2.2 MiB/s, 116 objects/s recovering 2024-06-27T21:23:51.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:51.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:51.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:51.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:51.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:51.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:51.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:51.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:23:51.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:51.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:51.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:51.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:52.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:52.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:52.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:52.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:52.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:52.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:52.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:52.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:52.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:52.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:52.606 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:23:52.606 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:23:52.606 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:23:52.606 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:23:52.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:52.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:52.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:52.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:52.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:52.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:52.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:53.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:53.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:53.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:53.233 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:53.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:53 smithi190 ceph-mon[32247]: pgmap v959: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 959 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 411 op/s; 1908/46002 objects degraded (4.148%); 2664/46002 objects misplaced (5.791%); 1.8 MiB/s, 158 objects/s recovering 2024-06-27T21:23:53.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 0 1 2024-06-27T21:23:53.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:53 smithi151 ceph-mon[28965]: pgmap v959: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 959 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 411 op/s; 1908/46002 objects degraded (4.148%); 2664/46002 objects misplaced (5.791%); 1.8 MiB/s, 158 objects/s recovering 2024-06-27T21:23:53.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:53.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:53.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:53.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:53.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:53 smithi192 ceph-mon[32956]: pgmap v959: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 959 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 411 op/s; 1908/46002 objects degraded (4.148%); 2664/46002 objects misplaced (5.791%); 1.8 MiB/s, 158 objects/s recovering 2024-06-27T21:23:53.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:53.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:53.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:53.742 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:23:53.742 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:23:53.742 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:23:53.742 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:23:53.742 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:23:53.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:53.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:53.871 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:53.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:53.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:54.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:54.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:54.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:54.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:54.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:54.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:54.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:54.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:23:54.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:54.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:54.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:54.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:54.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:54.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:54.813 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:54.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:54.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:54.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:55.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:55.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:55.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:55.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:55.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:55 smithi190 ceph-mon[32247]: pgmap v960: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 892 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 1908/42807 objects degraded (4.457%); 2256/42807 objects misplaced (5.270%); 2.6 MiB/s, 187 objects/s recovering 2024-06-27T21:23:55.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:55 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1908/42807 objects degraded (4.457%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:23:55.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:55.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:55.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:55.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:55 smithi151 ceph-mon[28965]: pgmap v960: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 892 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 1908/42807 objects degraded (4.457%); 2256/42807 objects misplaced (5.270%); 2.6 MiB/s, 187 objects/s recovering 2024-06-27T21:23:55.337 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:55 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1908/42807 objects degraded (4.457%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:23:55.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:55.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:23:55.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:55 smithi192 ceph-mon[32956]: pgmap v960: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 892 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 1908/42807 objects degraded (4.457%); 2256/42807 objects misplaced (5.270%); 2.6 MiB/s, 187 objects/s recovering 2024-06-27T21:23:55.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:55 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1908/42807 objects degraded (4.457%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:23:55.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:55.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:55.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:55.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:55.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:55.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:55.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:55.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:56.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:23:56.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:23:56.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:56.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:56.096 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:23:56.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:23:56.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:23:56.127 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:23:56.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:23:56.159 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:23:56.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/358524608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:56.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:23:56.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-06-27T21:23:56.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:23:56.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:23:56.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/358524608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:56.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:23:56.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:56.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:23:56.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:23:56.459 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:23:56.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:23:56.487 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:23:56.487 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:23:56.524 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:23:56.524 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:23:56.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/358524608' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:23:56.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:23:56.557 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:23:56.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:56.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:23:56.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:23:56.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:56.620 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:23:56.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:23:56.651 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:23:56.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:23:56.661 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:23:56.661 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:23:56.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:56.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:56.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:56.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:56.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:56.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:23:57.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:57.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:57.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:57.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:57.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:57 smithi190 ceph-mon[32247]: pgmap v961: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 740 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 1908/35496 objects degraded (5.375%); 2256/35496 objects misplaced (6.356%); 2.5 MiB/s, 252 objects/s recovering 2024-06-27T21:23:57.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:57.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:23:57.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:23:57.444 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:57 smithi192 ceph-mon[32956]: pgmap v961: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 740 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 1908/35496 objects degraded (5.375%); 2256/35496 objects misplaced (6.356%); 2.5 MiB/s, 252 objects/s recovering 2024-06-27T21:23:57.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:57 smithi151 ceph-mon[28965]: pgmap v961: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 740 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 1908/35496 objects degraded (5.375%); 2256/35496 objects misplaced (6.356%); 2.5 MiB/s, 252 objects/s recovering 2024-06-27T21:23:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:57.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:57.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:23:57.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:23:57.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:57.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:57.744 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:23:57.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:57.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:57.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:57.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:57.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:58.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:58.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:58.196 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:23:58.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:23:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:58.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:23:58.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:23:58.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:23:58.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:23:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:58.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:23:58.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:58.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:58.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:23:58.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:23:58.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:58.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:58.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:59.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:59.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:59.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:59.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:59.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:23:59 smithi190 ceph-mon[32247]: pgmap v962: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 712 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 1908/34155 objects degraded (5.586%); 2256/34155 objects misplaced (6.605%); 2.0 MiB/s, 234 objects/s recovering 2024-06-27T21:23:59.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:23:59.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:23:59.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:59.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:23:59 smithi151 ceph-mon[28965]: pgmap v962: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 712 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 1908/34155 objects degraded (5.586%); 2256/34155 objects misplaced (6.605%); 2.0 MiB/s, 234 objects/s recovering 2024-06-27T21:23:59.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:23:59.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:23:59.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:23:59 smithi192 ceph-mon[32956]: pgmap v962: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 712 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 1908/34155 objects degraded (5.586%); 2256/34155 objects misplaced (6.605%); 2.0 MiB/s, 234 objects/s recovering 2024-06-27T21:23:59.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:23:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:23:59.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:23:59.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:23:59.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:23:59.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:23:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:23:59.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:23:59.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:23:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:23:59.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:00.010 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:00.010 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:00.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:00.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:00.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:24:00.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:00.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:00.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:00 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1908/34155 objects degraded (5.586%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:00.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:00.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:00.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:00 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1908/34155 objects degraded (5.586%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:00.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:00.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:00.543 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:00 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1908/34155 objects degraded (5.586%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:00.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:00.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:00.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:00.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:24:00.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:00.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:00.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:00.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:00.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:01.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:01.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:01.158 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:24:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:01.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:01.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:01.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:01.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:01.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:01.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:01.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:01.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:01 smithi190 ceph-mon[32247]: pgmap v963: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 596 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 1908/28596 objects degraded (6.672%); 1590/28596 objects misplaced (5.560%); 2.1 MiB/s, 271 objects/s recovering 2024-06-27T21:24:01.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:01.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:01.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:01 smithi192 ceph-mon[32956]: pgmap v963: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 596 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 1908/28596 objects degraded (6.672%); 1590/28596 objects misplaced (5.560%); 2.1 MiB/s, 271 objects/s recovering 2024-06-27T21:24:01.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:01.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:01.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:01.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:01 smithi151 ceph-mon[28965]: pgmap v963: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 596 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 1908/28596 objects degraded (6.672%); 1590/28596 objects misplaced (5.560%); 2.1 MiB/s, 271 objects/s recovering 2024-06-27T21:24:01.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:01.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:02.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:02.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:02.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:02.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:02.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:02.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:02.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:02.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:02.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:02.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:02.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:02.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:02.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:02.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:02.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:02.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:02.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:03.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:03.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:03.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:03.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:03.266 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:03.266 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:03.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:03.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:03.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:03.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:03.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:03.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:03.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:03 smithi151 ceph-mon[28965]: pgmap v964: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 512 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 1908/24573 objects degraded (7.765%); 1590/24573 objects misplaced (6.471%); 1.9 MiB/s, 228 objects/s recovering 2024-06-27T21:24:03.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2382496899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:03.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:03.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:03.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:03.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:03.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:03.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:03.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:03 smithi190 ceph-mon[32247]: pgmap v964: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 512 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 1908/24573 objects degraded (7.765%); 1590/24573 objects misplaced (6.471%); 1.9 MiB/s, 228 objects/s recovering 2024-06-27T21:24:04.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2382496899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:04.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:03 smithi192 ceph-mon[32956]: pgmap v964: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 512 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 1908/24573 objects degraded (7.765%); 1590/24573 objects misplaced (6.471%); 1.9 MiB/s, 228 objects/s recovering 2024-06-27T21:24:04.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2382496899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:04.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:24:04.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:04.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:04.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:04.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:04.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:04.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:04.500 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:24:04.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:04.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:04.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:04.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:04.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:04.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:04.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:04 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:24:04.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:24:04.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:04 smithi190 ceph-mon[32247]: osdmap e367: 12 total, 12 up, 12 in 2024-06-27T21:24:04.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1908/24573 objects degraded (7.765%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:04.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:04.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:04 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:24:04.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:24:04.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:04 smithi151 ceph-mon[28965]: osdmap e367: 12 total, 12 up, 12 in 2024-06-27T21:24:04.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1908/24573 objects degraded (7.765%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:04.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:05.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:04 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:24:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:24:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:04 smithi192 ceph-mon[32956]: osdmap e367: 12 total, 12 up, 12 in 2024-06-27T21:24:05.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1908/24573 objects degraded (7.765%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:05.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:05.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:05.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:05.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:05.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:05.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:05.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:05.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:05.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:05.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:05.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:05.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:05.832 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:05 smithi190 ceph-mon[32247]: pgmap v966: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 440 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 1908/21084 objects degraded (9.050%); 930/21084 objects misplaced (4.411%); 2.8 MiB/s, 256 objects/s recovering 2024-06-27T21:24:05.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:05.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:05.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:05 smithi151 ceph-mon[28965]: pgmap v966: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 440 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 1908/21084 objects degraded (9.050%); 930/21084 objects misplaced (4.411%); 2.8 MiB/s, 256 objects/s recovering 2024-06-27T21:24:06.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:06.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:05 smithi192 ceph-mon[32956]: pgmap v966: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 440 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 1908/21084 objects degraded (9.050%); 930/21084 objects misplaced (4.411%); 2.8 MiB/s, 256 objects/s recovering 2024-06-27T21:24:06.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:06.097 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:24:06.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:06.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:06.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:06.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:06.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:06.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:06.465 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:06.465 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:06.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:06.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:06.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:06.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:06.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:06.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:06.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:06.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:06.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:07.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:07.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:07.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:07.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:07.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:07.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:07.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:07.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:07.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:07.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:07 smithi190 ceph-mon[32247]: pgmap v967: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 285 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 1908/13641 objects degraded (13.987%); 930/13641 objects misplaced (6.818%); 2.8 MiB/s, 264 objects/s recovering 2024-06-27T21:24:07.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:07.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:07.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:07.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:07.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:07.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:07.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:07.878 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:07.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:24:07.937 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:07 smithi192 ceph-mon[32956]: pgmap v967: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 285 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 1908/13641 objects degraded (13.987%); 930/13641 objects misplaced (6.818%); 2.8 MiB/s, 264 objects/s recovering 2024-06-27T21:24:07.937 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:07.937 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:07.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:07 smithi151 ceph-mon[28965]: pgmap v967: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 285 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 1908/13641 objects degraded (13.987%); 930/13641 objects misplaced (6.818%); 2.8 MiB/s, 264 objects/s recovering 2024-06-27T21:24:07.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:07.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:07.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:07.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:07.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:08.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:08.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:08.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:08.254 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:08.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:08.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:08.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:08.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:08.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:08.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:08.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:08.669 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:08 smithi192 ceph-mon[32956]: osdmap e368: 12 total, 12 up, 12 in 2024-06-27T21:24:08.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:08.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:08.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:08.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:08.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:08.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:08 smithi151 ceph-mon[28965]: osdmap e368: 12 total, 12 up, 12 in 2024-06-27T21:24:09.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:08 smithi190 ceph-mon[32247]: osdmap e368: 12 total, 12 up, 12 in 2024-06-27T21:24:09.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:09.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:09.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:09.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:09.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:09.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:09.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:09.411 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:09.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:09.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:09.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:09.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:09.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:09.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:09.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:09.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:09.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:09.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:09.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:09 smithi151 ceph-mon[28965]: pgmap v969: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 255 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 1908/12198 objects degraded (15.642%); 930/12198 objects misplaced (7.624%); 1.7 MiB/s, 254 objects/s recovering 2024-06-27T21:24:09.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:09 smithi151 ceph-mon[28965]: osdmap e369: 12 total, 12 up, 12 in 2024-06-27T21:24:09.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1908/12198 objects degraded (15.642%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:09.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:09 smithi151 ceph-mon[28965]: osdmap e370: 12 total, 12 up, 12 in 2024-06-27T21:24:09.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:09 smithi192 ceph-mon[32956]: pgmap v969: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 255 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 1908/12198 objects degraded (15.642%); 930/12198 objects misplaced (7.624%); 1.7 MiB/s, 254 objects/s recovering 2024-06-27T21:24:09.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:09 smithi192 ceph-mon[32956]: osdmap e369: 12 total, 12 up, 12 in 2024-06-27T21:24:09.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1908/12198 objects degraded (15.642%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:09.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:09 smithi192 ceph-mon[32956]: osdmap e370: 12 total, 12 up, 12 in 2024-06-27T21:24:10.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:09 smithi190 ceph-mon[32247]: pgmap v969: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 22 active+clean; 255 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 682 op/s; 1908/12198 objects degraded (15.642%); 930/12198 objects misplaced (7.624%); 1.7 MiB/s, 254 objects/s recovering 2024-06-27T21:24:10.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:09 smithi190 ceph-mon[32247]: osdmap e369: 12 total, 12 up, 12 in 2024-06-27T21:24:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1908/12198 objects degraded (15.642%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:09 smithi190 ceph-mon[32247]: osdmap e370: 12 total, 12 up, 12 in 2024-06-27T21:24:10.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:10.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:10.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:10.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:10.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:10.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:10.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:10.420 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:10.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:10.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:10.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:10.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:10.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:24:10.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:10.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:10.958 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:10.958 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:10.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:10.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:10.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:11.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:11.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:11.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:11.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:11.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:11.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:11.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:11 smithi192 ceph-mon[32956]: pgmap v972: 27 pgs: 1 activating, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 154 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 1908/7392 objects degraded (25.812%); 248/7392 objects misplaced (3.355%); 2.4 MiB/s, 261 objects/s recovering 2024-06-27T21:24:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:11 smithi192 ceph-mon[32956]: osdmap e371: 12 total, 12 up, 12 in 2024-06-27T21:24:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/372047127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:24:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:11.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:11.589 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-06-27T21:24:11.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:11.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:24:11.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:11.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:11.807 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:11.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:11.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:11.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:11.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:11 smithi151 ceph-mon[28965]: pgmap v972: 27 pgs: 1 activating, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 154 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 1908/7392 objects degraded (25.812%); 248/7392 objects misplaced (3.355%); 2.4 MiB/s, 261 objects/s recovering 2024-06-27T21:24:11.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:11 smithi151 ceph-mon[28965]: osdmap e371: 12 total, 12 up, 12 in 2024-06-27T21:24:11.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/372047127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:24:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:11 smithi190 ceph-mon[32247]: pgmap v972: 27 pgs: 1 activating, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 22 active+clean; 154 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 1908/7392 objects degraded (25.812%); 248/7392 objects misplaced (3.355%); 2.4 MiB/s, 261 objects/s recovering 2024-06-27T21:24:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:11 smithi190 ceph-mon[32247]: osdmap e371: 12 total, 12 up, 12 in 2024-06-27T21:24:12.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/372047127' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:24:12.013 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:12.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:12.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:12.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:12.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:12.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:12.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:12.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:12.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:12.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:12.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:12.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:12.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:12.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:12.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:12.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/372047127' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:24:12.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:12 smithi192 ceph-mon[32956]: osdmap e372: 12 total, 12 up, 12 in 2024-06-27T21:24:12.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:12.905 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 90736 objects 2024-06-27T21:24:12.906 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :118.294 2024-06-27T21:24:12.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:12.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:12.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/372047127' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:24:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:12 smithi151 ceph-mon[28965]: osdmap e372: 12 total, 12 up, 12 in 2024-06-27T21:24:13.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/372047127' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:24:13.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:12 smithi190 ceph-mon[32247]: osdmap e372: 12 total, 12 up, 12 in 2024-06-27T21:24:13.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:13.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:13.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:13.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:13.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:13.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:13.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:13.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:13.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:13.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:13.655 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:13.656 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:13 smithi192 ceph-mon[32956]: pgmap v975: 27 pgs: 1 activating, 4 active+recovering+undersized+remapped, 22 active+clean; 91 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 248/4371 objects misplaced (5.674%); 3.5 MiB/s, 328 objects/s recovering 2024-06-27T21:24:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:13 smithi192 ceph-mon[32956]: 4.0 scrub starts 2024-06-27T21:24:13.656 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:13 smithi192 ceph-mon[32956]: 4.0 scrub ok 2024-06-27T21:24:13.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:13.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:13.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:13.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:13.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:13.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:13.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:13.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:13 smithi151 ceph-mon[28965]: pgmap v975: 27 pgs: 1 activating, 4 active+recovering+undersized+remapped, 22 active+clean; 91 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 248/4371 objects misplaced (5.674%); 3.5 MiB/s, 328 objects/s recovering 2024-06-27T21:24:13.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:13 smithi151 ceph-mon[28965]: 4.0 scrub starts 2024-06-27T21:24:13.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:13 smithi151 ceph-mon[28965]: 4.0 scrub ok 2024-06-27T21:24:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:13 smithi190 ceph-mon[32247]: pgmap v975: 27 pgs: 1 activating, 4 active+recovering+undersized+remapped, 22 active+clean; 91 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 248/4371 objects misplaced (5.674%); 3.5 MiB/s, 328 objects/s recovering 2024-06-27T21:24:14.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:13 smithi190 ceph-mon[32247]: 4.0 scrub starts 2024-06-27T21:24:14.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:13 smithi190 ceph-mon[32247]: 4.0 scrub ok 2024-06-27T21:24:14.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:14.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:14.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:14.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:14.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:14.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:14.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:14.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:24:14.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:14.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:14.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:14.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:14.568 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:14.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.674 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:14.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:14.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:14.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:14.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:14.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:14.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:14.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:14 smithi151 ceph-mon[28965]: osdmap e373: 12 total, 12 up, 12 in 2024-06-27T21:24:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:14.964 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:14.964 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:14.964 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:14 smithi192 ceph-mon[32956]: osdmap e373: 12 total, 12 up, 12 in 2024-06-27T21:24:14.964 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:15.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:15.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:15.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:14 smithi190 ceph-mon[32247]: osdmap e373: 12 total, 12 up, 12 in 2024-06-27T21:24:15.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:15.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:15.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:15.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:15.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:15.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:15.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:15.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:15.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:15.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:15.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:15.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:15.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:15.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:15.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:15.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:15.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:15.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:15.935 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:15 smithi192 ceph-mon[32956]: pgmap v977: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 activating, 2 active+recovering+undersized+remapped, 22 active+clean; 55 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 2/2643 objects degraded (0.076%); 1.2 MiB/s, 338 objects/s recovering 2024-06-27T21:24:15.935 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:15 smithi192 ceph-mon[32956]: osdmap e374: 12 total, 12 up, 12 in 2024-06-27T21:24:15.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:15 smithi151 ceph-mon[28965]: pgmap v977: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 activating, 2 active+recovering+undersized+remapped, 22 active+clean; 55 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 2/2643 objects degraded (0.076%); 1.2 MiB/s, 338 objects/s recovering 2024-06-27T21:24:15.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:15 smithi151 ceph-mon[28965]: osdmap e374: 12 total, 12 up, 12 in 2024-06-27T21:24:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:15 smithi190 ceph-mon[32247]: pgmap v977: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 activating, 2 active+recovering+undersized+remapped, 22 active+clean; 55 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 2/2643 objects degraded (0.076%); 1.2 MiB/s, 338 objects/s recovering 2024-06-27T21:24:16.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:15 smithi190 ceph-mon[32247]: osdmap e374: 12 total, 12 up, 12 in 2024-06-27T21:24:16.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:16.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:16.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:16.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:16.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:16.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:16.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:16.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:16.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:16.453 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:16.453 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:16.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:16.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:16.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:16.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:16.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:16.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:16.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:16.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:16.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:16.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:16 smithi151 ceph-mon[28965]: 4.e scrub starts 2024-06-27T21:24:16.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:16 smithi151 ceph-mon[28965]: 4.e scrub ok 2024-06-27T21:24:17.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:16 smithi190 ceph-mon[32247]: 4.e scrub starts 2024-06-27T21:24:17.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:16 smithi190 ceph-mon[32247]: 4.e scrub ok 2024-06-27T21:24:17.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:17.036 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:17.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:16 smithi192 ceph-mon[32956]: 4.e scrub starts 2024-06-27T21:24:17.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:16 smithi192 ceph-mon[32956]: 4.e scrub ok 2024-06-27T21:24:17.085 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-06-27T21:24:17.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-06-27T21:24:17.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:17.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:24:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:17.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:17.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:17.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:17.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:17.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:17.439 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:17.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:17.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:17.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 1 1 2024-06-27T21:24:17.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:17.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:17.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:17.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:17.874 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:17 smithi192 ceph-mon[32956]: pgmap v979: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 347 op/s; 2/6 objects degraded (33.333%); 971 KiB/s, 304 objects/s recovering 2024-06-27T21:24:17.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:17.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:17.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:17.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:17 smithi151 ceph-mon[28965]: pgmap v979: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 347 op/s; 2/6 objects degraded (33.333%); 971 KiB/s, 304 objects/s recovering 2024-06-27T21:24:18.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:18.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:17 smithi190 ceph-mon[32247]: pgmap v979: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 347 op/s; 2/6 objects degraded (33.333%); 971 KiB/s, 304 objects/s recovering 2024-06-27T21:24:18.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:18.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:18.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:18.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:18.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:18.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:18.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:18.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:18.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:18.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:18.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:18.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:18.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:18.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:18.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:18.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:18.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:19.015 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:19.016 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:19.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:19.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:19.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:19.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:19.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:19.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:19.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:19.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:19.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:19.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:19.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:19.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:24:19.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:19.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:19.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:19.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:19.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:19.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:19 smithi151 ceph-mon[28965]: pgmap v980: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 227 op/s; 2/6 objects degraded (33.333%); 829 KiB/s, 189 objects/s recovering 2024-06-27T21:24:19.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:19.952 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:19.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:19 smithi190 ceph-mon[32247]: pgmap v980: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 227 op/s; 2/6 objects degraded (33.333%); 829 KiB/s, 189 objects/s recovering 2024-06-27T21:24:20.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:20.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:20.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:19 smithi192 ceph-mon[32956]: pgmap v980: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 227 op/s; 2/6 objects degraded (33.333%); 829 KiB/s, 189 objects/s recovering 2024-06-27T21:24:20.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:20.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:20.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:20.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:20.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:20.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:20.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:20.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:20.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:20.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:20.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:20.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:20.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:20.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:20.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:20.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:20.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:21.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:21.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:21.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:21.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.257 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:21.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:21.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:21.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:21.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:21.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:21.606 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:21.607 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:21.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:21.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:21.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:21.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:21.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:21 smithi192 ceph-mon[32956]: pgmap v981: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 181 op/s; 3/6 objects degraded (50.000%); 664 KiB/s, 204 objects/s recovering 2024-06-27T21:24:21.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2536015144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:21.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:21.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:21.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:21.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:21 smithi151 ceph-mon[28965]: pgmap v981: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 181 op/s; 3/6 objects degraded (50.000%); 664 KiB/s, 204 objects/s recovering 2024-06-27T21:24:21.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2536015144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:21.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:24:21.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:24:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:24:21.961 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:24:21.966 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:21.969 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:21.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:21.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:24:21.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:24:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:22.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:22.006 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:24:22.007 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:24:22.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:22.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:21 smithi190 ceph-mon[32247]: pgmap v981: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 181 op/s; 3/6 objects degraded (50.000%); 664 KiB/s, 204 objects/s recovering 2024-06-27T21:24:22.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2536015144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:22.027 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:24:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:24:22.048 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:24:22.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:24:22.069 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:24:22.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:24:22.093 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:24:22.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:24:22.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:22.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:24:22.126 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:24:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:22.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:22.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:24:22.152 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:24:22.180 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:24:22.208 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:24:22.208 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:24:22.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:22.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:22.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:24:22.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:22.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:22.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:22.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:22.563 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:22.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:22.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:22.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:22.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:22.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:22.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:22.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:22.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:22.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:23.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:23.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:23.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:23.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:23.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:23.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:23.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:23.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:23.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:23.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:23.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:23.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:23.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:23 smithi192 ceph-mon[32956]: pgmap v982: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 3/6 objects degraded (50.000%); 632 KiB/s, 227 objects/s recovering 2024-06-27T21:24:23.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:23.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:23.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:23 smithi151 ceph-mon[28965]: pgmap v982: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 3/6 objects degraded (50.000%); 632 KiB/s, 227 objects/s recovering 2024-06-27T21:24:23.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:23.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:23.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:24.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:23 smithi190 ceph-mon[32247]: pgmap v982: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 138 op/s; 3/6 objects degraded (50.000%); 632 KiB/s, 227 objects/s recovering 2024-06-27T21:24:24.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:24.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:24.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:24.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:24.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:24.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:24.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:24.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:24.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:24.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:24.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:24.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:24.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:24 smithi151 ceph-mon[28965]: osdmap e375: 12 total, 12 up, 12 in 2024-06-27T21:24:24.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/6 objects degraded (50.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:24.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:24 smithi192 ceph-mon[32956]: osdmap e375: 12 total, 12 up, 12 in 2024-06-27T21:24:24.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/6 objects degraded (50.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:24.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:24.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:24.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:24.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:24.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:24.985 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:24:24.985 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:24:24.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:24:24.985 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:24:24.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.985 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:653 4.6 4:697d5697:::benchmark_data_smithi192_88961_object652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.796944+0000", 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 221.188709625, 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25617681399999997, 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 653 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.986 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796944+0000", 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796944+0000", 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796945+0000", 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1500000000000003e-07 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796964+0000", 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8958e-05 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796974+0000", 2024-06-27T21:24:24.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0054999999999999e-05 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.796986+0000", 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2773e-05 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797028+0000", 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1832000000000002e-05 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797115+0000", 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6712999999999995e-05 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797195+0000", 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0402999999999996e-05 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.797550+0000", 2024-06-27T21:24:24.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035431999999999998 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.805651+0000", 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081016370000000001 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.053058+0000", 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.247406986 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.053093+0000", 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4881e-05 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.053121+0000", 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7341000000000001e-05 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:24.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:672 4.6 4:6f4ffc5a:::benchmark_data_smithi192_88961_object671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.804539+0000", 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 221.181114576, 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.25584938200000001, 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:24.993 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 672 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804539+0000", 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804539+0000", 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:24.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804540+0000", 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4419999999999999e-06 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804564+0000", 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3887999999999999e-05 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:24.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804577+0000", 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2992e-05 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804583+0000", 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0290000000000002e-06 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804639+0000", 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6381000000000001e-05 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804724+0000", 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4730999999999999e-05 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.804805+0000", 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1168999999999995e-05 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.805244+0000", 2024-06-27T21:24:24.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00043857899999999998 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.829263+0000", 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024019511 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.060311+0000", 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23104725100000001 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:24.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.060359+0000", 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7987e-05 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.060388+0000", 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9422000000000001e-05 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58291 4.6 4.30c53896 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:24:24.999 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.45548742899999, 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95344075399999995, 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.001 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58291 2024-06-27T21:24:25.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.001 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530167+0000", 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0860000000000001e-06 2024-06-27T21:24:25.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530189+0000", 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1943999999999999e-05 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530201+0000", 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1959e-05 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530208+0000", 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6880000000000002e-06 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530259+0000", 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0924000000000002e-05 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530267+0000", 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5790000000000004e-06 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483575+0000", 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95330775800000001 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483607+0000", 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1816000000000002e-05 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58315 4.6 4.d0954b56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.42373982399999, 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92179534699999999, 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58315 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.006 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561915+0000", 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.141e-06 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561933+0000", 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8337000000000001e-05 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561941+0000", 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7729999999999999e-06 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561946+0000", 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9230000000000001e-06 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.562002+0000", 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6212999999999999e-05 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.562008+0000", 2024-06-27T21:24:25.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5849999999999996e-06 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483698+0000", 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92168988600000001 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483709+0000", 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0489000000000001e-05 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58321 4.6 4.898f9226 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.41766923, 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91576660700000001, 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58321 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567986+0000", 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.516e-06 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568008+0000", 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2464e-05 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568022+0000", 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3726999999999999e-05 2024-06-27T21:24:25.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568028+0000", 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1530000000000002e-06 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568097+0000", 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8694000000000003e-05 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568110+0000", 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3739e-05 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483740+0000", 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91563002900000001 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483751+0000", 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0285000000000001e-05 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58331 4.6 4.e4097d56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.30258108999999, 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80079522000000003, 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58331 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683074+0000", 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5370000000000001e-06 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683096+0000", 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1880999999999999e-05 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683109+0000", 2024-06-27T21:24:25.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2858e-05 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683114+0000", 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7069999999999997e-06 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683168+0000", 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3949999999999997e-05 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683177+0000", 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4479999999999994e-06 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483855+0000", 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80067863299999997 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483867+0000", 2024-06-27T21:24:25.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2206e-05 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58364 4.6 4.a2603346 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.167308749, 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66557216799999996, 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.021 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58364 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818346+0000", 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.539e-06 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818373+0000", 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7081e-05 2024-06-27T21:24:25.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818392+0000", 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8879999999999999e-05 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818402+0000", 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7419999999999992e-06 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818457+0000", 2024-06-27T21:24:25.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5664000000000001e-05 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818466+0000", 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9029999999999993e-06 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483904+0000", 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66543803300000004 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483917+0000", 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2326e-05 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58368 4.6 4.bdd01c66 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 171.05415765999999, 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55245678099999995, 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.026 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58368 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931497+0000", 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3260000000000001e-06 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931517+0000", 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0378e-05 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931528+0000", 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.078e-05 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931534+0000", 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9429999999999997e-06 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931583+0000", 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9036000000000003e-05 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931592+0000", 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8230000000000001e-06 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483946+0000", 2024-06-27T21:24:25.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55235438299999995 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483952+0000", 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1120000000000001e-06 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:103650 4.d 4:b0eeaa89:::benchmark_data_smithi192_88961_object12911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:26.926193+0000", 2024-06-27T21:24:25.031 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 118.059460741, 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27303572399999998, 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 103650 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926193+0000", 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926193+0000", 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926194+0000", 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8640000000000001e-06 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926202+0000", 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6000000000000001e-06 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926221+0000", 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9207000000000001e-05 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926230+0000", 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8910000000000001e-06 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926300+0000", 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9486999999999994e-05 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926444+0000", 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014414199999999999 2024-06-27T21:24:25.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926571+0000", 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012735700000000001 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929247+0000", 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026755699999999999 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929348+0000", 2024-06-27T21:24:25.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010126 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199160+0000", 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26981249600000001 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199195+0000", 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4789000000000002e-05 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199228+0000", 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3061000000000001e-05 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:103650 4.d e335/301 ondisk, result = 0)", 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:26.929260+0000", 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 118.056393352, 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26997275199999998, 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.038 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929260+0000", 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929260+0000", 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929261+0000", 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9400000000000005e-07 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929265+0000", 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2679999999999996e-06 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199114+0000", 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26984877699999998 2024-06-27T21:24:25.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199119+0000", 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7060000000000001e-06 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199152+0000", 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3018999999999997e-05 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199156+0000", 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0160000000000002e-06 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199233+0000", 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6272000000000004e-05 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:143012 4.6 4:6ee82129:::benchmark_data_smithi192_88961_object52273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:16.299906+0000", 2024-06-27T21:24:25.042 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.685747149999997, 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.238634293, 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 143012 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299906+0000", 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299906+0000", 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299907+0000", 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8599999999999996e-07 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299911+0000", 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1479999999999999e-06 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299922+0000", 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0465999999999999e-05 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299927+0000", 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5990000000000001e-06 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.299966+0000", 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9008000000000003e-05 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.300047+0000", 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0975000000000003e-05 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2", 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.300114+0000", 2024-06-27T21:24:25.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6260000000000006e-05 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.303513+0000", 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003399216 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.304716+0000", 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012028849999999999 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.538450+0000", 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23373450500000001 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.538496+0000", 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5889000000000001e-05 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.538540+0000", 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4356e-05 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:143012 4.6 e352/335 ondisk, result = 0)", 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:16.303120+0000", 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.682532914999996, 2024-06-27T21:24:25.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23542490199999999, 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.303120+0000", 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.303120+0000", 2024-06-27T21:24:25.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.303121+0000", 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3600000000000002e-07 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.303125+0000", 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0029999999999996e-06 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.534375+0000", 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23125003 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.534383+0000", 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5889999999999997e-06 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.538436+0000", 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040533230000000002 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.538444+0000", 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7400000000000004e-06 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:16.538545+0000", 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101381 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.053 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDScrubReserve(1.6,QREQUEST,e:353,from:10,reservation_nonce:17)", 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:19.785054+0000", 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.200599081999997, 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33999884000000002, 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785054+0000", 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785054+0000", 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785055+0000", 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0270000000000001e-06 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785060+0000", 2024-06-27T21:24:25.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.437e-06 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124878+0000", 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33981821699999998 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124884+0000", 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0179999999999998e-06 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124930+0000", 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6521999999999999e-05 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124946+0000", 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5347999999999999e-05 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.125053+0000", 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107271 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163683 4.d 4.883d3e0d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.075334804999997, 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.612742381, 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.058 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163683 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910320+0000", 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.08e-06 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910324+0000", 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7899999999999999e-06 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910336+0000", 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1676e-05 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910341+0000", 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2360000000000004e-06 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910386+0000", 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4273000000000001e-05 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:24 smithi190 ceph-mon[32247]: osdmap e375: 12 total, 12 up, 12 in 2024-06-27T21:24:25.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/6 objects degraded (50.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910393+0000", 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2790000000000001e-06 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523031+0000", 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.612637722 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523061+0000", 2024-06-27T21:24:25.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0324999999999998e-05 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163697 4.d 4.ba6d2ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.052923307999997, 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59065456900000002, 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.064 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163697 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932731+0000", 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932735+0000", 2024-06-27T21:24:25.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3970000000000004e-06 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932745+0000", 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7929999999999992e-06 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932750+0000", 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2100000000000001e-06 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932794+0000", 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3946000000000002e-05 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932802+0000", 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5909999999999998e-06 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523377+0000", 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59057547099999996 2024-06-27T21:24:25.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523385+0000", 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2579999999999998e-06 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163710 4.d 4.4da514d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.039550611000003, 2024-06-27T21:24:25.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57733542599999998, 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163710 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:24:25.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946104+0000", 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.277e-06 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946109+0000", 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3809999999999998e-06 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946122+0000", 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3040000000000001e-05 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946130+0000", 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8469999999999996e-06 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946187+0000", 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7229999999999999e-05 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946199+0000", 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-05 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523432+0000", 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57723327000000002 2024-06-27T21:24:25.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523438+0000", 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8309999999999997e-06 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163683 4.d 4:b07cbc11:::benchmark_data_smithi192_88961_object72944:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 39.454744210000001, 2024-06-27T21:24:25.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46728570899999999, 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163683 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530911+0000", 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2199999999999999e-06 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530921+0000", 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1949999999999992e-06 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530939+0000", 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8541000000000001e-05 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530951+0000", 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.202e-05 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531024+0000", 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2605e-05 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531031+0000", 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6669999999999996e-06 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.980931+0000", 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44989959299999999 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.981077+0000", 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014631999999999999 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.981221+0000", 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000143874 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.987926+0000", 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006704738 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991110+0000", 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031838470000000001 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.997913+0000", 2024-06-27T21:24:25.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006803478 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.997982+0000", 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9077000000000003e-05 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.998195+0000", 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021253400000000001 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.081 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163697 4.d 4:b54b65d0:::benchmark_data_smithi192_88961_object72958:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 39.454664624999999, 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47026454699999998, 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.082 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163697 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530993+0000", 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5439999999999999e-06 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531002+0000", 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0939999999999997e-06 2024-06-27T21:24:25.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531007+0000", 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4369999999999998e-06 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531076+0000", 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8229e-05 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531084+0000", 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9260000000000006e-06 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984774+0000", 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45368975900000003 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984852+0000", 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7356999999999998e-05 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984929+0000", 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7101999999999995e-05 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991124+0000", 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0061952420000000001 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.995794+0000", 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004670319 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001042+0000", 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0052480169999999998 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001078+0000", 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5355000000000003e-05 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001253+0000", 2024-06-27T21:24:25.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017562099999999999 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163710 4.d 4:b28a5b20:::benchmark_data_smithi192_88961_object72971:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 39.454636567999998, 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47328603800000002, 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:25.089 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163710 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:24:25.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4299999999999999e-07 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531019+0000", 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3039999999999999e-06 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531025+0000", 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.874e-06 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531030+0000", 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2359999999999998e-06 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531108+0000", 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8033000000000003e-05 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531115+0000", 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1980000000000004e-06 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988002+0000", 2024-06-27T21:24:25.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45688679300000001 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988079+0000", 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7922000000000004e-05 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988158+0000", 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991184+0000", 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030261609999999999 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.995891+0000", 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004706422 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004153+0000", 2024-06-27T21:24:25.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082619319999999996 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004184+0000", 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1028999999999997e-05 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004303+0000", 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119159 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e375,346 4:2e81f327:::benchmark_data_smithi192_88961_object15186:head,336'6645)", 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:24.018765+0000", 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.96688786299999996, 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35042788800000002, 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018765+0000", 2024-06-27T21:24:25.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018765+0000", 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018766+0000", 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018770+0000", 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8680000000000001e-06 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369042+0000", 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35027164199999999 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369053+0000", 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0964e-05 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369116+0000", 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3647999999999998e-05 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369124+0000", 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1100000000000003e-06 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369193+0000", 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9045e-05 2024-06-27T21:24:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.101 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:25.101 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:25.101 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:24:25.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:25.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:25.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:25.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:25.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:25.370 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:25.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:25.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:25.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:25.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:25.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:25.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:25.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:24:25.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:25.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:25.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:25 smithi151 ceph-mon[28965]: pgmap v984: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 93 op/s; 0 B/s, 243 objects/s recovering 2024-06-27T21:24:25.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:25 smithi151 ceph-mon[28965]: osdmap e376: 12 total, 12 up, 12 in 2024-06-27T21:24:25.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:25.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:25.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:26.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:25 smithi190 ceph-mon[32247]: pgmap v984: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 93 op/s; 0 B/s, 243 objects/s recovering 2024-06-27T21:24:26.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:25 smithi190 ceph-mon[32247]: osdmap e376: 12 total, 12 up, 12 in 2024-06-27T21:24:26.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:25 smithi192 ceph-mon[32956]: pgmap v984: 27 pgs: 2 active+recovering+undersized+remapped, 25 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 93 op/s; 0 B/s, 243 objects/s recovering 2024-06-27T21:24:26.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:25 smithi192 ceph-mon[32956]: osdmap e376: 12 total, 12 up, 12 in 2024-06-27T21:24:26.090 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:26.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:26.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:26.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:26.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:26.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:26.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:26.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:26.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:26.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:26.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:26.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:26.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:26.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:26.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:26.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:26.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:26.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:27.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:27.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:27.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:27.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:27.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:27.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:27.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:27.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:27.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:27.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:27.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:27.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:27.737 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:24:27.737 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:24:27.738 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:24:27.738 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:24:27.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:27 smithi192 ceph-mon[32956]: pgmap v986: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 233 objects/s recovering 2024-06-27T21:24:27.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:27.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:24:27.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:27.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:27 smithi151 ceph-mon[28965]: pgmap v986: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 233 objects/s recovering 2024-06-27T21:24:27.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:27.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:27.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:28.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:27 smithi190 ceph-mon[32247]: pgmap v986: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 233 objects/s recovering 2024-06-27T21:24:28.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:28.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:28.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:28.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:28.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:28.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:28.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:24:28.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:28.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:28.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:28.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:28.658 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:28.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:28.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:28.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:28.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:28.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:28.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:28.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:28.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:29.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:29.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:29.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:29.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:29.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:29.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:29.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:29.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:29.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:29.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:29.634 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-06-27T21:24:29.634 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:24:29.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:29.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:29.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:29.837 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:29.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: pgmap v987: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 180 objects/s recovering 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: 4.b scrub starts 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: 4.b scrub ok 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1821201485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:29.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/260942511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:24:29.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:29.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:29.975 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: pgmap v987: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 180 objects/s recovering 2024-06-27T21:24:29.975 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: 4.b scrub starts 2024-06-27T21:24:29.975 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: 4.b scrub ok 2024-06-27T21:24:29.976 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:24:29.976 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:24:29.976 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1821201485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:24:29.976 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:29.976 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/260942511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: pgmap v987: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 180 objects/s recovering 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: 4.b scrub starts 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: 4.b scrub ok 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1821201485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/260942511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:24:30.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:30.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:30.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:30.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:30.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:30.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:30.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:30.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:30.493 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:24:30.493 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:24:30.493 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:24:30.494 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:24:30.494 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:24:30.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:30.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:30.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:30.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:30.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:30.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:30.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:30.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:30 smithi151 ceph-mon[28965]: 4.c scrub starts 2024-06-27T21:24:30.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:30 smithi151 ceph-mon[28965]: 4.c scrub ok 2024-06-27T21:24:30.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:30 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:24:30.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1821201485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:24:30.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/260942511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:24:30.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:30 smithi151 ceph-mon[28965]: osdmap e377: 12 total, 12 up, 12 in 2024-06-27T21:24:30.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:30.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:30 smithi190 ceph-mon[32247]: 4.c scrub starts 2024-06-27T21:24:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:30 smithi190 ceph-mon[32247]: 4.c scrub ok 2024-06-27T21:24:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:30 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:24:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1821201485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:24:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/260942511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:24:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:30 smithi190 ceph-mon[32247]: osdmap e377: 12 total, 12 up, 12 in 2024-06-27T21:24:31.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:30 smithi192 ceph-mon[32956]: 4.c scrub starts 2024-06-27T21:24:31.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:30 smithi192 ceph-mon[32956]: 4.c scrub ok 2024-06-27T21:24:31.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:30 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:24:31.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1821201485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:24:31.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/260942511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:24:31.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:30 smithi192 ceph-mon[32956]: osdmap e377: 12 total, 12 up, 12 in 2024-06-27T21:24:31.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:31.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:31.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:24:31.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:31.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:31.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:31.429 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:31.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:31.493 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:31.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:31.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:31.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:31.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:31.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:31.861 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:31 smithi192 ceph-mon[32956]: pgmap v989: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 227 objects/s recovering 2024-06-27T21:24:31.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:31.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:31.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:31 smithi151 ceph-mon[28965]: pgmap v989: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 227 objects/s recovering 2024-06-27T21:24:31.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:32.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:32.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:31 smithi190 ceph-mon[32247]: pgmap v989: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 227 objects/s recovering 2024-06-27T21:24:32.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:32.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:32.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:32.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:32.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:32.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:32.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:32.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:32.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:32.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:32.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:32.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:32.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:32.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:32.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:32.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:32.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:33.018 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:24:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:24:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:24:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:24:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "replica_scrub(pg: 1.7,from:0'0,to:0'0,epoch:202/163,start:1:e0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:16:46.004017+0000", 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 467.01262376099999, 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59493611400000002, 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.004017+0000", 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.004017+0000", 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.004018+0000", 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4300000000000003e-07 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.004022+0000", 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1679999999999997e-06 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.598694+0000", 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59467245400000002 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.598711+0000", 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6577000000000001e-05 2024-06-27T21:24:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.598892+0000", 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000181146 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.598925+0000", 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2638999999999997e-05 2024-06-27T21:24:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:16:46.598953+0000", 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8586999999999999e-05 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.15132.0:123914 3.1 e235/186 3:886387db:::benchmark_data_smithi192_74314_object17276:head v 235'7905, mlcod=235'7902)", 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:18:09.165686+0000", 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 383.85095474600001, 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.190831317, 2024-06-27T21:24:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.165686+0000", 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.165686+0000", 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.165687+0000", 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.159e-06 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.165693+0000", 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3739999999999997e-06 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.353724+0000", 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18803097199999999 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.353730+0000", 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2280000000000003e-06 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.353786+0000", 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6181000000000003e-05 2024-06-27T21:24:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.353802+0000", 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5637e-05 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.356483+0000", 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026813309999999999 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:18:09.356517+0000", 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4434999999999998e-05 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:686 4.7 e288/282 4:e23367be:::benchmark_data_smithi192_88961_object685:head v 288'39, mlcod=288'36)", 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:43.815439+0000", 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 229.20120177800001, 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.24340118499999999, 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.815439+0000", 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.815439+0000", 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.815440+0000", 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:43.815497+0000", 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7806000000000002e-05 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.058457+0000", 2024-06-27T21:24:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.242959815 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.058463+0000", 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3940000000000004e-06 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.058509+0000", 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6523000000000002e-05 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.058524+0000", 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4491999999999999e-05 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.058813+0000", 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028917199999999998 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:44.058840+0000", 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7409000000000001e-05 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:2737 4.5 e289/282 4:ab79fa01:::benchmark_data_smithi192_88961_object2736:head v 289'193, mlcod=289'190)", 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:45.530169+0000", 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 227.486471704, 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.220714667, 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.530169+0000", 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.530169+0000", 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.530170+0000", 2024-06-27T21:24:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4499999999999996e-07 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.530344+0000", 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017459800000000001 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.750347+0000", 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.220002953 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.750353+0000", 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7599999999999999e-06 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.750403+0000", 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9917999999999999e-05 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.750420+0000", 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6900000000000001e-05 2024-06-27T21:24:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.750858+0000", 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00043814400000000001 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.750884+0000", 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5649000000000001e-05 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:2789 4.e e289/282 4:7ce23414:::benchmark_data_smithi192_88961_object2788:head v 289'157, mlcod=289'154)", 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:45.566949+0000", 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 227.44969132700001, 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18576094900000001, 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.566949+0000", 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.566949+0000", 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.566950+0000", 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9400000000000005e-07 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.566985+0000", 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4671000000000001e-05 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.752163+0000", 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.18517789400000001 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.752167+0000", 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8790000000000002e-06 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.752221+0000", 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3319999999999998e-05 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.752239+0000", 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8057999999999999e-05 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.752660+0000", 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000420856 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:45.752710+0000", 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0677000000000003e-05 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(4.5 313/309 [PushOp(4:a4ec198a:::benchmark_data_smithi192_88961_object14492:head, version: 296'904, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a4ec198a:::benchmark_data_smithi192_88961_object14492:head@296'904, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:29.732847+0000", 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 183.28379412000001, 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.206312726, 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.732847+0000", 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.732847+0000", 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.732847+0000", 2024-06-27T21:24:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0999999999999997e-07 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.732897+0000", 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9190999999999997e-05 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.937945+0000", 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.205048543 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.937964+0000", 2024-06-27T21:24:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8655999999999999e-05 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.938088+0000", 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012372899999999999 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.938102+0000", 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4622e-05 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.939159+0000", 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001057175 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:54543 4.7 e313/309 4:e9e429f8:::benchmark_data_smithi192_88961_object54542:head v 313'3435, mlcod=294'668)", 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:29.750986+0000", 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 183.26565448900001, 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.19266193600000001, 2024-06-27T21:24:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.750986+0000", 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.750986+0000", 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.750987+0000", 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9800000000000003e-07 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.751038+0000", 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1230000000000002e-05 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.938816+0000", 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.187778052 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.938824+0000", 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0460000000000006e-06 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.938882+0000", 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7207999999999998e-05 2024-06-27T21:24:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.938907+0000", 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5451000000000001e-05 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.943613+0000", 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0047057389999999996 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:29.943648+0000", 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5411999999999999e-05 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(4.e 315/309 [PushOp(4:7bd01972:::benchmark_data_smithi192_88961_object1805:head, version: 289'99, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:7bd01972:::benchmark_data_smithi192_88961_object1805:head@289'99, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:34.823444+0000", 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 178.19319712800001, 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22804503000000001, 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.823444+0000", 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.823444+0000", 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.823445+0000", 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5629999999999999e-06 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.823508+0000", 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2415000000000005e-05 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:35.050342+0000", 2024-06-27T21:24:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.226834178 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:35.050356+0000", 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4267000000000001e-05 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:35.050459+0000", 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010276399999999999 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:35.050467+0000", 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2360000000000004e-06 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:35.051489+0000", 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010216069999999999 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(4.e 320/309 [PushOp(4:7fb3a793:::benchmark_data_smithi192_88961_object12169:head, version: 294'742, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:7fb3a793:::benchmark_data_smithi192_88961_object12169:head@294'742, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:45.174078+0000", 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 167.84256292200001, 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27256488600000001, 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.174078+0000", 2024-06-27T21:24:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.174078+0000", 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.174078+0000", 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.61e-07 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.174153+0000", 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4219000000000002e-05 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.444992+0000", 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27083928200000001 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.445004+0000", 2024-06-27T21:24:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1523e-05 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.445082+0000", 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8573999999999997e-05 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.445090+0000", 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1890000000000007e-06 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.446643+0000", 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001552438 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(4.7 320/309 [PushOp(4:e7339692:::benchmark_data_smithi192_88961_object31752:head, version: 305'1999, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:e7339692:::benchmark_data_smithi192_88961_object31752:head@305'1999, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:24:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:45.177485+0000", 2024-06-27T21:24:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 167.83915618399999, 2024-06-27T21:24:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27501013299999999, 2024-06-27T21:24:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.177485+0000", 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.177485+0000", 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.177485+0000", 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9100000000000002e-07 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.177537+0000", 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1826999999999997e-05 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.451267+0000", 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27372951600000001 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.451277+0000", 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0254e-05 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.451345+0000", 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8387999999999996e-05 2024-06-27T21:24:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.451354+0000", 2024-06-27T21:24:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8610000000000004e-06 2024-06-27T21:24:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.452495+0000", 2024-06-27T21:24:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011403959999999999 2024-06-27T21:24:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:121140 4.7 e342/323 4:e995ab46:::benchmark_data_smithi192_88961_object30401:head v 342'7678, mlcod=342'7675)", 2024-06-27T21:24:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:45.382529+0000", 2024-06-27T21:24:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 107.634111733, 2024-06-27T21:24:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22043254700000001, 2024-06-27T21:24:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.382529+0000", 2024-06-27T21:24:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.382529+0000", 2024-06-27T21:24:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.382530+0000", 2024-06-27T21:24:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0389999999999999e-06 2024-06-27T21:24:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.382537+0000", 2024-06-27T21:24:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-06-27T21:24:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.600519+0000", 2024-06-27T21:24:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.21798243 2024-06-27T21:24:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.600524+0000", 2024-06-27T21:24:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3270000000000002e-06 2024-06-27T21:24:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.600586+0000", 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2081000000000004e-05 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.600601+0000", 2024-06-27T21:24:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5041999999999999e-05 2024-06-27T21:24:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.602880+0000", 2024-06-27T21:24:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022789580000000002 2024-06-27T21:24:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:45.602962+0000", 2024-06-27T21:24:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1701999999999998e-05 2024-06-27T21:24:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "replica_scrub(pg: 4.7,from:0'0,to:344'8243,epoch:345/323,start:4:ecf72b89:::benchmark_data_smithi192_88961_object43974:0,end:4:ed73da31:::benchmark_data_smithi192_88961_object85114:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-27T21:24:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:56.557583+0000", 2024-06-27T21:24:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.459058210999999, 2024-06-27T21:24:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.451669074, 2024-06-27T21:24:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:24:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.557583+0000", 2024-06-27T21:24:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.557583+0000", 2024-06-27T21:24:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.557583+0000", 2024-06-27T21:24:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2200000000000004e-07 2024-06-27T21:24:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.557587+0000", 2024-06-27T21:24:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2119999999999997e-06 2024-06-27T21:24:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:57.009081+0000", 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45149388200000001 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:57.009090+0000", 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4719999999999995e-06 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:57.009194+0000", 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010445799999999999 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:57.009206+0000", 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2147000000000001e-05 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:57.009252+0000", 2024-06-27T21:24:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5281000000000002e-05 2024-06-27T21:24:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:148564 4.e 4:7883297f:::benchmark_data_smithi192_88961_object57825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-06-27T21:24:33.089 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:24.183633+0000", 2024-06-27T21:24:33.089 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.833007605999995, 2024-06-27T21:24:33.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.181166985, 2024-06-27T21:24:33.089 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.089 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.089 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148564 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183633+0000", 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183633+0000", 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183635+0000", 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183641+0000", 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9429999999999997e-06 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183655+0000", 2024-06-27T21:24:33.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4439e-05 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183663+0000", 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4290000000000004e-06 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183721+0000", 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7954000000000001e-05 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183813+0000", 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1935000000000007e-05 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6", 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.183881+0000", 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8663000000000003e-05 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.186810+0000", 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029290140000000002 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.364657+0000", 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.17784623999999999 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.364711+0000", 2024-06-27T21:24:33.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4429999999999999e-05 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.364800+0000", 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9078000000000003e-05 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:148570 4.e 4:71012da1:::benchmark_data_smithi192_88961_object57831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:24.198070+0000", 2024-06-27T21:24:33.096 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.818570417999993, 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.166889653, 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 148570 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198070+0000", 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198070+0000", 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198073+0000", 2024-06-27T21:24:33.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5450000000000002e-06 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198081+0000", 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2109999999999998e-06 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198102+0000", 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1316999999999999e-05 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198114+0000", 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1122e-05 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198177+0000", 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3911999999999994e-05 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198270+0000", 2024-06-27T21:24:33.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2346e-05 2024-06-27T21:24:33.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6", 2024-06-27T21:24:33.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.198351+0000", 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1697000000000001e-05 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.201288+0000", 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029368559999999998 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.364844+0000", 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.16355597099999999 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.364887+0000", 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2247999999999999e-05 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:24.364960+0000", 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3428000000000002e-05 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.103 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163681 4.e 4.cef3b3fe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.904855+0000", 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 49.111785888, 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.616111513, 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163681 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904855+0000", 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904855+0000", 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904857+0000", 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.821e-06 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904863+0000", 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2210000000000002e-06 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904878+0000", 2024-06-27T21:24:33.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5009e-05 2024-06-27T21:24:33.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904887+0000", 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8240000000000006e-06 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904959+0000", 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2492999999999999e-05 2024-06-27T21:24:33.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.904967+0000", 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0209999999999999e-06 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.520933+0000", 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61596539400000005 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.520966+0000", 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3729999999999997e-05 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163708 4.e 4.c8816cfe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.945905+0000", 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 49.070736171, 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57511373200000004, 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.113 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163708 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.945905+0000", 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.945905+0000", 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.945906+0000", 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.362e-06 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.945911+0000", 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0669999999999999e-06 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.945924+0000", 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2792000000000001e-05 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.945932+0000", 2024-06-27T21:24:33.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0399999999999993e-06 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946036+0000", 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104178 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946050+0000", 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4072e-05 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.521007+0000", 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57495701200000005 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.521018+0000", 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1209000000000001e-05 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.118 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163681 4.e 4:7fcdcf73:::benchmark_data_smithi192_88961_object72942:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.533324+0000", 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.483317200000002, 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44450116099999998, 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163681 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533324+0000", 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533324+0000", 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533326+0000", 2024-06-27T21:24:33.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533332+0000", 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0580000000000003e-06 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533345+0000", 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2721999999999999e-05 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533354+0000", 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2590000000000006e-06 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533445+0000", 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1316000000000005e-05 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533452+0000", 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3770000000000001e-06 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.962776+0000", 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42932388700000002 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.962910+0000", 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013433299999999999 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.963004+0000", 2024-06-27T21:24:33.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4338000000000004e-05 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.973416+0000", 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010411675 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977503+0000", 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040875529999999998 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977546+0000", 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2317000000000002e-05 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977583+0000", 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7199000000000001e-05 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977825+0000", 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024178399999999999 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.125 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163708 4.e 4:7f368113:::benchmark_data_smithi192_88961_object72969:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.533385+0000", 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.483255991999997, 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45622701999999998, 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.126 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163708 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533385+0000", 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533385+0000", 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533385+0000", 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.03e-07 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533388+0000", 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.587e-06 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533394+0000", 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8019999999999997e-06 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533399+0000", 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3530000000000002e-06 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533489+0000", 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0482999999999994e-05 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533498+0000", 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7430000000000008e-06 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.966295+0000", 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.432797288 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.966434+0000", 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013827300000000001 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.966584+0000", 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000149982 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977451+0000", 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010867546 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977883+0000", 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00043194199999999999 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.989337+0000", 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01145389 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.989376+0000", 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9171000000000002e-05 2024-06-27T21:24:33.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.989612+0000", 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00023555700000000001 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163722 4.e 4:711d24ac:::benchmark_data_smithi192_88961_object72983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.533408+0000", 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.483232979, 2024-06-27T21:24:33.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.456485382, 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163722 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533408+0000", 2024-06-27T21:24:33.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533408+0000", 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533408+0000", 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4099999999999999e-07 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533411+0000", 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.605e-06 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533418+0000", 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9229999999999996e-06 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533423+0000", 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9350000000000002e-06 2024-06-27T21:24:33.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533529+0000", 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106734 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533534+0000", 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2710000000000003e-06 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.970471+0000", 2024-06-27T21:24:33.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43693688800000002 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.970579+0000", 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001088 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.970665+0000", 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6054e-05 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977833+0000", 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071677779999999996 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.983112+0000", 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0052783129999999998 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.989664+0000", 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0065528369999999997 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.989719+0000", 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4211e-05 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.989893+0000", 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000174592 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.140 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163728 4.e 4:7c915199:::benchmark_data_smithi192_88961_object72989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.533439+0000", 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.483201545, 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45987196499999999, 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163728 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533439+0000", 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533439+0000", 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533440+0000", 2024-06-27T21:24:33.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533442+0000", 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3649999999999998e-06 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533449+0000", 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7270000000000001e-06 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:24:33.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533455+0000", 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0719999999999996e-06 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533560+0000", 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104947 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.533567+0000", 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5639999999999999e-06 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.973459+0000", 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43989167099999998 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.973591+0000", 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131662 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.973737+0000", 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014619200000000001 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.977899+0000", 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041621139999999997 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.983148+0000", 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0052492270000000004 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.993135+0000", 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0099864270000000008 2024-06-27T21:24:33.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.993176+0000", 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.087e-05 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.993311+0000", 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013565699999999999 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:24:33.147 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:24:33.148 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:24:33.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:33.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:33.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:33.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:33.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:33.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:33.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:33.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:33.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:33.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:33.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:33.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:33.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:33.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:33.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:33.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:33 smithi151 ceph-mon[28965]: pgmap v990: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 118 objects/s recovering 2024-06-27T21:24:34.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:33 smithi190 ceph-mon[32247]: pgmap v990: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 118 objects/s recovering 2024-06-27T21:24:34.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:34.015 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:33 smithi192 ceph-mon[32956]: pgmap v990: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 118 objects/s recovering 2024-06-27T21:24:34.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:34.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:24:34.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:34.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:34.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:34.285 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:34.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:34.338 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:34.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:34.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:34.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:34.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:34.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:34.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:34.718 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:34.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:34.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:34.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:34.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:34.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:34.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:35.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:35.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:35.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:35.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:35.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:35.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 3 1 2024-06-27T21:24:35.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:35.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:35.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:35.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:35.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:35.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:35.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:35.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:35.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:35.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:35.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:35.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:35 smithi190 ceph-mon[32247]: pgmap v991: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 143 objects/s recovering 2024-06-27T21:24:35.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:35.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:35.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:35 smithi151 ceph-mon[28965]: pgmap v991: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 143 objects/s recovering 2024-06-27T21:24:35.955 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:35 smithi192 ceph-mon[32956]: pgmap v991: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 143 objects/s recovering 2024-06-27T21:24:35.961 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:35.987 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:36.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:36.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:36.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:36.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:36.179 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:24:36.179 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:24:36.179 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:24:36.179 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:24:36.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:36.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:36.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:36.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:36.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:36.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:36.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:36.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:36.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:36.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:24:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:36.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:36.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:36.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:36.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:37.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:37.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:37.081 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:37.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:37.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:37.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:37.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:37.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:37.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:37.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:37.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:37.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:37.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:37.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:24:37.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:37.793 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:37 smithi192 ceph-mon[32956]: pgmap v992: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 97 objects/s recovering 2024-06-27T21:24:37.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:37.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:37.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:37.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:37.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:37 smithi151 ceph-mon[28965]: pgmap v992: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 97 objects/s recovering 2024-06-27T21:24:38.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:37 smithi190 ceph-mon[32247]: pgmap v992: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 97 objects/s recovering 2024-06-27T21:24:38.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:38.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:38.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:38.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:38.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:38.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:38.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:38.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:38.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:38.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:38.572 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:24:38.572 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:24:38.573 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:24:38.573 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:24:38.573 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:24:38.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:38.663 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_2' removed 2024-06-27T21:24:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:38.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:38.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:38.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/785335382' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:24:38.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:38.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:38.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:38.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:39.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/785335382' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:24:39.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:39.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/785335382' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:24:39.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:39.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:39.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:24:39.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:39.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:39.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:39.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:39.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:39.442 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:24:39.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:39.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:39.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:39.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:39.694 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:39 smithi192 ceph-mon[32956]: pgmap v993: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 97 objects/s recovering 2024-06-27T21:24:39.694 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/785335382' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:24:39.694 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:39 smithi192 ceph-mon[32956]: osdmap e378: 12 total, 12 up, 12 in 2024-06-27T21:24:39.694 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:39.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:39 smithi151 ceph-mon[28965]: pgmap v993: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 97 objects/s recovering 2024-06-27T21:24:39.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/785335382' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:24:39.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:39 smithi151 ceph-mon[28965]: osdmap e378: 12 total, 12 up, 12 in 2024-06-27T21:24:39.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:39.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:39.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:39.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:39.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:39.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:40.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:39 smithi190 ceph-mon[32247]: pgmap v993: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2/6 objects degraded (33.333%); 0 B/s, 97 objects/s recovering 2024-06-27T21:24:40.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/785335382' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:24:40.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:39 smithi190 ceph-mon[32247]: osdmap e378: 12 total, 12 up, 12 in 2024-06-27T21:24:40.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg undersized (PG_DEGRADED) 2024-06-27T21:24:40.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:40.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:40.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:40.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:40.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:40.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:40.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:40.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:40.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:40.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:40.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:40.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:40.672 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-06-27T21:24:40.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:40.708 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:24:40.709 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:24:40.710 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-06-27T21:24:40.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-06-27T21:24:40.770 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:40.771 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:40.771 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1099077376' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:24:40.771 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1589025947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:40.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:40.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:40.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:40.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:40.940 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:40.940 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:40.940 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1099077376' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:24:40.940 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1589025947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1099077376' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:24:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1589025947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:41.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:41.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:41.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:41.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:41.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:41.280 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:41.280 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:41.300 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:41.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:41.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:41.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 4 1 2024-06-27T21:24:41.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:24:41.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:24:41.520 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:24:41.521 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:24:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:24:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:24:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:24:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:24:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:24:41.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:41.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:41.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:41.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:41.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:41.768 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:41.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:41.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:41.883 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:41 smithi192 ceph-mon[32956]: pgmap v995: 11 pgs: 11 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:41.883 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:41.883 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:41.883 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:41 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg undersized) 2024-06-27T21:24:41.884 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1099077376' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:24:41.884 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:41 smithi192 ceph-mon[32956]: osdmap e379: 12 total, 12 up, 12 in 2024-06-27T21:24:41.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:41.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:24:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:41.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:41.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:41.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:41 smithi190 ceph-mon[32247]: pgmap v995: 11 pgs: 11 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:41.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:41.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:41.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:41 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg undersized) 2024-06-27T21:24:41.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1099077376' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:24:41.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:41 smithi190 ceph-mon[32247]: osdmap e379: 12 total, 12 up, 12 in 2024-06-27T21:24:41.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:41 smithi151 ceph-mon[28965]: pgmap v995: 11 pgs: 11 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:41.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:41.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:41.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:41 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg undersized) 2024-06-27T21:24:41.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1099077376' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:24:41.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:41 smithi151 ceph-mon[28965]: osdmap e379: 12 total, 12 up, 12 in 2024-06-27T21:24:42.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:42.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:42.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:42.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:42.169 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:24:42.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:42.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:42.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:42.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:42.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:42.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:42.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:42.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:42.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:42.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:42.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:42.874 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:42.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:42.913 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:42.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:42.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:43.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:43.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:43.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:43.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:43.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:43.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:43.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:43.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:43.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:43.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:43.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:43.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:43.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:43.776 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:43.776 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:43.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:43.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:43 smithi151 ceph-mon[28965]: pgmap v997: 11 pgs: 11 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:43.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:43.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:43.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:44.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:43 smithi190 ceph-mon[32247]: pgmap v997: 11 pgs: 11 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:44.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:44.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:44.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:43 smithi192 ceph-mon[32956]: pgmap v997: 11 pgs: 11 active+clean; 577 KiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:44.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:44.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:44.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:24:44.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:44.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:44.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:44.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:44.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:24:44.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:44.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:44.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:44.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:44.658 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:24:44.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:44.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:44.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:44.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:44.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:44.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=1627829847,config=2353167246,health=551204407,kv=1749082271}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3928690017,logm=2195729776}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640833082}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851017498}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761821877}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901577485}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900457952}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065646739}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4146233317}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405948878}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=380003174}) 2024-06-27T21:24:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3531280564}) 2024-06-27T21:24:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503928033}) 2024-06-27T21:24:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692621108}) 2024-06-27T21:24:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3398535731}) 2024-06-27T21:24:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419134673}) 2024-06-27T21:24:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956732850}) 2024-06-27T21:24:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185639675}) 2024-06-27T21:24:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541336244}) 2024-06-27T21:24:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459782476}) 2024-06-27T21:24:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2808586723}) 2024-06-27T21:24:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546918765}) 2024-06-27T21:24:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3442191505}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946878464}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138457015}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970204816}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573663580}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3941908053}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724492076}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2043835449}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913942537}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792432322}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2082392137}) 2024-06-27T21:24:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594296997}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868862736}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109542708}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1447525207}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461015737}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077461215}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648329058}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=1,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=65} crc {logm=3889938699,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=4128446811,mon_config_key=1936383811}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3687334432}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2491588863}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=26,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=56} crc {mon_config_key=2722354368,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=1604317357,osdmap=630051056}) 2024-06-27T21:24:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=769861923}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=1627829847,config=2353167246,health=551204407,kv=1749082271}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3928690017,logm=2195729776}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640833082}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851017498}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761821877}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901577485}) 2024-06-27T21:24:44.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900457952}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065646739}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4146233317}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405948878}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=380003174}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3531280564}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503928033}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692621108}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3398535731}) 2024-06-27T21:24:44.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419134673}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956732850}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185639675}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541336244}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459782476}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2808586723}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546918765}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3442191505}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946878464}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138457015}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970204816}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573663580}) 2024-06-27T21:24:44.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3941908053}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724492076}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2043835449}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913942537}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792432322}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2082392137}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594296997}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868862736}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109542708}) 2024-06-27T21:24:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1447525207}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461015737}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077461215}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648329058}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=1,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=65} crc {logm=3889938699,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=4128446811,mon_config_key=1936383811}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3687334432}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2491588863}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=26,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=56} crc {mon_config_key=2722354368,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=1604317357,osdmap=630051056}) 2024-06-27T21:24:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=769861923}) 2024-06-27T21:24:45.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:45.025 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:45.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=1627829847,config=2353167246,health=551204407,kv=1749082271}) 2024-06-27T21:24:45.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3928690017,logm=2195729776}) 2024-06-27T21:24:45.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640833082}) 2024-06-27T21:24:45.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851017498}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3761821877}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901577485}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=900457952}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1065646739}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4146233317}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405948878}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=380003174}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3531280564}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=503928033}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692621108}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3398535731}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419134673}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2956732850}) 2024-06-27T21:24:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185639675}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541336244}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=459782476}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2808586723}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3546918765}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3442191505}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1946878464}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138457015}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970204816}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573663580}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3941908053}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724492076}) 2024-06-27T21:24:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2043835449}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3913942537}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792432322}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2082392137}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594296997}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868862736}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109542708}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1447525207}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461015737}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077461215}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648329058}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=1,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=65} crc {logm=3889938699,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=4128446811,mon_config_key=1936383811}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3687334432}) 2024-06-27T21:24:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2491588863}) 2024-06-27T21:24:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=26,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=56} crc {mon_config_key=2722354368,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=1604317357,osdmap=630051056}) 2024-06-27T21:24:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=769861923}) 2024-06-27T21:24:45.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:45.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:45.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:45.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:45.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:45.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:45.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:45.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:45.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:45.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:45.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:45 smithi190 ceph-mon[32247]: pgmap v998: 11 pgs: 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:45.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:45.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:45.731 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:45 smithi151 ceph-mon[28965]: pgmap v998: 11 pgs: 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:45.731 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:45.731 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:45.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:45.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:45.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:45.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:45.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:46.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:46.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:46.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:45 smithi192 ceph-mon[32956]: pgmap v998: 11 pgs: 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:46.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:46.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:46.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:46.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:46.216 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:46.216 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:46.257 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:46.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:46.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:46.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:46.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:46.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:46.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:46.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:46.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:46.680 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-06-27T21:24:46.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:46.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:46.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:46.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:46 smithi151 ceph-mon[28965]: osdmap e380: 12 total, 12 up, 12 in 2024-06-27T21:24:46.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/78852330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:24:46.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:46.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:46.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:46.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:46 smithi190 ceph-mon[32247]: osdmap e380: 12 total, 12 up, 12 in 2024-06-27T21:24:46.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/78852330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:24:46.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:24:46.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:46.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:46.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:47.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:47.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:47.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:47.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:47.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:46 smithi192 ceph-mon[32956]: osdmap e380: 12 total, 12 up, 12 in 2024-06-27T21:24:47.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/78852330' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:24:47.101 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:24:47.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:47.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:47.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:47.291 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:47.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:47.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:47.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:47.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:47.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:47.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:47.695 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:24:47.695 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_3' created 2024-06-27T21:24:47.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:47.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:47.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:47.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:47.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:47.916 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: pgmap v1000: 11 pgs: 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:47.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/78852330' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:24:47.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: osdmap e381: 12 total, 12 up, 12 in 2024-06-27T21:24:47.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:47.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:47.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1483234635' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-27T21:24:47.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/405333210' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:47.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: pgmap v1000: 11 pgs: 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/78852330' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:24:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: osdmap e381: 12 total, 12 up, 12 in 2024-06-27T21:24:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1483234635' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-27T21:24:47.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/405333210' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:47.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 5 1 2024-06-27T21:24:48.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:48.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:48.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: pgmap v1000: 11 pgs: 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/78852330' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:24:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: osdmap e381: 12 total, 12 up, 12 in 2024-06-27T21:24:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1483234635' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-06-27T21:24:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/405333210' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:24:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:48.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:48.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:48.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:48.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:48.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:48.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:48.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:48.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:48.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:48.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:48.754 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:24:48.754 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:24:48.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:48.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:48.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:48.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:48.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:48.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:48 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:24:48.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1483234635' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-27T21:24:48.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/405333210' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:24:48.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:48 smithi151 ceph-mon[28965]: osdmap e382: 12 total, 12 up, 12 in 2024-06-27T21:24:48.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:24:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:48 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:24:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1483234635' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-27T21:24:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/405333210' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:24:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:48 smithi190 ceph-mon[32247]: osdmap e382: 12 total, 12 up, 12 in 2024-06-27T21:24:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:24:49.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:49.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:49.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:48 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:24:49.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1483234635' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-06-27T21:24:49.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/405333210' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:24:49.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:48 smithi192 ceph-mon[32956]: osdmap e382: 12 total, 12 up, 12 in 2024-06-27T21:24:49.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:24:49.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:49.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:49.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:49.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:49.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:24:49.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:49.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:49.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:49.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:49.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:49.657 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:49.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:49.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:49.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:49.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:49.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: pgmap v1003: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: osdmap e383: 12 total, 12 up, 12 in 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:50.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: pgmap v1003: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: osdmap e383: 12 total, 12 up, 12 in 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:50.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:50.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: pgmap v1003: 27 pgs: 3 creating+peering, 13 unknown, 11 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:50.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:24:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: osdmap e383: 12 total, 12 up, 12 in 2024-06-27T21:24:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:50.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-06-27T21:24:50.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:50.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:50.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:50.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:50.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:50.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:50.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:50.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:50.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:50.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:50.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:50.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:50.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:50.904 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:50.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:50.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:50 smithi151 ceph-mon[28965]: osdmap e384: 12 total, 12 up, 12 in 2024-06-27T21:24:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:24:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:24:50.993 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:51.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:51.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:50 smithi190 ceph-mon[32247]: osdmap e384: 12 total, 12 up, 12 in 2024-06-27T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:24:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:51.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:51.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:50 smithi192 ceph-mon[32956]: osdmap e384: 12 total, 12 up, 12 in 2024-06-27T21:24:51.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:51.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:51.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:24:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:24:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:24:51.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:51.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:51.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:51.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:51.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:51.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:24:51.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:51.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:51.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:24:51.590 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:24:51.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:24:51.611 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:24:51.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:24:51.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:24:51.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:51.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:24:51.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:24:51.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:51.679 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:24:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:24:51.702 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:24:51.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:24:51.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:51.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:51 smithi151 ceph-mon[28965]: pgmap v1006: 27 pgs: 5 creating+peering, 6 unknown, 16 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:51 smithi151 ceph-mon[28965]: osdmap e385: 12 total, 12 up, 12 in 2024-06-27T21:24:51.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3258974022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:51.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:51.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:51.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:51.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:24:51.926 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:24:51.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:24:51.948 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:24:51.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:24:51.970 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:24:51.992 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:24:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:24:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:51 smithi190 ceph-mon[32247]: pgmap v1006: 27 pgs: 5 creating+peering, 6 unknown, 16 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:51 smithi190 ceph-mon[32247]: osdmap e385: 12 total, 12 up, 12 in 2024-06-27T21:24:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3258974022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:52.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:24:52.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:24:52.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:52.037 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:24:52.037 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:24:52.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:51 smithi192 ceph-mon[32956]: pgmap v1006: 27 pgs: 5 creating+peering, 6 unknown, 16 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:51 smithi192 ceph-mon[32956]: osdmap e385: 12 total, 12 up, 12 in 2024-06-27T21:24:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3258974022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:24:52.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:52.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:52.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:52.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:52.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:52.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:52.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:52.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:52.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:52.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:52.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:52.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:52.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:52.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:52.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:53.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:53.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:53.108 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:53.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:53.179 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:53.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:53.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:53.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:53.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:53.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:53.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:53.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:53.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:53.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:53.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:53.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:53 smithi151 ceph-mon[28965]: pgmap v1008: 27 pgs: 5 creating+peering, 22 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:53.835 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:24:53.835 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:24:53.835 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:24:53.835 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:24:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:53.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:53.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:53.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:53 smithi190 ceph-mon[32247]: pgmap v1008: 27 pgs: 5 creating+peering, 22 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:54.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:53 smithi192 ceph-mon[32956]: pgmap v1008: 27 pgs: 5 creating+peering, 22 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:54.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:54.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:54.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:54.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:54.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:54.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:54.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:54.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:54.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:54.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:54.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:54.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:54.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:24:54.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:54.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:54.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:54.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:55.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:55.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:55.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:55.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:55.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:55.303 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:55.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:55.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:55.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:55.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:55.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:24:55.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:55.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:55.724 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:55.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:55.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:55.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:55.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:55.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:56.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:55 smithi190 ceph-mon[32247]: pgmap v1009: 27 pgs: 2 creating+peering, 25 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:56.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:56.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:55 smithi151 ceph-mon[28965]: pgmap v1009: 27 pgs: 2 creating+peering, 25 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:56.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:56.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:55 smithi192 ceph-mon[32956]: pgmap v1009: 27 pgs: 2 creating+peering, 25 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:56.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:24:56.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:24:56.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:56.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:56.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:56.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:56.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:56.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:56.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:56.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:56.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:56.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:56.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:56.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:56.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:56.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:56.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:57.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:24:57.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:24:57.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:24:57.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:24:57.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:57.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:57.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:57.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:57.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:57.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:57.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:57.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:57.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:57.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:57.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:57.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:57.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:57 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:57 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:57 smithi190 ceph-mon[32247]: pgmap v1010: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:57 smithi190 ceph-mon[32247]: osdmap e386: 12 total, 12 up, 12 in 2024-06-27T21:24:58.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:57 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:58.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:57 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:58.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:57 smithi192 ceph-mon[32956]: pgmap v1010: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:58.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:57 smithi192 ceph-mon[32956]: osdmap e386: 12 total, 12 up, 12 in 2024-06-27T21:24:58.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:58.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:57 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:58.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:57 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:58.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:57 smithi151 ceph-mon[28965]: pgmap v1010: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:58.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:57 smithi151 ceph-mon[28965]: osdmap e386: 12 total, 12 up, 12 in 2024-06-27T21:24:58.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:58.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:58.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:58.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:24:58.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:24:58.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:58.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:58.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:58.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:58.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:24:58.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:24:58.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:24:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:58.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:24:58.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:24:58.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:24:58.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:24:58.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:24:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:58.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:24:59.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:24:59.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:24:59.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:24:59.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:24:59.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:59.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:59.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:59.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:59.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:24:59.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:24:59.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:24:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:59.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:24:59.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:59.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:59.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:24:59.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:24:59.734 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-06-27T21:24:59.735 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:24:59.750 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:24:59.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:24:59.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:24:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:24:59.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:24:59.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:24:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:24:59.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:24:59.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:59 smithi151 ceph-mon[28965]: pgmap v1012: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:24:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:59 smithi151 ceph-mon[28965]: osdmap e387: 12 total, 12 up, 12 in 2024-06-27T21:24:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3232423313' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:24:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:24:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4109934744' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:25:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:59 smithi190 ceph-mon[32247]: pgmap v1012: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:59 smithi190 ceph-mon[32247]: osdmap e387: 12 total, 12 up, 12 in 2024-06-27T21:25:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3232423313' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:25:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:24:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4109934744' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:25:00.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:00.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:59 smithi192 ceph-mon[32956]: pgmap v1012: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:00.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:59 smithi192 ceph-mon[32956]: osdmap e387: 12 total, 12 up, 12 in 2024-06-27T21:25:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3232423313' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:25:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:24:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4109934744' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:25:00.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:00.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:00.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:00.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:00.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:00.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:00.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:00.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:00.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:00.603 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:00.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:00.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:00.735 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-06-27T21:25:00.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:00.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:00.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:00.955 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:00.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:00 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3232423313' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:25:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4109934744' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:25:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:00 smithi190 ceph-mon[32247]: osdmap e388: 12 total, 12 up, 12 in 2024-06-27T21:25:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/345050599' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:25:01.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:00 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3232423313' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:25:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4109934744' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:25:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:00 smithi192 ceph-mon[32956]: osdmap e388: 12 total, 12 up, 12 in 2024-06-27T21:25:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/345050599' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:25:01.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:01.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:01.104 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:00 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:01.104 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3232423313' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:25:01.104 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4109934744' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:25:01.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:00 smithi151 ceph-mon[28965]: osdmap e388: 12 total, 12 up, 12 in 2024-06-27T21:25:01.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/345050599' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:25:01.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:01.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:01.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:01.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:01.385 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-06-27T21:25:01.389 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:25:01.389 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:25:01.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:01.451 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:25:01.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:25:01.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_104167 2024-06-27T21:25:01.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:25:01.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:25:01.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:01.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 6 1 2024-06-27T21:25:01.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:01.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:01.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:01.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:01.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:01 smithi151 ceph-mon[28965]: pgmap v1015: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:01.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:01 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:25:01.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/345050599' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:25:01.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:01 smithi151 ceph-mon[28965]: osdmap e389: 12 total, 12 up, 12 in 2024-06-27T21:25:01.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:01.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:01.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:01.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:01.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:02.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:01 smithi190 ceph-mon[32247]: pgmap v1015: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:01 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:25:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/345050599' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:25:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:01 smithi190 ceph-mon[32247]: osdmap e389: 12 total, 12 up, 12 in 2024-06-27T21:25:02.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:01 smithi192 ceph-mon[32956]: pgmap v1015: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:02.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:01 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:25:02.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/345050599' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:25:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:01 smithi192 ceph-mon[32956]: osdmap e389: 12 total, 12 up, 12 in 2024-06-27T21:25:02.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:02.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:02.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:02.242 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:02.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:02.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:02.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 1806 1790 111.866 111.875 0.00163943 0.00836934 2024-06-27T21:25:02.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:02.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:02.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:02.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:02.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:02.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:02.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:02.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:02.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:02.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:03.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:03.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:03.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:03.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:03.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:03.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:03.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:03.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 3214 3198 99.9286 88 0.0646921 0.00946481 2024-06-27T21:25:03.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:03.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:03.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:03.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:03.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:03.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:03.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:03.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:03.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:03.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:04.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:03 smithi190 ceph-mon[32247]: pgmap v1017: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:03 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:25:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:03 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:25:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:03 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:25:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:03 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:25:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:03 smithi192 ceph-mon[32956]: pgmap v1017: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:03 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:25:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:03 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:25:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:03 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:25:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:03 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:25:04.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:04.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:04.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:03 smithi151 ceph-mon[28965]: pgmap v1017: 27 pgs: 27 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:25:04.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:03 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:25:04.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:03 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:25:04.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:03 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:25:04.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:03 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:25:04.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:04.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:04.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:04.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:04.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 15 4968 4953 103.179 109.688 0.00167639 0.00966993 2024-06-27T21:25:04.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:04.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:04.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:04.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:04.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:04.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:04.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:04.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:04.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:04.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:05.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:05.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:05.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:05.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:05.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:05.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:05.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:05.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 6593 6577 102.757 101.5 0.00248578 0.00972042 2024-06-27T21:25:05.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:05.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:05.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:05.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:05.701 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:25:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:25:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:25:05.702 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:25:05.702 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:25:05.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:05.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:05.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:05.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:05.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:06.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:06.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:05 smithi190 ceph-mon[32247]: pgmap v1018: 27 pgs: 27 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s wr, 142 op/s 2024-06-27T21:25:06.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:05 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:25:06.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:05 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:25:06.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:05 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:25:06.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:05 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:25:06.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:06.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:05 smithi192 ceph-mon[32956]: pgmap v1018: 27 pgs: 27 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s wr, 142 op/s 2024-06-27T21:25:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:05 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:25:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:05 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:25:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:05 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:25:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:05 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:25:06.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:06.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:05 smithi151 ceph-mon[28965]: pgmap v1018: 27 pgs: 27 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s wr, 142 op/s 2024-06-27T21:25:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:05 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:25:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:05 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:25:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:05 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:25:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:05 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:25:06.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:06.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:06.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:25:06.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:25:06.372 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:25:06.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:06.391 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:06.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:06.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:25:06.411 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:25:06.446 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:25:06.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:25:06.452 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 7921 7905 98.8043 83 0.00213814 0.0101108 2024-06-27T21:25:06.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:25:06.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:25:06.504 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:25:06.504 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:25:06.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:06.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:06.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:25:06.531 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:25:06.559 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:25:06.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:25:06.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:25:06.589 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:25:06.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:25:06.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:25:06.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:06.639 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:25:06.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:25:06.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:06.663 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:25:06.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:25:06.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:25:06.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:25:06.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:06.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/164044760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:06.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:06.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:06.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:06.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:06.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:07.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:07.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/164044760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:07.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:07.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/164044760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:07.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:07.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:07.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:07.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:07.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:07.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:25:07.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:07.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 9528 9512 99.0752 100.438 0.00508067 0.0100742 2024-06-27T21:25:07.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:07.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:07.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:07.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:07.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:07.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:07.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:07.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:07.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:07 smithi190 ceph-mon[32247]: pgmap v1019: 27 pgs: 27 active+clean; 310 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 679 op/s 2024-06-27T21:25:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:07 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:25:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:07 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:25:08.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:08.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:07 smithi192 ceph-mon[32956]: pgmap v1019: 27 pgs: 27 active+clean; 310 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 679 op/s 2024-06-27T21:25:08.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:07 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:25:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:07 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:25:08.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:08.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:07 smithi151 ceph-mon[28965]: pgmap v1019: 27 pgs: 27 active+clean; 310 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 679 op/s 2024-06-27T21:25:08.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:07 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:25:08.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:07 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:25:08.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:08.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:08.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:08.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:08.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:08.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:08.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 10919 10903 97.3404 86.9375 0.00463538 0.0102353 2024-06-27T21:25:08.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:08.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:08.615 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:08.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:08.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:08.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:08.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:08.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:08.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:08 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:25:08.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:08 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:25:08.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:08.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:08 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:25:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:08 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:25:09.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:09.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:08 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:25:09.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:08 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:25:09.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:09.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.190 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:09.190 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:09.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:09.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:09.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:09.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:09.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 12144 12128 94.7424 76.5625 0.00328786 0.010536 2024-06-27T21:25:09.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:09.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:09.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:09.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:09.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:09.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:09.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:09.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:09.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:09.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:09 smithi190 ceph-mon[32247]: pgmap v1020: 27 pgs: 27 active+clean; 400 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 772 op/s 2024-06-27T21:25:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:09 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:25:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:09 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:25:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:09 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:25:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:09 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:25:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:09 smithi192 ceph-mon[32956]: pgmap v1020: 27 pgs: 27 active+clean; 400 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 772 op/s 2024-06-27T21:25:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:09 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:25:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:09 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:25:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:09 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:25:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:09 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:25:10.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:10.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:09 smithi151 ceph-mon[28965]: pgmap v1020: 27 pgs: 27 active+clean; 400 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 772 op/s 2024-06-27T21:25:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:09 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:25:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:09 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:25:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:09 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:25:10.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:09 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:25:10.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:10.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:10.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:10.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:10.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:10.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:10.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 13518 13502 93.7564 85.875 0.00651736 0.0106462 2024-06-27T21:25:10.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:10.577 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:10.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:25:10.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:10.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:10.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:10.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:10.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:10.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:10 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:25:10.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:10 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:25:10.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:10.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:11.011 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:11.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:10 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:25:11.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:10 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:25:11.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:10 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:25:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:10 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:25:11.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:11.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:11.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:11.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:11.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:11.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:11.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:11.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 14917 14901 93.1238 87.4375 0.0125495 0.010722 2024-06-27T21:25:11.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:11.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:11.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:11.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:11.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:11.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:11.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:11.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:11.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:11.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:11.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:12.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:11 smithi192 ceph-mon[32956]: pgmap v1021: 27 pgs: 27 active+clean; 600 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-06-27T21:25:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:11 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:25:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:11 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:25:12.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:12.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:12.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:12.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:11 smithi151 ceph-mon[28965]: pgmap v1021: 27 pgs: 27 active+clean; 600 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-06-27T21:25:12.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:11 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:25:12.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:11 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:25:12.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:12.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:11 smithi190 ceph-mon[32247]: pgmap v1021: 27 pgs: 27 active+clean; 600 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s 2024-06-27T21:25:12.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:11 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:25:12.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:11 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:25:12.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:12.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:12.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:12.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 16421 16405 93.2028 94 0.00835458 0.0107172 2024-06-27T21:25:12.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:12.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:12.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:12.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:12.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:12.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:12.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:12.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:12.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:12.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:12.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:13.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:13.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:13.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:13.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:13.256 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:25:13.256 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:25:13.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:13.355 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:13.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 17758 17742 92.3988 83.5625 0.013078 0.0108141 2024-06-27T21:25:13.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:13.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:13.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:13.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:13.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:13.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:13.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:13.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:13.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:13 smithi151 ceph-mon[28965]: pgmap v1022: 27 pgs: 27 active+clean; 770 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:13.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:13.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:13.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:14.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:13 smithi192 ceph-mon[32956]: pgmap v1022: 27 pgs: 27 active+clean; 770 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:14.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:14.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:14.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:14.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:14.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:13 smithi190 ceph-mon[32247]: pgmap v1022: 27 pgs: 27 active+clean; 770 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:14.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:14.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:14.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 19098 19082 91.7327 83.75 0.0247533 0.0108732 2024-06-27T21:25:14.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:14.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:14.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:14.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:14.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:14.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:14.772 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:25:14.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:14.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:14.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:14.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:15.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:15.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3906986567' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:25:15.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:15.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:15.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3906986567' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:25:15.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:25:15.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:15.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:15.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3906986567' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:25:15.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:15.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:15.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 20380 20364 90.903 80.125 0.00165788 0.0109855 2024-06-27T21:25:15.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:15.508 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:15.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:15.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:15.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:15.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:15.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:15.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:15.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:15.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:16.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:16.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: pgmap v1023: 27 pgs: 27 active+clean; 901 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:25:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:25:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:25:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3906986567' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:25:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: osdmap e390: 12 total, 12 up, 12 in 2024-06-27T21:25:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:25:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: pgmap v1023: 27 pgs: 27 active+clean; 901 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3906986567' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: osdmap e390: 12 total, 12 up, 12 in 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:25:16.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:25:16.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:16.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:16.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: pgmap v1023: 27 pgs: 27 active+clean; 901 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3906986567' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: osdmap e390: 12 total, 12 up, 12 in 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:25:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:25:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:16.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:16.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:16.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:16.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 21287 21271 88.6217 56.6875 0.0522028 0.0112704 2024-06-27T21:25:16.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:16.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:16.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:16.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:16.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:16.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:16.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:16.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:16.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:16.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:16 smithi151 ceph-mon[28965]: osdmap e391: 12 total, 12 up, 12 in 2024-06-27T21:25:16.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:17.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:17.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:16 smithi192 ceph-mon[32956]: osdmap e391: 12 total, 12 up, 12 in 2024-06-27T21:25:17.086 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:17.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:17.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:17.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:16 smithi190 ceph-mon[32247]: osdmap e391: 12 total, 12 up, 12 in 2024-06-27T21:25:17.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:17.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:17.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:17.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:17.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 22636 22620 88.352 84.3125 0.0153297 0.0113021 2024-06-27T21:25:17.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:17.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:17.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:17.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:17.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:17.778 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-06-27T21:25:17.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:17.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:17.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:17.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:17.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:17 smithi151 ceph-mon[28965]: pgmap v1026: 27 pgs: 27 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-27T21:25:17.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:17 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2911897924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:25:18.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:17 smithi192 ceph-mon[32956]: pgmap v1026: 27 pgs: 27 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-27T21:25:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:17 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2911897924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:25:18.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:18.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:18.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:18.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:17 smithi190 ceph-mon[32247]: pgmap v1026: 27 pgs: 27 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-27T21:25:18.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:17 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2911897924' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:25:18.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:18.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:18.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:18.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:18.453 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 24002 23986 88.1765 85.375 0.0579284 0.0113034 2024-06-27T21:25:18.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:25:18.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:18.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:18.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:18.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:18.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:18.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:18.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:18.850 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:25:18.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:25:18.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:25:18.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:25:18.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:25:18.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:25:18.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:25:18.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:25:18.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:18.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:25:18.937 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:25:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:18.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:18.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:18.958 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:25:18.958 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:25:18.980 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:25:18.981 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:25:19.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:25:19.002 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:25:19.024 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:25:19.024 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:25:19.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:19.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2911897924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:25:19.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:18 smithi192 ceph-mon[32956]: osdmap e392: 12 total, 12 up, 12 in 2024-06-27T21:25:19.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:25:19.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:25:19.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2911897924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:25:19.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:18 smithi151 ceph-mon[28965]: osdmap e392: 12 total, 12 up, 12 in 2024-06-27T21:25:19.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:19.078 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:25:19.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:25:19.109 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:25:19.109 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:25:19.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:19.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:19.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2911897924' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:25:19.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:18 smithi190 ceph-mon[32247]: osdmap e392: 12 total, 12 up, 12 in 2024-06-27T21:25:19.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:19.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:19.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:19.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:19.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 15 25112 25097 87.1351 69.4375 0.00739738 0.0114637 2024-06-27T21:25:19.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:19.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 7 1 2024-06-27T21:25:19.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:19.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:19.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:19.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:19.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:19.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:19.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:19.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:19 smithi192 ceph-mon[32956]: pgmap v1028: 27 pgs: 27 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-27T21:25:20.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2849665153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:20.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:19 smithi151 ceph-mon[28965]: pgmap v1028: 27 pgs: 27 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-27T21:25:20.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2849665153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:20.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:20.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:20.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:20.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:20.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:19 smithi190 ceph-mon[32247]: pgmap v1028: 27 pgs: 27 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-06-27T21:25:20.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2849665153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:20.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:20.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:20.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:20.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 26362 26346 86.6572 78.0625 0.00244195 0.0115289 2024-06-27T21:25:20.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:20.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:20.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:20.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:20.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:20.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:20.823 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25615.0:26934 5.0 e392/382 5:03911812:::benchmark_data_smithi192_104167_object26933:head v 392'1695, mlcod=392'1692)", 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:20.822479+0000", 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00080697600000000005, 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083631000000000005, 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:25:20.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822479+0000", 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822479+0000", 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822481+0000", 2024-06-27T21:25:20.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.13e-06 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822605+0000", 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012454999999999999 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822637+0000", 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1952000000000002e-05 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822647+0000", 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8660000000000001e-06 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822696+0000", 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8952000000000002e-05 2024-06-27T21:25:20.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:20.822711+0000", 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4887e-05 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-06-27T21:25:20.828 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:25:20.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:20.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:20.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:20.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:21.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:21.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:21.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:21.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:21.283 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:21.283 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:21.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:21.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:21.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:21.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:25:21.455953+0000 min lat: 0.00102517 max lat: 0.278582 avg lat: 0.0114604 2024-06-27T21:25:21.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:25:21.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 27885 27869 87.0832 95.1875 0.0302754 0.0114604 2024-06-27T21:25:21.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:21.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:21.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:21.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:21.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:21.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:21.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:25:21.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:21.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:21.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:21.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:21.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:21.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:21 smithi151 ceph-mon[28965]: pgmap v1029: 27 pgs: 27 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.40k op/s 2024-06-27T21:25:22.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:22.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:22.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:22.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:22.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:21 smithi190 ceph-mon[32247]: pgmap v1029: 27 pgs: 27 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.40k op/s 2024-06-27T21:25:22.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:21 smithi192 ceph-mon[32956]: pgmap v1029: 27 pgs: 27 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.40k op/s 2024-06-27T21:25:22.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:22.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:22.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 15 28894 28879 85.942 63.125 0.00564104 0.0116291 2024-06-27T21:25:22.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:22.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:22.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:25:22.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:22.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:22.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:22.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:22.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:22.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:22.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:22.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:23.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:23.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:23.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:23.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:23.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:23.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:23.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:23.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 30432 30416 86.4017 96.0625 0.0187317 0.0115675 2024-06-27T21:25:23.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:23.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:23.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:23.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:23.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:23.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:23.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:23.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:23.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:23.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:24.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:24.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:23 smithi151 ceph-mon[28965]: pgmap v1030: 27 pgs: 27 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-27T21:25:24.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:24.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:24.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:24.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:23 smithi190 ceph-mon[32247]: pgmap v1030: 27 pgs: 27 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-27T21:25:24.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:24.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:23 smithi192 ceph-mon[32956]: pgmap v1030: 27 pgs: 27 active+clean; 1.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-27T21:25:24.338 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:24.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:24.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:24.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 31734 31718 86.1829 81.375 0.0103216 0.0115863 2024-06-27T21:25:24.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:24.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:24.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:24.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:24.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:24.798 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-06-27T21:25:24.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:24.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:24.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:24.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:25.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:25.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:25.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:25.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:25.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2885308500' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:25:25.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2885308500' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:25:25.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:25.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2885308500' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:25:25.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:25.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:25.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:25.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 16 32770 32754 85.2896 64.75 0.0287222 0.0117056 2024-06-27T21:25:25.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:25.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:25.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:25.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 8 1 2024-06-27T21:25:25.688 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:25.718 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:25.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:25.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:25.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:25.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:25.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:25.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:25 smithi151 ceph-mon[28965]: pgmap v1031: 27 pgs: 27 active+clean; 1.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:25:25.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2885308500' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:25:25.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:25 smithi151 ceph-mon[28965]: osdmap e393: 12 total, 12 up, 12 in 2024-06-27T21:25:25.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:25 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:25:25.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:25 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:25:26.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:26.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:26.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:26.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:25 smithi190 ceph-mon[32247]: pgmap v1031: 27 pgs: 27 active+clean; 1.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:25:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2885308500' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:25:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:25 smithi190 ceph-mon[32247]: osdmap e393: 12 total, 12 up, 12 in 2024-06-27T21:25:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:25 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:25:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:25 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:25:26.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:25 smithi192 ceph-mon[32956]: pgmap v1031: 27 pgs: 27 active+clean; 1.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:25:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2885308500' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:25:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:25 smithi192 ceph-mon[32956]: osdmap e393: 12 total, 12 up, 12 in 2024-06-27T21:25:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:25 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:25:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:25 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:25:26.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:26.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:26.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:26.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 33687 33671 84.1703 57.3125 0.00358224 0.0118757 2024-06-27T21:25:26.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:26.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:26.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:26.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:26.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:26.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:26.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:26.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:26.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:27.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:26 smithi151 ceph-mon[28965]: osdmap e394: 12 total, 12 up, 12 in 2024-06-27T21:25:27.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:27.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:27.195 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:26 smithi192 ceph-mon[32956]: osdmap e394: 12 total, 12 up, 12 in 2024-06-27T21:25:27.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:27.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:26 smithi190 ceph-mon[32247]: osdmap e394: 12 total, 12 up, 12 in 2024-06-27T21:25:27.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:27.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:27.346 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:27.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:27.454 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 34937 34921 83.9376 78.125 0.00373567 0.0119053 2024-06-27T21:25:27.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:27.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:27.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:27.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:27.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:27.819 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:25:27.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:27.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:27.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:27.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:27.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:28.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:28.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:28.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:27 smithi151 ceph-mon[28965]: pgmap v1034: 27 pgs: 27 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:25:28.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3253232484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:25:28.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:28.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:27 smithi190 ceph-mon[32247]: pgmap v1034: 27 pgs: 27 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:25:28.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3253232484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:25:28.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:27 smithi192 ceph-mon[32956]: pgmap v1034: 27 pgs: 27 active+clean; 1.9 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:25:28.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3253232484' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:25:28.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:28.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:28.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:28.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 16 36195 36179 83.7406 78.625 0.00218354 0.0119341 2024-06-27T21:25:28.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:28.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:28.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:28.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:28.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:28.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:28.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:28.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:28.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:28.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:28.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:29.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:28 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3253232484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:25:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:28 smithi151 ceph-mon[28965]: osdmap e395: 12 total, 12 up, 12 in 2024-06-27T21:25:29.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:29.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:29.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:28 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:29.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3253232484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:25:29.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:28 smithi190 ceph-mon[32247]: osdmap e395: 12 total, 12 up, 12 in 2024-06-27T21:25:29.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:28 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:29.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3253232484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:25:29.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:28 smithi192 ceph-mon[32956]: osdmap e395: 12 total, 12 up, 12 in 2024-06-27T21:25:29.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:29.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:29.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 37572 37556 83.8232 86.0625 0.00240974 0.0119182 2024-06-27T21:25:29.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:29.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:29.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:29.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:29.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:29.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:29.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:29.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:29.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:29.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:30.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:30.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:30.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:30.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:29 smithi151 ceph-mon[28965]: pgmap v1036: 27 pgs: 27 active+clean; 2.0 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:25:30.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:29 smithi151 ceph-mon[28965]: osdmap e396: 12 total, 12 up, 12 in 2024-06-27T21:25:30.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:30.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:29 smithi190 ceph-mon[32247]: pgmap v1036: 27 pgs: 27 active+clean; 2.0 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:25:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:29 smithi190 ceph-mon[32247]: osdmap e396: 12 total, 12 up, 12 in 2024-06-27T21:25:30.263 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:30.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:29 smithi192 ceph-mon[32956]: pgmap v1036: 27 pgs: 27 active+clean; 2.0 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:25:30.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:29 smithi192 ceph-mon[32956]: osdmap e396: 12 total, 12 up, 12 in 2024-06-27T21:25:30.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:30.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:30.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 38847 38831 83.6803 79.6875 0.00498689 0.0119302 2024-06-27T21:25:30.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:30.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:30.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:30.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:30.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:30.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:30.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:30.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:30.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:31.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:31.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:31.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:31.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:31.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:25:31.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:31.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:31.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:31.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:31.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 40375 40359 84.0741 95.5 0.0403145 0.0118851 2024-06-27T21:25:31.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:31.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:31.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:31.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:31.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:31.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:31.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:31.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:31.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:32.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:32.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:32.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:31 smithi151 ceph-mon[28965]: pgmap v1038: 27 pgs: 27 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-27T21:25:32.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:32.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:32.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:31 smithi190 ceph-mon[32247]: pgmap v1038: 27 pgs: 27 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-27T21:25:32.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:31 smithi192 ceph-mon[32956]: pgmap v1038: 27 pgs: 27 active+clean; 2.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-27T21:25:32.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:32.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:32.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:32.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:32.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 41725 41709 84.0836 84.375 0.00778265 0.0118859 2024-06-27T21:25:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:32.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:32.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:32.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:32.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:32.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:32.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:32.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:32.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:32.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:33.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:33.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:33.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:33.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:33.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:33.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:33.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:33.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 16 42948 42932 83.8444 76.4375 0.00881708 0.0119188 2024-06-27T21:25:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:33.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:33.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:33.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:33.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:33.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:33.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:33.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:33.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:33.957 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:25:33.957 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:25:33.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:33.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:33.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:25:33.988 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:25:34.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:25:34.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:25:34.041 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:25:34.042 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:25:34.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:25:34.072 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:25:34.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:34.105 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:25:34.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:25:34.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:33 smithi151 ceph-mon[28965]: pgmap v1039: 27 pgs: 27 active+clean; 2.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:25:34.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:34.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:25:34.131 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:25:34.157 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:25:34.157 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:25:34.182 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:25:34.182 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:25:34.185 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:25:34.186 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:25:34.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:25:34.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:25:34.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:34.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:25:34.229 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:25:34.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:34.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:25:34.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:25:34.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:33 smithi190 ceph-mon[32247]: pgmap v1039: 27 pgs: 27 active+clean; 2.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:25:34.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:33 smithi192 ceph-mon[32956]: pgmap v1039: 27 pgs: 27 active+clean; 2.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:25:34.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:34.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:34.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 15 44013 43998 83.3225 66.625 0.00658646 0.0119968 2024-06-27T21:25:34.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:34.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:34.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:34.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:34.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:34.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:34.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:34.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:34.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:34.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:35.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:35.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:35.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3362819235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:35.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:34 smithi151 ceph-mon[28965]: osdmap e397: 12 total, 12 up, 12 in 2024-06-27T21:25:35.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:35.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:35.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3362819235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:35.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:34 smithi190 ceph-mon[32247]: osdmap e397: 12 total, 12 up, 12 in 2024-06-27T21:25:35.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:35.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3362819235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:35.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:34 smithi192 ceph-mon[32956]: osdmap e397: 12 total, 12 up, 12 in 2024-06-27T21:25:35.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:35.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:35.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:35.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 45605 45589 83.7962 99.4375 0.0039224 0.0119278 2024-06-27T21:25:35.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:35.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:35.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:25:35.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:35.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:35.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:35.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:35.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:35.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:35.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:36.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:36.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:36.110 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:36.115 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:36.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:36.170 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:35 smithi151 ceph-mon[28965]: pgmap v1040: 27 pgs: 27 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:25:36.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:35 smithi190 ceph-mon[32247]: pgmap v1040: 27 pgs: 27 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:25:36.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:36.293 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:35 smithi192 ceph-mon[32956]: pgmap v1040: 27 pgs: 27 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:25:36.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:36.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:36.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:36.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 15 47014 46999 83.9197 88.125 0.0308079 0.0119068 2024-06-27T21:25:36.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:36.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:36.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:36.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:36.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:36.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:36.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:36.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:36.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:37.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:37.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:37.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:37.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:25:37.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:37.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:36 smithi151 ceph-mon[28965]: pgmap v1042: 27 pgs: 27 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:25:37.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:36 smithi190 ceph-mon[32247]: pgmap v1042: 27 pgs: 27 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:25:37.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:37.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:36 smithi192 ceph-mon[32956]: pgmap v1042: 27 pgs: 27 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:25:37.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:37.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:37.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:37.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 48293 48277 83.8072 79.875 0.00253456 0.0119241 2024-06-27T21:25:37.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:37.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:37.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:37.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:37.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:37.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:37.860 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-06-27T21:25:37.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:37.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:37.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:38.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:38.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:38.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:38.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/631589436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:25:38.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:38.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/631589436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:25:38.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/631589436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:25:38.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:38.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:38.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:38.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 49484 49468 83.5538 74.4375 0.00298563 0.0119624 2024-06-27T21:25:38.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:38.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:38.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:38.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:38.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:38.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:38.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:38.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:38.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:38.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:38.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:39.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:39.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:39.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/631589436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:25:39.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:38 smithi151 ceph-mon[28965]: osdmap e398: 12 total, 12 up, 12 in 2024-06-27T21:25:39.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:38 smithi151 ceph-mon[28965]: pgmap v1044: 27 pgs: 27 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:25:39.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:39.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/631589436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:25:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:38 smithi190 ceph-mon[32247]: osdmap e398: 12 total, 12 up, 12 in 2024-06-27T21:25:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:38 smithi190 ceph-mon[32247]: pgmap v1044: 27 pgs: 27 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:25:39.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/631589436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:25:39.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:38 smithi192 ceph-mon[32956]: osdmap e398: 12 total, 12 up, 12 in 2024-06-27T21:25:39.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:38 smithi192 ceph-mon[32956]: pgmap v1044: 27 pgs: 27 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:25:39.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:39.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:39.455 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 50674 50658 83.3121 74.375 0.0305142 0.0119669 2024-06-27T21:25:39.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:39.487 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:39.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:39.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:39.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:39.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:39.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:39.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:39.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:39.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:40.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:40.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:40.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:40.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:40.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:40.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:40.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:40.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:40.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 52041 52025 83.3664 85.4375 0.00839889 0.0119903 2024-06-27T21:25:40.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:40.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:40.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 9 1 2024-06-27T21:25:40.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:40.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:40.729 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:25:40.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:40.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:40.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:40.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:40.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:40.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:40 smithi151 ceph-mon[28965]: osdmap e399: 12 total, 12 up, 12 in 2024-06-27T21:25:40.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3570352403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:25:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:40 smithi190 ceph-mon[32247]: osdmap e399: 12 total, 12 up, 12 in 2024-06-27T21:25:41.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3570352403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:25:41.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:41.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:40 smithi192 ceph-mon[32956]: osdmap e399: 12 total, 12 up, 12 in 2024-06-27T21:25:41.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3570352403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:25:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:41.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:41.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:41.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:41.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:41.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:41.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:41.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:41.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:25:41.457596+0000 min lat: 0.00102517 max lat: 0.321278 avg lat: 0.0120074 2024-06-27T21:25:41.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:25:41.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 53296 53280 83.243 78.4375 0.0198809 0.0120074 2024-06-27T21:25:41.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:41.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:41.644 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:25:41.645 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:25:41.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:41.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:41.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:41.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:41.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:41.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:41 smithi151 ceph-mon[28965]: pgmap v1046: 27 pgs: 27 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-27T21:25:41.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:41 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:41.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3570352403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:25:41.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:41 smithi151 ceph-mon[28965]: osdmap e400: 12 total, 12 up, 12 in 2024-06-27T21:25:41.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:41.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:42.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:41 smithi190 ceph-mon[32247]: pgmap v1046: 27 pgs: 27 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-27T21:25:42.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:41 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:42.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3570352403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:25:42.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:41 smithi190 ceph-mon[32247]: osdmap e400: 12 total, 12 up, 12 in 2024-06-27T21:25:42.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:41 smithi192 ceph-mon[32956]: pgmap v1046: 27 pgs: 27 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-06-27T21:25:42.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:41 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:42.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3570352403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:25:42.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:41 smithi192 ceph-mon[32956]: osdmap e400: 12 total, 12 up, 12 in 2024-06-27T21:25:42.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:42.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:42.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:42.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:42.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:42.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:42.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:25:42.416 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:42.417 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:42.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:42.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 16 54552 54536 83.1272 78.5 0.00901898 0.012024 2024-06-27T21:25:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:42.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:42.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:42.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:42.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:42.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:42.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:42.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:42.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:42.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:42.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:43.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:43.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:43.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:43.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:43.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:43.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:43.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:43.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 55392 55376 82.3977 52.5 0.00276019 0.0121116 2024-06-27T21:25:43.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:43.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:25:43.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:43.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:43.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:43.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:43.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:43.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:43.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:43.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:43 smithi151 ceph-mon[28965]: pgmap v1048: 27 pgs: 27 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:43.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:43.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:44.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:43 smithi190 ceph-mon[32247]: pgmap v1048: 27 pgs: 27 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:44.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:43 smithi192 ceph-mon[32956]: pgmap v1048: 27 pgs: 27 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:44.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:44.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:44.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:44.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:44.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:44.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:44.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 15 56730 56715 82.4275 83.6875 0.00248269 0.0121251 2024-06-27T21:25:44.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:44.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:44.531 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:44.561 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:44.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:44.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:44.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:44.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:44.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:44.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:44.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:44.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:45.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:45.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:45.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:45.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:45.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:45.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:45.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 57789 57773 82.057 66.125 0.00232451 0.0121747 2024-06-27T21:25:45.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:45.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:45.570 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-06-27T21:25:45.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:45.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:45.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:45.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:45.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:25:45.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:25:45.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:45 smithi151 ceph-mon[28965]: pgmap v1049: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:25:45.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:45 smithi151 ceph-mon[28965]: osdmap e401: 12 total, 12 up, 12 in 2024-06-27T21:25:45.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3696697367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:25:45.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:45.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:46.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:46.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:25:46.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:25:46.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:45 smithi190 ceph-mon[32247]: pgmap v1049: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:25:46.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:45 smithi190 ceph-mon[32247]: osdmap e401: 12 total, 12 up, 12 in 2024-06-27T21:25:46.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3696697367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:25:46.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:46.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:25:46.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:25:46.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:45 smithi192 ceph-mon[32956]: pgmap v1049: 27 pgs: 27 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:25:46.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:45 smithi192 ceph-mon[32956]: osdmap e401: 12 total, 12 up, 12 in 2024-06-27T21:25:46.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3696697367' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:25:46.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:46.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:46.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:46.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:46.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:46.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:46.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 58955 58939 81.8528 72.875 0.00274571 0.0122091 2024-06-27T21:25:46.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:46.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:46.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:46.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:46.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3696697367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:25:46.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:46 smithi151 ceph-mon[28965]: osdmap e402: 12 total, 12 up, 12 in 2024-06-27T21:25:46.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.823 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:46.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:46.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:46.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:46.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:47.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3696697367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:25:47.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:46 smithi190 ceph-mon[32247]: osdmap e402: 12 total, 12 up, 12 in 2024-06-27T21:25:47.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 11 1 2024-06-27T21:25:47.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3696697367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:25:47.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:46 smithi192 ceph-mon[32956]: osdmap e402: 12 total, 12 up, 12 in 2024-06-27T21:25:47.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:47.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:47.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:47.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:47.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:47.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:47.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:47.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 60417 60401 82.0596 91.375 0.01083 0.0121816 2024-06-27T21:25:47.473 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:47.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:47.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:47.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:47.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:47.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:47.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:47.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:47 smithi151 ceph-mon[28965]: pgmap v1052: 27 pgs: 27 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:25:47.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:47.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:47.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:48.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:47 smithi190 ceph-mon[32247]: pgmap v1052: 27 pgs: 27 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:25:48.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:47 smithi192 ceph-mon[32956]: pgmap v1052: 27 pgs: 27 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:25:48.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:48.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:48.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:48.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:48.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:48.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:48.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 61519 61503 81.779 68.875 0.0398988 0.0122224 2024-06-27T21:25:48.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:48.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:48.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:48.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:48.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:48.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:48.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:48.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:48.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:48.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:49.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:49.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:49.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:49.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:49.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:49.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:49.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:49.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 15 62934 62919 81.9188 88.5 0.0107703 0.0122024 2024-06-27T21:25:49.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:49.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:49.528 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:25:49.528 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:25:49.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:25:49.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:25:49.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:25:49.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:25:49.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:25:49.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:25:49.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:49 smithi151 ceph-mon[28965]: pgmap v1053: 27 pgs: 27 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:49.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/84468946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:49.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:49.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:49.652 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:25:49.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:25:49.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:25:49.686 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:25:49.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:25:49.747 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:25:49.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:49.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:49.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:25:49.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:25:49.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:49.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:49.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:50.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:49 smithi190 ceph-mon[32247]: pgmap v1053: 27 pgs: 27 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/84468946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:50.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:25:50.028 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:25:50.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:49 smithi192 ceph-mon[32956]: pgmap v1053: 27 pgs: 27 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:25:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/84468946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:25:50.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:25:50.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:25:50.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:25:50.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:25:50.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:25:50.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:25:50.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:50.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:50.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:50.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:50.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:50.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:50.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 16 64364 64348 82.0696 89.3125 0.0139002 0.0121798 2024-06-27T21:25:50.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:50.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:50.591 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:25:50.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:50.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:50.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:50.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:50.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:50.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:50.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/300417810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:25:50.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:25:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:50.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/300417810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:25:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:25:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/300417810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:25:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:25:51.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:51.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:51.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:51.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:51.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:51.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:51.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:51.456 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 15 65798 65783 82.2218 89.6875 0.0327005 0.0121564 2024-06-27T21:25:51.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:51.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:51.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:51.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:51.728 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:51.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:51.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:51.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: pgmap v1054: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:25:51.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:51.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/300417810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:25:51.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: osdmap e403: 12 total, 12 up, 12 in 2024-06-27T21:25:51.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:25:51.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:25:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:51.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:51.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:51.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: pgmap v1054: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:25:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/300417810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:25:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: osdmap e403: 12 total, 12 up, 12 in 2024-06-27T21:25:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:25:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:25:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: pgmap v1054: 27 pgs: 27 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/300417810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: osdmap e403: 12 total, 12 up, 12 in 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:25:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:25:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:25:52.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:52.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:52.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:52.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:52.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:52.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:52.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 67059 67043 82.1536 78.75 0.00649832 0.0121668 2024-06-27T21:25:52.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:52.483 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:52.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:52.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:52.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:52.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:52 smithi151 ceph-mon[28965]: osdmap e404: 12 total, 12 up, 12 in 2024-06-27T21:25:52.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:52.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:52.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:52.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:52.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:52.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:52.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:52 smithi190 ceph-mon[32247]: osdmap e404: 12 total, 12 up, 12 in 2024-06-27T21:25:53.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:52 smithi192 ceph-mon[32956]: osdmap e404: 12 total, 12 up, 12 in 2024-06-27T21:25:53.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:53.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:53.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:25:53.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:25:53.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:53.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:53.446 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:25:53.446 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:25:53.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.446 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:68142 5.a 5:53bdd156:::benchmark_data_smithi192_104167_object68141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-06-27T21:25:53.447 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:53.415365+0000", 2024-06-27T21:25:53.447 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.029459909999999999, 2024-06-27T21:25:53.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.029492343000000001, 2024-06-27T21:25:53.447 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:25:53.447 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:25:53.448 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:25:53.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:25:53.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:25:53.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:25:53.448 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 68142 2024-06-27T21:25:53.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.449 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:25:53.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:25:53.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.415365+0000", 2024-06-27T21:25:53.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:53.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:25:53.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.415365+0000", 2024-06-27T21:25:53.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:25:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.415366+0000", 2024-06-27T21:25:53.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6899999999999996e-07 2024-06-27T21:25:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:25:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.415441+0000", 2024-06-27T21:25:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5110000000000004e-05 2024-06-27T21:25:53.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:25:53.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.415450+0000", 2024-06-27T21:25:53.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3910000000000004e-06 2024-06-27T21:25:53.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:25:53.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.415457+0000", 2024-06-27T21:25:53.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7109999999999998e-06 2024-06-27T21:25:53.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:25:53.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.417980+0000", 2024-06-27T21:25:53.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025229229999999998 2024-06-27T21:25:53.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:25:53.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.418122+0000", 2024-06-27T21:25:53.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014203299999999999 2024-06-27T21:25:53.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,10", 2024-06-27T21:25:53.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.418288+0000", 2024-06-27T21:25:53.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016631999999999999 2024-06-27T21:25:53.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:25:53.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.420250+0000", 2024-06-27T21:25:53.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019619569999999999 2024-06-27T21:25:53.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.427343+0000", 2024-06-27T21:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070928199999999997 2024-06-27T21:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:68166 5.6 5:66d54a2d:::benchmark_data_smithi192_104167_object68165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-06-27T21:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:53.437714+0000", 2024-06-27T21:25:53.461 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0071109260000000001, 2024-06-27T21:25:53.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.007228979, 2024-06-27T21:25:53.461 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:25:53.461 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:25:53.461 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:25:53.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:25:53.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:25:53.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:25:53.462 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 68166 2024-06-27T21:25:53.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.462 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:25:53.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:25:53.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437714+0000", 2024-06-27T21:25:53.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:53.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:25:53.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437714+0000", 2024-06-27T21:25:53.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:53.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:25:53.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437716+0000", 2024-06-27T21:25:53.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8509999999999999e-06 2024-06-27T21:25:53.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:25:53.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437751+0000", 2024-06-27T21:25:53.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4647000000000002e-05 2024-06-27T21:25:53.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:25:53.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437773+0000", 2024-06-27T21:25:53.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2561999999999998e-05 2024-06-27T21:25:53.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:25:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437783+0000", 2024-06-27T21:25:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9070000000000002e-06 2024-06-27T21:25:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:25:53.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.437854+0000", 2024-06-27T21:25:53.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0862e-05 2024-06-27T21:25:53.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:25:53.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.438029+0000", 2024-06-27T21:25:53.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175203 2024-06-27T21:25:53.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:25:53.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.438200+0000", 2024-06-27T21:25:53.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017048399999999999 2024-06-27T21:25:53.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:25:53.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.440263+0000", 2024-06-27T21:25:53.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020638169999999999 2024-06-27T21:25:53.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:25:53.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.441253+0000", 2024-06-27T21:25:53.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00098957899999999998 2024-06-27T21:25:53.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:25:53.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.473 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25615.0:68172 5.4 e405/382 5:25b56385:::benchmark_data_smithi192_104167_object68171:head v 405'4324, mlcod=405'4321)", 2024-06-27T21:25:53.474 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:53.443725+0000", 2024-06-27T21:25:53.474 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001100041, 2024-06-27T21:25:53.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001287393, 2024-06-27T21:25:53.474 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:25:53.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:25:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:25:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:25:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:25:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443725+0000", 2024-06-27T21:25:53.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:53.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:25:53.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443725+0000", 2024-06-27T21:25:53.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:25:53.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:25:53.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443726+0000", 2024-06-27T21:25:53.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.342e-06 2024-06-27T21:25:53.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:25:53.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443810+0000", 2024-06-27T21:25:53.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3409000000000001e-05 2024-06-27T21:25:53.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:25:53.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443836+0000", 2024-06-27T21:25:53.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6058e-05 2024-06-27T21:25:53.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:25:53.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443844+0000", 2024-06-27T21:25:53.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0320000000000003e-06 2024-06-27T21:25:53.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:25:53.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443905+0000", 2024-06-27T21:25:53.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1379999999999998e-05 2024-06-27T21:25:53.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:25:53.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:25:53.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:25:53.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:53.443938+0000", 2024-06-27T21:25:53.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2539000000000001e-05 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-06-27T21:25:53.483 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:25:53.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:53.485 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 68191 68175 81.9342 70.75 0.00549126 0.012199 2024-06-27T21:25:53.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:53.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:25:53.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:53.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:53.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:53 smithi151 ceph-mon[28965]: pgmap v1057: 27 pgs: 27 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:25:53.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:53 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:25:53.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:53 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:25:53.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:53 smithi151 ceph-mon[28965]: osdmap e405: 12 total, 12 up, 12 in 2024-06-27T21:25:53.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:53.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:53.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:53.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:53.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:54.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:53 smithi190 ceph-mon[32247]: pgmap v1057: 27 pgs: 27 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:25:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:53 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:25:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:53 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:25:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:53 smithi190 ceph-mon[32247]: osdmap e405: 12 total, 12 up, 12 in 2024-06-27T21:25:54.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:53 smithi192 ceph-mon[32956]: pgmap v1057: 27 pgs: 27 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:25:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:53 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:25:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:53 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:25:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:53 smithi192 ceph-mon[32956]: osdmap e405: 12 total, 12 up, 12 in 2024-06-27T21:25:54.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:54.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:54.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:54.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:54.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:54.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:54.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:25:54.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 69345 69329 81.749 72.125 0.0231805 0.012227 2024-06-27T21:25:54.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:54.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:54.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:54.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:54.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:25:54.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:54.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:54 smithi151 ceph-mon[28965]: 5.b scrub starts 2024-06-27T21:25:54.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:54 smithi151 ceph-mon[28965]: 5.b scrub ok 2024-06-27T21:25:54.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:54 smithi151 ceph-mon[28965]: osdmap e406: 12 total, 12 up, 12 in 2024-06-27T21:25:54.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:54.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:54.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:54.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:55.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:54 smithi190 ceph-mon[32247]: 5.b scrub starts 2024-06-27T21:25:55.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:54 smithi190 ceph-mon[32247]: 5.b scrub ok 2024-06-27T21:25:55.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:54 smithi190 ceph-mon[32247]: osdmap e406: 12 total, 12 up, 12 in 2024-06-27T21:25:55.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:55.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:54 smithi192 ceph-mon[32956]: 5.b scrub starts 2024-06-27T21:25:55.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:54 smithi192 ceph-mon[32956]: 5.b scrub ok 2024-06-27T21:25:55.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:54 smithi192 ceph-mon[32956]: osdmap e406: 12 total, 12 up, 12 in 2024-06-27T21:25:55.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:55.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:55.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:55.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:55.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:55.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:55.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 70645 70629 81.7396 81.25 0.0152075 0.0122294 2024-06-27T21:25:55.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:55.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:55.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:55.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:55.763 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:55.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:55.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:55.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:55 smithi151 ceph-mon[28965]: pgmap v1059: 27 pgs: 27 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:25:55.900 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:55 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:25:55.900 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:55 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:25:55.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:55.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:55.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:56.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:55 smithi190 ceph-mon[32247]: pgmap v1059: 27 pgs: 27 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:25:56.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:55 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:25:56.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:55 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:25:56.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:55 smithi192 ceph-mon[32956]: pgmap v1059: 27 pgs: 27 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:25:56.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:55 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:25:56.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:55 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:25:56.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:56.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:56.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:56.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:56.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:56.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:56.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:56.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 71719 71703 81.4738 67.125 0.00859457 0.012268 2024-06-27T21:25:56.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:56.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:56.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:56.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:56.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:56.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:25:56.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:25:56.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:56.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:56 smithi151 ceph-mon[28965]: 5.8 scrub starts 2024-06-27T21:25:56.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:56 smithi151 ceph-mon[28965]: 5.6 scrub starts 2024-06-27T21:25:56.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:56.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:57.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:56 smithi190 ceph-mon[32247]: 5.8 scrub starts 2024-06-27T21:25:57.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:56 smithi190 ceph-mon[32247]: 5.6 scrub starts 2024-06-27T21:25:57.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:56 smithi192 ceph-mon[32956]: 5.8 scrub starts 2024-06-27T21:25:57.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:56 smithi192 ceph-mon[32956]: 5.6 scrub starts 2024-06-27T21:25:57.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:57.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:57.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:57.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:57.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:57.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:57.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:25:57.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 72512 72496 80.9039 49.5625 0.16574 0.0123452 2024-06-27T21:25:57.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:25:57.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:25:57.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:25:57.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:57.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:57.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:57.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:57.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:57 smithi192 ceph-mon[32956]: pgmap v1061: 27 pgs: 27 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:25:57.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:57 smithi192 ceph-mon[32956]: 5.8 scrub ok 2024-06-27T21:25:57.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:57 smithi192 ceph-mon[32956]: 5.a scrub starts 2024-06-27T21:25:57.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:57 smithi151 ceph-mon[28965]: pgmap v1061: 27 pgs: 27 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:25:57.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:57 smithi151 ceph-mon[28965]: 5.8 scrub ok 2024-06-27T21:25:57.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:57 smithi151 ceph-mon[28965]: 5.a scrub starts 2024-06-27T21:25:57.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:57.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:57.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:57 smithi190 ceph-mon[32247]: pgmap v1061: 27 pgs: 27 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:25:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:57 smithi190 ceph-mon[32247]: 5.8 scrub ok 2024-06-27T21:25:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:57 smithi190 ceph-mon[32247]: 5.a scrub starts 2024-06-27T21:25:58.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:58.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:58.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:58.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:58.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:58.358 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:58.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 73664 73648 80.7477 72 0.0035357 0.0123796 2024-06-27T21:25:58.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:25:58.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:25:58.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:25:58.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:25:58.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:58.725 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:25:58.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:58.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:58.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:58 smithi151 ceph-mon[28965]: 5.6 scrub ok 2024-06-27T21:25:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:58 smithi151 ceph-mon[28965]: 5.a scrub ok 2024-06-27T21:25:58.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:25:58.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:25:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:58.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:25:59.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:58 smithi190 ceph-mon[32247]: 5.6 scrub ok 2024-06-27T21:25:59.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:58 smithi190 ceph-mon[32247]: 5.a scrub ok 2024-06-27T21:25:59.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:58 smithi192 ceph-mon[32956]: 5.6 scrub ok 2024-06-27T21:25:59.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:58 smithi192 ceph-mon[32956]: 5.a scrub ok 2024-06-27T21:25:59.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:25:59.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:25:59.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:59.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:59.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:59.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:59.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 74615 74599 80.3802 59.4375 0.0164473 0.0124333 2024-06-27T21:25:59.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:25:59.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:25:59.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:25:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:25:59.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:25:59.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:25:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:59.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:25:59.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:25:59.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:25:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:25:59.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:25:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:59 smithi151 ceph-mon[28965]: pgmap v1062: 27 pgs: 27 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:25:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:25:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:25:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:59 smithi151 ceph-mon[28965]: osdmap e407: 12 total, 12 up, 12 in 2024-06-27T21:25:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:25:59 smithi151 ceph-mon[28965]: osdmap e408: 12 total, 12 up, 12 in 2024-06-27T21:25:59.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:59 smithi190 ceph-mon[32247]: pgmap v1062: 27 pgs: 27 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:26:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:59 smithi190 ceph-mon[32247]: osdmap e407: 12 total, 12 up, 12 in 2024-06-27T21:26:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:25:59 smithi190 ceph-mon[32247]: osdmap e408: 12 total, 12 up, 12 in 2024-06-27T21:26:00.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:59 smithi192 ceph-mon[32956]: pgmap v1062: 27 pgs: 27 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:26:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:59 smithi192 ceph-mon[32956]: osdmap e407: 12 total, 12 up, 12 in 2024-06-27T21:26:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:25:59 smithi192 ceph-mon[32956]: osdmap e408: 12 total, 12 up, 12 in 2024-06-27T21:26:00.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:00.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:00.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:00.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:00.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:00.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:00.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 75855 75839 80.3313 77.5 0.020353 0.0124433 2024-06-27T21:26:00.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:00.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:00.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:00.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:00.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:00.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:00.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:00.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:00.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:00.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:00 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:00.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:00 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:00.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:00 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:00.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:00 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:00 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:00 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:00 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:01.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:00 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:01.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:00 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:00 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:00 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:00 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:01.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:01.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:01.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:01.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:01.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:01.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:01.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:26:01.459225+0000 min lat: 0.00102517 max lat: 0.321278 avg lat: 0.0124729 2024-06-27T21:26:01.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:26:01.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 76954 76938 80.1371 68.6875 0.00924814 0.0124729 2024-06-27T21:26:01.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:01.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:01.617 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:26:01.617 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:26:01.649 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:26:01.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:26:01.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:01.683 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:26:01.683 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:26:01.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:01.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:26:01.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:26:01.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:01 smithi151 ceph-mon[28965]: pgmap v1065: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-06-27T21:26:01.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:01 smithi151 ceph-mon[28965]: osdmap e409: 12 total, 12 up, 12 in 2024-06-27T21:26:01.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:01 smithi151 ceph-mon[28965]: 5.f scrub starts 2024-06-27T21:26:01.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4140199928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:01.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:26:01.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:26:01.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:01.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:01.835 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:26:01.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:26:01.859 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:26:01.860 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:26:01.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:26:01.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:26:01.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:26:01.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:26:01.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:01.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:26:01.948 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:26:01.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:01.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:01.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:26:01.979 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:26:02.012 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:26:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:26:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:01 smithi190 ceph-mon[32247]: pgmap v1065: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-06-27T21:26:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:01 smithi190 ceph-mon[32247]: osdmap e409: 12 total, 12 up, 12 in 2024-06-27T21:26:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:01 smithi190 ceph-mon[32247]: 5.f scrub starts 2024-06-27T21:26:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4140199928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:01 smithi192 ceph-mon[32956]: pgmap v1065: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-06-27T21:26:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:01 smithi192 ceph-mon[32956]: osdmap e409: 12 total, 12 up, 12 in 2024-06-27T21:26:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:01 smithi192 ceph-mon[32956]: 5.f scrub starts 2024-06-27T21:26:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4140199928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:02.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:02.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:02.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:02.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:02.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:02.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:02.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:02.457 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 78138 78122 80.0363 74 0.00355499 0.01248 2024-06-27T21:26:02.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:02.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:02.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:02.639 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-06-27T21:26:02.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:02.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:02.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:02.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:02.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2688901297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:26:02.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2688901297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:26:02.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:02.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:02.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:03.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2688901297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:26:03.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:03.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:03.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:03.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:03.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:03.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:03.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 79212 79196 79.828 67.125 0.0029955 0.0125217 2024-06-27T21:26:03.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:03.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:03.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:03.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:03.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:03.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:03.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 10 1 2024-06-27T21:26:03.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:03.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:03.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:03 smithi151 ceph-mon[28965]: 5.4 scrub starts 2024-06-27T21:26:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:03 smithi151 ceph-mon[28965]: pgmap v1067: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s 2024-06-27T21:26:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:03 smithi151 ceph-mon[28965]: 5.f scrub ok 2024-06-27T21:26:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2688901297' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:26:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:03 smithi151 ceph-mon[28965]: osdmap e410: 12 total, 12 up, 12 in 2024-06-27T21:26:03.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:03 smithi190 ceph-mon[32247]: 5.4 scrub starts 2024-06-27T21:26:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:03 smithi190 ceph-mon[32247]: pgmap v1067: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s 2024-06-27T21:26:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:03 smithi190 ceph-mon[32247]: 5.f scrub ok 2024-06-27T21:26:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2688901297' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:26:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:03 smithi190 ceph-mon[32247]: osdmap e410: 12 total, 12 up, 12 in 2024-06-27T21:26:04.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:03 smithi192 ceph-mon[32956]: 5.4 scrub starts 2024-06-27T21:26:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:03 smithi192 ceph-mon[32956]: pgmap v1067: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s 2024-06-27T21:26:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:03 smithi192 ceph-mon[32956]: 5.f scrub ok 2024-06-27T21:26:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2688901297' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:26:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:03 smithi192 ceph-mon[32956]: osdmap e410: 12 total, 12 up, 12 in 2024-06-27T21:26:04.085 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:04.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:04.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:04.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:04.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:04.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:04.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:04.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 80462 80446 79.8008 78.125 0.0534368 0.012526 2024-06-27T21:26:04.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:04.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:04.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:04.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:04.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:04.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:04 smithi151 ceph-mon[28965]: 5.4 scrub ok 2024-06-27T21:26:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:04.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:04.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:04.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:04.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:05.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:04 smithi190 ceph-mon[32247]: 5.4 scrub ok 2024-06-27T21:26:05.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:04 smithi192 ceph-mon[32956]: 5.4 scrub ok 2024-06-27T21:26:05.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:05.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:05.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:05.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:05.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:05.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:05.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:05.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:05.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 81404 81388 79.4738 58.875 0.0867113 0.0125611 2024-06-27T21:26:05.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:05.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:05.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:05.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:05.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:05.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:05 smithi151 ceph-mon[28965]: pgmap v1069: 27 pgs: 27 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-06-27T21:26:05.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:05.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:05.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:05.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:06.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:05 smithi190 ceph-mon[32247]: pgmap v1069: 27 pgs: 27 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-06-27T21:26:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:05 smithi192 ceph-mon[32956]: pgmap v1069: 27 pgs: 27 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s 2024-06-27T21:26:06.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:06.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:06.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:26:06.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:06.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:06.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:06.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:06.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 82481 82465 79.2866 67.3125 0.0111071 0.0126073 2024-06-27T21:26:06.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:06.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:06.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:06.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:06.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:06.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:06.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:06.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:06.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:06.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:07.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:07.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:06 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:07.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:06 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:07.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:06 smithi190 ceph-mon[32247]: osdmap e411: 12 total, 12 up, 12 in 2024-06-27T21:26:07.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:06 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:07.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:06 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:07.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:06 smithi192 ceph-mon[32956]: osdmap e411: 12 total, 12 up, 12 in 2024-06-27T21:26:07.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:06 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:07.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:06 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:07.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:06 smithi151 ceph-mon[28965]: osdmap e411: 12 total, 12 up, 12 in 2024-06-27T21:26:07.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:07.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:07.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:07.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:07.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:07.313 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:26:07.313 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:26:07.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:07.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:07.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 83684 83668 79.2244 75.1875 0.0111028 0.0126176 2024-06-27T21:26:07.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:07.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:07.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:07.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:07.718 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:26:07.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:07.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:07.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:07.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:07.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: pgmap v1071: 27 pgs: 27 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: osdmap e412: 12 total, 12 up, 12 in 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/962251783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:07.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: pgmap v1071: 27 pgs: 27 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-27T21:26:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: osdmap e412: 12 total, 12 up, 12 in 2024-06-27T21:26:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/962251783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:26:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: pgmap v1071: 27 pgs: 27 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: osdmap e412: 12 total, 12 up, 12 in 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/962251783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:08.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:08.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:08.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:08.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:08.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:08.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:08.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:08.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 16 85032 85016 79.2993 84.25 0.0195291 0.0126056 2024-06-27T21:26:08.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:08.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:08.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:26:08.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:08.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.736 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:08.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:08.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:08.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:08.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:08 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:08 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/962251783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:08 smithi151 ceph-mon[28965]: osdmap e413: 12 total, 12 up, 12 in 2024-06-27T21:26:08.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:08.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:08.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:08 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:26:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:08 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:26:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/962251783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:26:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:08 smithi190 ceph-mon[32247]: osdmap e413: 12 total, 12 up, 12 in 2024-06-27T21:26:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:08 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:26:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:08 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:26:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/962251783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:26:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:08 smithi192 ceph-mon[32956]: osdmap e413: 12 total, 12 up, 12 in 2024-06-27T21:26:09.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:09.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:09.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:09.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:09.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:09.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:09.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:09.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 16 86317 86301 79.3141 80.3125 0.00686301 0.0126029 2024-06-27T21:26:09.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:09.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:09.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:09.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:09.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:09.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:09.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:09.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:09.894 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:09.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:09.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:09 smithi190 ceph-mon[32247]: pgmap v1074: 27 pgs: 27 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:26:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:09 smithi190 ceph-mon[32247]: 5.1 scrub starts 2024-06-27T21:26:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:09 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:26:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:09 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:26:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:10.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:10.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:09 smithi151 ceph-mon[28965]: pgmap v1074: 27 pgs: 27 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:26:10.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:09 smithi151 ceph-mon[28965]: 5.1 scrub starts 2024-06-27T21:26:10.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:09 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:26:10.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:09 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:26:10.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:10.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:09 smithi192 ceph-mon[32956]: pgmap v1074: 27 pgs: 27 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:09 smithi192 ceph-mon[32956]: 5.1 scrub starts 2024-06-27T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:09 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:09 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:10.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:10.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:10.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:10.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:10.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:10.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:10.456 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:10.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 87303 87287 79.0577 61.625 0.0114773 0.0126437 2024-06-27T21:26:10.500 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:10.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:10.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:10.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:10.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:10.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:10.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:10.855 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:10 smithi151 ceph-mon[28965]: 5.1 scrub ok 2024-06-27T21:26:10.855 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:10 smithi151 ceph-mon[28965]: 5.2 scrub starts 2024-06-27T21:26:10.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:10 smithi151 ceph-mon[28965]: osdmap e414: 12 total, 12 up, 12 in 2024-06-27T21:26:10.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:10 smithi151 ceph-mon[28965]: 5.9 scrub starts 2024-06-27T21:26:10.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:10.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:10.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:11.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:10 smithi190 ceph-mon[32247]: 5.1 scrub ok 2024-06-27T21:26:11.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:10 smithi190 ceph-mon[32247]: 5.2 scrub starts 2024-06-27T21:26:11.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:10 smithi190 ceph-mon[32247]: osdmap e414: 12 total, 12 up, 12 in 2024-06-27T21:26:11.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:10 smithi190 ceph-mon[32247]: 5.9 scrub starts 2024-06-27T21:26:11.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:10 smithi192 ceph-mon[32956]: 5.1 scrub ok 2024-06-27T21:26:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:10 smithi192 ceph-mon[32956]: 5.2 scrub starts 2024-06-27T21:26:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:10 smithi192 ceph-mon[32956]: osdmap e414: 12 total, 12 up, 12 in 2024-06-27T21:26:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:10 smithi192 ceph-mon[32956]: 5.9 scrub starts 2024-06-27T21:26:11.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:11.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:11.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:11.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:11.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:11.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:11.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:11.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 88506 88490 79.0024 75.1875 0.0111617 0.0126533 2024-06-27T21:26:11.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:11.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:11.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:11.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:11.766 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-06-27T21:26:11.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:11.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:11.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:11 smithi151 ceph-mon[28965]: pgmap v1076: 27 pgs: 27 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:26:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1697465966' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:26:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:11 smithi151 ceph-mon[28965]: 5.2 scrub ok 2024-06-27T21:26:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:11 smithi151 ceph-mon[28965]: 5.9 scrub ok 2024-06-27T21:26:11.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:26:11.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:11.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:12.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:11 smithi190 ceph-mon[32247]: pgmap v1076: 27 pgs: 27 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:26:12.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1697465966' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:26:12.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:11 smithi190 ceph-mon[32247]: 5.2 scrub ok 2024-06-27T21:26:12.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:11 smithi190 ceph-mon[32247]: 5.9 scrub ok 2024-06-27T21:26:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:11 smithi192 ceph-mon[32956]: pgmap v1076: 27 pgs: 27 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:26:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1697465966' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:26:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:11 smithi192 ceph-mon[32956]: 5.2 scrub ok 2024-06-27T21:26:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:11 smithi192 ceph-mon[32956]: 5.9 scrub ok 2024-06-27T21:26:12.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:12.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:12.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:12.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:12.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:12.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:12.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:12.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 89540 89524 78.7998 64.625 0.0177449 0.0126848 2024-06-27T21:26:12.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:12.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:12.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:12.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:12.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:12.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:12.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:12.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 2 1 2024-06-27T21:26:12.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:12.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:12.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:13.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1697465966' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:26:13.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:12 smithi192 ceph-mon[32956]: osdmap e415: 12 total, 12 up, 12 in 2024-06-27T21:26:13.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:12 smithi192 ceph-mon[32956]: 5.7 scrub starts 2024-06-27T21:26:13.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:13.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:13.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1697465966' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:26:13.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:12 smithi151 ceph-mon[28965]: osdmap e415: 12 total, 12 up, 12 in 2024-06-27T21:26:13.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:12 smithi151 ceph-mon[28965]: 5.7 scrub starts 2024-06-27T21:26:13.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:13.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1697465966' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:26:13.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:12 smithi190 ceph-mon[32247]: osdmap e415: 12 total, 12 up, 12 in 2024-06-27T21:26:13.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:12 smithi190 ceph-mon[32247]: 5.7 scrub starts 2024-06-27T21:26:13.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:13.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:13.458 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 90390 90374 78.4431 53.125 0.0304088 0.0127377 2024-06-27T21:26:13.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:13.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:13.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:13.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:13.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:13.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:13.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:13.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:13.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:13.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:13.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:14.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:13 smithi151 ceph-mon[28965]: pgmap v1078: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-27T21:26:14.029 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:13 smithi151 ceph-mon[28965]: 5.c deep-scrub starts 2024-06-27T21:26:14.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:13 smithi192 ceph-mon[32956]: pgmap v1078: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-27T21:26:14.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:13 smithi192 ceph-mon[32956]: 5.c deep-scrub starts 2024-06-27T21:26:14.067 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:26:14.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:26:14.100 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:26:14.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:26:14.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:14.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:14.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:26:14.131 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:26:14.162 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:26:14.162 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:26:14.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:13 smithi190 ceph-mon[32247]: pgmap v1078: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-27T21:26:14.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:13 smithi190 ceph-mon[32247]: 5.c deep-scrub starts 2024-06-27T21:26:14.195 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:26:14.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:26:14.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:26:14.222 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:26:14.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:14.252 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:26:14.252 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:26:14.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:14.280 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:26:14.280 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:26:14.310 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:26:14.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:26:14.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:26:14.336 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:26:14.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:14.363 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:26:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:26:14.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:14.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:26:14.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:26:14.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 91162 91146 78.0295 48.25 0.00334945 0.0128091 2024-06-27T21:26:14.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:14.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:14.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:14.623 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:14.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:14.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:14.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:14.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:14.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:14.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:14 smithi151 ceph-mon[28965]: 5.7 scrub ok 2024-06-27T21:26:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:14 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:26:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:14 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:26:14.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1998404800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:14.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:15.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:14 smithi192 ceph-mon[32956]: 5.7 scrub ok 2024-06-27T21:26:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:14 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:26:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:14 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:26:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1998404800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:15.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:15.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:15.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:15.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:14 smithi190 ceph-mon[32247]: 5.7 scrub ok 2024-06-27T21:26:15.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:14 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:26:15.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:14 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:26:15.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1998404800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:15.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:15.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:15.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:15.422 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:15.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 91966 91950 77.6541 50.25 0.100103 0.0128706 2024-06-27T21:26:15.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:15.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:15.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:15.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:15.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:15.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:15.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:15.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:15.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:15.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:15 smithi151 ceph-mon[28965]: pgmap v1079: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:26:15.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:15 smithi151 ceph-mon[28965]: 5.5 scrub starts 2024-06-27T21:26:16.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.15132.0:39557 3.5 e201/186 3:ae7c1835:::benchmark_data_smithi192_74314_object39556:head v 201'2491, mlcod=201'2487)", 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:16:45.036962+0000", 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 570.98106594800004, 2024-06-27T21:26:16.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18553656499999999, 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.036962+0000", 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.036962+0000", 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.036963+0000", 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.158e-06 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.037062+0000", 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8716999999999997e-05 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.221253+0000", 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18419126199999999 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.221259+0000", 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4640000000000003e-06 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.221311+0000", 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1267999999999998e-05 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.221327+0000", 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6594000000000001e-05 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.222453+0000", 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011258049999999999 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:16:45.222498+0000", 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5296999999999997e-05 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:121138 4.6 e342/335 4:6d6f429b:::benchmark_data_smithi192_88961_object30399:head v 342'7543, mlcod=342'7540)", 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:45.379583+0000", 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 210.63844442800001, 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24418741099999999, 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.379583+0000", 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.379583+0000", 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.379584+0000", 2024-06-27T21:26:16.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.119e-06 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.379613+0000", 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8572999999999999e-05 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.620546+0000", 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24093287299999999 2024-06-27T21:26:16.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.620554+0000", 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9820000000000008e-06 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.620639+0000", 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5321999999999994e-05 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.620666+0000", 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6619999999999999e-05 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.623725+0000", 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030595319999999998 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:45.623771+0000", 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5389999999999997e-05 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:123647 4.6 e343/335 4:6df1f70d:::benchmark_data_smithi192_88961_object32908:head v 343'7689, mlcod=343'7685)", 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:48.371525+0000", 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 207.64650327800001, 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17694386500000001, 2024-06-27T21:26:16.032 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.371525+0000", 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.371525+0000", 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.371525+0000", 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000005e-07 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.371531+0000", 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6930000000000004e-06 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.545817+0000", 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17428591299999999 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.545828+0000", 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1318000000000001e-05 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.545931+0000", 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102358 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.545951+0000", 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0071000000000001e-05 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.548422+0000", 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024714979999999999 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.548468+0000", 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6264999999999998e-05 2024-06-27T21:26:16.036 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:123650 4.6 e343/335 4:6c662c5e:::benchmark_data_smithi192_88961_object32911:head v 343'7690, mlcod=343'7686)", 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:48.380181+0000", 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 207.637847115, 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.223958349, 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.380181+0000", 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.380181+0000", 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.380181+0000", 2024-06-27T21:26:16.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0500000000000002e-07 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.380188+0000", 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0900000000000001e-06 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.596314+0000", 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21612638000000001 2024-06-27T21:26:16.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.596323+0000", 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1160000000000005e-06 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.596387+0000", 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3566999999999996e-05 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.596427+0000", 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0382000000000001e-05 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.604048+0000", 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076212850000000002 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:48.604139+0000", 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0723999999999994e-05 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:3189 5.a 5:570007c4:::benchmark_data_smithi192_104167_object3188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.333508+0000", 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.684519727999998, 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18376176299999999, 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 3189 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333508+0000", 2024-06-27T21:26:16.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333508+0000", 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333509+0000", 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6600000000000005e-07 2024-06-27T21:26:16.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333587+0000", 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8129999999999999e-05 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333598+0000", 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.064e-05 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333606+0000", 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8960000000000003e-06 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333646+0000", 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0059999999999999e-05 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333733+0000", 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7045000000000004e-05 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,10", 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.333837+0000", 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104478 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.334196+0000", 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035924900000000002 2024-06-27T21:26:16.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.336291+0000", 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020949559999999998 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517172+0000", 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.180880174 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517225+0000", 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3745000000000001e-05 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517270+0000", 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4524000000000003e-05 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:3192 5.a 5:5837920d:::benchmark_data_smithi192_104167_object3191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.335072+0000", 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.682956191000002, 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18485825, 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 3192 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.050 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335072+0000", 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335072+0000", 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335072+0000", 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0399999999999995e-07 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335124+0000", 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1360000000000003e-05 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335133+0000", 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5140000000000007e-06 2024-06-27T21:26:16.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335139+0000", 2024-06-27T21:26:16.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8250000000000001e-06 2024-06-27T21:26:16.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.054 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:15 smithi192 ceph-mon[32956]: pgmap v1079: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:26:16.054 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:15 smithi192 ceph-mon[32956]: 5.5 scrub starts 2024-06-27T21:26:16.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335173+0000", 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3905999999999997e-05 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335292+0000", 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119154 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,10", 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335451+0000", 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015898 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.335820+0000", 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036918399999999999 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.344219+0000", 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0083982620000000001 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.519831+0000", 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.175612139 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.519874+0000", 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3259e-05 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.519930+0000", 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5962999999999999e-05 2024-06-27T21:26:16.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:3189 5.a e389/382 ondisk, result = 0)", 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.336235+0000", 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.681792831999999, 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18104177399999999, 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.058 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.336235+0000", 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.336235+0000", 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.336236+0000", 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-07 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.336240+0000", 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1350000000000002e-06 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517049+0000", 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.180809413 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517059+0000", 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3400000000000004e-06 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517157+0000", 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8000999999999999e-05 2024-06-27T21:26:16.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517165+0000", 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9190000000000005e-06 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.517277+0000", 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011130599999999999 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:3207 5.6 5:66b001b7:::benchmark_data_smithi192_104167_object3206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.342159+0000", 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.675868962999999, 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20351281900000001, 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.063 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 3207 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342159+0000", 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342159+0000", 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342160+0000", 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3600000000000002e-07 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342209+0000", 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9444000000000003e-05 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342221+0000", 2024-06-27T21:26:16.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1307e-05 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342226+0000", 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.789e-06 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342270+0000", 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3668999999999998e-05 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342357+0000", 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7343999999999996e-05 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342461+0000", 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103724 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342806+0000", 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034485100000000002 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545526+0000", 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.202720025 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545576+0000", 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9938999999999997e-05 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545624+0000", 2024-06-27T21:26:16.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8349000000000001e-05 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545672+0000", 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7441999999999997e-05 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:3209 5.a 5:510c4346:::benchmark_data_smithi192_104167_object3208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-06-27T21:26:16.069 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.342824+0000", 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.675203361000001, 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.180211542, 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 3209 2024-06-27T21:26:16.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342824+0000", 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342824+0000", 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342825+0000", 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5300000000000004e-07 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342908+0000", 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3084000000000006e-05 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342917+0000", 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.439e-06 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342944+0000", 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7823000000000001e-05 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.342988+0000", 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3949999999999998e-05 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343126+0000", 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013776700000000001 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,10", 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343289+0000", 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000162468 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343677+0000", 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038885500000000002 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.519787+0000", 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17610932800000001 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.522963+0000", 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031763260000000001 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.523003+0000", 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0201999999999999e-05 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.523036+0000", 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2746999999999997e-05 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:3207 5.6 e389/382 ondisk, result = 0)", 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.343085+0000", 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.674942365000007, 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.202593626, 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343085+0000", 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343085+0000", 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343086+0000", 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6000000000000004e-07 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343093+0000", 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5919999999999997e-06 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545502+0000", 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20240955099999999 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545511+0000", 2024-06-27T21:26:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9309999999999997e-06 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545562+0000", 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1053000000000001e-05 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545572+0000", 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5210000000000008e-06 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545679+0000", 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010741799999999999 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:3207 5.6 e389/382 ondisk, result = 0)", 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:03.343155+0000", 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 72.674873294999998, 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.202388447, 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.081 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343155+0000", 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343155+0000", 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343155+0000", 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.63e-07 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.343158+0000", 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8009999999999999e-06 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545444+0000", 2024-06-27T21:26:16.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20228653999999999 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545452+0000", 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7570000000000004e-06 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545500+0000", 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7521999999999997e-05 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545517+0000", 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7578999999999999e-05 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:03.545543+0000", 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5785000000000001e-05 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:32173 5.6 5:62a6bfa6:::benchmark_data_smithi192_104167_object32172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e393)", 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:24.864269+0000", 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 51.153758709999998, 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21303149399999999, 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 32173 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.086 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864269+0000", 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864269+0000", 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864270+0000", 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.39e-07 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864296+0000", 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6883e-05 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864303+0000", 2024-06-27T21:26:16.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1600000000000003e-06 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864307+0000", 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5240000000000002e-06 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864341+0000", 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3596000000000002e-05 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864457+0000", 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011639800000000001 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.864560+0000", 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102459 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.865508+0000", 2024-06-27T21:26:16.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00094825200000000001 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.865856+0000", 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034807499999999999 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077210+0000", 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.211354028 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077255+0000", 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4712000000000002e-05 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077301+0000", 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5967999999999998e-05 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:32173 5.6 e393/382 ondisk, result = 0)", 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:24.865875+0000", 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 51.152152901999997, 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21143204600000001, 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.865875+0000", 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.865875+0000", 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.865875+0000", 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.77e-07 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:24.865878+0000", 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3479999999999998e-06 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077155+0000", 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.211276883 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077158+0000", 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9689999999999996e-06 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077199+0000", 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0210000000000003e-05 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077205+0000", 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7669999999999998e-06 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.077307+0000", 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101492 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:32753 5.6 5:654f1891:::benchmark_data_smithi192_104167_object32752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:25.417246+0000", 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.600782258999999, 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18615187, 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.097 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 32753 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417246+0000", 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417246+0000", 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417246+0000", 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4099999999999998e-07 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417283+0000", 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6377000000000001e-05 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417292+0000", 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5219999999999996e-06 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417298+0000", 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4720000000000003e-06 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417335+0000", 2024-06-27T21:26:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:16.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:16.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7718e-05 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417425+0000", 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0032999999999997e-05 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.417527+0000", 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101693 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418237+0000", 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070952400000000001 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603272+0000", 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18503519099999999 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603309+0000", 2024-06-27T21:26:16.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7054000000000001e-05 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603352+0000", 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.367e-05 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603397+0000", 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4975000000000002e-05 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:32753 5.6 e393/382 ondisk, result = 0)", 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:25.418505+0000", 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.599522946999997, 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.184772732, 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418505+0000", 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418505+0000", 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418505+0000", 2024-06-27T21:26:16.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6600000000000002e-07 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418507+0000", 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.261e-06 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.602797+0000", 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.184289279 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.602802+0000", 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4219999999999999e-06 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603260+0000", 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045799499999999999 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603266+0000", 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2090000000000002e-06 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603278+0000", 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1199999999999999e-05 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.110 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:32753 5.6 e393/382 ondisk, result = 0)", 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:25.418541+0000", 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.599486992999999, 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18486359899999999, 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418541+0000", 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418541+0000", 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418541+0000", 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4999999999999999e-07 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.418543+0000", 2024-06-27T21:26:16.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.545e-06 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.602816+0000", 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18427345000000001 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.602819+0000", 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0979999999999998e-06 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603295+0000", 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047590700000000002 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603302+0000", 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4969999999999998e-06 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:25.603404+0000", 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102852 2024-06-27T21:26:16.114 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:34790 5.6 5:600a8cee:::benchmark_data_smithi192_104167_object34789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:27.171147+0000", 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 48.846880556000002, 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.209233381, 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.115 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 34790 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171147+0000", 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171147+0000", 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171148+0000", 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6899999999999997e-07 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171206+0000", 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7773999999999999e-05 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171214+0000", 2024-06-27T21:26:16.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5480000000000002e-06 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171219+0000", 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6199999999999998e-06 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171249+0000", 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0399000000000002e-05 2024-06-27T21:26:16.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171331+0000", 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1279000000000006e-05 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.171427+0000", 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6335000000000005e-05 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.172075+0000", 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064787500000000001 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.172466+0000", 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039147500000000002 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380301+0000", 2024-06-27T21:26:16.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20783512800000001 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380344+0000", 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2786e-05 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380381+0000", 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6492999999999998e-05 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:34790 5.6 e394/382 ondisk, result = 0)", 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:27.172440+0000", 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 48.845587498999997, 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20794591500000001, 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.172440+0000", 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.172440+0000", 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.172441+0000", 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.29e-07 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.172442+0000", 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8640000000000001e-06 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380243+0000", 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.207800755 2024-06-27T21:26:16.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380250+0000", 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8889999999999996e-06 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380291+0000", 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0535999999999999e-05 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380297+0000", 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5119999999999997e-06 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:27.380386+0000", 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9129999999999995e-05 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.126 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:42965 5.a 5:5a6dbdaa:::benchmark_data_smithi192_104167_object42964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:33.473339+0000", 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.544688346000001, 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.199310868, 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 42965 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.473339+0000", 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.473339+0000", 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.473341+0000", 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9850000000000001e-06 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.473387+0000", 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5222000000000002e-05 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.541383+0000", 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.067996510999999996 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.541410+0000", 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6427e-05 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.541509+0000", 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9476999999999996e-05 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.541632+0000", 2024-06-27T21:26:16.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012306700000000001 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,10", 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.541764+0000", 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013176799999999999 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.542992+0000", 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00122836 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.543388+0000", 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000395664 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.672559+0000", 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12917065 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.672612+0000", 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3223999999999997e-05 2024-06-27T21:26:16.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:33.672650+0000", 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8513000000000002e-05 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:72448 5.6 5:6c840317:::benchmark_data_smithi192_104167_object72447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:57.223544+0000", 2024-06-27T21:26:16.133 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.794484214000001, 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16634939200000001, 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 72448 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:16.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223544+0000", 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223544+0000", 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:16.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223545+0000", 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.540000000000001e-07 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223591+0000", 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6564999999999999e-05 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223602+0000", 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1243999999999999e-05 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223608+0000", 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8300000000000001e-06 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223652+0000", 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4029000000000001e-05 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223770+0000", 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117916 2024-06-27T21:26:16.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.223880+0000", 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011017499999999999 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.225755+0000", 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018745280000000001 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.226201+0000", 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044627300000000001 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.389764+0000", 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16356246999999999 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.389833+0000", 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9332000000000006e-05 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:16.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:57.389893+0000", 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0075999999999999e-05 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:16.140 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:26:16.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:16.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:15 smithi190 ceph-mon[32247]: pgmap v1079: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:26:16.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:15 smithi190 ceph-mon[32247]: 5.5 scrub starts 2024-06-27T21:26:16.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:16.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:16.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 92814 92798 77.3253 53 0.0264111 0.0129268 2024-06-27T21:26:16.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:16.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:16.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:16.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:16.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:16.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:16.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:16.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:16.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:17.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:17.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:16 smithi151 ceph-mon[28965]: 5.e scrub starts 2024-06-27T21:26:17.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:16 smithi151 ceph-mon[28965]: 5.5 scrub ok 2024-06-27T21:26:17.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:16 smithi192 ceph-mon[32956]: 5.e scrub starts 2024-06-27T21:26:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:16 smithi192 ceph-mon[32956]: 5.5 scrub ok 2024-06-27T21:26:17.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:17.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:17.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:16 smithi190 ceph-mon[32247]: 5.e scrub starts 2024-06-27T21:26:17.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:16 smithi190 ceph-mon[32247]: 5.5 scrub ok 2024-06-27T21:26:17.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:17.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:17.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:26:17.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:17.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:17.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 93899 93883 77.2 67.8125 0.0181678 0.0129481 2024-06-27T21:26:17.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:17.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:17.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:17.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:17.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:17.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:17.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:17.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:17.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:18.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:17 smithi192 ceph-mon[32956]: pgmap v1080: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:26:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:17 smithi192 ceph-mon[32956]: 5.e scrub ok 2024-06-27T21:26:18.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:17 smithi151 ceph-mon[28965]: pgmap v1080: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:26:18.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:17 smithi151 ceph-mon[28965]: 5.e scrub ok 2024-06-27T21:26:18.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:26:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:18.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:18.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:18.202 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:18.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:17 smithi190 ceph-mon[32247]: pgmap v1080: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:26:18.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:17 smithi190 ceph-mon[32247]: 5.e scrub ok 2024-06-27T21:26:18.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:18.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:18.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:18.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:18.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 94746 94730 76.8849 52.9375 0.00537969 0.0129984 2024-06-27T21:26:18.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:18.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:18.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:18.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:18.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:18.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:18.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:18 smithi151 ceph-mon[28965]: 5.0 scrub starts 2024-06-27T21:26:18.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:18 smithi151 ceph-mon[28965]: 5.3 scrub starts 2024-06-27T21:26:18.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:18.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:18.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:19.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:19.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:19.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:19.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:19.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:18 smithi190 ceph-mon[32247]: 5.0 scrub starts 2024-06-27T21:26:19.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:18 smithi190 ceph-mon[32247]: 5.3 scrub starts 2024-06-27T21:26:19.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:18 smithi192 ceph-mon[32956]: 5.0 scrub starts 2024-06-27T21:26:19.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:18 smithi192 ceph-mon[32956]: 5.3 scrub starts 2024-06-27T21:26:19.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:19.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:19.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 95444 95428 76.4584 43.625 0.00388054 0.0130511 2024-06-27T21:26:19.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:19.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:19.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:19.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:19.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:19.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:19.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:19.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:19.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:19.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:20.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:20.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:20.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:19 smithi151 ceph-mon[28965]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s 2024-06-27T21:26:20.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:19 smithi151 ceph-mon[28965]: 5.d scrub starts 2024-06-27T21:26:20.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:19 smithi151 ceph-mon[28965]: 5.3 scrub ok 2024-06-27T21:26:20.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:19 smithi151 ceph-mon[28965]: 5.0 scrub ok 2024-06-27T21:26:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:20.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:20.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:20.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:20.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:19 smithi190 ceph-mon[32247]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s 2024-06-27T21:26:20.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:19 smithi190 ceph-mon[32247]: 5.d scrub starts 2024-06-27T21:26:20.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:19 smithi190 ceph-mon[32247]: 5.3 scrub ok 2024-06-27T21:26:20.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:19 smithi190 ceph-mon[32247]: 5.0 scrub ok 2024-06-27T21:26:20.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:19 smithi192 ceph-mon[32956]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s 2024-06-27T21:26:20.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:19 smithi192 ceph-mon[32956]: 5.d scrub starts 2024-06-27T21:26:20.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:19 smithi192 ceph-mon[32956]: 5.3 scrub ok 2024-06-27T21:26:20.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:19 smithi192 ceph-mon[32956]: 5.0 scrub ok 2024-06-27T21:26:20.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:20.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:20.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 96229 96213 76.1116 49.0625 0.0148107 0.0131322 2024-06-27T21:26:20.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:20.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:20.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:20.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:20.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:20.810 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-06-27T21:26:20.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:20.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:20.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:20 smithi151 ceph-mon[28965]: 5.d scrub ok 2024-06-27T21:26:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4284008351' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:26:20.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:20 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:26:21.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:21.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:21.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:21.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:21.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:20 smithi190 ceph-mon[32247]: 5.d scrub ok 2024-06-27T21:26:21.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4284008351' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:26:21.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:20 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:26:21.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:21.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:20 smithi192 ceph-mon[32956]: 5.d scrub ok 2024-06-27T21:26:21.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4284008351' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:26:21.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:20 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:26:21.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:21.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:21.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:21.450 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:21.451 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:21.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:26:21.460875+0000 min lat: 0.00102517 max lat: 0.349423 avg lat: 0.0131629 2024-06-27T21:26:21.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:26:21.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 97234 97218 75.9453 62.8125 0.0199771 0.0131629 2024-06-27T21:26:21.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:21.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:21.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:21.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:21.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:21.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:21.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:21.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:21.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:21 smithi151 ceph-mon[28965]: pgmap v1082: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 991 op/s 2024-06-27T21:26:21.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:21 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:26:21.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:21 smithi151 ceph-mon[28965]: osdmap e416: 12 total, 12 up, 12 in 2024-06-27T21:26:22.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:22.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:22.074 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-27T21:26:22.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:26:22.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:22.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:22.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:21 smithi190 ceph-mon[32247]: pgmap v1082: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 991 op/s 2024-06-27T21:26:22.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:21 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:26:22.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:21 smithi190 ceph-mon[32247]: osdmap e416: 12 total, 12 up, 12 in 2024-06-27T21:26:22.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:21 smithi192 ceph-mon[32956]: pgmap v1082: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 991 op/s 2024-06-27T21:26:22.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:21 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:26:22.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:21 smithi192 ceph-mon[32956]: osdmap e416: 12 total, 12 up, 12 in 2024-06-27T21:26:22.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:22.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:22.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:22.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 98111 98095 75.6843 54.8125 0.0064242 0.0132068 2024-06-27T21:26:22.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:22.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:22.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:22.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:22.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:22.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:22.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:22.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:22.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:22.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:23.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:23.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:23.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:23.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:23.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:23.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:23.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 98945 98929 75.3969 52.125 0.0138034 0.0132581 2024-06-27T21:26:23.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:23.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:23.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:23.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:23.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:23.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:23.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:23.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:23.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:23.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:24.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:24.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:24.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:24.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:24.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:23 smithi151 ceph-mon[28965]: pgmap v1084: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s 2024-06-27T21:26:24.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:23 smithi151 ceph-mon[28965]: 1.7 deep-scrub starts 2024-06-27T21:26:24.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:23 smithi151 ceph-mon[28965]: 1.7 deep-scrub ok 2024-06-27T21:26:24.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:24.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:23 smithi190 ceph-mon[32247]: pgmap v1084: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s 2024-06-27T21:26:24.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:23 smithi190 ceph-mon[32247]: 1.7 deep-scrub starts 2024-06-27T21:26:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:23 smithi190 ceph-mon[32247]: 1.7 deep-scrub ok 2024-06-27T21:26:24.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:24.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:23 smithi192 ceph-mon[32956]: pgmap v1084: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s 2024-06-27T21:26:24.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:23 smithi192 ceph-mon[32956]: 1.7 deep-scrub starts 2024-06-27T21:26:24.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:23 smithi192 ceph-mon[32956]: 1.7 deep-scrub ok 2024-06-27T21:26:24.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:24.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:24.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 100030 100014 75.3055 67.8125 0.00403892 0.0132732 2024-06-27T21:26:24.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:24.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:24.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:24.678 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:24.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:24.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:24.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:24.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:24.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:25.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:25.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:25.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:25.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:25.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:25.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:25.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:25.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:25.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 100866 100850 75.031 52.25 0.0313033 0.0133228 2024-06-27T21:26:25.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:25.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:25.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:25.708 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:25.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:25.812 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:26:25.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:25.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:25.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:25.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:26.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:26.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:25 smithi151 ceph-mon[28965]: pgmap v1085: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 882 op/s 2024-06-27T21:26:26.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1654172106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:26:26.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:26.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:26.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:25 smithi190 ceph-mon[32247]: pgmap v1085: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 882 op/s 2024-06-27T21:26:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1654172106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:26:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:25 smithi192 ceph-mon[32956]: pgmap v1085: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 882 op/s 2024-06-27T21:26:26.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1654172106' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:26:26.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:26.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:26.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:26.459 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 101726 101710 74.7806 53.75 0.0072374 0.0133665 2024-06-27T21:26:26.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:26.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:26.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:26.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:26.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:26.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:26.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:26.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:26:26.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:26:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:26:26.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:26:26.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:26.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:26.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:27.057 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:26:27.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:26:27.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:27.092 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:26:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:26:27.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:26 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:27.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1654172106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:26:27.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:26 smithi151 ceph-mon[28965]: osdmap e417: 12 total, 12 up, 12 in 2024-06-27T21:26:27.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1719645756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:27.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:27.125 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:26:27.126 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:26:27.153 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:26:27.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:26:27.181 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:26:27.181 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:26:27.194 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:26 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:27.194 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1654172106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:26:27.195 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:26 smithi192 ceph-mon[32956]: osdmap e417: 12 total, 12 up, 12 in 2024-06-27T21:26:27.195 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1719645756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:27.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:27.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:26 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1654172106' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:26:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:26 smithi190 ceph-mon[32247]: osdmap e417: 12 total, 12 up, 12 in 2024-06-27T21:26:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1719645756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:27.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:27.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:27.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 102594 102578 74.5418 54.25 0.004495 0.0134102 2024-06-27T21:26:27.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:27.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:27.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:27.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:27.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:27.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:27.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:27.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:27.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:27.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:28.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:28.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:28.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:28.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:27 smithi151 ceph-mon[28965]: pgmap v1087: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-27T21:26:28.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:28.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:27 smithi190 ceph-mon[32247]: pgmap v1087: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-27T21:26:28.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:28.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:27 smithi192 ceph-mon[32956]: pgmap v1087: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-27T21:26:28.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:28.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:28.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 103498 103482 74.3344 56.5 0.045409 0.0134444 2024-06-27T21:26:28.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:28.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:28.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:28.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:28.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:28.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:28.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:28.906 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:28.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:29.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:29.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:29.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:29.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:29.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:29.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:29.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:29.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:29.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 104452 104436 74.1671 59.625 0.0413937 0.013478 2024-06-27T21:26:29.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:29.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:29.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:29.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:29.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:29.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:29.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:29.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:29.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:30.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:30.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:30.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:30.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:30.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:29 smithi151 ceph-mon[28965]: pgmap v1088: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-06-27T21:26:30.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:30.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:29 smithi190 ceph-mon[32247]: pgmap v1088: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-06-27T21:26:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:30.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:30.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:29 smithi192 ceph-mon[32956]: pgmap v1088: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-06-27T21:26:30.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:30.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:30.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:30.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:30.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 16 105266 105250 73.9054 50.875 0.00423914 0.0135235 2024-06-27T21:26:30.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:30.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:30.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:30.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:30.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:30.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:30.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:30.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:31.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:30 smithi151 ceph-mon[28965]: osdmap e418: 12 total, 12 up, 12 in 2024-06-27T21:26:31.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:31.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:31.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:31.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:31.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:30 smithi190 ceph-mon[32247]: osdmap e418: 12 total, 12 up, 12 in 2024-06-27T21:26:31.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:30 smithi192 ceph-mon[32956]: osdmap e418: 12 total, 12 up, 12 in 2024-06-27T21:26:31.318 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:31.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:31.330 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:26:31.330 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:26:31.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:31.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:31.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 9 106038 106029 73.6251 48.6875 0.0103772 0.0135765 2024-06-27T21:26:31.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:31.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:31.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:26:31.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:31.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:31.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:31.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:31.911 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:31 smithi151 ceph-mon[28965]: pgmap v1090: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s 2024-06-27T21:26:31.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:31.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:31.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:32.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:32.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:32.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:31 smithi190 ceph-mon[32247]: pgmap v1090: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s 2024-06-27T21:26:32.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:32.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:32.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:31 smithi192 ceph-mon[32956]: pgmap v1090: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 904 op/s 2024-06-27T21:26:32.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:32.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:32.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:32.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.6378 2024-06-27T21:26:32.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 106038 2024-06-27T21:26:32.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:26:32.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:26:32.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 73.1193 2024-06-27T21:26:32.460 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 15.342 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 111.875 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 43.625 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1169 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 245.472 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 1790 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 698 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0136022 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0198115 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 0.671198 2024-06-27T21:26:32.461 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00102517 2024-06-27T21:26:32.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:32.467 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:26:32.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:32.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:32.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:32.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:32.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:26:32.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:32.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:32.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:32.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:32.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:33.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:33.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:33.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:26:33.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.257 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:33.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:33.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:33.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:33.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:33.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:33.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:33.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:33.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:33.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:33.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:33.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:33.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:34.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:34.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:34.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:34.163 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:26:34.163 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:26:34.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:33 smithi151 ceph-mon[28965]: pgmap v1091: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-06-27T21:26:34.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:34.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:33 smithi190 ceph-mon[32247]: pgmap v1091: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-06-27T21:26:34.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:34.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:33 smithi192 ceph-mon[32956]: pgmap v1091: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 942 op/s 2024-06-27T21:26:34.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:34.355 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:34.409 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:34.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:34.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:34.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:34.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:34.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:34.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:34.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:34.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:34.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:34.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:34.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:34.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:35.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:35.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:35.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:35.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:35.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:35.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:35.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:35.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:35.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:35.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:35.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:35.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:35.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:35.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:35.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:35.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:35.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:36.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:36.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:35 smithi151 ceph-mon[28965]: pgmap v1092: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 776 op/s 2024-06-27T21:26:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:35 smithi151 ceph-mon[28965]: from='client.16023 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:36.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:36.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:35 smithi190 ceph-mon[32247]: pgmap v1092: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 776 op/s 2024-06-27T21:26:36.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:35 smithi190 ceph-mon[32247]: from='client.16023 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:36.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:36.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:35 smithi192 ceph-mon[32956]: pgmap v1092: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 776 op/s 2024-06-27T21:26:36.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:35 smithi192 ceph-mon[32956]: from='client.16023 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:36.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:36.388 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:36.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:36.396 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1092,"stamp":"2024-06-27T21:26:34.025785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6889865760,"num_objects":105124,"num_object_clones":0,"num_object_copies":315372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105510,"num_write_kb":6740224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109303,"ondisk_log_size":109303,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":23702308,"kb_used_data":20153904,"kb_used_omap":110,"kb_used_meta":3548177,"kb_avail":1101239516,"statfs":{"total":1151940427776,"available":1127669264384,"internally_reserved":0,"allocated":20637597696,"data_stored":20634232956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":3633334231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":392101888,"num_objects":5983,"num_object_clones":0,"num_object_copies":17949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6371,"num_write_kb":395328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.209535"},"pg_stats":[{"pgid":"5.f","version":"418'6792","reported_seq":6934,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:32.733438+0000","last_change":"2024-06-27T21:26:02.177610+0000","last_active":"2024-06-27T21:26:32.733438+0000","last_peered":"2024-06-27T21:26:32.733438+0000","last_clean":"2024-06-27T21:26:32.733438+0000","last_became_active":"2024-06-27T21:24:48.707635+0000","last_became_peered":"2024-06-27T21:24:48.707635+0000","last_unstale":"2024-06-27T21:26:32.733438+0000","last_undegraded":"2024-06-27T21:26:32.733438+0000","last_fullsized":"2024-06-27T21:26:32.733438+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'4999","last_scrub_stamp":"2024-06-27T21:26:02.177486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:02.177486+0000","objects_scrubbed":4951,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:15.211969+0000","scrub_duration":1413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441843712,"num_objects":6742,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"418'6602","reported_seq":6735,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:30.225498+0000","last_change":"2024-06-27T21:25:54.256382+0000","last_active":"2024-06-27T21:26:30.225498+0000","last_peered":"2024-06-27T21:26:30.225498+0000","last_clean":"2024-06-27T21:26:30.225498+0000","last_became_active":"2024-06-27T21:24:48.708225+0000","last_became_peered":"2024-06-27T21:24:48.708225+0000","last_unstale":"2024-06-27T21:26:30.225498+0000","last_undegraded":"2024-06-27T21:26:30.225498+0000","last_fullsized":"2024-06-27T21:26:30.225498+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'4335","last_scrub_stamp":"2024-06-27T21:25:54.256168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:54.256168+0000","objects_scrubbed":4294,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:16.380086+0000","scrub_duration":869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":422528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"418'6723","reported_seq":6859,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:33.327757+0000","last_change":"2024-06-27T21:25:58.409829+0000","last_active":"2024-06-27T21:26:33.327757+0000","last_peered":"2024-06-27T21:26:33.327757+0000","last_clean":"2024-06-27T21:26:33.327757+0000","last_became_active":"2024-06-27T21:24:48.708150+0000","last_became_peered":"2024-06-27T21:24:48.708150+0000","last_unstale":"2024-06-27T21:26:33.327757+0000","last_undegraded":"2024-06-27T21:26:33.327757+0000","last_fullsized":"2024-06-27T21:26:33.327757+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4607","last_scrub_stamp":"2024-06-27T21:25:58.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:58.409408+0000","objects_scrubbed":4559,"log_size":6723,"log_dups_size":0,"ondisk_log_size":6723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:22.763094+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433520640,"num_objects":6615,"num_object_clones":0,"num_object_copies":19845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"417'6555","reported_seq":6697,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.356200+0000","last_change":"2024-06-27T21:26:11.531967+0000","last_active":"2024-06-27T21:26:29.356200+0000","last_peered":"2024-06-27T21:26:29.356200+0000","last_clean":"2024-06-27T21:26:29.356200+0000","last_became_active":"2024-06-27T21:24:48.708687+0000","last_became_peered":"2024-06-27T21:24:48.708687+0000","last_unstale":"2024-06-27T21:26:29.356200+0000","last_undegraded":"2024-06-27T21:26:29.356200+0000","last_fullsized":"2024-06-27T21:26:29.356200+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5559","last_scrub_stamp":"2024-06-27T21:26:11.531764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:11.531764+0000","objects_scrubbed":5513,"log_size":6555,"log_dups_size":0,"ondisk_log_size":6555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:26.815955+0000","scrub_duration":1548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429588480,"num_objects":6555,"num_object_clones":0,"num_object_copies":19665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6555,"num_write_kb":419520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"417'6534","reported_seq":6666,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.624398+0000","last_change":"2024-06-27T21:25:57.075916+0000","last_active":"2024-06-27T21:26:29.624398+0000","last_peered":"2024-06-27T21:26:29.624398+0000","last_clean":"2024-06-27T21:26:29.624398+0000","last_became_active":"2024-06-27T21:24:48.708111+0000","last_became_peered":"2024-06-27T21:24:48.708111+0000","last_unstale":"2024-06-27T21:26:29.624398+0000","last_undegraded":"2024-06-27T21:26:29.624398+0000","last_fullsized":"2024-06-27T21:26:29.624398+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4478","last_scrub_stamp":"2024-06-27T21:25:57.075739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:57.075739+0000","objects_scrubbed":4444,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:19.549115+0000","scrub_duration":1535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428212224,"num_objects":6534,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":418176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.4","version":"417'6572","reported_seq":6708,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.584405+0000","last_change":"2024-06-27T21:26:03.238316+0000","last_active":"2024-06-27T21:26:29.584405+0000","last_peered":"2024-06-27T21:26:29.584405+0000","last_clean":"2024-06-27T21:26:29.584405+0000","last_became_active":"2024-06-27T21:24:48.709413+0000","last_became_peered":"2024-06-27T21:24:48.709413+0000","last_unstale":"2024-06-27T21:26:29.584405+0000","last_undegraded":"2024-06-27T21:26:29.584405+0000","last_fullsized":"2024-06-27T21:26:29.584405+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'4970","last_scrub_stamp":"2024-06-27T21:26:03.238159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:03.238159+0000","objects_scrubbed":4924,"log_size":6572,"log_dups_size":0,"ondisk_log_size":6572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:15.739270+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430702592,"num_objects":6572,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":864,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.818210+0000","last_change":"2024-06-27T21:25:08.952316+0000","last_active":"2024-06-27T21:26:25.818210+0000","last_peered":"2024-06-27T21:26:25.818210+0000","last_clean":"2024-06-27T21:26:25.818210+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:26:25.818210+0000","last_undegraded":"2024-06-27T21:26:25.818210+0000","last_fullsized":"2024-06-27T21:26:25.818210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:08.952246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:08.952246+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:29.226596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"418'6585","reported_seq":6733,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:32.734257+0000","last_change":"2024-06-27T21:26:13.463329+0000","last_active":"2024-06-27T21:26:32.734257+0000","last_peered":"2024-06-27T21:26:32.734257+0000","last_clean":"2024-06-27T21:26:32.734257+0000","last_became_active":"2024-06-27T21:24:48.885413+0000","last_became_peered":"2024-06-27T21:24:48.885413+0000","last_unstale":"2024-06-27T21:26:32.734257+0000","last_undegraded":"2024-06-27T21:26:32.734257+0000","last_fullsized":"2024-06-27T21:26:32.734257+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5601","last_scrub_stamp":"2024-06-27T21:26:13.463234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:13.463234+0000","objects_scrubbed":5547,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:21.762213+0000","scrub_duration":1698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429064192,"num_objects":6547,"num_object_clones":0,"num_object_copies":19641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6585,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":461,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.852815+0000","last_change":"2024-06-27T21:25:08.804533+0000","last_active":"2024-06-27T21:26:29.852815+0000","last_peered":"2024-06-27T21:26:29.852815+0000","last_clean":"2024-06-27T21:26:29.852815+0000","last_became_active":"2024-06-27T21:23:44.524342+0000","last_became_peered":"2024-06-27T21:23:44.524342+0000","last_unstale":"2024-06-27T21:26:29.852815+0000","last_undegraded":"2024-06-27T21:26:29.852815+0000","last_fullsized":"2024-06-27T21:26:29.852815+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:08.804415+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:25:08.804415+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:33.348751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"418'6710","reported_seq":6846,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:33.332304+0000","last_change":"2024-06-27T21:25:58.105417+0000","last_active":"2024-06-27T21:26:33.332304+0000","last_peered":"2024-06-27T21:26:33.332304+0000","last_clean":"2024-06-27T21:26:33.332304+0000","last_became_active":"2024-06-27T21:24:48.710086+0000","last_became_peered":"2024-06-27T21:24:48.710086+0000","last_unstale":"2024-06-27T21:26:33.332304+0000","last_undegraded":"2024-06-27T21:26:33.332304+0000","last_fullsized":"2024-06-27T21:26:33.332304+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4594","last_scrub_stamp":"2024-06-27T21:25:58.105207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:58.105207+0000","objects_scrubbed":4550,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:09.223215+0000","scrub_duration":1746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431751168,"num_objects":6588,"num_object_clones":0,"num_object_copies":19764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":877,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.818272+0000","last_change":"2024-06-27T21:25:07.953798+0000","last_active":"2024-06-27T21:26:25.818272+0000","last_peered":"2024-06-27T21:26:25.818272+0000","last_clean":"2024-06-27T21:26:25.818272+0000","last_became_active":"2024-06-27T21:23:44.529886+0000","last_became_peered":"2024-06-27T21:23:44.529886+0000","last_unstale":"2024-06-27T21:26:25.818272+0000","last_undegraded":"2024-06-27T21:26:25.818272+0000","last_fullsized":"2024-06-27T21:26:25.818272+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:07.953724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:07.953724+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:34.698359+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"418'6475","reported_seq":6621,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:30.983007+0000","last_change":"2024-06-27T21:26:15.938541+0000","last_active":"2024-06-27T21:26:30.983007+0000","last_peered":"2024-06-27T21:26:30.983007+0000","last_clean":"2024-06-27T21:26:30.983007+0000","last_became_active":"2024-06-27T21:24:48.885372+0000","last_became_peered":"2024-06-27T21:24:48.885372+0000","last_unstale":"2024-06-27T21:26:30.983007+0000","last_undegraded":"2024-06-27T21:26:30.983007+0000","last_fullsized":"2024-06-27T21:26:30.983007+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5629","last_scrub_stamp":"2024-06-27T21:26:15.938322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:15.938322+0000","objects_scrubbed":5594,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:38.993790+0000","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424345600,"num_objects":6475,"num_object_clones":0,"num_object_copies":19425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6475,"num_write_kb":414400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":271,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.823572+0000","last_change":"2024-06-27T21:25:10.180661+0000","last_active":"2024-06-27T21:26:25.823572+0000","last_peered":"2024-06-27T21:26:25.823572+0000","last_clean":"2024-06-27T21:26:25.823572+0000","last_became_active":"2024-06-27T21:21:18.602559+0000","last_became_peered":"2024-06-27T21:21:18.602559+0000","last_unstale":"2024-06-27T21:26:25.823572+0000","last_undegraded":"2024-06-27T21:26:25.823572+0000","last_fullsized":"2024-06-27T21:26:25.823572+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:10.180545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:10.180545+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:34.511896+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"418'6540","reported_seq":6688,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:30.912767+0000","last_change":"2024-06-27T21:26:19.240429+0000","last_active":"2024-06-27T21:26:30.912767+0000","last_peered":"2024-06-27T21:26:30.912767+0000","last_clean":"2024-06-27T21:26:30.912767+0000","last_became_active":"2024-06-27T21:24:48.710822+0000","last_became_peered":"2024-06-27T21:24:48.710822+0000","last_unstale":"2024-06-27T21:26:30.912767+0000","last_undegraded":"2024-06-27T21:26:30.912767+0000","last_fullsized":"2024-06-27T21:26:30.912767+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5920","last_scrub_stamp":"2024-06-27T21:26:19.240187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.240187+0000","objects_scrubbed":5882,"log_size":6540,"log_dups_size":0,"ondisk_log_size":6540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:33.458729+0000","scrub_duration":1408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428605440,"num_objects":6540,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6540,"num_write_kb":418560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":608,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.852781+0000","last_change":"2024-06-27T21:26:22.770901+0000","last_active":"2024-06-27T21:26:29.852781+0000","last_peered":"2024-06-27T21:26:29.852781+0000","last_clean":"2024-06-27T21:26:29.852781+0000","last_became_active":"2024-06-27T21:21:18.602789+0000","last_became_peered":"2024-06-27T21:21:18.602789+0000","last_unstale":"2024-06-27T21:26:29.852781+0000","last_undegraded":"2024-06-27T21:26:29.852781+0000","last_fullsized":"2024-06-27T21:26:29.852781+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:26:22.770782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:31.213714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"418'6535","reported_seq":6679,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:30.548664+0000","last_change":"2024-06-27T21:26:11.195361+0000","last_active":"2024-06-27T21:26:30.548664+0000","last_peered":"2024-06-27T21:26:30.548664+0000","last_clean":"2024-06-27T21:26:30.548664+0000","last_became_active":"2024-06-27T21:24:48.708618+0000","last_became_peered":"2024-06-27T21:24:48.708618+0000","last_unstale":"2024-06-27T21:26:30.548664+0000","last_undegraded":"2024-06-27T21:26:30.548664+0000","last_fullsized":"2024-06-27T21:26:30.548664+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5451","last_scrub_stamp":"2024-06-27T21:26:11.195038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:11.195038+0000","objects_scrubbed":5384,"log_size":6535,"log_dups_size":0,"ondisk_log_size":6535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:29.760754+0000","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428277760,"num_objects":6535,"num_object_clones":0,"num_object_copies":19605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6535,"num_write_kb":418240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":271,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.823669+0000","last_change":"2024-06-27T21:25:10.874252+0000","last_active":"2024-06-27T21:26:25.823669+0000","last_peered":"2024-06-27T21:26:25.823669+0000","last_clean":"2024-06-27T21:26:25.823669+0000","last_became_active":"2024-06-27T21:21:18.602398+0000","last_became_peered":"2024-06-27T21:21:18.602398+0000","last_unstale":"2024-06-27T21:26:25.823669+0000","last_undegraded":"2024-06-27T21:26:25.823669+0000","last_fullsized":"2024-06-27T21:26:25.823669+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:10.874135+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:25:10.874135+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:40.653889+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"418'6602","reported_seq":6745,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:32.872820+0000","last_change":"2024-06-27T21:26:09.408114+0000","last_active":"2024-06-27T21:26:32.872820+0000","last_peered":"2024-06-27T21:26:32.872820+0000","last_clean":"2024-06-27T21:26:32.872820+0000","last_became_active":"2024-06-27T21:24:48.709002+0000","last_became_peered":"2024-06-27T21:24:48.709002+0000","last_unstale":"2024-06-27T21:26:32.872820+0000","last_undegraded":"2024-06-27T21:26:32.872820+0000","last_fullsized":"2024-06-27T21:26:32.872820+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'5325","last_scrub_stamp":"2024-06-27T21:26:09.407899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:09.407899+0000","objects_scrubbed":5258,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:36.464873+0000","scrub_duration":1191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428081152,"num_objects":6532,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":259,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.823623+0000","last_change":"2024-06-27T21:25:04.736172+0000","last_active":"2024-06-27T21:26:25.823623+0000","last_peered":"2024-06-27T21:26:25.823623+0000","last_clean":"2024-06-27T21:26:25.823623+0000","last_became_active":"2024-06-27T21:21:34.494291+0000","last_became_peered":"2024-06-27T21:21:34.494291+0000","last_unstale":"2024-06-27T21:26:25.823623+0000","last_undegraded":"2024-06-27T21:26:25.823623+0000","last_fullsized":"2024-06-27T21:26:25.823623+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:04.736074+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:25:04.736074+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"417'6426","reported_seq":6570,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.585874+0000","last_change":"2024-06-27T21:26:19.252916+0000","last_active":"2024-06-27T21:26:29.585874+0000","last_peered":"2024-06-27T21:26:29.585874+0000","last_clean":"2024-06-27T21:26:29.585874+0000","last_became_active":"2024-06-27T21:24:48.885324+0000","last_became_peered":"2024-06-27T21:24:48.885324+0000","last_unstale":"2024-06-27T21:26:29.585874+0000","last_undegraded":"2024-06-27T21:26:29.585874+0000","last_fullsized":"2024-06-27T21:26:29.585874+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5908","last_scrub_stamp":"2024-06-27T21:26:19.252716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.252716+0000","objects_scrubbed":5869,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:22.654032+0000","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421134336,"num_objects":6426,"num_object_clones":0,"num_object_copies":19278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":411264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":680,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.819270+0000","last_change":"2024-06-27T21:26:08.543233+0000","last_active":"2024-06-27T21:26:25.819270+0000","last_peered":"2024-06-27T21:26:25.819270+0000","last_clean":"2024-06-27T21:26:25.819270+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:26:25.819270+0000","last_undegraded":"2024-06-27T21:26:25.819270+0000","last_fullsized":"2024-06-27T21:26:25.819270+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:08.543015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:08.543015+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:23.153986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"418'6620","reported_seq":6769,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:30.151814+0000","last_change":"2024-06-27T21:26:16.853366+0000","last_active":"2024-06-27T21:26:30.151814+0000","last_peered":"2024-06-27T21:26:30.151814+0000","last_clean":"2024-06-27T21:26:30.151814+0000","last_became_active":"2024-06-27T21:24:48.707990+0000","last_became_peered":"2024-06-27T21:24:48.707990+0000","last_unstale":"2024-06-27T21:26:30.151814+0000","last_undegraded":"2024-06-27T21:26:30.151814+0000","last_fullsized":"2024-06-27T21:26:30.151814+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5897","last_scrub_stamp":"2024-06-27T21:26:16.853222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:16.853222+0000","objects_scrubbed":5847,"log_size":6620,"log_dups_size":0,"ondisk_log_size":6620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:34.301452+0000","scrub_duration":1472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433848320,"num_objects":6620,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6620,"num_write_kb":423680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":580,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.846937+0000","last_change":"2024-06-27T21:25:02.630434+0000","last_active":"2024-06-27T21:26:29.846937+0000","last_peered":"2024-06-27T21:26:29.846937+0000","last_clean":"2024-06-27T21:26:29.846937+0000","last_became_active":"2024-06-27T21:23:44.529707+0000","last_became_peered":"2024-06-27T21:23:44.529707+0000","last_unstale":"2024-06-27T21:26:29.846937+0000","last_undegraded":"2024-06-27T21:26:29.846937+0000","last_fullsized":"2024-06-27T21:26:29.846937+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:02.630318+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:25:02.630318+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:31.347552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"417'6498","reported_seq":6644,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.590793+0000","last_change":"2024-06-27T21:26:19.426445+0000","last_active":"2024-06-27T21:26:29.590793+0000","last_peered":"2024-06-27T21:26:29.590793+0000","last_clean":"2024-06-27T21:26:29.590793+0000","last_became_active":"2024-06-27T21:24:48.955695+0000","last_became_peered":"2024-06-27T21:24:48.955695+0000","last_unstale":"2024-06-27T21:26:29.590793+0000","last_undegraded":"2024-06-27T21:26:29.590793+0000","last_fullsized":"2024-06-27T21:26:29.590793+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5936","last_scrub_stamp":"2024-06-27T21:26:19.426355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.426355+0000","objects_scrubbed":5918,"log_size":6498,"log_dups_size":0,"ondisk_log_size":6498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:41.889854+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":415872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":888,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-06-27T21:26:25.818306+0000","last_change":"2024-06-27T21:26:14.033864+0000","last_active":"2024-06-27T21:26:25.818306+0000","last_peered":"2024-06-27T21:26:25.818306+0000","last_clean":"2024-06-27T21:26:25.818306+0000","last_became_active":"2024-06-27T21:21:34.485764+0000","last_became_peered":"2024-06-27T21:21:34.485764+0000","last_unstale":"2024-06-27T21:26:25.818306+0000","last_undegraded":"2024-06-27T21:26:25.818306+0000","last_fullsized":"2024-06-27T21:26:25.818306+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:14.033749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:14.033749+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:38.288649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"418'9522","reported_seq":9773,"reported_epoch":418,"state":"active+clean+scrubbing+deep","last_fresh":"2024-06-27T21:26:30.979016+0000","last_change":"2024-06-27T21:26:12.708076+0000","last_active":"2024-06-27T21:26:30.979016+0000","last_peered":"2024-06-27T21:26:30.979016+0000","last_clean":"2024-06-27T21:26:30.979016+0000","last_became_active":"2024-06-27T21:24:48.885320+0000","last_became_peered":"2024-06-27T21:24:48.885320+0000","last_unstale":"2024-06-27T21:26:30.979016+0000","last_undegraded":"2024-06-27T21:26:30.979016+0000","last_fullsized":"2024-06-27T21:26:30.979016+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:24:47.689585+0000","objects_scrubbed":4136,"log_size":9522,"log_dups_size":0,"ondisk_log_size":9522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441778176,"num_objects":6741,"num_object_clones":0,"num_object_copies":20223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6741,"num_write_kb":431424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":613,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-06-27T21:26:29.847873+0000","last_change":"2024-06-27T21:25:54.786735+0000","last_active":"2024-06-27T21:26:29.847873+0000","last_peered":"2024-06-27T21:26:29.847873+0000","last_clean":"2024-06-27T21:26:29.847873+0000","last_became_active":"2024-06-27T21:21:34.487122+0000","last_became_peered":"2024-06-27T21:21:34.487122+0000","last_unstale":"2024-06-27T21:26:29.847873+0000","last_undegraded":"2024-06-27T21:26:29.847873+0000","last_fullsized":"2024-06-27T21:26:29.847873+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:54.786656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:54.786656+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:12.608584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6889275392,"num_objects":105122,"num_object_clones":0,"num_object_copies":315366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105510,"num_write_kb":6740224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20668223488,"data_stored":20668219416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108291,"ondisk_log_size":108291,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2096404,"kb_used_data":1682060,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91648748,"statfs":{"total":95995035648,"available":93848317952,"internally_reserved":0,"allocated":1722429440,"data_stored":1722152182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514707,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2342804,"kb_used_data":2077004,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91402348,"statfs":{"total":95995035648,"available":93596004352,"internally_reserved":0,"allocated":2126852096,"data_stored":2126574838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743639,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1561300,"kb_used_data":1255244,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92183852,"statfs":{"total":95995035648,"available":94396264448,"internally_reserved":0,"allocated":1285369856,"data_stored":1285092598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972572,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058020,"kb_used_data":856088,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92687132,"statfs":{"total":95995035648,"available":94911623168,"internally_reserved":0,"allocated":876634112,"data_stored":876352356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2010900,"kb_used_data":1659020,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91734252,"statfs":{"total":95995035648,"available":93935874048,"internally_reserved":0,"allocated":1698836480,"data_stored":1698559222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528550,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690964,"kb_used_data":1250252,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":92054188,"statfs":{"total":95995035648,"available":94263488512,"internally_reserved":0,"allocated":1280258048,"data_stored":1279980790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":451279302},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757484,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3226600,"kb_used_data":2974688,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90518552,"statfs":{"total":95995035648,"available":92690997248,"internally_reserved":0,"allocated":3046080512,"data_stored":3045795204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161187,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2224484,"kb_used_data":2119064,"kb_used_omap":92,"kb_used_meta":105315,"kb_avail":91520668,"statfs":{"total":95995035648,"available":93717164032,"internally_reserved":0,"allocated":2169921536,"data_stored":2169639780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":107842658},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182644,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727276,"kb_used_data":421216,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93017876,"statfs":{"total":95995035648,"available":95250305024,"internally_reserved":0,"allocated":431325184,"data_stored":431039876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":313391559},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411577,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458408,"kb_used_data":1278688,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92286744,"statfs":{"total":95995035648,"available":94501625856,"internally_reserved":0,"allocated":1309376512,"data_stored":1309091204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575101,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2021016,"kb_used_data":1642764,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91724136,"statfs":{"total":95995035648,"available":93925515264,"internally_reserved":0,"allocated":1682190336,"data_stored":1681913078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738626,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3284132,"kb_used_data":2937816,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":90461020,"statfs":{"total":95995035648,"available":92632084480,"internally_reserved":0,"allocated":3008323584,"data_stored":3008041828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3028615168,"data_stored":3028615168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1694236672,"data_stored":1694236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1296433152,"data_stored":1296433152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":430374912,"data_stored":430374912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2159411200,"data_stored":2159411200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3036352512,"data_stored":3036348440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1288503296,"data_stored":1288503296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1706950656,"data_stored":1706950656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":875560960,"data_stored":875560960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1292566528,"data_stored":1292566528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2137391104,"data_stored":2137391104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1721827328,"data_stored":1721827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:26:36.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:26:36.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:36.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:36.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:36.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:36.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:36.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:36.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:36.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:36.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:36.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:37.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:37.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:37.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:37.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:37.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:37.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:37.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:37.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:37.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:37.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:37.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:37.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:37.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:37.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:37.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:37.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:38.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:38.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:38.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:37 smithi151 ceph-mon[28965]: pgmap v1093: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 38 MiB/s wr, 797 op/s 2024-06-27T21:26:38.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:38.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:38.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:37 smithi190 ceph-mon[32247]: pgmap v1093: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 38 MiB/s wr, 797 op/s 2024-06-27T21:26:38.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:38.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:37 smithi192 ceph-mon[32956]: pgmap v1093: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 38 MiB/s wr, 797 op/s 2024-06-27T21:26:38.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:38.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:38.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:38.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:38.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:38.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:38.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:38.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:38.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:38.863 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:26:38.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:38.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:38.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:39.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:39.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:39.130 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:39.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:39.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:39.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1094078550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:26:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1094078550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:26:39.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:39.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1094078550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:26:39.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:39.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:39.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:39.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:39.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:39.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:39.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:39.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:39.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:39.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:39 smithi151 ceph-mon[28965]: pgmap v1094: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 33 MiB/s wr, 796 op/s 2024-06-27T21:26:39.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:39 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:39.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1094078550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:26:39.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:39 smithi151 ceph-mon[28965]: osdmap e419: 12 total, 12 up, 12 in 2024-06-27T21:26:39.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:39.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:39.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:40.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:40.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:40.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:40.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:40.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:39 smithi190 ceph-mon[32247]: pgmap v1094: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 33 MiB/s wr, 796 op/s 2024-06-27T21:26:40.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:39 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:40.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1094078550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:26:40.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:39 smithi190 ceph-mon[32247]: osdmap e419: 12 total, 12 up, 12 in 2024-06-27T21:26:40.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:39 smithi192 ceph-mon[32956]: pgmap v1094: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 33 MiB/s wr, 796 op/s 2024-06-27T21:26:40.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:39 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:40.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1094078550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:26:40.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:39 smithi192 ceph-mon[32956]: osdmap e419: 12 total, 12 up, 12 in 2024-06-27T21:26:40.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:40.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:40.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:40.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:40.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:40.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:40.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:40.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:40.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:40.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:40.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:40.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:40.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:41.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:41.039 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:26:41.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:26:41.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:26:41.065 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:26:41.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:26:41.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:26:41.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:41.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:26:41.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:26:41.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:41.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:41.141 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:26:41.141 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:26:41.171 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:26:41.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:26:41.202 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:26:41.203 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:26:41.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:41.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:26:41.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:26:41.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:41.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:26:41.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:26:41.286 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:26:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:26:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:26:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:26:41.286 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:26:41.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:26:41.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:26:41.320 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:26:41.320 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:26:41.344 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:26:41.344 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:26:41.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:41.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:41.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:41.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:41.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:41.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:41.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:26:41.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:41.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:41.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:41.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:41.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:42.020 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:42.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:42.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:41 smithi151 ceph-mon[28965]: pgmap v1096: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 789 op/s 2024-06-27T21:26:42.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/870557874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:42.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:42.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:42.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:41 smithi190 ceph-mon[32247]: pgmap v1096: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 789 op/s 2024-06-27T21:26:42.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/870557874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:42.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:42.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:41 smithi192 ceph-mon[32956]: pgmap v1096: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 6.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 21 MiB/s wr, 789 op/s 2024-06-27T21:26:42.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/870557874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:42.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:42.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:42.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:42.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:42.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:42.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:42.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:42.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:42.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:42.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:42.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:42.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:43.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:43.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.076 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:43.156 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:43.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:42 smithi151 ceph-mon[28965]: pgmap v1097: 27 pgs: 27 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 10 MiB/s wr, 789 op/s 2024-06-27T21:26:43.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:26:43.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:43.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:43.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:42 smithi190 ceph-mon[32247]: pgmap v1097: 27 pgs: 27 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 10 MiB/s wr, 789 op/s 2024-06-27T21:26:43.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:42 smithi192 ceph-mon[32956]: pgmap v1097: 27 pgs: 27 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 10 MiB/s wr, 789 op/s 2024-06-27T21:26:43.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:43.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:43.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:43.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:43.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:43.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:43.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:43.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:43.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:43.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:43.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:43.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:43.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:44.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:44.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:44.191 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:26:44.192 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:26:44.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:26:44.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:44.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:44.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:44.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:44.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:44.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:44.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:44.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:44.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:44.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:44.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:44.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:44.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:44.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:45.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:45.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:45.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:45.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:45.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:45.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.416 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:45.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:45.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:45 smithi151 ceph-mon[28965]: pgmap v1098: 27 pgs: 27 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.5 MiB/s wr, 821 op/s 2024-06-27T21:26:45.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:45 smithi151 ceph-mon[28965]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:45.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:45 smithi190 ceph-mon[32247]: pgmap v1098: 27 pgs: 27 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.5 MiB/s wr, 821 op/s 2024-06-27T21:26:45.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:45 smithi190 ceph-mon[32247]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:45.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:45.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:45 smithi192 ceph-mon[32956]: pgmap v1098: 27 pgs: 27 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.5 MiB/s wr, 821 op/s 2024-06-27T21:26:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:45 smithi192 ceph-mon[32956]: from='client.16041 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:45.637 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1098,"stamp":"2024-06-27T21:26:44.028931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6446318136,"num_objects":98357,"num_object_clones":0,"num_object_copies":295071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98357,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113723,"num_write_kb":6786433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117538,"ondisk_log_size":117538,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":22637856,"kb_used_data":19033776,"kb_used_omap":110,"kb_used_meta":3603729,"kb_avail":1102303968,"statfs":{"total":1151940427776,"available":1128759263232,"internally_reserved":0,"allocated":19490586624,"data_stored":19487141748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":3690219479},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-443547624,"num_objects":-6767,"num_object_clones":0,"num_object_copies":-20301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8213,"num_write_kb":46209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003145"},"pg_stats":[{"pgid":"5.f","version":"419'7360","reported_seq":7504,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:42.730079+0000","last_change":"2024-06-27T21:26:02.177610+0000","last_active":"2024-06-27T21:26:42.730079+0000","last_peered":"2024-06-27T21:26:42.730079+0000","last_clean":"2024-06-27T21:26:42.730079+0000","last_became_active":"2024-06-27T21:24:48.707635+0000","last_became_peered":"2024-06-27T21:24:48.707635+0000","last_unstale":"2024-06-27T21:26:42.730079+0000","last_undegraded":"2024-06-27T21:26:42.730079+0000","last_fullsized":"2024-06-27T21:26:42.730079+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'4999","last_scrub_stamp":"2024-06-27T21:26:02.177486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:02.177486+0000","objects_scrubbed":4951,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:15.211969+0000","scrub_duration":1413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404619264,"num_objects":6174,"num_object_clones":0,"num_object_copies":18522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"419'7088","reported_seq":7222,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:40.221337+0000","last_change":"2024-06-27T21:25:54.256382+0000","last_active":"2024-06-27T21:26:40.221337+0000","last_peered":"2024-06-27T21:26:40.221337+0000","last_clean":"2024-06-27T21:26:40.221337+0000","last_became_active":"2024-06-27T21:24:48.708225+0000","last_became_peered":"2024-06-27T21:24:48.708225+0000","last_unstale":"2024-06-27T21:26:40.221337+0000","last_undegraded":"2024-06-27T21:26:40.221337+0000","last_fullsized":"2024-06-27T21:26:40.221337+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'4335","last_scrub_stamp":"2024-06-27T21:25:54.256168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:54.256168+0000","objects_scrubbed":4294,"log_size":7088,"log_dups_size":0,"ondisk_log_size":7088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:16.380086+0000","scrub_duration":869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410255360,"num_objects":6260,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7088,"num_write_kb":427136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"419'7264","reported_seq":7402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:43.318861+0000","last_change":"2024-06-27T21:25:58.409829+0000","last_active":"2024-06-27T21:26:43.318861+0000","last_peered":"2024-06-27T21:26:43.318861+0000","last_clean":"2024-06-27T21:26:43.318861+0000","last_became_active":"2024-06-27T21:24:48.708150+0000","last_became_peered":"2024-06-27T21:24:48.708150+0000","last_unstale":"2024-06-27T21:26:43.318861+0000","last_undegraded":"2024-06-27T21:26:43.318861+0000","last_fullsized":"2024-06-27T21:26:43.318861+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4607","last_scrub_stamp":"2024-06-27T21:25:58.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:58.409408+0000","objects_scrubbed":4559,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:22.763094+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398065664,"num_objects":6074,"num_object_clones":0,"num_object_copies":18222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7264,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"419'7059","reported_seq":7205,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:39.407439+0000","last_change":"2024-06-27T21:26:11.531967+0000","last_active":"2024-06-27T21:26:39.407439+0000","last_peered":"2024-06-27T21:26:39.407439+0000","last_clean":"2024-06-27T21:26:39.407439+0000","last_became_active":"2024-06-27T21:24:48.708687+0000","last_became_peered":"2024-06-27T21:24:48.708687+0000","last_unstale":"2024-06-27T21:26:39.407439+0000","last_undegraded":"2024-06-27T21:26:39.407439+0000","last_fullsized":"2024-06-27T21:26:39.407439+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5559","last_scrub_stamp":"2024-06-27T21:26:11.531764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:11.531764+0000","objects_scrubbed":5513,"log_size":7059,"log_dups_size":0,"ondisk_log_size":7059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:26.815955+0000","scrub_duration":1548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410058752,"num_objects":6257,"num_object_clones":0,"num_object_copies":18771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7059,"num_write_kb":426112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"419'7041","reported_seq":7178,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:39.585494+0000","last_change":"2024-06-27T21:25:57.075916+0000","last_active":"2024-06-27T21:26:39.585494+0000","last_peered":"2024-06-27T21:26:39.585494+0000","last_clean":"2024-06-27T21:26:39.585494+0000","last_became_active":"2024-06-27T21:24:48.708111+0000","last_became_peered":"2024-06-27T21:24:48.708111+0000","last_unstale":"2024-06-27T21:26:39.585494+0000","last_undegraded":"2024-06-27T21:26:39.585494+0000","last_fullsized":"2024-06-27T21:26:39.585494+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4478","last_scrub_stamp":"2024-06-27T21:25:57.075739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:57.075739+0000","objects_scrubbed":4444,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:19.549115+0000","scrub_duration":1535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410058752,"num_objects":6257,"num_object_clones":0,"num_object_copies":18771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.4","version":"419'7056","reported_seq":7196,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:39.622054+0000","last_change":"2024-06-27T21:26:03.238316+0000","last_active":"2024-06-27T21:26:39.622054+0000","last_peered":"2024-06-27T21:26:39.622054+0000","last_clean":"2024-06-27T21:26:39.622054+0000","last_became_active":"2024-06-27T21:24:48.709413+0000","last_became_peered":"2024-06-27T21:24:48.709413+0000","last_unstale":"2024-06-27T21:26:39.622054+0000","last_undegraded":"2024-06-27T21:26:39.622054+0000","last_fullsized":"2024-06-27T21:26:39.622054+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'4970","last_scrub_stamp":"2024-06-27T21:26:03.238159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:03.238159+0000","objects_scrubbed":4924,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:15.739270+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408813568,"num_objects":6238,"num_object_clones":0,"num_object_copies":18714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7056,"num_write_kb":425408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":868,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.881846+0000","last_change":"2024-06-27T21:25:08.952316+0000","last_active":"2024-06-27T21:26:38.881846+0000","last_peered":"2024-06-27T21:26:38.881846+0000","last_clean":"2024-06-27T21:26:38.881846+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:26:38.881846+0000","last_undegraded":"2024-06-27T21:26:38.881846+0000","last_fullsized":"2024-06-27T21:26:38.881846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:08.952246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:08.952246+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:42.890900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"419'7156","reported_seq":7305,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:42.683156+0000","last_change":"2024-06-27T21:26:13.463329+0000","last_active":"2024-06-27T21:26:42.683156+0000","last_peered":"2024-06-27T21:26:42.683156+0000","last_clean":"2024-06-27T21:26:42.683156+0000","last_became_active":"2024-06-27T21:24:48.885413+0000","last_became_peered":"2024-06-27T21:24:48.885413+0000","last_unstale":"2024-06-27T21:26:42.683156+0000","last_undegraded":"2024-06-27T21:26:42.683156+0000","last_fullsized":"2024-06-27T21:26:42.683156+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5601","last_scrub_stamp":"2024-06-27T21:26:13.463234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:13.463234+0000","objects_scrubbed":5547,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:21.762213+0000","scrub_duration":1698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391643136,"num_objects":5976,"num_object_clones":0,"num_object_copies":17928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7156,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":463,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.865383+0000","last_change":"2024-06-27T21:25:08.804533+0000","last_active":"2024-06-27T21:26:38.865383+0000","last_peered":"2024-06-27T21:26:38.865383+0000","last_clean":"2024-06-27T21:26:38.865383+0000","last_became_active":"2024-06-27T21:23:44.524342+0000","last_became_peered":"2024-06-27T21:23:44.524342+0000","last_unstale":"2024-06-27T21:26:38.865383+0000","last_undegraded":"2024-06-27T21:26:38.865383+0000","last_fullsized":"2024-06-27T21:26:38.865383+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:08.804415+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:25:08.804415+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:42.734017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"419'7277","reported_seq":7415,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:43.329557+0000","last_change":"2024-06-27T21:25:58.105417+0000","last_active":"2024-06-27T21:26:43.329557+0000","last_peered":"2024-06-27T21:26:43.329557+0000","last_clean":"2024-06-27T21:26:43.329557+0000","last_became_active":"2024-06-27T21:24:48.710086+0000","last_became_peered":"2024-06-27T21:24:48.710086+0000","last_unstale":"2024-06-27T21:26:43.329557+0000","last_undegraded":"2024-06-27T21:26:43.329557+0000","last_fullsized":"2024-06-27T21:26:43.329557+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4594","last_scrub_stamp":"2024-06-27T21:25:58.105207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:58.105207+0000","objects_scrubbed":4550,"log_size":7277,"log_dups_size":0,"ondisk_log_size":7277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:09.223215+0000","scrub_duration":1746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394592256,"num_objects":6021,"num_object_clones":0,"num_object_copies":18063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7277,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":881,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.881858+0000","last_change":"2024-06-27T21:25:07.953798+0000","last_active":"2024-06-27T21:26:38.881858+0000","last_peered":"2024-06-27T21:26:38.881858+0000","last_clean":"2024-06-27T21:26:38.881858+0000","last_became_active":"2024-06-27T21:23:44.529886+0000","last_became_peered":"2024-06-27T21:23:44.529886+0000","last_unstale":"2024-06-27T21:26:38.881858+0000","last_undegraded":"2024-06-27T21:26:38.881858+0000","last_fullsized":"2024-06-27T21:26:38.881858+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:07.953724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:07.953724+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:42.890942+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"419'6979","reported_seq":7127,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:41.001943+0000","last_change":"2024-06-27T21:26:15.938541+0000","last_active":"2024-06-27T21:26:41.001943+0000","last_peered":"2024-06-27T21:26:41.001943+0000","last_clean":"2024-06-27T21:26:41.001943+0000","last_became_active":"2024-06-27T21:24:48.885372+0000","last_became_peered":"2024-06-27T21:24:48.885372+0000","last_unstale":"2024-06-27T21:26:41.001943+0000","last_undegraded":"2024-06-27T21:26:41.001943+0000","last_fullsized":"2024-06-27T21:26:41.001943+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5629","last_scrub_stamp":"2024-06-27T21:26:15.938322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:15.938322+0000","objects_scrubbed":5594,"log_size":6979,"log_dups_size":0,"ondisk_log_size":6979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:38.993790+0000","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394067968,"num_objects":6013,"num_object_clones":0,"num_object_copies":18039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6979,"num_write_kb":415744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":275,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.868466+0000","last_change":"2024-06-27T21:25:10.180661+0000","last_active":"2024-06-27T21:26:38.868466+0000","last_peered":"2024-06-27T21:26:38.868466+0000","last_clean":"2024-06-27T21:26:38.868466+0000","last_became_active":"2024-06-27T21:21:18.602559+0000","last_became_peered":"2024-06-27T21:21:18.602559+0000","last_unstale":"2024-06-27T21:26:38.868466+0000","last_undegraded":"2024-06-27T21:26:38.868466+0000","last_fullsized":"2024-06-27T21:26:38.868466+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:10.180545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:10.180545+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:41.125951+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"419'7041","reported_seq":7192,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:40.971575+0000","last_change":"2024-06-27T21:26:19.240429+0000","last_active":"2024-06-27T21:26:40.971575+0000","last_peered":"2024-06-27T21:26:40.971575+0000","last_clean":"2024-06-27T21:26:40.971575+0000","last_became_active":"2024-06-27T21:24:48.710822+0000","last_became_peered":"2024-06-27T21:24:48.710822+0000","last_unstale":"2024-06-27T21:26:40.971575+0000","last_undegraded":"2024-06-27T21:26:40.971575+0000","last_fullsized":"2024-06-27T21:26:40.971575+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5920","last_scrub_stamp":"2024-06-27T21:26:19.240187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.240187+0000","objects_scrubbed":5882,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:33.458729+0000","scrub_duration":1408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399441920,"num_objects":6095,"num_object_clones":0,"num_object_copies":18285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":610,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.865439+0000","last_change":"2024-06-27T21:26:22.770901+0000","last_active":"2024-06-27T21:26:38.865439+0000","last_peered":"2024-06-27T21:26:38.865439+0000","last_clean":"2024-06-27T21:26:38.865439+0000","last_became_active":"2024-06-27T21:21:18.602789+0000","last_became_peered":"2024-06-27T21:21:18.602789+0000","last_unstale":"2024-06-27T21:26:38.865439+0000","last_undegraded":"2024-06-27T21:26:38.865439+0000","last_fullsized":"2024-06-27T21:26:38.865439+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:26:22.770782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:31.213714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"419'7002","reported_seq":7148,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:40.513642+0000","last_change":"2024-06-27T21:26:11.195361+0000","last_active":"2024-06-27T21:26:40.513642+0000","last_peered":"2024-06-27T21:26:40.513642+0000","last_clean":"2024-06-27T21:26:40.513642+0000","last_became_active":"2024-06-27T21:24:48.708618+0000","last_became_peered":"2024-06-27T21:24:48.708618+0000","last_unstale":"2024-06-27T21:26:40.513642+0000","last_undegraded":"2024-06-27T21:26:40.513642+0000","last_fullsized":"2024-06-27T21:26:40.513642+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5451","last_scrub_stamp":"2024-06-27T21:26:11.195038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:11.195038+0000","objects_scrubbed":5384,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:29.760754+0000","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402653184,"num_objects":6144,"num_object_clones":0,"num_object_copies":18432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7002,"num_write_kb":420672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":275,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.868551+0000","last_change":"2024-06-27T21:25:10.874252+0000","last_active":"2024-06-27T21:26:38.868551+0000","last_peered":"2024-06-27T21:26:38.868551+0000","last_clean":"2024-06-27T21:26:38.868551+0000","last_became_active":"2024-06-27T21:21:18.602398+0000","last_became_peered":"2024-06-27T21:21:18.602398+0000","last_unstale":"2024-06-27T21:26:38.868551+0000","last_undegraded":"2024-06-27T21:26:38.868551+0000","last_fullsized":"2024-06-27T21:26:38.868551+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:10.874135+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:25:10.874135+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:40.653889+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"419'7172","reported_seq":7318,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:42.870408+0000","last_change":"2024-06-27T21:26:09.408114+0000","last_active":"2024-06-27T21:26:42.870408+0000","last_peered":"2024-06-27T21:26:42.870408+0000","last_clean":"2024-06-27T21:26:42.870408+0000","last_became_active":"2024-06-27T21:24:48.709002+0000","last_became_peered":"2024-06-27T21:24:48.709002+0000","last_unstale":"2024-06-27T21:26:42.870408+0000","last_undegraded":"2024-06-27T21:26:42.870408+0000","last_fullsized":"2024-06-27T21:26:42.870408+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'5325","last_scrub_stamp":"2024-06-27T21:26:09.407899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:09.407899+0000","objects_scrubbed":5258,"log_size":7172,"log_dups_size":0,"ondisk_log_size":7172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:36.464873+0000","scrub_duration":1191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390725632,"num_objects":5962,"num_object_clones":0,"num_object_copies":17886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":263,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.868498+0000","last_change":"2024-06-27T21:25:04.736172+0000","last_active":"2024-06-27T21:26:38.868498+0000","last_peered":"2024-06-27T21:26:38.868498+0000","last_clean":"2024-06-27T21:26:38.868498+0000","last_became_active":"2024-06-27T21:21:34.494291+0000","last_became_peered":"2024-06-27T21:21:34.494291+0000","last_unstale":"2024-06-27T21:26:38.868498+0000","last_undegraded":"2024-06-27T21:26:38.868498+0000","last_fullsized":"2024-06-27T21:26:38.868498+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:04.736074+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:25:04.736074+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:41.125881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"419'6917","reported_seq":7066,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:39.594655+0000","last_change":"2024-06-27T21:26:19.252916+0000","last_active":"2024-06-27T21:26:39.594655+0000","last_peered":"2024-06-27T21:26:39.594655+0000","last_clean":"2024-06-27T21:26:39.594655+0000","last_became_active":"2024-06-27T21:24:48.885324+0000","last_became_peered":"2024-06-27T21:24:48.885324+0000","last_unstale":"2024-06-27T21:26:39.594655+0000","last_undegraded":"2024-06-27T21:26:39.594655+0000","last_fullsized":"2024-06-27T21:26:39.594655+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5908","last_scrub_stamp":"2024-06-27T21:26:19.252716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.252716+0000","objects_scrubbed":5869,"log_size":6917,"log_dups_size":0,"ondisk_log_size":6917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:22.654032+0000","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399835136,"num_objects":6101,"num_object_clones":0,"num_object_copies":18303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6917,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":684,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.865877+0000","last_change":"2024-06-27T21:26:08.543233+0000","last_active":"2024-06-27T21:26:38.865877+0000","last_peered":"2024-06-27T21:26:38.865877+0000","last_clean":"2024-06-27T21:26:38.865877+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:26:38.865877+0000","last_undegraded":"2024-06-27T21:26:38.865877+0000","last_fullsized":"2024-06-27T21:26:38.865877+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:08.543015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:08.543015+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:23.153986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"419'7118","reported_seq":7268,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:40.176715+0000","last_change":"2024-06-27T21:26:16.853366+0000","last_active":"2024-06-27T21:26:40.176715+0000","last_peered":"2024-06-27T21:26:40.176715+0000","last_clean":"2024-06-27T21:26:40.176715+0000","last_became_active":"2024-06-27T21:24:48.707990+0000","last_became_peered":"2024-06-27T21:24:48.707990+0000","last_unstale":"2024-06-27T21:26:40.176715+0000","last_undegraded":"2024-06-27T21:26:40.176715+0000","last_fullsized":"2024-06-27T21:26:40.176715+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5897","last_scrub_stamp":"2024-06-27T21:26:16.853222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:16.853222+0000","objects_scrubbed":5847,"log_size":7118,"log_dups_size":0,"ondisk_log_size":7118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:34.301452+0000","scrub_duration":1472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410517504,"num_objects":6264,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":428224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":582,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.870466+0000","last_change":"2024-06-27T21:25:02.630434+0000","last_active":"2024-06-27T21:26:38.870466+0000","last_peered":"2024-06-27T21:26:38.870466+0000","last_clean":"2024-06-27T21:26:38.870466+0000","last_became_active":"2024-06-27T21:23:44.529707+0000","last_became_peered":"2024-06-27T21:23:44.529707+0000","last_unstale":"2024-06-27T21:26:38.870466+0000","last_undegraded":"2024-06-27T21:26:38.870466+0000","last_fullsized":"2024-06-27T21:26:38.870466+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:02.630318+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:25:02.630318+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:41.410773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"419'6960","reported_seq":7112,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:39.593650+0000","last_change":"2024-06-27T21:26:19.426445+0000","last_active":"2024-06-27T21:26:39.593650+0000","last_peered":"2024-06-27T21:26:39.593650+0000","last_clean":"2024-06-27T21:26:39.593650+0000","last_became_active":"2024-06-27T21:24:48.955695+0000","last_became_peered":"2024-06-27T21:24:48.955695+0000","last_unstale":"2024-06-27T21:26:39.593650+0000","last_undegraded":"2024-06-27T21:26:39.593650+0000","last_fullsized":"2024-06-27T21:26:39.593650+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5936","last_scrub_stamp":"2024-06-27T21:26:19.426355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.426355+0000","objects_scrubbed":5918,"log_size":6960,"log_dups_size":0,"ondisk_log_size":6960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:41.889854+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408354840,"num_objects":6232,"num_object_clones":0,"num_object_copies":18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6960,"num_write_kb":422081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":892,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.884675+0000","last_change":"2024-06-27T21:26:14.033864+0000","last_active":"2024-06-27T21:26:38.884675+0000","last_peered":"2024-06-27T21:26:38.884675+0000","last_clean":"2024-06-27T21:26:38.884675+0000","last_became_active":"2024-06-27T21:21:34.485764+0000","last_became_peered":"2024-06-27T21:21:34.485764+0000","last_unstale":"2024-06-27T21:26:38.884675+0000","last_undegraded":"2024-06-27T21:26:38.884675+0000","last_fullsized":"2024-06-27T21:26:38.884675+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:14.033749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:14.033749+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:38.288649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"419'11037","reported_seq":11347,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:40.981322+0000","last_change":"2024-06-27T21:26:38.958637+0000","last_active":"2024-06-27T21:26:40.981322+0000","last_peered":"2024-06-27T21:26:40.981322+0000","last_clean":"2024-06-27T21:26:40.981322+0000","last_became_active":"2024-06-27T21:24:48.885320+0000","last_became_peered":"2024-06-27T21:24:48.885320+0000","last_unstale":"2024-06-27T21:26:40.981322+0000","last_undegraded":"2024-06-27T21:26:40.981322+0000","last_fullsized":"2024-06-27T21:26:40.981322+0000","mapping_epoch":382,"log_start":"389'1001","ondisk_log_start":"389'1001","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:24:47.689585+0000","objects_scrubbed":5498,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:26:43.957992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024832,"num_objects":6287,"num_object_clones":0,"num_object_copies":18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":432640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":615,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-06-27T21:26:38.869472+0000","last_change":"2024-06-27T21:25:54.786735+0000","last_active":"2024-06-27T21:26:38.869472+0000","last_peered":"2024-06-27T21:26:38.869472+0000","last_clean":"2024-06-27T21:26:38.869472+0000","last_became_active":"2024-06-27T21:21:34.487122+0000","last_became_peered":"2024-06-27T21:21:34.487122+0000","last_unstale":"2024-06-27T21:26:38.869472+0000","last_undegraded":"2024-06-27T21:26:38.869472+0000","last_fullsized":"2024-06-27T21:26:38.869472+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:54.786656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:54.786656+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:12.608584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6445727768,"num_objects":98355,"num_object_clones":0,"num_object_copies":295065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113723,"num_write_kb":6786433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19332673536,"data_stored":19332661320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116526,"ondisk_log_size":116526,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253072,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2028644,"kb_used_data":1622620,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91716508,"statfs":{"total":95995035648,"available":93917704192,"internally_reserved":0,"allocated":1661562880,"data_stored":1661277052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514709,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2293424,"kb_used_data":2001252,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91451728,"statfs":{"total":95995035648,"available":93646569472,"internally_reserved":0,"allocated":2049282048,"data_stored":2048995744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743641,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1514160,"kb_used_data":1203940,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92230992,"statfs":{"total":95995035648,"available":94444535808,"internally_reserved":0,"allocated":1232834560,"data_stored":1232548256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":317651392},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972574,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979820,"kb_used_data":790372,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92765332,"statfs":{"total":95995035648,"available":94991699968,"internally_reserved":0,"allocated":809340928,"data_stored":809054624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1968100,"kb_used_data":1607900,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91777052,"statfs":{"total":95995035648,"available":93979701248,"internally_reserved":0,"allocated":1646489600,"data_stored":1646203772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368835018},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528552,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1620656,"kb_used_data":1192420,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92124496,"statfs":{"total":95995035648,"available":94335483904,"internally_reserved":0,"allocated":1221038080,"data_stored":1220751776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757486,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2984120,"kb_used_data":2732080,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90761032,"statfs":{"total":95995035648,"available":92939296768,"internally_reserved":0,"allocated":2797649920,"data_stored":2797355992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161189,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062380,"kb_used_data":1938916,"kb_used_omap":92,"kb_used_meta":123363,"kb_avail":91682772,"statfs":{"total":95995035648,"available":93883158528,"internally_reserved":0,"allocated":1985449984,"data_stored":1985163680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":126323810},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182646,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695924,"kb_used_data":385708,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93049228,"statfs":{"total":95995035648,"available":95282409472,"internally_reserved":0,"allocated":394964992,"data_stored":394675136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":317651399},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411579,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339956,"kb_used_data":1168556,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92405196,"statfs":{"total":95995035648,"available":94622920704,"internally_reserved":0,"allocated":1196601344,"data_stored":1196311488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575103,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2001568,"kb_used_data":1605272,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91743584,"statfs":{"total":95995035648,"available":93945430016,"internally_reserved":0,"allocated":1643798528,"data_stored":1643516772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738628,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3149104,"kb_used_data":2784740,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":90596048,"statfs":{"total":95995035648,"available":92770353152,"internally_reserved":0,"allocated":2851573760,"data_stored":2851287456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2824470528,"data_stored":2824470528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1632108544,"data_stored":1632108544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1184759808,"data_stored":1184759808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":392101888,"data_stored":392101888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1970929664,"data_stored":1970929664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2775781376,"data_stored":2775777304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1208942592,"data_stored":1208942592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1635454976,"data_stored":1635450904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":803471360,"data_stored":803471360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1221328896,"data_stored":1221328896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2037907456,"data_stored":2037907456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1645416448,"data_stored":1645412376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:26:45.638 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-27T21:26:45.639 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:26:45.639 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-06-27T21:26:45.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:26:45.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:45.666 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:45.668 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:45.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:45.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:45.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:45.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:45.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:45.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:46.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:46.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:46.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:46.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:46.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:46.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:46.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:46.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:46.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:46.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:46.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:46.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:46.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:46.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:46.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:46.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:46.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:47.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:47.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:47.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:47.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:47.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:47.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:47.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:47 smithi151 ceph-mon[28965]: pgmap v1099: 27 pgs: 27 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s 2024-06-27T21:26:47.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:47.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:47.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:47 smithi190 ceph-mon[32247]: pgmap v1099: 27 pgs: 27 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s 2024-06-27T21:26:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:47 smithi192 ceph-mon[32956]: pgmap v1099: 27 pgs: 27 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s 2024-06-27T21:26:47.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:47.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:47.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:47.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:47.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:47.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:47.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:47.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:48.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:48.097 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:26:48.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:48.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:48.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:48.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:48.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:48.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:48.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3674979681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:26:48.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:48 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:26:48.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:48.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:48.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:48.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3674979681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:26:48.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:48 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:26:48.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:48.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:48.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3674979681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:26:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:48 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:26:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:48.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:48.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:48.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:48.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:48.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:48.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:48.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:49.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:49.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:49.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:49.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:49.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:49.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:49.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:49.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:49.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:49 smithi151 ceph-mon[28965]: pgmap v1100: 27 pgs: 27 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s 2024-06-27T21:26:49.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:49 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:49.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:26:49.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:49 smithi151 ceph-mon[28965]: osdmap e420: 12 total, 12 up, 12 in 2024-06-27T21:26:49.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:49 smithi151 ceph-mon[28965]: 5.c scrub starts 2024-06-27T21:26:49.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:49 smithi190 ceph-mon[32247]: pgmap v1100: 27 pgs: 27 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s 2024-06-27T21:26:49.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:49 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:49.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:26:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:49 smithi190 ceph-mon[32247]: osdmap e420: 12 total, 12 up, 12 in 2024-06-27T21:26:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:49 smithi190 ceph-mon[32247]: 5.c scrub starts 2024-06-27T21:26:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:49 smithi192 ceph-mon[32956]: pgmap v1100: 27 pgs: 27 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s 2024-06-27T21:26:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:49 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:26:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:26:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:49 smithi192 ceph-mon[32956]: osdmap e420: 12 total, 12 up, 12 in 2024-06-27T21:26:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:49 smithi192 ceph-mon[32956]: 5.c scrub starts 2024-06-27T21:26:49.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:49.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:49.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:49.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:49.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:49.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:49.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:49.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:49.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:50.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:50.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:50.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:50.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:50.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:50.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:50.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:50.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:50.458 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:50.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:50.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:50.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:26:50.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:50 smithi151 ceph-mon[28965]: 5.c scrub ok 2024-06-27T21:26:50.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:50 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:26:50.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:50 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:26:50.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:26:50.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:50.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:50.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:50.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:50.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:50 smithi190 ceph-mon[32247]: 5.c scrub ok 2024-06-27T21:26:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:50 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:26:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:50 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:26:51.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:26:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:50 smithi192 ceph-mon[32956]: 5.c scrub ok 2024-06-27T21:26:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:50 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:26:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:50 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:26:51.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:51.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:51.116 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:66816 4.c 4:3ddaefad:::benchmark_data_smithi192_88961_object66815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:44.600428+0000", 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 306.47534817100001, 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27979604200000002, 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 66816 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600428+0000", 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600428+0000", 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600440+0000", 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1620999999999999e-05 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600469+0000", 2024-06-27T21:26:51.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8926000000000001e-05 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600482+0000", 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2865e-05 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600489+0000", 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9299999999999997e-06 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600530+0000", 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1480999999999998e-05 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600628+0000", 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8002000000000001e-05 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.600722+0000", 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3893000000000004e-05 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.603805+0000", 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030831090000000001 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880106+0000", 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27630123099999998 2024-06-27T21:26:51.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880142+0000", 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5559000000000003e-05 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880188+0000", 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5874000000000002e-05 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880224+0000", 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6551000000000003e-05 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:66816 4.c e320/301 ondisk, result = 0)", 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:44.602810+0000", 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 306.47296601199997, 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27741933200000002, 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.124 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.602810+0000", 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.602810+0000", 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.602811+0000", 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4700000000000001e-07 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.602814+0000", 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.633e-06 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880066+0000", 2024-06-27T21:26:51.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27725232799999999 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880071+0000", 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8550000000000001e-06 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880133+0000", 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2292000000000006e-05 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880138+0000", 2024-06-27T21:26:51.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1139999999999997e-06 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.880230+0000", 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1463000000000002e-05 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:66838 4.0 4:062d9aad:::benchmark_data_smithi192_88961_object66837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:44.643125+0000", 2024-06-27T21:26:51.128 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 306.43265107000002, 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32717901500000002, 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 66838 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643125+0000", 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643125+0000", 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643127+0000", 2024-06-27T21:26:51.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5039999999999999e-06 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643164+0000", 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6983000000000001e-05 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643179+0000", 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4953000000000001e-05 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643186+0000", 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4279999999999999e-06 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643239+0000", 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2936999999999998e-05 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643349+0000", 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109891 2024-06-27T21:26:51.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.643447+0000", 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7779e-05 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.645412+0000", 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001964773 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.646105+0000", 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00069336300000000001 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970211+0000", 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.324105743 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970255+0000", 2024-06-27T21:26:51.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4304999999999997e-05 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970304+0000", 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9356e-05 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.135 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:66838 4.0 e320/282 ondisk, result = 0)", 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:44.645737+0000", 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 306.43003955699999, 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32457361099999998, 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.645737+0000", 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.645737+0000", 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.645738+0000", 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9099999999999991e-07 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.645746+0000", 2024-06-27T21:26:51.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8690000000000004e-06 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970130+0000", 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32438418299999999 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970137+0000", 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8589999999999999e-06 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970202+0000", 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4931999999999994e-05 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970207+0000", 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3990000000000003e-06 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.970310+0000", 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103378 2024-06-27T21:26:51.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:66852 4.c 4:35d59555:::benchmark_data_smithi192_88961_object66851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:44.877155+0000", 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 306.19862137400003, 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29932344, 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 66852 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877155+0000", 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877155+0000", 2024-06-27T21:26:51.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877156+0000", 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4440000000000001e-06 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877196+0000", 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9801000000000001e-05 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877208+0000", 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1572e-05 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877214+0000", 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4999999999999996e-06 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877263+0000", 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8319999999999998e-05 2024-06-27T21:26:51.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877357+0000", 2024-06-27T21:26:51.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4435e-05 2024-06-27T21:26:51.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.877447+0000", 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9572000000000002e-05 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.879428+0000", 2024-06-27T21:26:51.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019815520000000001 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:44.981478+0000", 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.102049448 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:45.176398+0000", 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19492047600000001 2024-06-27T21:26:51.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:45.176447+0000", 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8634000000000002e-05 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:45.176478+0000", 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1686000000000001e-05 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:130426 4.0 4:06e924af:::benchmark_data_smithi192_88961_object39687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:55.734748+0000", 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 235.34102870500001, 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51450275300000003, 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 130426 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734748+0000", 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734748+0000", 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734750+0000", 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8330000000000001e-06 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734756+0000", 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.348e-06 2024-06-27T21:26:51.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734773+0000", 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6968000000000001e-05 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734783+0000", 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0101999999999999e-05 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734846+0000", 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3546000000000004e-05 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.734987+0000", 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014087099999999999 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.735107+0000", 2024-06-27T21:26:51.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119721 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.738233+0000", 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003125657 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.738567+0000", 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033432600000000002 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249115+0000", 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51054771700000001 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249180+0000", 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5382000000000005e-05 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249250+0000", 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0282000000000002e-05 2024-06-27T21:26:51.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:130426 4.0 e345/282 ondisk, result = 0)", 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:55.739293+0000", 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 235.336483001, 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50996267900000003, 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.157 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.739293+0000", 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.739293+0000", 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.739294+0000", 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5300000000000004e-07 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:55.739298+0000", 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8149999999999999e-06 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249015+0000", 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50971681599999996 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249024+0000", 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7729999999999994e-06 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249097+0000", 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3034000000000006e-05 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249108+0000", 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0764999999999999e-05 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.249256+0000", 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147823 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.8,RELEASE,e:351,from:5,reservation_nonce:0)", 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:13.600626+0000", 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.475150332, 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58558301400000001, 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:13.600626+0000", 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:13.600626+0000", 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:13.600626+0000", 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4499999999999998e-07 2024-06-27T21:26:51.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:13.600629+0000", 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.153e-06 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:14.186120+0000", 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58549143699999995 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:14.186124+0000", 2024-06-27T21:26:51.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.177e-06 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:14.186173+0000", 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690999999999998e-05 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:14.186191+0000", 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8070000000000001e-05 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:14.186209+0000", 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8141000000000001e-05 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163686 4.c 4.a540257c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.912645+0000", 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 187.16313090599999, 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.608717866, 2024-06-27T21:26:51.166 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163686 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912645+0000", 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912645+0000", 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912646+0000", 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6100000000000002e-07 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912648+0000", 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.39e-06 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912658+0000", 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6379999999999999e-06 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912662+0000", 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.777e-06 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912727+0000", 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5489999999999998e-05 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.912739+0000", 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2289999999999999e-05 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521330+0000", 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60859015999999999 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521363+0000", 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3760000000000002e-05 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163689 4.c 4.b7b9d05c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.920296+0000", 2024-06-27T21:26:51.172 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 187.155480165, 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60110333699999996, 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163689 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.173 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920296+0000", 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920296+0000", 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920298+0000", 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.739e-06 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920305+0000", 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1319999999999996e-06 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920322+0000", 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7050000000000001e-05 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920330+0000", 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1640000000000001e-06 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920402+0000", 2024-06-27T21:26:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1965999999999995e-05 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.920416+0000", 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3356999999999999e-05 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521393+0000", 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60097717299999998 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521400+0000", 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7560000000000002e-06 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163699 4.c 4.f3b0a6bc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.934059+0000", 2024-06-27T21:26:51.178 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 187.14171772700001, 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.587368102, 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163699 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934059+0000", 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934059+0000", 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934060+0000", 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5689999999999999e-06 2024-06-27T21:26:51.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934067+0000", 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.114e-06 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934085+0000", 2024-06-27T21:26:51.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7401000000000002e-05 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934092+0000", 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5830000000000001e-06 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934166+0000", 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3368999999999995e-05 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.934179+0000", 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3277e-05 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521422+0000", 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58724276399999997 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521427+0000", 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0250000000000002e-06 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.183 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163712 4.0 4:0319ddeb:::benchmark_data_smithi192_88961_object72973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.950211+0000", 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 187.12556540700001, 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56581035599999996, 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163712 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950211+0000", 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950211+0000", 2024-06-27T21:26:51.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950213+0000", 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.637e-06 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950219+0000", 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2380000000000002e-06 2024-06-27T21:26:51.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950233+0000", 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3988999999999999e-05 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950241+0000", 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.089e-06 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950293+0000", 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.164e-05 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950384+0000", 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0937e-05 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-06-27T21:26:51.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950462+0000", 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8368999999999995e-05 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954100+0000", 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003638044 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954598+0000", 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000498009 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.515953+0000", 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56135471800000003 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.515990+0000", 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6990000000000003e-05 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.516021+0000", 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1696000000000003e-05 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163713 4.c 4.13785a2c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.951525+0000", 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 187.12425096699999, 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.569927709, 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163713 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951525+0000", 2024-06-27T21:26:51.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951525+0000", 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951526+0000", 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951530+0000", 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0589999999999996e-06 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951542+0000", 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1815e-05 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951549+0000", 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0500000000000003e-06 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951614+0000", 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5302999999999995e-05 2024-06-27T21:26:51.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.951626+0000", 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1666e-05 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521448+0000", 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56982208999999995 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521453+0000", 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8829999999999997e-06 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.197 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163714 4.c 4.cb0e241c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.954754+0000", 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 187.12102229600001, 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56672829300000005, 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163714 2024-06-27T21:26:51.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954754+0000", 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954754+0000", 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954755+0000", 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954759+0000", 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.476e-06 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954771+0000", 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1302e-05 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954778+0000", 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7710000000000006e-06 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954823+0000", 2024-06-27T21:26:51.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4242000000000001e-05 2024-06-27T21:26:51.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.954830+0000", 2024-06-27T21:26:51.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4460000000000004e-06 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521476+0000", 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56664631200000004 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.521482+0000", 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8860000000000001e-06 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.203 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163686 4.c 4:3ea402a5:::benchmark_data_smithi192_88961_object72947:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.637840+0000", 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 185.437936855, 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35358037599999997, 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.204 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163686 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637840+0000", 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637840+0000", 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637841+0000", 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.53e-06 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637847+0000", 2024-06-27T21:26:51.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0159999999999997e-06 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637860+0000", 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2847000000000001e-05 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637870+0000", 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6209999999999999e-06 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637935+0000", 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5331000000000002e-05 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637947+0000", 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2439999999999999e-05 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.971759+0000", 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33381170300000002 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.971882+0000", 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012267300000000001 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.971982+0000", 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100525 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.990796+0000", 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018813955 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.990942+0000", 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014539199999999999 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991081+0000", 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013972400000000001 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991147+0000", 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5925000000000003e-05 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991420+0000", 2024-06-27T21:26:51.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027269399999999997 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163689 4.c 4:3a0b9ded:::benchmark_data_smithi192_88961_object72950:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.637890+0000", 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 185.437886883, 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37365030100000002, 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.212 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163689 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637890+0000", 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637890+0000", 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637890+0000", 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1699999999999999e-07 2024-06-27T21:26:51.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637892+0000", 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.013e-06 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637896+0000", 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4819999999999996e-06 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637900+0000", 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3759999999999999e-06 2024-06-27T21:26:51.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637989+0000", 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9370000000000007e-05 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638001+0000", 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1589e-05 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.976032+0000", 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33803094099999997 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.976172+0000", 2024-06-27T21:26:51.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014048399999999999 2024-06-27T21:26:51.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.976317+0000", 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014486800000000001 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.990883+0000", 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0145662 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991011+0000", 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012827499999999999 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011206+0000", 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020194456999999999 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011265+0000", 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9537999999999998e-05 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011540+0000", 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027439100000000002 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.221 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163699 4.c 4:3d650dcf:::benchmark_data_smithi192_88961_object72960:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.637934+0000", 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 185.43784263200001, 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37388843399999999, 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163699 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.222 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637934+0000", 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637934+0000", 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637934+0000", 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8000000000000002e-07 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637936+0000", 2024-06-27T21:26:51.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.272e-06 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637941+0000", 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8840000000000002e-06 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637944+0000", 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1319999999999998e-06 2024-06-27T21:26:51.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638042+0000", 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8029999999999995e-05 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638050+0000", 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6869999999999994e-06 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.981282+0000", 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34323151600000001 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.981459+0000", 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017763100000000001 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.981612+0000", 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015248700000000001 2024-06-27T21:26:51.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.990889+0000", 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.009277492 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.999781+0000", 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0088922510000000003 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011611+0000", 2024-06-27T21:26:51.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011829087 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011660+0000", 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9064999999999999e-05 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011822+0000", 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016262000000000001 2024-06-27T21:26:51.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163713 4.c 4:345a1ec8:::benchmark_data_smithi192_88961_object72974:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.637949+0000", 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 185.43782709000001, 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37409626800000001, 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.230 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163713 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637949+0000", 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637949+0000", 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637950+0000", 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4400000000000001e-07 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637951+0000", 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6920000000000001e-06 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637955+0000", 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3100000000000001e-06 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637957+0000", 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9000000000000002e-06 2024-06-27T21:26:51.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638086+0000", 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128913 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638095+0000", 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4959999999999996e-06 2024-06-27T21:26:51.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.985774+0000", 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.347678969 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.985898+0000", 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124015 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.986018+0000", 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120016 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991433+0000", 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054152169999999999 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.002348+0000", 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010914493000000001 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011863+0000", 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0095153189999999995 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.011908+0000", 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5247000000000002e-05 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.012046+0000", 2024-06-27T21:26:51.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013743699999999999 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163714 4.c 4:382470d3:::benchmark_data_smithi192_88961_object72975:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.637963+0000", 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 185.43781318699999, 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38999152300000001, 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.238 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163714 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637963+0000", 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637963+0000", 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637964+0000", 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1699999999999999e-07 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637966+0000", 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.227e-06 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637972+0000", 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2609999999999999e-06 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.637977+0000", 2024-06-27T21:26:51.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0209999999999999e-06 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638131+0000", 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000153506 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638137+0000", 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1670000000000004e-06 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991461+0000", 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35332473599999997 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991553+0000", 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1203000000000001e-05 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-06-27T21:26:51.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.991644+0000", 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1520999999999993e-05 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.999733+0000", 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080884479999999998 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.002403+0000", 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026701110000000002 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.027722+0000", 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025319443000000001 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.027781+0000", 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9265000000000003e-05 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.027955+0000", 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017329699999999999 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25615.0:28432 5.1 e392/382 5:896e3337:::benchmark_data_smithi192_104167_object28431:head v 392'1777, mlcod=392'1774)", 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:25:22.060228+0000", 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.015548635000002, 2024-06-27T21:26:51.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216261391, 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.060228+0000", 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:26:51.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.060228+0000", 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.060228+0000", 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.37e-07 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.060280+0000", 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1286999999999999e-05 2024-06-27T21:26:51.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.274230+0000", 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21395012399999999 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.274238+0000", 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7370000000000006e-06 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:26:51.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.274306+0000", 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8392000000000005e-05 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.274333+0000", 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7081e-05 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.276436+0000", 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021024669999999998 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:25:22.276489+0000", 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3566000000000002e-05 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:26:51.251 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:26:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:51.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:51.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:51.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:51.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:51.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:51.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:51.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:51 smithi151 ceph-mon[28965]: pgmap v1102: 27 pgs: 27 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-06-27T21:26:51.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:51 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:26:51.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:51 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:26:51.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:51.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:51.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:51.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:51.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:51.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:51 smithi190 ceph-mon[32247]: pgmap v1102: 27 pgs: 27 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-06-27T21:26:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:51 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:26:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:51 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:26:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:51 smithi192 ceph-mon[32956]: pgmap v1102: 27 pgs: 27 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-06-27T21:26:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:51 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:26:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:51 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:26:52.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:52.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:52.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:52.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:52.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:52.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:52.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:52.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:52.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:52.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:52.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:52.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:52.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:52.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:52.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:52 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:26:52.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:52 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:26:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:26:52.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:52.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:52.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:52 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:26:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:52 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:26:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:26:53.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:52 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:26:53.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:52 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:26:53.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:26:53.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:53.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:53.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:53.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:53.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:53.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:53.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:53.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:53.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:53.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:53.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:53.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:53.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:53.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:53.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:53 smithi151 ceph-mon[28965]: pgmap v1103: 27 pgs: 27 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-06-27T21:26:53.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:53 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:26:53.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:53 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:26:53.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:53.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:53.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:53 smithi190 ceph-mon[32247]: pgmap v1103: 27 pgs: 27 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-06-27T21:26:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:53 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:26:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:53 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:26:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:53 smithi192 ceph-mon[32956]: pgmap v1103: 27 pgs: 27 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-06-27T21:26:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:53 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:26:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:53 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:26:54.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:26:54.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:54.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:54.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:54.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:54.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:54.365 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:26:54.366 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:26:54.366 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:26:54.367 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:26:54.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:54.400 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:26:54.405 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:26:54.405 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:26:54.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:26:54.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:26:54.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:26:54.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:26:54.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:54.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:26:54.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:26:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:54.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:54.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:26:54.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:26:54.564 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:26:54.564 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:26:54.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:26:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:26:54.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:54.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:26:54.629 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:26:54.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:54.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:26:54.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:26:54.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:26:54.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:26:54.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:54.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:54.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: osdmap e421: 12 total, 12 up, 12 in 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1245771856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:26:54.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: osdmap e421: 12 total, 12 up, 12 in 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1245771856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:26:54.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:54.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:26:54.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:54.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:54.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:55.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:55.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:55.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:26:55.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: osdmap e421: 12 total, 12 up, 12 in 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1245771856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:26:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:26:55.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:55.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:55.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:55.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:55.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:55.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:55.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:55.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:55.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:55.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:55.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:55.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:55.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:55.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:55.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:56.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:56.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:56.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:56.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:56.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:56.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:56.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:56.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:56.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:56.628 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:56.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:56.753 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:56.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:56.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:56.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:56.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:57.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:57.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:57.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:57.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:57.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:57.336 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:57 smithi190 ceph-mon[32247]: pgmap v1106: 27 pgs: 27 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-06-27T21:26:57.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:57.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:57.431 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:26:57.431 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:26:57.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:57 smithi151 ceph-mon[28965]: pgmap v1106: 27 pgs: 27 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-06-27T21:26:57.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:57 smithi192 ceph-mon[32956]: pgmap v1106: 27 pgs: 27 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-06-27T21:26:57.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:57.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:57.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:57.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:57.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:57.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:57.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:57.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:58.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:26:58.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:26:58.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:26:58.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:26:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:58.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:26:58.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:26:58.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:26:58.324 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:26:58.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:58.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:58.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:58 smithi151 ceph-mon[28965]: osdmap e422: 12 total, 12 up, 12 in 2024-06-27T21:26:58.325 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:26:58.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:26:58.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:58.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:58 smithi190 ceph-mon[32247]: osdmap e422: 12 total, 12 up, 12 in 2024-06-27T21:26:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:58 smithi192 ceph-mon[32956]: osdmap e422: 12 total, 12 up, 12 in 2024-06-27T21:26:58.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:26:58.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:26:58.646 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:26:58.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:26:58.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:26:58.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:58.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:58.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:58.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:58.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:59.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:26:59.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:26:59.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:26:59.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:26:59.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:59 smithi151 ceph-mon[28965]: pgmap v1108: 27 pgs: 27 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s 2024-06-27T21:26:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:59 smithi151 ceph-mon[28965]: osdmap e423: 12 total, 12 up, 12 in 2024-06-27T21:26:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:26:59 smithi151 ceph-mon[28965]: from='client.16056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:59.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:26:59.226 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1109,"stamp":"2024-06-27T21:26:58.102351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5477564984,"num_objects":83575,"num_object_clones":0,"num_object_copies":250725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83575,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128505,"num_write_kb":6786433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131420,"ondisk_log_size":131420,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19753148,"kb_used_data":16130668,"kb_used_omap":110,"kb_used_meta":3621777,"kb_avail":1105188676,"statfs":{"total":1151940427776,"available":1131713204224,"internally_reserved":0,"allocated":16517804032,"data_stored":16514665766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":3708700631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-373161984,"num_objects":-5694,"num_object_clones":0,"num_object_copies":-17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5694,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.071757"},"pg_stats":[{"pgid":"5.f","version":"422'8316","reported_seq":8465,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-06-27T21:26:57.752674+0000","last_change":"2024-06-27T21:26:02.177610+0000","last_active":"2024-06-27T21:26:57.752674+0000","last_peered":"2024-06-27T21:26:57.752674+0000","last_clean":"2024-06-27T21:26:57.752674+0000","last_became_active":"2024-06-27T21:24:48.707635+0000","last_became_peered":"2024-06-27T21:24:48.707635+0000","last_unstale":"2024-06-27T21:26:57.752674+0000","last_undegraded":"2024-06-27T21:26:57.752674+0000","last_fullsized":"2024-06-27T21:26:57.752674+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'4999","last_scrub_stamp":"2024-06-27T21:26:02.177486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:02.177486+0000","objects_scrubbed":4951,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:15.211969+0000","scrub_duration":1413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341966848,"num_objects":5218,"num_object_clones":0,"num_object_copies":15654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8316,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"421'8056","reported_seq":8195,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:55.231356+0000","last_change":"2024-06-27T21:25:54.256382+0000","last_active":"2024-06-27T21:26:55.231356+0000","last_peered":"2024-06-27T21:26:55.231356+0000","last_clean":"2024-06-27T21:26:55.231356+0000","last_became_active":"2024-06-27T21:24:48.708225+0000","last_became_peered":"2024-06-27T21:24:48.708225+0000","last_unstale":"2024-06-27T21:26:55.231356+0000","last_undegraded":"2024-06-27T21:26:55.231356+0000","last_fullsized":"2024-06-27T21:26:55.231356+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'4335","last_scrub_stamp":"2024-06-27T21:25:54.256168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:54.256168+0000","objects_scrubbed":4294,"log_size":8056,"log_dups_size":0,"ondisk_log_size":8056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:16.380086+0000","scrub_duration":869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346816512,"num_objects":5292,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":427136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"420'7896","reported_seq":8036,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.225913+0000","last_change":"2024-06-27T21:25:58.409829+0000","last_active":"2024-06-27T21:26:53.225913+0000","last_peered":"2024-06-27T21:26:53.225913+0000","last_clean":"2024-06-27T21:26:53.225913+0000","last_became_active":"2024-06-27T21:24:48.708150+0000","last_became_peered":"2024-06-27T21:24:48.708150+0000","last_unstale":"2024-06-27T21:26:53.225913+0000","last_undegraded":"2024-06-27T21:26:53.225913+0000","last_fullsized":"2024-06-27T21:26:53.225913+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4607","last_scrub_stamp":"2024-06-27T21:25:58.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:58.409408+0000","objects_scrubbed":4559,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:22.763094+0000","scrub_duration":1066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356646912,"num_objects":5442,"num_object_clones":0,"num_object_copies":16326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7896,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"421'7993","reported_seq":8144,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:54.411788+0000","last_change":"2024-06-27T21:26:11.531967+0000","last_active":"2024-06-27T21:26:54.411788+0000","last_peered":"2024-06-27T21:26:54.411788+0000","last_clean":"2024-06-27T21:26:54.411788+0000","last_became_active":"2024-06-27T21:24:48.708687+0000","last_became_peered":"2024-06-27T21:24:48.708687+0000","last_unstale":"2024-06-27T21:26:54.411788+0000","last_undegraded":"2024-06-27T21:26:54.411788+0000","last_fullsized":"2024-06-27T21:26:54.411788+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5559","last_scrub_stamp":"2024-06-27T21:26:11.531764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:11.531764+0000","objects_scrubbed":5513,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:26.815955+0000","scrub_duration":1548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348848128,"num_objects":5323,"num_object_clones":0,"num_object_copies":15969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":426112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"421'7967","reported_seq":8107,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:54.592184+0000","last_change":"2024-06-27T21:25:57.075916+0000","last_active":"2024-06-27T21:26:54.592184+0000","last_peered":"2024-06-27T21:26:54.592184+0000","last_clean":"2024-06-27T21:26:54.592184+0000","last_became_active":"2024-06-27T21:24:48.708111+0000","last_became_peered":"2024-06-27T21:24:48.708111+0000","last_unstale":"2024-06-27T21:26:54.592184+0000","last_undegraded":"2024-06-27T21:26:54.592184+0000","last_fullsized":"2024-06-27T21:26:54.592184+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4478","last_scrub_stamp":"2024-06-27T21:25:57.075739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:57.075739+0000","objects_scrubbed":4444,"log_size":7967,"log_dups_size":0,"ondisk_log_size":7967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:19.549115+0000","scrub_duration":1535,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349372416,"num_objects":5331,"num_object_clones":0,"num_object_copies":15993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.4","version":"421'8068","reported_seq":8212,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:54.622460+0000","last_change":"2024-06-27T21:26:03.238316+0000","last_active":"2024-06-27T21:26:54.622460+0000","last_peered":"2024-06-27T21:26:54.622460+0000","last_clean":"2024-06-27T21:26:54.622460+0000","last_became_active":"2024-06-27T21:24:48.709413+0000","last_became_peered":"2024-06-27T21:24:48.709413+0000","last_unstale":"2024-06-27T21:26:54.622460+0000","last_undegraded":"2024-06-27T21:26:54.622460+0000","last_fullsized":"2024-06-27T21:26:54.622460+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'4970","last_scrub_stamp":"2024-06-27T21:26:03.238159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:03.238159+0000","objects_scrubbed":4924,"log_size":8068,"log_dups_size":0,"ondisk_log_size":8068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:15.739270+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342491136,"num_objects":5226,"num_object_clones":0,"num_object_copies":15678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8068,"num_write_kb":425408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":880,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.602814+0000","last_change":"2024-06-27T21:26:52.781849+0000","last_active":"2024-06-27T21:26:53.602814+0000","last_peered":"2024-06-27T21:26:53.602814+0000","last_clean":"2024-06-27T21:26:53.602814+0000","last_became_active":"2024-06-27T21:14:53.704228+0000","last_became_peered":"2024-06-27T21:14:53.704228+0000","last_unstale":"2024-06-27T21:26:53.602814+0000","last_undegraded":"2024-06-27T21:26:53.602814+0000","last_fullsized":"2024-06-27T21:26:53.602814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:52.781728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:52.781728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:28:09.792381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"422'8106","reported_seq":8261,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-06-27T21:26:57.706403+0000","last_change":"2024-06-27T21:26:13.463329+0000","last_active":"2024-06-27T21:26:57.706403+0000","last_peered":"2024-06-27T21:26:57.706403+0000","last_clean":"2024-06-27T21:26:57.706403+0000","last_became_active":"2024-06-27T21:24:48.885413+0000","last_became_peered":"2024-06-27T21:24:48.885413+0000","last_unstale":"2024-06-27T21:26:57.706403+0000","last_undegraded":"2024-06-27T21:26:57.706403+0000","last_fullsized":"2024-06-27T21:26:57.706403+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5601","last_scrub_stamp":"2024-06-27T21:26:13.463234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:13.463234+0000","objects_scrubbed":5547,"log_size":8106,"log_dups_size":0,"ondisk_log_size":8106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:21.762213+0000","scrub_duration":1698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329383936,"num_objects":5026,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8106,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.608026+0000","last_change":"2024-06-27T21:26:52.953192+0000","last_active":"2024-06-27T21:26:53.608026+0000","last_peered":"2024-06-27T21:26:53.608026+0000","last_clean":"2024-06-27T21:26:53.608026+0000","last_became_active":"2024-06-27T21:23:44.524342+0000","last_became_peered":"2024-06-27T21:23:44.524342+0000","last_unstale":"2024-06-27T21:26:53.608026+0000","last_undegraded":"2024-06-27T21:26:53.608026+0000","last_fullsized":"2024-06-27T21:26:53.608026+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:52.953082+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:26:52.953082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:28:01.646736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"420'7878","reported_seq":8018,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.277043+0000","last_change":"2024-06-27T21:25:58.105417+0000","last_active":"2024-06-27T21:26:53.277043+0000","last_peered":"2024-06-27T21:26:53.277043+0000","last_clean":"2024-06-27T21:26:53.277043+0000","last_became_active":"2024-06-27T21:24:48.710086+0000","last_became_peered":"2024-06-27T21:24:48.710086+0000","last_unstale":"2024-06-27T21:26:53.277043+0000","last_undegraded":"2024-06-27T21:26:53.277043+0000","last_fullsized":"2024-06-27T21:26:53.277043+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'4594","last_scrub_stamp":"2024-06-27T21:25:58.105207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:25:58.105207+0000","objects_scrubbed":4550,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:09.223215+0000","scrub_duration":1746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355205120,"num_objects":5420,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":885,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.602842+0000","last_change":"2024-06-27T21:25:07.953798+0000","last_active":"2024-06-27T21:26:53.602842+0000","last_peered":"2024-06-27T21:26:53.602842+0000","last_clean":"2024-06-27T21:26:53.602842+0000","last_became_active":"2024-06-27T21:23:44.529886+0000","last_became_peered":"2024-06-27T21:23:44.529886+0000","last_unstale":"2024-06-27T21:26:53.602842+0000","last_undegraded":"2024-06-27T21:26:53.602842+0000","last_fullsized":"2024-06-27T21:26:53.602842+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:07.953724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:07.953724+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"421'7935","reported_seq":8088,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:56.000959+0000","last_change":"2024-06-27T21:26:15.938541+0000","last_active":"2024-06-27T21:26:56.000959+0000","last_peered":"2024-06-27T21:26:56.000959+0000","last_clean":"2024-06-27T21:26:56.000959+0000","last_became_active":"2024-06-27T21:24:48.885372+0000","last_became_peered":"2024-06-27T21:24:48.885372+0000","last_unstale":"2024-06-27T21:26:56.000959+0000","last_undegraded":"2024-06-27T21:26:56.000959+0000","last_fullsized":"2024-06-27T21:26:56.000959+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5629","last_scrub_stamp":"2024-06-27T21:26:15.938322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:15.938322+0000","objects_scrubbed":5594,"log_size":7935,"log_dups_size":0,"ondisk_log_size":7935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:38.993790+0000","scrub_duration":1216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331415552,"num_objects":5057,"num_object_clones":0,"num_object_copies":15171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":415744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":287,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.603880+0000","last_change":"2024-06-27T21:26:51.114393+0000","last_active":"2024-06-27T21:26:53.603880+0000","last_peered":"2024-06-27T21:26:53.603880+0000","last_clean":"2024-06-27T21:26:53.603880+0000","last_became_active":"2024-06-27T21:21:18.602559+0000","last_became_peered":"2024-06-27T21:21:18.602559+0000","last_unstale":"2024-06-27T21:26:53.603880+0000","last_undegraded":"2024-06-27T21:26:53.603880+0000","last_fullsized":"2024-06-27T21:26:53.603880+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:51.114285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:51.114285+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:28:19.109466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"421'8070","reported_seq":8224,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:56.001195+0000","last_change":"2024-06-27T21:26:19.240429+0000","last_active":"2024-06-27T21:26:56.001195+0000","last_peered":"2024-06-27T21:26:56.001195+0000","last_clean":"2024-06-27T21:26:56.001195+0000","last_became_active":"2024-06-27T21:24:48.710822+0000","last_became_peered":"2024-06-27T21:24:48.710822+0000","last_unstale":"2024-06-27T21:26:56.001195+0000","last_undegraded":"2024-06-27T21:26:56.001195+0000","last_fullsized":"2024-06-27T21:26:56.001195+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5920","last_scrub_stamp":"2024-06-27T21:26:19.240187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.240187+0000","objects_scrubbed":5882,"log_size":8070,"log_dups_size":0,"ondisk_log_size":8070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:33.458729+0000","scrub_duration":1408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332005376,"num_objects":5066,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":420352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":614,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.607846+0000","last_change":"2024-06-27T21:26:22.770901+0000","last_active":"2024-06-27T21:26:53.607846+0000","last_peered":"2024-06-27T21:26:53.607846+0000","last_clean":"2024-06-27T21:26:53.607846+0000","last_became_active":"2024-06-27T21:21:18.602789+0000","last_became_peered":"2024-06-27T21:21:18.602789+0000","last_unstale":"2024-06-27T21:26:53.607846+0000","last_undegraded":"2024-06-27T21:26:53.607846+0000","last_fullsized":"2024-06-27T21:26:53.607846+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:26:22.770782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:31.213714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"421'7962","reported_seq":8110,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:55.574683+0000","last_change":"2024-06-27T21:26:11.195361+0000","last_active":"2024-06-27T21:26:55.574683+0000","last_peered":"2024-06-27T21:26:55.574683+0000","last_clean":"2024-06-27T21:26:55.574683+0000","last_became_active":"2024-06-27T21:24:48.708618+0000","last_became_peered":"2024-06-27T21:24:48.708618+0000","last_unstale":"2024-06-27T21:26:55.574683+0000","last_undegraded":"2024-06-27T21:26:55.574683+0000","last_fullsized":"2024-06-27T21:26:55.574683+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'5451","last_scrub_stamp":"2024-06-27T21:26:11.195038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:11.195038+0000","objects_scrubbed":5384,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:29.760754+0000","scrub_duration":1547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339738624,"num_objects":5184,"num_object_clones":0,"num_object_copies":15552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7962,"num_write_kb":420672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":287,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.603945+0000","last_change":"2024-06-27T21:26:53.115946+0000","last_active":"2024-06-27T21:26:53.603945+0000","last_peered":"2024-06-27T21:26:53.603945+0000","last_clean":"2024-06-27T21:26:53.603945+0000","last_became_active":"2024-06-27T21:21:18.602398+0000","last_became_peered":"2024-06-27T21:21:18.602398+0000","last_unstale":"2024-06-27T21:26:53.603945+0000","last_undegraded":"2024-06-27T21:26:53.603945+0000","last_fullsized":"2024-06-27T21:26:53.603945+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:53.115844+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:26:53.115844+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:53.446216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"422'8132","reported_seq":8285,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-06-27T21:26:57.840718+0000","last_change":"2024-06-27T21:26:09.408114+0000","last_active":"2024-06-27T21:26:57.840718+0000","last_peered":"2024-06-27T21:26:57.840718+0000","last_clean":"2024-06-27T21:26:57.840718+0000","last_became_active":"2024-06-27T21:24:48.709002+0000","last_became_peered":"2024-06-27T21:24:48.709002+0000","last_unstale":"2024-06-27T21:26:57.840718+0000","last_undegraded":"2024-06-27T21:26:57.840718+0000","last_fullsized":"2024-06-27T21:26:57.840718+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'5325","last_scrub_stamp":"2024-06-27T21:26:09.407899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:09.407899+0000","objects_scrubbed":5258,"log_size":8132,"log_dups_size":0,"ondisk_log_size":8132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:36.464873+0000","scrub_duration":1191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327811072,"num_objects":5002,"num_object_clones":0,"num_object_copies":15006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":275,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.603916+0000","last_change":"2024-06-27T21:26:50.122844+0000","last_active":"2024-06-27T21:26:53.603916+0000","last_peered":"2024-06-27T21:26:53.603916+0000","last_clean":"2024-06-27T21:26:53.603916+0000","last_became_active":"2024-06-27T21:21:34.494291+0000","last_became_peered":"2024-06-27T21:21:34.494291+0000","last_unstale":"2024-06-27T21:26:53.603916+0000","last_undegraded":"2024-06-27T21:26:53.603916+0000","last_fullsized":"2024-06-27T21:26:53.603916+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:50.122401+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:26:50.122401+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:28:05.562001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"421'7906","reported_seq":8061,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:54.628643+0000","last_change":"2024-06-27T21:26:19.252916+0000","last_active":"2024-06-27T21:26:54.628643+0000","last_peered":"2024-06-27T21:26:54.628643+0000","last_clean":"2024-06-27T21:26:54.628643+0000","last_became_active":"2024-06-27T21:24:48.885324+0000","last_became_peered":"2024-06-27T21:24:48.885324+0000","last_unstale":"2024-06-27T21:26:54.628643+0000","last_undegraded":"2024-06-27T21:26:54.628643+0000","last_fullsized":"2024-06-27T21:26:54.628643+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5908","last_scrub_stamp":"2024-06-27T21:26:19.252716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.252716+0000","objects_scrubbed":5869,"log_size":7906,"log_dups_size":0,"ondisk_log_size":7906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:22.654032+0000","scrub_duration":1421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335020032,"num_objects":5112,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7906,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":688,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.599395+0000","last_change":"2024-06-27T21:26:08.543233+0000","last_active":"2024-06-27T21:26:53.599395+0000","last_peered":"2024-06-27T21:26:53.599395+0000","last_clean":"2024-06-27T21:26:53.599395+0000","last_became_active":"2024-06-27T21:14:55.709807+0000","last_became_peered":"2024-06-27T21:14:55.709807+0000","last_unstale":"2024-06-27T21:26:53.599395+0000","last_undegraded":"2024-06-27T21:26:53.599395+0000","last_fullsized":"2024-06-27T21:26:53.599395+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:08.543015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:08.543015+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:23.153986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.e","version":"421'8092","reported_seq":8247,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:55.193230+0000","last_change":"2024-06-27T21:26:16.853366+0000","last_active":"2024-06-27T21:26:55.193230+0000","last_peered":"2024-06-27T21:26:55.193230+0000","last_clean":"2024-06-27T21:26:55.193230+0000","last_became_active":"2024-06-27T21:24:48.707990+0000","last_became_peered":"2024-06-27T21:24:48.707990+0000","last_unstale":"2024-06-27T21:26:55.193230+0000","last_undegraded":"2024-06-27T21:26:55.193230+0000","last_fullsized":"2024-06-27T21:26:55.193230+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5897","last_scrub_stamp":"2024-06-27T21:26:16.853222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:16.853222+0000","objects_scrubbed":5847,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:34.301452+0000","scrub_duration":1472,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346685440,"num_objects":5290,"num_object_clones":0,"num_object_copies":15870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8092,"num_write_kb":428224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":594,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.610963+0000","last_change":"2024-06-27T21:26:50.130517+0000","last_active":"2024-06-27T21:26:53.610963+0000","last_peered":"2024-06-27T21:26:53.610963+0000","last_clean":"2024-06-27T21:26:53.610963+0000","last_became_active":"2024-06-27T21:23:44.529707+0000","last_became_peered":"2024-06-27T21:23:44.529707+0000","last_unstale":"2024-06-27T21:26:53.610963+0000","last_undegraded":"2024-06-27T21:26:53.610963+0000","last_fullsized":"2024-06-27T21:26:53.610963+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:50.130443+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:26:50.130443+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:56.575564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"421'7947","reported_seq":8103,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:54.482015+0000","last_change":"2024-06-27T21:26:19.426445+0000","last_active":"2024-06-27T21:26:54.482015+0000","last_peered":"2024-06-27T21:26:54.482015+0000","last_clean":"2024-06-27T21:26:54.482015+0000","last_became_active":"2024-06-27T21:24:48.955695+0000","last_became_peered":"2024-06-27T21:24:48.955695+0000","last_unstale":"2024-06-27T21:26:54.482015+0000","last_undegraded":"2024-06-27T21:26:54.482015+0000","last_fullsized":"2024-06-27T21:26:54.482015+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'5936","last_scrub_stamp":"2024-06-27T21:26:19.426355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:19.426355+0000","objects_scrubbed":5918,"log_size":7947,"log_dups_size":0,"ondisk_log_size":7947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:41.889854+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343670808,"num_objects":5245,"num_object_clones":0,"num_object_copies":15735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7947,"num_write_kb":422081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":896,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:53.603474+0000","last_change":"2024-06-27T21:26:14.033864+0000","last_active":"2024-06-27T21:26:53.603474+0000","last_peered":"2024-06-27T21:26:53.603474+0000","last_clean":"2024-06-27T21:26:53.603474+0000","last_became_active":"2024-06-27T21:21:34.485764+0000","last_became_peered":"2024-06-27T21:21:34.485764+0000","last_unstale":"2024-06-27T21:26:53.603474+0000","last_undegraded":"2024-06-27T21:26:53.603474+0000","last_fullsized":"2024-06-27T21:26:53.603474+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:26:14.033749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:26:14.033749+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:38.288649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"421'11985","reported_seq":12365,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-06-27T21:26:55.996025+0000","last_change":"2024-06-27T21:26:49.566978+0000","last_active":"2024-06-27T21:26:55.996025+0000","last_peered":"2024-06-27T21:26:55.996025+0000","last_clean":"2024-06-27T21:26:55.996025+0000","last_became_active":"2024-06-27T21:24:48.885320+0000","last_became_peered":"2024-06-27T21:24:48.885320+0000","last_unstale":"2024-06-27T21:26:55.996025+0000","last_undegraded":"2024-06-27T21:26:55.996025+0000","last_fullsized":"2024-06-27T21:26:55.996025+0000","mapping_epoch":382,"log_start":"392'1901","ondisk_log_start":"392'1901","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'11584","last_scrub_stamp":"2024-06-27T21:26:49.566877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:24:47.689585+0000","last_clean_scrub_stamp":"2024-06-27T21:26:49.566877+0000","objects_scrubbed":5779,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:49.647178+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349896704,"num_objects":5339,"num_object_clones":0,"num_object_copies":16017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":432640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":621,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-06-27T21:26:57.098903+0000","last_change":"2024-06-27T21:25:54.786735+0000","last_active":"2024-06-27T21:26:57.098903+0000","last_peered":"2024-06-27T21:26:57.098903+0000","last_clean":"2024-06-27T21:26:57.098903+0000","last_became_active":"2024-06-27T21:21:34.487122+0000","last_became_peered":"2024-06-27T21:21:34.487122+0000","last_unstale":"2024-06-27T21:26:57.098903+0000","last_undegraded":"2024-06-27T21:26:57.098903+0000","last_fullsized":"2024-06-27T21:26:57.098903+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:25:54.786656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:25:54.786656+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:27:12.608584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5476974616,"num_objects":83573,"num_object_clones":0,"num_object_copies":250719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128505,"num_write_kb":6786433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16374968320,"data_stored":16374956104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130408,"ondisk_log_size":130408,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253075,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1798796,"kb_used_data":1376008,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":91946356,"statfs":{"total":95995035648,"available":94153068544,"internally_reserved":0,"allocated":1409032192,"data_stored":1408774741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514712,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1971532,"kb_used_data":1691844,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91773620,"statfs":{"total":95995035648,"available":93976186880,"internally_reserved":0,"allocated":1732448256,"data_stored":1732194877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":286390720},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743644,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1333644,"kb_used_data":1009540,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92411508,"statfs":{"total":95995035648,"available":94629384192,"internally_reserved":0,"allocated":1033768960,"data_stored":1033515581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972577,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":868044,"kb_used_data":670276,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92877108,"statfs":{"total":95995035648,"available":95106158592,"internally_reserved":0,"allocated":686362624,"data_stored":686109245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266917,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1723084,"kb_used_data":1362760,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92022068,"statfs":{"total":95995035648,"available":94230597632,"internally_reserved":0,"allocated":1395466240,"data_stored":1395208789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":368835018},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1427208,"kb_used_data":1003012,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92317944,"statfs":{"total":95995035648,"available":94533574656,"internally_reserved":0,"allocated":1027084288,"data_stored":1026830909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757489,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2549656,"kb_used_data":2301904,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91195496,"statfs":{"total":95995035648,"available":93384187904,"internally_reserved":0,"allocated":2357149696,"data_stored":2356888693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161192,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1760524,"kb_used_data":1637060,"kb_used_omap":92,"kb_used_meta":123363,"kb_avail":91984628,"statfs":{"total":95995035648,"available":94192259072,"internally_reserved":0,"allocated":1676349440,"data_stored":1676096061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":126323810},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182649,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637972,"kb_used_data":323596,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":93107180,"statfs":{"total":95995035648,"available":95341752320,"internally_reserved":0,"allocated":331362304,"data_stored":331105373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":321911239},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411581,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1223744,"kb_used_data":1044024,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92521408,"statfs":{"total":95995035648,"available":94741921792,"internally_reserved":0,"allocated":1069080576,"data_stored":1068786222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1752312,"kb_used_data":1360176,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91992840,"statfs":{"total":95995035648,"available":94200668160,"internally_reserved":0,"allocated":1392820224,"data_stored":1392529422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738631,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2706632,"kb_used_data":2350468,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91038520,"statfs":{"total":95995035648,"available":93223444480,"internally_reserved":0,"allocated":2406879232,"data_stored":2406625853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2382757888,"data_stored":2382757888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1380974592,"data_stored":1380974592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1058996224,"data_stored":1058996224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":329056256,"data_stored":329056256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1658191872,"data_stored":1658191872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2333151232,"data_stored":2333147160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1017577472,"data_stored":1017577472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1385566208,"data_stored":1385562136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":679149568,"data_stored":679149568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1032716288,"data_stored":1032716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1722220544,"data_stored":1722220544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1394610176,"data_stored":1394606104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:26:59.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-06-27T21:26:59.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:26:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:26:59.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:59.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:59.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:26:59.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:26:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:26:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:26:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:26:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:59 smithi190 ceph-mon[32247]: pgmap v1108: 27 pgs: 27 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s 2024-06-27T21:26:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:59 smithi190 ceph-mon[32247]: osdmap e423: 12 total, 12 up, 12 in 2024-06-27T21:26:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:26:59 smithi190 ceph-mon[32247]: from='client.16056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:26:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:26:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:59 smithi192 ceph-mon[32956]: pgmap v1108: 27 pgs: 27 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s 2024-06-27T21:26:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:59 smithi192 ceph-mon[32956]: osdmap e423: 12 total, 12 up, 12 in 2024-06-27T21:26:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:26:59 smithi192 ceph-mon[32956]: from='client.16056 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:26:59.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:26:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:59.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:26:59.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:26:59.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:26:59.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:26:59.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:26:59.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:26:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:26:59.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:26:59.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:00.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:00.085 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:00.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:00.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:00.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:00.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:00.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:00.405 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:00.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:00.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:00.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:00.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:00.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:00.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:00.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:00.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:00.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:01.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:01.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:01.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:01.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:01.194 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:01.194 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:01.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:01.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:01.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:01.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:01.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:01.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:01.634 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:01.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:01.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:01.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:01 smithi151 ceph-mon[28965]: pgmap v1110: 27 pgs: 27 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-06-27T21:27:01.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:01.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:01.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:01 smithi190 ceph-mon[32247]: pgmap v1110: 27 pgs: 27 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-06-27T21:27:01.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:01.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:27:01.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:01.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:01 smithi192 ceph-mon[32956]: pgmap v1110: 27 pgs: 27 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-06-27T21:27:02.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:02.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:02.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:02.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:02.280 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:02.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:02.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:02.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:02.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:02.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:02.587 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:27:02.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:02.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1290632959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:02.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:02.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:02.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:02.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:02.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1290632959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:02.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:02.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:03.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1290632959' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:03.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:03.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:03.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:03.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:03.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:03.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:03.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:03.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:03.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:03.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:03.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:03.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:03.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:27:03.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: pgmap v1111: 27 pgs: 27 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-27T21:27:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:27:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:27:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:27:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1290632959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:03 smithi190 ceph-mon[32247]: osdmap e424: 12 total, 12 up, 12 in 2024-06-27T21:27:03.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:03.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:03.892 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:27:03.892 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: pgmap v1111: 27 pgs: 27 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-27T21:27:03.892 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:27:03.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:27:03.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:27:03.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1290632959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:03.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:03 smithi151 ceph-mon[28965]: osdmap e424: 12 total, 12 up, 12 in 2024-06-27T21:27:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:03.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:03.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:04.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:27:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: pgmap v1111: 27 pgs: 27 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-06-27T21:27:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:27:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:27:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:27:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1290632959' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:03 smithi192 ceph-mon[32956]: osdmap e424: 12 total, 12 up, 12 in 2024-06-27T21:27:04.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:04.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:04.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:04.249 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:04.249 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:04.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:04.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:04.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:04.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:04.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:04.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:04.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:04.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:04.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:04.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:04.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:04.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:04.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:04.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:27:05.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:05.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:05.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:05.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:05.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:05.362 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:05.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:05.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:05.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:05.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:05.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:05.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:05.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:05 smithi190 ceph-mon[32247]: pgmap v1113: 27 pgs: 27 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s 2024-06-27T21:27:05.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:27:05.824 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:27:05.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:05.854 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:27:05.855 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:27:05.855 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:05 smithi151 ceph-mon[28965]: pgmap v1113: 27 pgs: 27 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s 2024-06-27T21:27:05.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:05.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:05.888 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:27:05.889 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:27:05.920 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:27:05.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:27:05.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:27:05.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:27:05.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:05.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:27:05.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:27:06.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:06.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:27:06.021 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:27:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:05 smithi192 ceph-mon[32956]: pgmap v1113: 27 pgs: 27 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s 2024-06-27T21:27:06.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:27:06.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:27:06.080 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:27:06.081 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:27:06.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:06.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:27:06.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:27:06.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:06.143 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:27:06.143 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:27:06.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:27:06.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:27:06.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:06.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:06.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:06.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:06.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:27:06.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:06.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:06.609 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 5 pgp_num to 16 2024-06-27T21:27:06.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:06.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:06.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:06.785 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:06.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:06 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:06.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:06 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:06.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:06 smithi151 ceph-mon[28965]: osdmap e425: 12 total, 12 up, 12 in 2024-06-27T21:27:06.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/23364388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:06.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2623044172' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:27:06.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:06 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:27:06.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:06.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:06 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:06.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:06 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:06.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:06 smithi190 ceph-mon[32247]: osdmap e425: 12 total, 12 up, 12 in 2024-06-27T21:27:06.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/23364388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:06.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2623044172' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:27:06.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:06 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:27:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:06.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:06.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:07.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:07.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:07.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:06 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:07.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:06 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:07.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:06 smithi192 ceph-mon[32956]: osdmap e425: 12 total, 12 up, 12 in 2024-06-27T21:27:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/23364388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2623044172' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:27:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:06 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:27:07.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:07.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:07.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:07.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:07.343 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:27:07.343 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:27:07.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.343 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:139913 5.2 5:442182a0:::benchmark_data_smithi192_104167_object33872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-06-27T21:27:07.343 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:07.320206+0000", 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.022660573, 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.022692497999999998, 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 139913 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.320206+0000", 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.320206+0000", 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.320207+0000", 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1480000000000001e-06 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.320215+0000", 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9750000000000007e-06 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.320229+0000", 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4226000000000001e-05 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.320236+0000", 2024-06-27T21:27:07.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5039999999999996e-06 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.323295+0000", 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003058304 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.323420+0000", 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000125153 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,10", 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.323534+0000", 2024-06-27T21:27:07.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113848 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.331030+0000", 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007496015 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.331071+0000", 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1059000000000001e-05 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:07.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:139933 5.3 e426/382)", 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:07.342870+0000", 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4294967295.9999962, 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100035, 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.342870+0000", 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.342870+0000", 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.342871+0000", 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7519999999999999e-06 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.342879+0000", 2024-06-27T21:27:07.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6490000000000008e-06 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.342920+0000", 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1224999999999999e-05 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:07.342928+0000", 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7430000000000002e-06 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:07.351 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-06-27T21:27:07.352 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 2 2024-06-27T21:27:07.352 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:27:07.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:07.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:07.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:07.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:07.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:07.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:07.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:07.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:07 smithi151 ceph-mon[28965]: pgmap v1115: 27 pgs: 27 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-06-27T21:27:07.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:07 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:27:07.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:07 smithi151 ceph-mon[28965]: osdmap e426: 12 total, 12 up, 12 in 2024-06-27T21:27:07.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:07 smithi151 ceph-mon[28965]: 5.f scrub starts 2024-06-27T21:27:07.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:07.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:07.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:07.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:07.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:07.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:07 smithi190 ceph-mon[32247]: pgmap v1115: 27 pgs: 27 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-06-27T21:27:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:07 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:27:08.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:07 smithi190 ceph-mon[32247]: osdmap e426: 12 total, 12 up, 12 in 2024-06-27T21:27:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:07 smithi190 ceph-mon[32247]: 5.f scrub starts 2024-06-27T21:27:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:07 smithi192 ceph-mon[32956]: pgmap v1115: 27 pgs: 27 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s 2024-06-27T21:27:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:07 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:27:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:07 smithi192 ceph-mon[32956]: osdmap e426: 12 total, 12 up, 12 in 2024-06-27T21:27:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:07 smithi192 ceph-mon[32956]: 5.f scrub starts 2024-06-27T21:27:08.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:08.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:27:08.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:08.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:08.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:08.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:08.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:08.409 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:08.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:08.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:08.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:08.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:08.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:08.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:08.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:08.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:08.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:08 smithi190 ceph-mon[32247]: osdmap e427: 12 total, 12 up, 12 in 2024-06-27T21:27:08.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:08 smithi190 ceph-mon[32247]: 5.8 scrub starts 2024-06-27T21:27:08.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:08 smithi190 ceph-mon[32247]: 5.a deep-scrub starts 2024-06-27T21:27:08.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:08.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:08.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:27:08.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:08.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:08.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:08 smithi151 ceph-mon[28965]: osdmap e427: 12 total, 12 up, 12 in 2024-06-27T21:27:08.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:08 smithi151 ceph-mon[28965]: 5.8 scrub starts 2024-06-27T21:27:08.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:08 smithi151 ceph-mon[28965]: 5.a deep-scrub starts 2024-06-27T21:27:08.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:08.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:08.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:08 smithi192 ceph-mon[32956]: osdmap e427: 12 total, 12 up, 12 in 2024-06-27T21:27:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:08 smithi192 ceph-mon[32956]: 5.8 scrub starts 2024-06-27T21:27:09.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:08 smithi192 ceph-mon[32956]: 5.a deep-scrub starts 2024-06-27T21:27:09.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:09.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:09.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:09.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:09.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:09.259 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:09.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:09.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:09.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:09.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:09.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:09.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:09.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:09.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:09.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:09 smithi151 ceph-mon[28965]: 5.f scrub ok 2024-06-27T21:27:09.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:09 smithi151 ceph-mon[28965]: pgmap v1118: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-06-27T21:27:09.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:09 smithi151 ceph-mon[28965]: 5.b scrub starts 2024-06-27T21:27:09.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:09 smithi151 ceph-mon[28965]: 5.8 scrub ok 2024-06-27T21:27:09.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:09 smithi151 ceph-mon[28965]: 5.b scrub ok 2024-06-27T21:27:09.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:09 smithi190 ceph-mon[32247]: 5.f scrub ok 2024-06-27T21:27:09.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:09 smithi190 ceph-mon[32247]: pgmap v1118: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-06-27T21:27:09.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:09 smithi190 ceph-mon[32247]: 5.b scrub starts 2024-06-27T21:27:09.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:09 smithi190 ceph-mon[32247]: 5.8 scrub ok 2024-06-27T21:27:09.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:09 smithi190 ceph-mon[32247]: 5.b scrub ok 2024-06-27T21:27:09.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:09.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:09.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:10.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:10.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:09 smithi192 ceph-mon[32956]: 5.f scrub ok 2024-06-27T21:27:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:09 smithi192 ceph-mon[32956]: pgmap v1118: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 4.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-06-27T21:27:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:09 smithi192 ceph-mon[32956]: 5.b scrub starts 2024-06-27T21:27:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:09 smithi192 ceph-mon[32956]: 5.8 scrub ok 2024-06-27T21:27:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:09 smithi192 ceph-mon[32956]: 5.b scrub ok 2024-06-27T21:27:10.123 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:27:10.123 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:27:10.123 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:27:10.123 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:27:10.123 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:27:10.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:10.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:10.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:10.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:10.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:10.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:10.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:10.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:10.627 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:27:10.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:10.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:10.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:10 smithi151 ceph-mon[28965]: 5.4 scrub starts 2024-06-27T21:27:10.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2812963888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:27:10.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:10.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:10.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:27:10.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:10 smithi190 ceph-mon[32247]: 5.4 scrub starts 2024-06-27T21:27:10.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2812963888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:27:10.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:10.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:10.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:11.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:10 smithi192 ceph-mon[32956]: 5.4 scrub starts 2024-06-27T21:27:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2812963888' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:27:11.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:11.153 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:11.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:11.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:11.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:11.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:11.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:11.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:11.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:11.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:11.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:11.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:11.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:11.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:11.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:11 smithi190 ceph-mon[32247]: 5.4 scrub ok 2024-06-27T21:27:11.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:11 smithi190 ceph-mon[32247]: pgmap v1119: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:27:11.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:11 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:11.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2812963888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:27:11.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:11 smithi190 ceph-mon[32247]: osdmap e428: 12 total, 12 up, 12 in 2024-06-27T21:27:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:11 smithi151 ceph-mon[28965]: 5.4 scrub ok 2024-06-27T21:27:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:11 smithi151 ceph-mon[28965]: pgmap v1119: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:27:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:11 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2812963888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:27:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:11 smithi151 ceph-mon[28965]: osdmap e428: 12 total, 12 up, 12 in 2024-06-27T21:27:11.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:11.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:11.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:11 smithi192 ceph-mon[32956]: 5.4 scrub ok 2024-06-27T21:27:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:11 smithi192 ceph-mon[32956]: pgmap v1119: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:27:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:11 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2812963888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:27:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:11 smithi192 ceph-mon[32956]: osdmap e428: 12 total, 12 up, 12 in 2024-06-27T21:27:12.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:12.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:12.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:12.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:12.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:12.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:12.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:12.486 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:27:12.486 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:27:12.486 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.22763417364247593 2024-06-27T21:27:12.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 3 0.22763417364247593 2024-06-27T21:27:12.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:12.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:27:12.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:27:12.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:27:12.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:27:12.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:12.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:12.625 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:27:12.625 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:27:12.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:27:12.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:27:12.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:27:12.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:27:12.693 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1327632417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:12.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:12.726 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:27:12.727 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:27:12.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:12.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:27:12.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:27:12.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1327632417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:12.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:27:12.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:27:12.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:27:12.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:27:12.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:12.855 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:27:12.856 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:27:12.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:12.900 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:27:12.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:27:12.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:12.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:12.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:27:12.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:27:13.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:13.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:13.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1327632417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:13.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:13.183 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:13.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:13.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:13.401 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:13.401 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:13.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:13.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:13.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:13.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:13.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:13.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:13.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:13.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:13.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:13.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:13 smithi151 ceph-mon[28965]: pgmap v1121: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s 2024-06-27T21:27:13.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:13.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:13.961 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:13 smithi190 ceph-mon[32247]: pgmap v1121: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s 2024-06-27T21:27:14.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:13 smithi192 ceph-mon[32956]: pgmap v1121: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 727 op/s 2024-06-27T21:27:14.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:14.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:14.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:27:14.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:27:14.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:14.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:14.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:14.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:14.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:14.475 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:14.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:14.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:14.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:14.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:14.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:14.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:14.870 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:14.870 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:14.870 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:14 smithi190 ceph-mon[32247]: osdmap e429: 12 total, 12 up, 12 in 2024-06-27T21:27:14.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:14.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:14.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:14 smithi151 ceph-mon[28965]: osdmap e429: 12 total, 12 up, 12 in 2024-06-27T21:27:14.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:14.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:14.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:14 smithi192 ceph-mon[32956]: osdmap e429: 12 total, 12 up, 12 in 2024-06-27T21:27:15.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:15.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:15.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:15.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:15.302 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:15.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:15.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:15.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:15.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:15.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:15.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:15.651 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 0.227634 (3a46) 2024-06-27T21:27:15.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:27:15.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:15.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:15.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:15 smithi151 ceph-mon[28965]: pgmap v1123: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s 2024-06-27T21:27:15.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:15.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:15.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:15 smithi151 ceph-mon[28965]: osdmap e430: 12 total, 12 up, 12 in 2024-06-27T21:27:15.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2940032314' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]: dispatch 2024-06-27T21:27:15.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:15 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]: dispatch 2024-06-27T21:27:15.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:15.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:15.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:15.985 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:15 smithi190 ceph-mon[32247]: pgmap v1123: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s 2024-06-27T21:27:15.985 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:15.985 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:15.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:15 smithi190 ceph-mon[32247]: osdmap e430: 12 total, 12 up, 12 in 2024-06-27T21:27:15.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2940032314' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]: dispatch 2024-06-27T21:27:15.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:15 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]: dispatch 2024-06-27T21:27:15.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:16.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:15 smithi192 ceph-mon[32956]: pgmap v1123: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 861 op/s 2024-06-27T21:27:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:15 smithi192 ceph-mon[32956]: osdmap e430: 12 total, 12 up, 12 in 2024-06-27T21:27:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2940032314' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]: dispatch 2024-06-27T21:27:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:15 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]: dispatch 2024-06-27T21:27:16.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:16.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:16.238 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:16.238 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:16.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:16.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:16.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:16.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:16.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:16.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:16.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:16.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:16.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:16.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:16.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:16.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]': finished 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: osdmap e431: 12 total, 12 up, 12 in 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:27:16.877 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:16 smithi190 ceph-mon[32247]: osdmap e432: 12 total, 12 up, 12 in 2024-06-27T21:27:16.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:16.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:16.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]': finished 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: osdmap e431: 12 total, 12 up, 12 in 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:27:16.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:16 smithi151 ceph-mon[28965]: osdmap e432: 12 total, 12 up, 12 in 2024-06-27T21:27:16.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:17.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.22763417364247593}]': finished 2024-06-27T21:27:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: osdmap e431: 12 total, 12 up, 12 in 2024-06-27T21:27:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:27:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:27:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:27:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:27:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:16 smithi192 ceph-mon[32956]: osdmap e432: 12 total, 12 up, 12 in 2024-06-27T21:27:17.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:17.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:17.195 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:17.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:17.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:17.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:17.372 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:17.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:17.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:17.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:17.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:17.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:17.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:17.752 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:17.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:17.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:17.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:17 smithi151 ceph-mon[28965]: pgmap v1126: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-27T21:27:17.897 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:17 smithi151 ceph-mon[28965]: osdmap e433: 12 total, 12 up, 12 in 2024-06-27T21:27:17.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:17.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:17.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:18.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:17 smithi190 ceph-mon[32247]: pgmap v1126: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-27T21:27:18.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:17 smithi190 ceph-mon[32247]: osdmap e433: 12 total, 12 up, 12 in 2024-06-27T21:27:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:17 smithi192 ceph-mon[32956]: pgmap v1126: 27 pgs: 1 active+clean+scrubbing+deep, 26 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-06-27T21:27:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:17 smithi192 ceph-mon[32956]: osdmap e433: 12 total, 12 up, 12 in 2024-06-27T21:27:18.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:18.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:18.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:18.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:18.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:18.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:18.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:18.514 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:18.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:18.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:18.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:18.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:18.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:18.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:18.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:18.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:18 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 6567/199533 objects degraded (3.291%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:18.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:19.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:18 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 6567/199533 objects degraded (3.291%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:19.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:19.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:18 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 6567/199533 objects degraded (3.291%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:19.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:19.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:19.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:19.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:19.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:19.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:19.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:19.507 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:19.507 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:19.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:19.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:19.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:19.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:19.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:19.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:19.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:19.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:19.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:19.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:19 smithi151 ceph-mon[28965]: pgmap v1129: 27 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 6567/199533 objects degraded (3.291%); 49 KiB/s, 0 objects/s recovering 2024-06-27T21:27:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:19 smithi190 ceph-mon[32247]: pgmap v1129: 27 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 6567/199533 objects degraded (3.291%); 49 KiB/s, 0 objects/s recovering 2024-06-27T21:27:20.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:20.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:19 smithi192 ceph-mon[32956]: pgmap v1129: 27 pgs: 1 activating+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 6567/199533 objects degraded (3.291%); 49 KiB/s, 0 objects/s recovering 2024-06-27T21:27:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:20.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:20.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:20.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:20.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:20.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:27:20.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:20.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:20.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:20.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:20.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:20.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:20.629 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:27:20.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:20.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:20.719 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:27:20.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/338571082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:20.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:20 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:20.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:20.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:20.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:20.954 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:27:20.954 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:27:20.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:20.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:20.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:27:20.988 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:27:21.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/338571082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:21.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:20 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:21.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:27:21.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:27:21.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/338571082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:21.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:20 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:21.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:27:21.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:27:21.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:21.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:27:21.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:27:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:21.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:21.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:27:21.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:27:21.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:27:21.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:27:21.167 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:27:21.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:27:21.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:27:21.199 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:27:21.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:21.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:27:21.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:27:21.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:21.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:27:21.255 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:27:21.289 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:27:21.289 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:27:21.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:21.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:21.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:21.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:21.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:21.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:21.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:21.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:21.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:21.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:21.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: pgmap v1130: 27 pgs: 1 active+clean+scrubbing+deep, 1 activating+undersized+degraded+remapped, 25 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 6567/197469 objects degraded (3.326%); 36 KiB/s, 0 objects/s recovering 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: osdmap e434: 12 total, 12 up, 12 in 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1276935231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:21.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:22.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: pgmap v1130: 27 pgs: 1 active+clean+scrubbing+deep, 1 activating+undersized+degraded+remapped, 25 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 6567/197469 objects degraded (3.326%); 36 KiB/s, 0 objects/s recovering 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: osdmap e434: 12 total, 12 up, 12 in 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1276935231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: pgmap v1130: 27 pgs: 1 active+clean+scrubbing+deep, 1 activating+undersized+degraded+remapped, 25 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 6567/197469 objects degraded (3.326%); 36 KiB/s, 0 objects/s recovering 2024-06-27T21:27:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: osdmap e434: 12 total, 12 up, 12 in 2024-06-27T21:27:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1276935231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:22.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:22.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:22.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:22.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:22.361 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:27:22.361 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:27:22.361 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10, 2] 2024-06-27T21:27:22.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 7 2024-06-27T21:27:22.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:22.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:22.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:22.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:22.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:22.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:22.690 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:27:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:27:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:27:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15132.0:123874 3.d 3:bf6c1432:::benchmark_data_smithi192_74314_object17236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:18:09.115076+0000", 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 553.57439912500001, 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28677490999999999, 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15132", 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:27:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 123874 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.115076+0000", 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.115076+0000", 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.115077+0000", 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8999999999999995e-07 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.115081+0000", 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9509999999999999e-06 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.115090+0000", 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1859999999999998e-06 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.115097+0000", 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.238e-06 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.118719+0000", 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003621272 2024-06-27T21:27:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.118873+0000", 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000154041 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 5,9", 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.118994+0000", 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121128 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.122905+0000", 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039110560000000004 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.122923+0000", 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7586000000000001e-05 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.401741+0000", 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27881805900000001 2024-06-27T21:27:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.401800+0000", 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9225999999999998e-05 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:18:09.401851+0000", 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1377e-05 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:9521 4.3 e292/282 4:c314fc9a:::benchmark_data_smithi192_88961_object9520:head v 292'597, mlcod=291'593)", 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:50.566137+0000", 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 392.12333860899997, 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.53457432599999999, 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:50.566137+0000", 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:50.566137+0000", 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:50.566138+0000", 2024-06-27T21:27:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.09e-07 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:50.566193+0000", 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5238000000000002e-05 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:51.100038+0000", 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.533845605 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:51.100048+0000", 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2410000000000001e-06 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:51.100107+0000", 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9771000000000001e-05 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:51.100123+0000", 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5696999999999999e-05 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:51.100681+0000", 2024-06-27T21:27:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000558089 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:51.100711+0000", 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9876e-05 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:66878 4.3 e320/282 4:c506cff3:::benchmark_data_smithi192_88961_object66877:head v 320'4172, mlcod=320'4170)", 2024-06-27T21:27:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:21:44.971075+0000", 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 337.71840064399998, 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33489006599999999, 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971075+0000", 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971075+0000", 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971076+0000", 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5300000000000003e-07 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971165+0000", 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9655999999999997e-05 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.303997+0000", 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33283127899999998 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.304004+0000", 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9970000000000001e-06 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.304059+0000", 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5937999999999999e-05 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.304077+0000", 2024-06-27T21:27:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7231000000000002e-05 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.305933+0000", 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001856555 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.305965+0000", 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1557000000000003e-05 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130343 4.4 4.aee468b4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.588913+0000", 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 267.10056224700003, 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97634029499999997, 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130343 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588913+0000", 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588913+0000", 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588914+0000", 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1519999999999999e-06 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588918+0000", 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1679999999999997e-06 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588926+0000", 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.34e-06 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588931+0000", 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8879999999999996e-06 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588989+0000", 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8211999999999998e-05 2024-06-27T21:27:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588997+0000", 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2830000000000001e-06 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565243+0000", 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97624575099999999 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565253+0000", 2024-06-27T21:27:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0501e-05 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130350 4.4 4.db94d154 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.592713+0000", 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 267.09676244899998, 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97257341200000003, 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130350 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592713+0000", 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592713+0000", 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592714+0000", 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.288e-06 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592723+0000", 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2110000000000004e-06 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592737+0000", 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4038e-05 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592745+0000", 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2189999999999998e-06 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592788+0000", 2024-06-27T21:27:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.367e-05 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592794+0000", 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5029999999999997e-06 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565279+0000", 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97248522199999998 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565286+0000", 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2609999999999996e-06 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130380 4.4 4.f594c0b4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.633122+0000", 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 267.05635329699999, 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.93219419800000003, 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130380 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633122+0000", 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633122+0000", 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633123+0000", 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.141e-06 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633127+0000", 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7249999999999999e-06 2024-06-27T21:27:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633139+0000", 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1657e-05 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633144+0000", 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4500000000000003e-06 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633192+0000", 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8196999999999999e-05 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633199+0000", 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5409999999999997e-06 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565309+0000", 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.93211017699999998 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565316+0000", 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3100000000000003e-06 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130386 4.4 4.b7dc4824 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.641448+0000", 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 267.04802750099998, 2024-06-27T21:27:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.92389724399999995, 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130386 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641448+0000", 2024-06-27T21:27:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641448+0000", 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641449+0000", 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0869999999999999e-06 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641454+0000", 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641466+0000", 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2486999999999999e-05 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641474+0000", 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1300000000000001e-06 2024-06-27T21:27:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641529+0000", 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4432000000000003e-05 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641535+0000", 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5030000000000002e-06 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565338+0000", 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.923803072 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565345+0000", 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8129999999999998e-06 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130390 4.4 4.72e9fe74 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.647900+0000", 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 267.04157503699997, 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91747367300000005, 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130390 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647900+0000", 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647900+0000", 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647901+0000", 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647905+0000", 2024-06-27T21:27:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6969999999999999e-06 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647915+0000", 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0122000000000001e-05 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647922+0000", 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0090000000000001e-06 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647985+0000", 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3336000000000005e-05 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647990+0000", 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5680000000000001e-06 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565367+0000", 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.917377263 2024-06-27T21:27:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565374+0000", 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9630000000000001e-06 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130428 4.4 4.d1ff1df4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.739569+0000", 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.94990597499998, 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82583467799999999, 2024-06-27T21:27:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130428 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739569+0000", 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739569+0000", 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739571+0000", 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2559999999999999e-06 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739575+0000", 2024-06-27T21:27:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4560000000000002e-06 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739586+0000", 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0891e-05 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739591+0000", 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4820000000000002e-06 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739629+0000", 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7082000000000002e-05 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739634+0000", 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3709999999999999e-06 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565397+0000", 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82576353800000002 2024-06-27T21:27:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565404+0000", 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6019999999999996e-06 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130448 4.4 4.cedc1aa4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.821951+0000", 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.86752477900001, 2024-06-27T21:27:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.74348186000000005, 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130448 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821951+0000", 2024-06-27T21:27:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821951+0000", 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821952+0000", 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5230000000000001e-06 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821958+0000", 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1670000000000004e-06 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821974+0000", 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5716e-05 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821997+0000", 2024-06-27T21:27:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3374999999999999e-05 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.822057+0000", 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9338e-05 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.822063+0000", 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2879999999999998e-06 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565426+0000", 2024-06-27T21:27:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.74336267600000006 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565432+0000", 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7769999999999997e-06 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130449 4.4 4.c6957ab4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.833860+0000", 2024-06-27T21:27:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.85561502000002, 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.731599997, 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130449 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833860+0000", 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833860+0000", 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833862+0000", 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833872+0000", 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6770000000000007e-06 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833899+0000", 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7248999999999999e-05 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833914+0000", 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.521e-05 2024-06-27T21:27:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833999+0000", 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4529000000000003e-05 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.834005+0000", 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0920000000000003e-06 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565453+0000", 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.731448443 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565460+0000", 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8639999999999998e-06 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130454 4.4 4.c3c86de4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.871420+0000", 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.81805563699999, 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.694069504, 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130454 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871420+0000", 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871420+0000", 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871421+0000", 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.33e-06 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871425+0000", 2024-06-27T21:27:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1189999999999999e-06 2024-06-27T21:27:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871436+0000", 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0814e-05 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871458+0000", 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2430999999999999e-05 2024-06-27T21:27:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871507+0000", 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8266000000000002e-05 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871512+0000", 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3469999999999998e-06 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565482+0000", 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69397007799999999 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565489+0000", 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1189999999999999e-06 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130455 4.4 4.76b76934 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.896512+0000", 2024-06-27T21:27:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.79296342100002, 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.66900683900000002, 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130455 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896512+0000", 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896512+0000", 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896514+0000", 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9130000000000001e-06 2024-06-27T21:27:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896523+0000", 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1370000000000008e-06 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896550+0000", 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7118999999999999e-05 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896564+0000", 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3423e-05 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896698+0000", 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000134177 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896703+0000", 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.733e-06 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565512+0000", 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.66880835599999999 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565519+0000", 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9809999999999997e-06 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:130464 4.8 e345/282 4:198f8843:::benchmark_data_smithi192_88961_object39725:head v 345'8149, mlcod=345'8147)", 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.71903363899997, 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28325987400000002, 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:27:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8800000000000002e-07 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970447+0000", 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3109999999999999e-06 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250043+0000", 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.279596446 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250057+0000", 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4209000000000001e-05 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250125+0000", 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7310000000000004e-05 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250142+0000", 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7373000000000001e-05 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.253653+0000", 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035112680000000001 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.253702+0000", 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8368999999999997e-05 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130472 4.4 4.6db037f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:56.012165+0000", 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.67731020600002, 2024-06-27T21:27:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.55339389400000005, 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130472 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012165+0000", 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012165+0000", 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012167+0000", 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.801e-06 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012173+0000", 2024-06-27T21:27:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0889999999999996e-06 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012188+0000", 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5378000000000001e-05 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012212+0000", 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3447000000000001e-05 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012265+0000", 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3480999999999998e-05 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012273+0000", 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8029999999999997e-06 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565548+0000", 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.55327492599999994 2024-06-27T21:27:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565559+0000", 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0969e-05 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130474 4.4 4.bfd3c974 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:56.018145+0000", 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.67133063699998, 2024-06-27T21:27:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54743817800000005, 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130474 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018145+0000", 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018145+0000", 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018147+0000", 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9269999999999999e-06 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018156+0000", 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6660000000000003e-06 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018193+0000", 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6958000000000001e-05 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018198+0000", 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9949999999999996e-06 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018258+0000", 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0081999999999998e-05 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018263+0000", 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4959999999999998e-06 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565576+0000", 2024-06-27T21:27:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54731361000000001 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565583+0000", 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4439999999999996e-06 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:28d715ef:::benchmark_data_smithi192_88961_object27315:head,341'7381)", 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:24:30.681611+0000", 2024-06-27T21:27:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 172.00786454499999, 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83259944799999996, 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681611+0000", 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681611+0000", 2024-06-27T21:27:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681618+0000", 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6900000000000003e-06 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681628+0000", 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0316000000000001e-05 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.513986+0000", 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83235856100000005 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514007+0000", 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0356e-05 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514135+0000", 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012872599999999999 2024-06-27T21:27:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514143+0000", 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5970000000000003e-06 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514210+0000", 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7201999999999998e-05 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:34690 5.9 5:962f63b1:::benchmark_data_smithi192_104167_object34689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:27.105162+0000", 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 115.584312892, 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26655124400000002, 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:27:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 34690 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105162+0000", 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105162+0000", 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105164+0000", 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.903e-06 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105201+0000", 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6452999999999998e-05 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105213+0000", 2024-06-27T21:27:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.254e-05 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105220+0000", 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3439999999999997e-06 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105277+0000", 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7569999999999999e-05 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105420+0000", 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014268700000000001 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.105548+0000", 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012790699999999999 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.106983+0000", 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001435165 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.107375+0000", 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039175100000000002 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.371643+0000", 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26426819299999998 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.371682+0000", 2024-06-27T21:27:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8797000000000001e-05 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.371714+0000", 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1934000000000003e-05 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:72404 5.9 5:98cfc148:::benchmark_data_smithi192_104167_object72403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-27T21:27:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:57.149102+0000", 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 85.540373549999998, 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26547631399999999, 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 72404 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149102+0000", 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149102+0000", 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149104+0000", 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4260000000000002e-06 2024-06-27T21:27:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149152+0000", 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8004999999999999e-05 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149166+0000", 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3242e-05 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149172+0000", 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1149999999999999e-06 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149254+0000", 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2378000000000003e-05 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149396+0000", 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142378 2024-06-27T21:27:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.149554+0000", 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015807899999999999 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.151588+0000", 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002033857 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.151826+0000", 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023792099999999999 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.414514+0000", 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26268807900000002 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.414550+0000", 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6066999999999999e-05 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.414578+0000", 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7767e-05 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:72417 5.9 5:9fcfca4f:::benchmark_data_smithi192_104167_object72416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:57.166396+0000", 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 85.523079437999996, 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.287432464, 2024-06-27T21:27:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 72417 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166396+0000", 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166396+0000", 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166397+0000", 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.114e-06 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166449+0000", 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1529000000000001e-05 2024-06-27T21:27:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166460+0000", 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1270999999999999e-05 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166466+0000", 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9410000000000004e-06 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166509+0000", 2024-06-27T21:27:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3612999999999997e-05 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166602+0000", 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2732000000000006e-05 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166692+0000", 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0198999999999995e-05 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.168629+0000", 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019362610000000001 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.169390+0000", 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00076179500000000001 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.453695+0000", 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28430495 2024-06-27T21:27:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.453774+0000", 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8935000000000002e-05 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.453828+0000", 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4123999999999999e-05 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:27:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:27:22.801 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:27:22.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:22.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:22 smithi151 ceph-mon[28965]: osdmap e435: 12 total, 12 up, 12 in 2024-06-27T21:27:22.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:22 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:22.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:22 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:22.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:27:22.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:27:22.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:22 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:22.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:22.873 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:22.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:22.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:22.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:22 smithi190 ceph-mon[32247]: osdmap e435: 12 total, 12 up, 12 in 2024-06-27T21:27:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:22 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:22 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:27:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:27:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:22 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:22 smithi192 ceph-mon[32956]: osdmap e435: 12 total, 12 up, 12 in 2024-06-27T21:27:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:22 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:22 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:27:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:27:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:22 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:23.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:23.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:23.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:23.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:23.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:23.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:23.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:23.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:23.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:23.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:23.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:23.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:23.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:23.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:23.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:23.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:24.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:24.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:23 smithi190 ceph-mon[32247]: pgmap v1133: 27 pgs: 1 active+clean+scrubbing+deep, 1 activating+undersized+degraded+remapped, 25 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 6567/194202 objects degraded (3.382%); 36 KiB/s, 0 objects/s recovering 2024-06-27T21:27:24.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:23 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:24.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:23 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:24.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:23 smithi192 ceph-mon[32956]: pgmap v1133: 27 pgs: 1 active+clean+scrubbing+deep, 1 activating+undersized+degraded+remapped, 25 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 6567/194202 objects degraded (3.382%); 36 KiB/s, 0 objects/s recovering 2024-06-27T21:27:24.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:23 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:24.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:23 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:24.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:24.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:24.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:23 smithi151 ceph-mon[28965]: pgmap v1133: 27 pgs: 1 active+clean+scrubbing+deep, 1 activating+undersized+degraded+remapped, 25 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 6567/194202 objects degraded (3.382%); 36 KiB/s, 0 objects/s recovering 2024-06-27T21:27:24.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:23 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:24.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:23 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:24.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:24.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:24.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:27:24.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:24.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:24.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:24.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:24.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:24.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:24.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:27:24.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:24.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:24.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:27:24.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:24.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:24.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:25.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:25.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:24 smithi190 ceph-mon[32247]: osdmap e436: 12 total, 12 up, 12 in 2024-06-27T21:27:25.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6567/194202 objects degraded (3.382%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:25.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:25.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:24 smithi192 ceph-mon[32956]: osdmap e436: 12 total, 12 up, 12 in 2024-06-27T21:27:25.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6567/194202 objects degraded (3.382%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:25.045 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:25.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:24 smithi151 ceph-mon[28965]: osdmap e436: 12 total, 12 up, 12 in 2024-06-27T21:27:25.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6567/194202 objects degraded (3.382%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:25.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:25.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:25.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:25.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:25.350 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:25.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:25.402 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:25.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:25.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:25.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:25.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:25.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:25.777 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.7. 2024-06-27T21:27:25.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:25.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:25.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:25.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:25 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:27:25.646+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e436 definitely_dead 0 2024-06-27T21:27:25.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:25 smithi151 ceph-mon[28965]: pgmap v1135: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 88/190743 objects degraded (0.046%); 3823/190743 objects misplaced (2.004%); 1.6 MiB/s, 25 objects/s recovering 2024-06-27T21:27:25.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3167717090' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:27:25.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:25 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:27:25.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:25 smithi151 ceph-mon[28965]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-27T21:27:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:25.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:25.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:26.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:25 smithi192 ceph-mon[32956]: pgmap v1135: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 88/190743 objects degraded (0.046%); 3823/190743 objects misplaced (2.004%); 1.6 MiB/s, 25 objects/s recovering 2024-06-27T21:27:26.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3167717090' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:27:26.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:25 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:27:26.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:25 smithi192 ceph-mon[32956]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-27T21:27:26.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:26.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:26.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:26.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:26.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:25 smithi190 ceph-mon[32247]: pgmap v1135: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 88/190743 objects degraded (0.046%); 3823/190743 objects misplaced (2.004%); 1.6 MiB/s, 25 objects/s recovering 2024-06-27T21:27:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3167717090' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:27:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:25 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:27:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:25 smithi190 ceph-mon[32247]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-27T21:27:26.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:26.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:26.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:26.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:26.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:26.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:26.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:26.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:26.787 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:27:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:27:26.787 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:27:26.787 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:27:26.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:26.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:26.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:26.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:26.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:27.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:27.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:26 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-27T21:27:27.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:26 smithi192 ceph-mon[32956]: osdmap e437: 12 total, 12 up, 11 in 2024-06-27T21:27:27.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:27.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:27.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:27.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:26 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-27T21:27:27.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:26 smithi151 ceph-mon[28965]: osdmap e437: 12 total, 12 up, 11 in 2024-06-27T21:27:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:26 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-27T21:27:27.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:26 smithi190 ceph-mon[32247]: osdmap e437: 12 total, 12 up, 11 in 2024-06-27T21:27:27.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:27.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:27.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:27.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:27:27.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:27.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:27.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:27.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:27.669 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:27.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:27.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:27.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:27.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:27.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:27.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:27.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:27 smithi151 ceph-mon[28965]: pgmap v1137: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 88/184662 objects degraded (0.048%); 3823/184662 objects misplaced (2.070%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:27:27.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:27 smithi151 ceph-mon[28965]: osdmap e438: 12 total, 12 up, 11 in 2024-06-27T21:27:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:27 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:27:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:27 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:27:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:27 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:27:28.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:28.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:27 smithi192 ceph-mon[32956]: pgmap v1137: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 88/184662 objects degraded (0.048%); 3823/184662 objects misplaced (2.070%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:27:28.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:27 smithi192 ceph-mon[32956]: osdmap e438: 12 total, 12 up, 11 in 2024-06-27T21:27:28.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:27 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:27:28.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:27 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:27:28.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:27 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:27:28.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:28.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:28.190 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:28.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:27 smithi190 ceph-mon[32247]: pgmap v1137: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 88/184662 objects degraded (0.048%); 3823/184662 objects misplaced (2.070%); 1.8 MiB/s, 28 objects/s recovering 2024-06-27T21:27:28.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:27 smithi190 ceph-mon[32247]: osdmap e438: 12 total, 12 up, 11 in 2024-06-27T21:27:28.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:27 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:27:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:27 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:27:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:27 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:27:28.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:28.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:28.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:28.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:28.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:28.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:28.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:28.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:28.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:28.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:28.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:28.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:29.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:29.040 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:28 smithi192 ceph-mon[32956]: osdmap e439: 12 total, 12 up, 11 in 2024-06-27T21:27:29.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:29.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:29.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:28 smithi151 ceph-mon[28965]: osdmap e439: 12 total, 12 up, 11 in 2024-06-27T21:27:29.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:29.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:29.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:28 smithi190 ceph-mon[32247]: osdmap e439: 12 total, 12 up, 11 in 2024-06-27T21:27:29.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:29.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:29.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:29.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:29.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:29.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:29.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:29.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:29 smithi192 ceph-mon[32956]: pgmap v1140: 27 pgs: 1 activating+undersized+remapped, 1 peering, 25 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 12893/180945 objects degraded (7.125%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:27:29.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:29 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:27:29.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:29 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/184662 objects degraded (0.048%), 1 pg degraded) 2024-06-27T21:27:29.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:29.888 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:27:29.888 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:27:29.888 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:27:29.888 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:27:29.889 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:27:29.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:29.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:29.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:29 smithi151 ceph-mon[28965]: pgmap v1140: 27 pgs: 1 activating+undersized+remapped, 1 peering, 25 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 12893/180945 objects degraded (7.125%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:27:29.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:29 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:27:29.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:29 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/184662 objects degraded (0.048%), 1 pg degraded) 2024-06-27T21:27:30.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:30.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:30.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:30.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:30.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:29 smithi190 ceph-mon[32247]: pgmap v1140: 27 pgs: 1 activating+undersized+remapped, 1 peering, 25 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 12893/180945 objects degraded (7.125%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:27:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:29 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:27:30.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:29 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 88/184662 objects degraded (0.048%), 1 pg degraded) 2024-06-27T21:27:30.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:30.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:30.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:30.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:30.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:30.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:27:30.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:30.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:30.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:30.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:30.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:30.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:30.960 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:31.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:31.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:31.098 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:30 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 12915/178803 objects degraded (7.223%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:27:31.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:30 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 12915/178803 objects degraded (7.223%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:27:31.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:31.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:31.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:31.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:30 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 12915/178803 objects degraded (7.223%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:27:31.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:31.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:31.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:31.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:31.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:31.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:31.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:31.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:31.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:31.849 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:27:31.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:31.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:31.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:31.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:32.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:32.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:32.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:31 smithi190 ceph-mon[32247]: pgmap v1141: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 peering, 23 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 12915/178803 objects degraded (7.223%); 7553/178803 objects misplaced (4.224%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:27:32.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3414546116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:32.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/92336046' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:27:32.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:31 smithi192 ceph-mon[32956]: pgmap v1141: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 peering, 23 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 12915/178803 objects degraded (7.223%); 7553/178803 objects misplaced (4.224%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:27:32.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3414546116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:32.180 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/92336046' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:27:32.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:31 smithi151 ceph-mon[28965]: pgmap v1141: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 1 activating+undersized+remapped, 1 peering, 23 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 12915/178803 objects degraded (7.223%); 7553/178803 objects misplaced (4.224%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:27:32.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3414546116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:32.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/92336046' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:27:32.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:32.232 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:27:32.232 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:27:32.233 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-06-27T21:27:32.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 7 2024-06-27T21:27:32.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:32.303 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:27:32.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:27:32.343 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:27:32.344 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:27:32.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:32.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:32.379 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:27:32.379 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:27:32.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:27:32.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:27:32.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:27:32.435 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:27:32.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:27:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:27:32.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:32.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:32.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:27:32.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:27:32.542 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:27:32.542 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:27:32.571 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:27:32.571 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:27:32.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:27:32.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:27:32.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:32.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:27:32.632 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:27:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:32.670 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:27:32.671 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:27:32.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:32.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:32.803 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:32.803 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:32.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:32.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:32.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:33.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:33.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:32 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/92336046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:27:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:32 smithi192 ceph-mon[32956]: osdmap e440: 12 total, 12 up, 11 in 2024-06-27T21:27:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:32 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:27:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:32 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:27:33.112 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:32 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:33.112 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/92336046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:27:33.112 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:32 smithi190 ceph-mon[32247]: osdmap e440: 12 total, 12 up, 11 in 2024-06-27T21:27:33.112 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:32 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:27:33.112 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:32 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:27:33.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:33.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:33.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:32 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:33.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/92336046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:27:33.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:32 smithi151 ceph-mon[28965]: osdmap e440: 12 total, 12 up, 11 in 2024-06-27T21:27:33.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:32 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:27:33.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:32 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:27:33.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:33.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:33.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:33.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:33.454 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:33.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:33.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:27:33.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:33.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:33.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:33.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:33.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:33.899 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:33.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:33.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:33.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:34.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:34.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:33 smithi151 ceph-mon[28965]: pgmap v1143: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 1 activating+undersized+remapped, 23 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 460 op/s; 12956/176370 objects degraded (7.346%); 11223/176370 objects misplaced (6.363%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:27:34.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:33 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:27:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:33 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:27:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:33 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:27:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:33 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:27:34.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:33 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:27:34.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:34.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:33 smithi190 ceph-mon[32247]: pgmap v1143: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 1 activating+undersized+remapped, 23 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 460 op/s; 12956/176370 objects degraded (7.346%); 11223/176370 objects misplaced (6.363%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:27:34.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:33 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:27:34.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:33 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:27:34.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:33 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:27:34.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:33 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:27:34.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:33 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:27:34.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:34.188 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:33 smithi192 ceph-mon[32956]: pgmap v1143: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 1 activating+undersized+remapped, 23 active+clean; 3.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 460 op/s; 12956/176370 objects degraded (7.346%); 11223/176370 objects misplaced (6.363%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:27:34.188 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:33 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:27:34.188 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:33 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:27:34.188 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:33 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:27:34.188 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:33 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:27:34.189 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:33 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:27:34.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:34.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:34.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:34.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:34.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:34.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:34.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:34.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:27:34.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:34.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:34.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:34.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:34.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:34.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:34.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:35.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:35.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:35.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:35.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:35.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:35.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:27:35.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:35.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:35.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:35.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:35.723 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:35.723 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:35.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:35.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:35.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:35.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:35.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:36.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:36.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:35 smithi192 ceph-mon[32956]: pgmap v1144: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 407 op/s; 129/173478 objects degraded (0.074%); 17976/173478 objects misplaced (10.362%); 4.5 MiB/s, 71 objects/s recovering 2024-06-27T21:27:36.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:36.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:36.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:36.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:36.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:35 smithi151 ceph-mon[28965]: pgmap v1144: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 407 op/s; 129/173478 objects degraded (0.074%); 17976/173478 objects misplaced (10.362%); 4.5 MiB/s, 71 objects/s recovering 2024-06-27T21:27:36.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:35 smithi190 ceph-mon[32247]: pgmap v1144: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 407 op/s; 129/173478 objects degraded (0.074%); 17976/173478 objects misplaced (10.362%); 4.5 MiB/s, 71 objects/s recovering 2024-06-27T21:27:36.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:36.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:36.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:36.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:27:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:36.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:36.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:36.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:36.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:36.680 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:36.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:36.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:36.835 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:36.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:36.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:36.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:36.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:37.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:37.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:37.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:36 smithi151 ceph-mon[28965]: pgmap v1145: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 229/169026 objects degraded (0.135%); 17566/169026 objects misplaced (10.392%); 6.3 MiB/s, 100 objects/s recovering 2024-06-27T21:27:37.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:36 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 229/169026 objects degraded (0.135%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:27:37.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:37.213 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:36 smithi192 ceph-mon[32956]: pgmap v1145: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 229/169026 objects degraded (0.135%); 17566/169026 objects misplaced (10.392%); 6.3 MiB/s, 100 objects/s recovering 2024-06-27T21:27:37.213 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:36 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 229/169026 objects degraded (0.135%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:27:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:37.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:37.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:36 smithi190 ceph-mon[32247]: pgmap v1145: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 229/169026 objects degraded (0.135%); 17566/169026 objects misplaced (10.392%); 6.3 MiB/s, 100 objects/s recovering 2024-06-27T21:27:37.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:36 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 229/169026 objects degraded (0.135%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:27:37.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:37.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:37.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:37.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:37.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:37.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:37.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:37.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:37.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:37.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:37.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:37.923 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.7. 2024-06-27T21:27:37.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:38.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:38.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:38.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:38.181 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2491571955' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:27:38.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:37 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:27:37.853+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e440 definitely_dead 0 2024-06-27T21:27:38.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2491571955' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:27:38.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:38.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:38.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2491571955' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:27:38.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:38.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:38.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:38.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:38.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:38.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:38.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:38.741 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:38.753 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-06-27T21:27:38.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:38.788 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:38.788 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:38.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:38.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:38.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:38.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:39.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:39.034 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2491571955' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-27T21:27:39.034 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:38 smithi192 ceph-mon[32956]: osdmap e441: 12 total, 12 up, 12 in 2024-06-27T21:27:39.034 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:38 smithi192 ceph-mon[32956]: pgmap v1147: 27 pgs: 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 541 op/s; 225/165816 objects degraded (0.136%); 10608/165816 objects misplaced (6.397%); 6.9 MiB/s, 110 objects/s recovering 2024-06-27T21:27:39.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2491571955' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-27T21:27:39.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:38 smithi151 ceph-mon[28965]: osdmap e441: 12 total, 12 up, 12 in 2024-06-27T21:27:39.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:38 smithi151 ceph-mon[28965]: pgmap v1147: 27 pgs: 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 541 op/s; 225/165816 objects degraded (0.136%); 10608/165816 objects misplaced (6.397%); 6.9 MiB/s, 110 objects/s recovering 2024-06-27T21:27:39.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:39.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:39.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2491571955' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-27T21:27:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:38 smithi190 ceph-mon[32247]: osdmap e441: 12 total, 12 up, 12 in 2024-06-27T21:27:39.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:38 smithi190 ceph-mon[32247]: pgmap v1147: 27 pgs: 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 541 op/s; 225/165816 objects degraded (0.136%); 10608/165816 objects misplaced (6.397%); 6.9 MiB/s, 110 objects/s recovering 2024-06-27T21:27:39.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:39.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:39.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:39.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:39.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:39.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:39.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:39.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:27:39.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:39.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:39.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:39.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:39.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:40.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:40.048 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:40.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:40.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:40.178 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 5.e scrub starts 2024-06-27T21:27:40.178 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 5.e scrub ok 2024-06-27T21:27:40.178 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 5.a scrub starts 2024-06-27T21:27:40.179 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 5.2 scrub starts 2024-06-27T21:27:40.179 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: osdmap e442: 12 total, 12 up, 12 in 2024-06-27T21:27:40.179 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 5.0 scrub starts 2024-06-27T21:27:40.179 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:27:40.179 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:27:40.179 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:39 smithi192 ceph-mon[32956]: 5.a scrub ok 2024-06-27T21:27:40.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:40.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 5.e scrub starts 2024-06-27T21:27:40.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 5.e scrub ok 2024-06-27T21:27:40.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 5.a scrub starts 2024-06-27T21:27:40.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 5.2 scrub starts 2024-06-27T21:27:40.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: osdmap e442: 12 total, 12 up, 12 in 2024-06-27T21:27:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 5.0 scrub starts 2024-06-27T21:27:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:27:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:27:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:39 smithi151 ceph-mon[28965]: 5.a scrub ok 2024-06-27T21:27:40.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 5.e scrub starts 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 5.e scrub ok 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 5.a scrub starts 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 5.2 scrub starts 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: osdmap e442: 12 total, 12 up, 12 in 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 5.0 scrub starts 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:27:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:39 smithi190 ceph-mon[32247]: 5.a scrub ok 2024-06-27T21:27:40.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:40.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:40.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:40.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:40.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:40.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:40.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:40.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:40.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:40.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:40.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:40.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:40.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:41.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:41.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:41.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:41.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:40 smithi151 ceph-mon[28965]: 5.2 scrub ok 2024-06-27T21:27:41.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:40 smithi151 ceph-mon[28965]: pgmap v1149: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 22 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 102/162828 objects degraded (0.063%); 3459/162828 objects misplaced (2.124%); 8.2 MiB/s, 137 objects/s recovering 2024-06-27T21:27:41.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:40 smithi151 ceph-mon[28965]: 5.0 scrub ok 2024-06-27T21:27:41.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:40 smithi192 ceph-mon[32956]: 5.2 scrub ok 2024-06-27T21:27:41.204 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:40 smithi192 ceph-mon[32956]: pgmap v1149: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 22 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 102/162828 objects degraded (0.063%); 3459/162828 objects misplaced (2.124%); 8.2 MiB/s, 137 objects/s recovering 2024-06-27T21:27:41.204 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:40 smithi192 ceph-mon[32956]: 5.0 scrub ok 2024-06-27T21:27:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:41.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:41.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:40 smithi190 ceph-mon[32247]: 5.2 scrub ok 2024-06-27T21:27:41.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:40 smithi190 ceph-mon[32247]: pgmap v1149: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 22 active+clean; 3.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 102/162828 objects degraded (0.063%); 3459/162828 objects misplaced (2.124%); 8.2 MiB/s, 137 objects/s recovering 2024-06-27T21:27:41.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:40 smithi190 ceph-mon[32247]: 5.0 scrub ok 2024-06-27T21:27:41.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:41.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.358 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:41.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:27:41.454 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:27:41.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:41.483 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:27:41.483 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:27:41.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:41.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:27:41.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:27:41.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:27:41.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:27:41.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:27:41.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:27:41.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:41.595 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:27:41.595 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:27:41.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:41.625 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:27:41.625 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:27:41.655 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:27:41.655 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:27:41.682 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:27:41.682 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:27:41.706 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:27:41.706 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:27:41.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:27:41.733 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:27:41.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:41.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:27:41.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:27:41.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:41.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:41.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:41.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:41.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:42.025 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2262562657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:42.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:42.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58291 4.6 4.30c53896 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 368.616405649, 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95344075399999995, 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.148 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58291 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530167+0000", 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0860000000000001e-06 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530189+0000", 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1943999999999999e-05 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530201+0000", 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1959e-05 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530208+0000", 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6880000000000002e-06 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530259+0000", 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0924000000000002e-05 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530267+0000", 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5790000000000004e-06 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483575+0000", 2024-06-27T21:27:42.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95330775800000001 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483607+0000", 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1816000000000002e-05 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58315 4.6 4.d0954b56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 368.58465804399998, 2024-06-27T21:27:42.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92179534699999999, 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58315 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561915+0000", 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.141e-06 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561933+0000", 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8337000000000001e-05 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561941+0000", 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7729999999999999e-06 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561946+0000", 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9230000000000001e-06 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.562002+0000", 2024-06-27T21:27:42.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6212999999999999e-05 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.562008+0000", 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5849999999999996e-06 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483698+0000", 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92168988600000001 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483709+0000", 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0489000000000001e-05 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58321 4.6 4.898f9226 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 368.57858744999999, 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91576660700000001, 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.159 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58321 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567986+0000", 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.516e-06 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568008+0000", 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2464e-05 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568022+0000", 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3726999999999999e-05 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568028+0000", 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1530000000000002e-06 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568097+0000", 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8694000000000003e-05 2024-06-27T21:27:42.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568110+0000", 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3739e-05 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483740+0000", 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91563002900000001 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483751+0000", 2024-06-27T21:27:42.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0285000000000001e-05 2024-06-27T21:27:42.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.164 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58331 4.6 4.e4097d56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 368.46349930999997, 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80079522000000003, 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.165 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58331 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683074+0000", 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5370000000000001e-06 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683096+0000", 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1880999999999999e-05 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683109+0000", 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2858e-05 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683114+0000", 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7069999999999997e-06 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683168+0000", 2024-06-27T21:27:42.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3949999999999997e-05 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683177+0000", 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4479999999999994e-06 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483855+0000", 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80067863299999997 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483867+0000", 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2206e-05 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58364 4.6 4.a2603346 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 368.32822696900001, 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66557216799999996, 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.170 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58364 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818346+0000", 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.539e-06 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818373+0000", 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7081e-05 2024-06-27T21:27:42.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818392+0000", 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8879999999999999e-05 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818402+0000", 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7419999999999992e-06 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818457+0000", 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5664000000000001e-05 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818466+0000", 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9029999999999993e-06 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483904+0000", 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66543803300000004 2024-06-27T21:27:42.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483917+0000", 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2326e-05 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58368 4.6 4.bdd01c66 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 368.21507587999997, 2024-06-27T21:27:42.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55245678099999995, 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58368 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:27:42.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931497+0000", 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3260000000000001e-06 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931517+0000", 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0378e-05 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931528+0000", 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.078e-05 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931534+0000", 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9429999999999997e-06 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931583+0000", 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9036000000000003e-05 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931592+0000", 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8230000000000001e-06 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483946+0000", 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55235438299999995 2024-06-27T21:27:42.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483952+0000", 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1120000000000001e-06 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:103650 4.d 4:b0eeaa89:::benchmark_data_smithi192_88961_object12911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:26.926193+0000", 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 315.22037896099999, 2024-06-27T21:27:42.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27303572399999998, 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 103650 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926193+0000", 2024-06-27T21:27:42.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926193+0000", 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926194+0000", 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8640000000000001e-06 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926202+0000", 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6000000000000001e-06 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926221+0000", 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9207000000000001e-05 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926230+0000", 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8910000000000001e-06 2024-06-27T21:27:42.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926300+0000", 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9486999999999994e-05 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926444+0000", 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014414199999999999 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:27:42.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.926571+0000", 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012735700000000001 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929247+0000", 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026755699999999999 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.929348+0000", 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010126 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199160+0000", 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.26981249600000001 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199195+0000", 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4789000000000002e-05 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.199228+0000", 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3061000000000001e-05 2024-06-27T21:27:42.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDScrubReserve(1.6,QREQUEST,e:353,from:10,reservation_nonce:17)", 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:19.785054+0000", 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 262.36151730199998, 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33999884000000002, 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785054+0000", 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785054+0000", 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785055+0000", 2024-06-27T21:27:42.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0270000000000001e-06 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:19.785060+0000", 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.437e-06 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124878+0000", 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33981821699999998 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124884+0000", 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0179999999999998e-06 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124930+0000", 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6521999999999999e-05 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.124946+0000", 2024-06-27T21:27:42.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5347999999999999e-05 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:20.125053+0000", 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107271 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163683 4.d 4.883d3e0d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:27:42.191 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 238.236253025, 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.612742381, 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163683 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:27:42.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910320+0000", 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.08e-06 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910324+0000", 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7899999999999999e-06 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910336+0000", 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1676e-05 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910341+0000", 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2360000000000004e-06 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910386+0000", 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4273000000000001e-05 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910393+0000", 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2790000000000001e-06 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523031+0000", 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.612637722 2024-06-27T21:27:42.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523061+0000", 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0324999999999998e-05 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163697 4.d 4.ba6d2ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:27:42.196 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 238.21384152799999, 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59065456900000002, 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163697 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932731+0000", 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-06-27T21:27:42.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932735+0000", 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3970000000000004e-06 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932745+0000", 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7929999999999992e-06 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932750+0000", 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2100000000000001e-06 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932794+0000", 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3946000000000002e-05 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932802+0000", 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5909999999999998e-06 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523377+0000", 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59057547099999996 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523385+0000", 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2579999999999998e-06 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.201 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163710 4.d 4.4da514d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 238.20046883099999, 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57733542599999998, 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163710 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.202 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946104+0000", 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.277e-06 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946109+0000", 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3809999999999998e-06 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946122+0000", 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3040000000000001e-05 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946130+0000", 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8469999999999996e-06 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946187+0000", 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7229999999999999e-05 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946199+0000", 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-05 2024-06-27T21:27:42.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523432+0000", 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57723327000000002 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523438+0000", 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8309999999999997e-06 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163683 4.d 4:b07cbc11:::benchmark_data_smithi192_88961_object72944:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 236.61566242999999, 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46728570899999999, 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163683 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.207 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530911+0000", 2024-06-27T21:27:42.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2199999999999999e-06 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530921+0000", 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1949999999999992e-06 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2262562657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:42.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530939+0000", 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8541000000000001e-05 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530951+0000", 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.202e-05 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531024+0000", 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2605e-05 2024-06-27T21:27:42.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531031+0000", 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6669999999999996e-06 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.980931+0000", 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44989959299999999 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.981077+0000", 2024-06-27T21:27:42.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014631999999999999 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.981221+0000", 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000143874 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.987926+0000", 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006704738 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991110+0000", 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031838470000000001 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.997913+0000", 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006803478 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.997982+0000", 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9077000000000003e-05 2024-06-27T21:27:42.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.998195+0000", 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021253400000000001 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163697 4.d 4:b54b65d0:::benchmark_data_smithi192_88961_object72958:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 236.61558284500001, 2024-06-27T21:27:42.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47026454699999998, 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163697 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:27:42.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530993+0000", 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5439999999999999e-06 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531002+0000", 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0939999999999997e-06 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531007+0000", 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4369999999999998e-06 2024-06-27T21:27:42.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531076+0000", 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8229e-05 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531084+0000", 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9260000000000006e-06 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984774+0000", 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45368975900000003 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984852+0000", 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7356999999999998e-05 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984929+0000", 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7101999999999995e-05 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991124+0000", 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0061952420000000001 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.995794+0000", 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004670319 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001042+0000", 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0052480169999999998 2024-06-27T21:27:42.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001078+0000", 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5355000000000003e-05 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001253+0000", 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017562099999999999 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163710 4.d 4:b28a5b20:::benchmark_data_smithi192_88961_object72971:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 236.615554788, 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47328603800000002, 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163710 2024-06-27T21:27:42.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4299999999999999e-07 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531019+0000", 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3039999999999999e-06 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531025+0000", 2024-06-27T21:27:42.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.874e-06 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531030+0000", 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2359999999999998e-06 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531108+0000", 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8033000000000003e-05 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531115+0000", 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1980000000000004e-06 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988002+0000", 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45688679300000001 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988079+0000", 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7922000000000004e-05 2024-06-27T21:27:42.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988158+0000", 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991184+0000", 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030261609999999999 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.995891+0000", 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004706422 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004153+0000", 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082619319999999996 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004184+0000", 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1028999999999997e-05 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004303+0000", 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119159 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e375,346 4:2e81f327:::benchmark_data_smithi192_88961_object15186:head,336'6645)", 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:24.018765+0000", 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 198.127806083, 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35042788800000002, 2024-06-27T21:27:42.230 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018765+0000", 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018765+0000", 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018766+0000", 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.018770+0000", 2024-06-27T21:27:42.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8680000000000001e-06 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369042+0000", 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35027164199999999 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369053+0000", 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0964e-05 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369116+0000", 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3647999999999998e-05 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369124+0000", 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1100000000000003e-06 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:24.369193+0000", 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9045e-05 2024-06-27T21:27:42.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e376,346 4:26f4a754:::benchmark_data_smithi192_88961_object19828:head,337'6948)", 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:26.386183+0000", 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 195.76038825399999, 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51000209500000004, 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386183+0000", 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386183+0000", 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386184+0000", 2024-06-27T21:27:42.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1699999999999997e-07 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386188+0000", 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9559999999999999e-06 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896011+0000", 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50982296199999999 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896020+0000", 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4910000000000005e-06 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896110+0000", 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0766000000000005e-05 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896122+0000", 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2082000000000001e-05 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896185+0000", 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2921000000000003e-05 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e376,346 4:238bb66b:::benchmark_data_smithi192_88961_object20912:head,338'7005)", 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:27.447281+0000", 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 194.69929046199999, 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.273843911, 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.239 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.447281+0000", 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.447281+0000", 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.447283+0000", 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.57e-06 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.447293+0000", 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0096e-05 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.720957+0000", 2024-06-27T21:27:42.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27366473899999999 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.720967+0000", 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.533e-06 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.721057+0000", 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0208000000000001e-05 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.721065+0000", 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0600000000000008e-06 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:27.721125+0000", 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9704999999999998e-05 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.243 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:2b2a0b1c:::benchmark_data_smithi192_88961_object26504:head,341'7333)", 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:29.727966+0000", 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.41860526299999, 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67969687300000003, 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.727966+0000", 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.727966+0000", 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.727968+0000", 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5030000000000001e-06 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.728000+0000", 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2143e-05 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407501+0000", 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67950098000000003 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407512+0000", 2024-06-27T21:27:42.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.065e-05 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407586+0000", 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4127000000000004e-05 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407598+0000", 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1833e-05 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407663+0000", 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5636999999999995e-05 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:27ab21ae:::benchmark_data_smithi192_88961_object28689:head,342'7457)", 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:32.115635+0000", 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 190.03093687000001, 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61824737100000005, 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.248 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115635+0000", 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115635+0000", 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115636+0000", 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.483e-06 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115651+0000", 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4844000000000001e-05 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733722+0000", 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.618070546 2024-06-27T21:27:42.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733731+0000", 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3570000000000004e-06 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733813+0000", 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1669999999999996e-05 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733819+0000", 2024-06-27T21:27:42.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8970000000000002e-06 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733882+0000", 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2574000000000001e-05 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:2acc291c:::benchmark_data_smithi192_88961_object33232:head,343'7743)", 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:36.282894+0000", 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 185.86367766500001, 2024-06-27T21:27:42.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63189966099999995, 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282894+0000", 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282894+0000", 2024-06-27T21:27:42.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282895+0000", 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7599999999999997e-07 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282899+0000", 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.729e-06 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:27:42.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914615+0000", 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63171538800000004 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914642+0000", 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7322e-05 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914707+0000", 2024-06-27T21:27:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4821999999999997e-05 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914717+0000", 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0548e-05 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914794+0000", 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6175999999999997e-05 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.256 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.257 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.257 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:27:42.257 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:27:42.257 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:27:42.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:42.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2262562657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:42.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:42.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:42.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:42.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:42.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:42.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:42.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:42.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:42.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:42.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:42.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:42.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:43.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:43.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:43.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:27:43.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:42 smithi151 ceph-mon[28965]: pgmap v1150: 27 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 22 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 582 op/s; 6.3 MiB/s, 119 objects/s recovering 2024-06-27T21:27:43.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:43.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:43.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:42 smithi190 ceph-mon[32247]: pgmap v1150: 27 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 22 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 582 op/s; 6.3 MiB/s, 119 objects/s recovering 2024-06-27T21:27:43.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:42 smithi192 ceph-mon[32956]: pgmap v1150: 27 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 22 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 582 op/s; 6.3 MiB/s, 119 objects/s recovering 2024-06-27T21:27:43.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:43.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:43.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:43.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:43.501 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:43.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:43.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:43.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:43.753 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:27:43.754 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:27:43.754 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-06-27T21:27:43.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-06-27T21:27:43.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:43.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:43.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:43.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:43.968 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:27:43.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:44.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:44.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:44.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:43 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 102/162828 objects degraded (0.063%), 1 pg degraded) 2024-06-27T21:27:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2263347642' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:44.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:43 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 102/162828 objects degraded (0.063%), 1 pg degraded) 2024-06-27T21:27:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2263347642' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:44.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:44.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:43 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 102/162828 objects degraded (0.063%), 1 pg degraded) 2024-06-27T21:27:44.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2263347642' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:27:44.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:44.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:44.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:44.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:44.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.678 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:44.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:44.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:44.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:27:44.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:44.844 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:44.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:44.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:44.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:45.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:45.059 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:45.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:45.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:45.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:44 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:27:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:44 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:27:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2263347642' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:44 smithi190 ceph-mon[32247]: osdmap e443: 12 total, 12 up, 12 in 2024-06-27T21:27:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:44 smithi190 ceph-mon[32247]: pgmap v1152: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 22 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 6172/156990 objects degraded (3.931%); 6.8 MiB/s, 182 objects/s recovering 2024-06-27T21:27:45.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:44 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:27:45.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:44 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:27:45.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2263347642' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:45.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:44 smithi151 ceph-mon[28965]: osdmap e443: 12 total, 12 up, 12 in 2024-06-27T21:27:45.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:44 smithi151 ceph-mon[28965]: pgmap v1152: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 22 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 6172/156990 objects degraded (3.931%); 6.8 MiB/s, 182 objects/s recovering 2024-06-27T21:27:45.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:44 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:27:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:44 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:27:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2263347642' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:27:45.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:44 smithi192 ceph-mon[32956]: osdmap e443: 12 total, 12 up, 12 in 2024-06-27T21:27:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:44 smithi192 ceph-mon[32956]: pgmap v1152: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 22 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 6172/156990 objects degraded (3.931%); 6.8 MiB/s, 182 objects/s recovering 2024-06-27T21:27:45.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:45.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:45.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:45.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:45.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:45.650 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:45.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:45.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:45.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:45.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:45.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:45.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:45.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:46.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:46.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:46.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:45 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:27:46.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:45 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:27:46.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:45 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 6172/156990 objects degraded (3.931%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:46.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:45 smithi151 ceph-mon[28965]: osdmap e444: 12 total, 12 up, 12 in 2024-06-27T21:27:46.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:46.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:45 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:27:46.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:45 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:27:46.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:45 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 6172/156990 objects degraded (3.931%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:46.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:45 smithi190 ceph-mon[32247]: osdmap e444: 12 total, 12 up, 12 in 2024-06-27T21:27:46.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:46.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:45 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:27:46.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:45 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:27:46.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:45 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 6172/156990 objects degraded (3.931%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:27:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:45 smithi192 ceph-mon[32956]: osdmap e444: 12 total, 12 up, 12 in 2024-06-27T21:27:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:46.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:46.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:46.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:46.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:46.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:46.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:46.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:27:46.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:46.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:46.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:46.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:46.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:47.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:47.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:47.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:47.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:47.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:47.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:47.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:47.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:47.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:47.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:47.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:47.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:47.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:47 smithi192 ceph-mon[32956]: pgmap v1154: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 23 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 6172/152019 objects degraded (4.060%); 3.0 MiB/s, 153 objects/s recovering 2024-06-27T21:27:47.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:47 smithi192 ceph-mon[32956]: osdmap e445: 12 total, 12 up, 12 in 2024-06-27T21:27:47.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:47 smithi192 ceph-mon[32956]: 5.6 scrub starts 2024-06-27T21:27:47.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:47 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:27:47.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:47 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:27:47.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:47.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:47.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:47.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:47 smithi151 ceph-mon[28965]: pgmap v1154: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 23 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 6172/152019 objects degraded (4.060%); 3.0 MiB/s, 153 objects/s recovering 2024-06-27T21:27:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:47 smithi151 ceph-mon[28965]: osdmap e445: 12 total, 12 up, 12 in 2024-06-27T21:27:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:47 smithi151 ceph-mon[28965]: 5.6 scrub starts 2024-06-27T21:27:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:47 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:27:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:47 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:27:47.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:48.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:47 smithi190 ceph-mon[32247]: pgmap v1154: 27 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 23 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 6172/152019 objects degraded (4.060%); 3.0 MiB/s, 153 objects/s recovering 2024-06-27T21:27:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:47 smithi190 ceph-mon[32247]: osdmap e445: 12 total, 12 up, 12 in 2024-06-27T21:27:48.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:47 smithi190 ceph-mon[32247]: 5.6 scrub starts 2024-06-27T21:27:48.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:47 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:27:48.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:47 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:27:48.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:48.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:48.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:48.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:48.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:48.378 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:27:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:27:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:27:48.378 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:27:48.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:48.460 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:48.477 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:27:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:27:48.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:48.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:48.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:48.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:48.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:48 smithi151 ceph-mon[28965]: 5.5 deep-scrub starts 2024-06-27T21:27:48.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:27:48.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:27:48.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:48 smithi151 ceph-mon[28965]: 5.6 scrub ok 2024-06-27T21:27:48.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:48.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:48.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:48.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:48.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:48 smithi190 ceph-mon[32247]: 5.5 deep-scrub starts 2024-06-27T21:27:49.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:27:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:27:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:48 smithi190 ceph-mon[32247]: 5.6 scrub ok 2024-06-27T21:27:49.015 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:48 smithi192 ceph-mon[32956]: 5.5 deep-scrub starts 2024-06-27T21:27:49.015 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:27:49.015 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:27:49.015 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:48 smithi192 ceph-mon[32956]: 5.6 scrub ok 2024-06-27T21:27:49.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:49.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:49.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:27:49.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:49.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:49.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:49.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:49.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:49.422 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:49.438 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:49.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:49.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:49.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:49.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:49.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:49.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:49.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6172/152019 objects degraded (4.060%), 1 pg degraded) 2024-06-27T21:27:49.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:27:49.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: osdmap e446: 12 total, 12 up, 12 in 2024-06-27T21:27:49.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:27:49.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:27:49.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:27:49.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:27:49.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:27:49.828 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:27:49.829 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:49 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:49.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:49.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6172/152019 objects degraded (4.060%), 1 pg degraded) 2024-06-27T21:27:49.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:27:49.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: osdmap e446: 12 total, 12 up, 12 in 2024-06-27T21:27:49.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:27:49.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:27:49.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:27:49.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:27:49.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:27:49.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:27:49.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:49 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:49.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:49.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6172/152019 objects degraded (4.060%), 1 pg degraded) 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: osdmap e446: 12 total, 12 up, 12 in 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:27:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:49 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:27:50.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:50.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:27:50.142 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:27:50.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-06-27T21:27:50.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:50.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:50.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:50.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:50.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:50.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:50.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:50.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:50.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:50.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:50.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:50.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:50.923 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:50 smithi192 ceph-mon[32956]: 5.7 deep-scrub starts 2024-06-27T21:27:50.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:50 smithi151 ceph-mon[28965]: 5.7 deep-scrub starts 2024-06-27T21:27:50.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:50.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:50.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:51.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:50 smithi190 ceph-mon[32247]: 5.7 deep-scrub starts 2024-06-27T21:27:51.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:51.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:51.192 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:27:51.192 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:27:51.192 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:27:51.192 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:27:51.192 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:27:51.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:51.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:51.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:51.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:51.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:51.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:51.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:51.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:27:51.612 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:27:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:51.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:51.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:27:51.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:27:51.683 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:27:51.684 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:27:51.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:27:51.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:27:51.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:51.748 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:27:51.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:27:51.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:51.787 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:27:51.787 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:27:51.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:27:51.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:27:51.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:51.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:51 smithi151 ceph-mon[28965]: pgmap v1158: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s; 2953/143691 objects misplaced (2.055%); 1.9 MiB/s, 149 objects/s recovering 2024-06-27T21:27:51.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:51 smithi151 ceph-mon[28965]: 5.3 scrub starts 2024-06-27T21:27:51.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/719042283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:51.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:27:51.866 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:27:51.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:51.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:51.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:27:51.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:27:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:51 smithi192 ceph-mon[32956]: pgmap v1158: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s; 2953/143691 objects misplaced (2.055%); 1.9 MiB/s, 149 objects/s recovering 2024-06-27T21:27:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:51 smithi192 ceph-mon[32956]: 5.3 scrub starts 2024-06-27T21:27:51.910 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/719042283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:51.941 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:27:51.942 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:27:51.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:27:51.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:27:51.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:27:51.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:51 smithi190 ceph-mon[32247]: pgmap v1158: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 738 op/s; 2953/143691 objects misplaced (2.055%); 1.9 MiB/s, 149 objects/s recovering 2024-06-27T21:27:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:51 smithi190 ceph-mon[32247]: 5.3 scrub starts 2024-06-27T21:27:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/719042283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:27:52.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:52.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:27:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:27:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:52.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:52.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:52.246 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:52.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:52.274 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:52.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:52.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:52.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:52.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:52.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:52.637 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:27:52.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:52.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:52 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:27:52.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:52 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:27:52.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:52 smithi151 ceph-mon[28965]: 5.3 scrub ok 2024-06-27T21:27:52.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1596736308' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:27:52.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:52.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:52.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:52.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:52.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:52.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:52 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:27:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:52 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:27:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:52 smithi190 ceph-mon[32247]: 5.3 scrub ok 2024-06-27T21:27:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1596736308' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:27:53.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:52 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:27:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:52 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:27:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:52 smithi192 ceph-mon[32956]: 5.3 scrub ok 2024-06-27T21:27:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1596736308' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:27:53.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:53.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:53.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:53.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:53.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:53.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:53.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:53.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:53.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:53.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:53.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:53.771 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:53 smithi192 ceph-mon[32956]: pgmap v1159: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 637 op/s; 2953/140424 objects misplaced (2.103%); 1.8 MiB/s, 93 objects/s recovering 2024-06-27T21:27:53.771 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:53 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:53.771 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1596736308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:27:53.771 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:53 smithi192 ceph-mon[32956]: osdmap e447: 12 total, 12 up, 12 in 2024-06-27T21:27:53.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:53.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:53 smithi151 ceph-mon[28965]: pgmap v1159: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 637 op/s; 2953/140424 objects misplaced (2.103%); 1.8 MiB/s, 93 objects/s recovering 2024-06-27T21:27:53.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:53 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:53.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1596736308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:27:53.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:53 smithi151 ceph-mon[28965]: osdmap e447: 12 total, 12 up, 12 in 2024-06-27T21:27:53.850 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:53.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:53.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:53.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:53 smithi190 ceph-mon[32247]: pgmap v1159: 27 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 637 op/s; 2953/140424 objects misplaced (2.103%); 1.8 MiB/s, 93 objects/s recovering 2024-06-27T21:27:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:53 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:27:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1596736308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:27:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:53 smithi190 ceph-mon[32247]: osdmap e447: 12 total, 12 up, 12 in 2024-06-27T21:27:54.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:54.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:54.178 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:54.178 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:54.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:54.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:54.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:54.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:54.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:54.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:54.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:54.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:54.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:54.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:54.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:54.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:27:54.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:54.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:54.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:27:54.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:55.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:55.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:55.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:55.186 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:55.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:55.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:55.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:55.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:55.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:55.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:55.575 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:55.592 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:27:55.592 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:27:55.592 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:27:55.592 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:27:55.593 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:27:55.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:27:55.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:55.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:55.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:55.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:55.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:55.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:55.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:55 smithi151 ceph-mon[28965]: pgmap v1161: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 23 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 2733/137844 objects misplaced (1.983%); 3.4 MiB/s, 110 objects/s recovering 2024-06-27T21:27:56.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:55 smithi190 ceph-mon[32247]: pgmap v1161: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 23 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 2733/137844 objects misplaced (1.983%); 3.4 MiB/s, 110 objects/s recovering 2024-06-27T21:27:56.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:56.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:55 smithi192 ceph-mon[32956]: pgmap v1161: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 23 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 2733/137844 objects misplaced (1.983%); 3.4 MiB/s, 110 objects/s recovering 2024-06-27T21:27:56.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:27:56.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:27:56.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:56.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:56.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:56.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:56.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:56.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:56.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:56.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:56.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:56.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:56.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:56.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:56.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:57.005 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:27:57.005 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:27:57.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:57.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:57.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:27:57.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:27:57.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:57.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:57.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:27:57.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:27:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:27:57.471 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:27:57.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-06-27T21:27:57.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:57.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:57.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:27:57.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:27:57.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:27:57.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:57.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:57.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:57 smithi151 ceph-mon[28965]: pgmap v1162: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 23 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 2733/134055 objects misplaced (2.039%); 1.7 MiB/s, 79 objects/s recovering 2024-06-27T21:27:57.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:57.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:57.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:57 smithi190 ceph-mon[32247]: pgmap v1162: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 23 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 2733/134055 objects misplaced (2.039%); 1.7 MiB/s, 79 objects/s recovering 2024-06-27T21:27:58.021 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:58.034 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:27:58.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:27:58.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:57 smithi192 ceph-mon[32956]: pgmap v1162: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 23 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 2733/134055 objects misplaced (2.039%); 1.7 MiB/s, 79 objects/s recovering 2024-06-27T21:27:58.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:27:58.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:27:58.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:58.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:58.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:58.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:58.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:27:58.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:27:58.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:27:58.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:27:58.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:58.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:58.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:58.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:58.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:58.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:59.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:59.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:59.146 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:59.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:27:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:59.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:27:59.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:59.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:59.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:27:59.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:27:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:27:59.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:27:59.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:27:59.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:27:59.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:27:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:27:59.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:27:59.922 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:27:59 smithi192 ceph-mon[32956]: pgmap v1163: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 2555/131379 objects misplaced (1.945%); 2.6 MiB/s, 75 objects/s recovering 2024-06-27T21:27:59.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:27:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:27:59.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:27:59.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:27:59 smithi151 ceph-mon[28965]: pgmap v1163: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 2555/131379 objects misplaced (1.945%); 2.6 MiB/s, 75 objects/s recovering 2024-06-27T21:28:00.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:27:59 smithi190 ceph-mon[32247]: pgmap v1163: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 2555/131379 objects misplaced (1.945%); 2.6 MiB/s, 75 objects/s recovering 2024-06-27T21:28:00.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:00.064 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:28:00.064 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:28:00.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:00.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:00.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:00.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:00.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:00.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:00.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:00.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:00.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:00.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:00.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:00.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:00.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:28:00.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:00.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:00.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:00.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:00.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:01.072 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:28:01.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:01.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:01.156 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:28:01.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:28:01.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:28:01.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:28:01.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:01.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:28:01.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:28:01.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:01.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:28:01.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:28:01.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:28:01.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:28:01.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:28:01.325 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:28:01.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:01.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:28:01.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:28:01.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:01.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:01.396 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:28:01.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:28:01.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:28:01.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:28:01.463 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:28:01.464 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:28:01.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:01.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:28:01.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:28:01.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:01.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:28:01.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:28:01.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:01.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:01.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:01.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:01.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:01 smithi192 ceph-mon[32956]: pgmap v1164: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 2555/128163 objects misplaced (1.994%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:28:01.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4193650456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:01.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:01.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:01.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:01.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:01 smithi151 ceph-mon[28965]: pgmap v1164: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 2555/128163 objects misplaced (1.994%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:28:01.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4193650456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:01.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:02.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:01 smithi190 ceph-mon[32247]: pgmap v1164: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 2555/128163 objects misplaced (1.994%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:28:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4193650456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:02.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:02.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:02.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:02.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:02.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:02.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:02.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:02.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:02.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:02.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:02.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:02.658 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:28:02.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:02.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:02.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/105416220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:02.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:02 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.15132.0:213264 3.4 3:20d1c801:::benchmark_data_smithi192_74314_object106626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:07.087204+0000", 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 475.73734815199998, 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.380334114, 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.827 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.15132", 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2143916733", 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 213264 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087204+0000", 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087204+0000", 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087206+0000", 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1829999999999999e-06 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087210+0000", 2024-06-27T21:28:02.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3950000000000003e-06 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087221+0000", 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0711e-05 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087226+0000", 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2989999999999996e-06 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087272+0000", 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5618000000000003e-05 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087358+0000", 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6531000000000002e-05 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.087434+0000", 2024-06-27T21:28:02.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5946999999999996e-05 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.091650+0000", 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042159090000000003 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.102657+0000", 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011007082 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.467432+0000", 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36477513700000003 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.467494+0000", 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1838e-05 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:07.467539+0000", 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4464e-05 2024-06-27T21:28:02.833 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:9486 4.3 4:c476da72:::benchmark_data_smithi192_88961_object9485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 432.43553097199998, 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.31799833300000002, 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.834 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 9486 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389022+0000", 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4700000000000001e-07 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389039+0000", 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6892999999999999e-05 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389049+0000", 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9620000000000005e-06 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389055+0000", 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3770000000000004e-06 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389102+0000", 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7261e-05 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389216+0000", 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000114268 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389358+0000", 2024-06-27T21:28:02.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014186 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.389912+0000", 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00055357200000000001 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.396996+0000", 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070839789999999998 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.706949+0000", 2024-06-27T21:28:02.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30995374999999997 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.706995+0000", 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6125000000000003e-05 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.707020+0000", 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4539e-05 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:9521 4.3 4:c314fc9a:::benchmark_data_smithi192_88961_object9520:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:20:50.565666+0000", 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 432.258886631, 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53533422500000005, 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.841 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 9521 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565666+0000", 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565666+0000", 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565667+0000", 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1800000000000005e-07 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565706+0000", 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9322000000000001e-05 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565715+0000", 2024-06-27T21:28:02.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3780000000000007e-06 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565720+0000", 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.656e-06 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565770+0000", 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9424e-05 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.565887+0000", 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011731900000000001 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.566033+0000", 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014573599999999999 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.566532+0000", 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049953499999999995 2024-06-27T21:28:02.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:50.657244+0000", 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.090711438000000005 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:51.100924+0000", 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.443680713 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:51.100959+0000", 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4941000000000003e-05 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:20:51.101000+0000", 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0945000000000002e-05 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:20969 4.9 e301/282 ondisk, result = 0)", 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:28:02.847 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 421.28130953700003, 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42756724600000001, 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:28:02.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543243+0000", 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5199999999999998e-07 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.543245+0000", 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.815e-06 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970713+0000", 2024-06-27T21:28:02.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42746736499999999 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970738+0000", 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5360000000000001e-05 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970792+0000", 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4327999999999999e-05 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970800+0000", 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4120000000000004e-06 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.970810+0000", 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0614e-05 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.851 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:20969 4.9 e301/282 ondisk, result = 0)", 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 421.20456451699999, 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.378104671, 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619988+0000", 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9400000000000001e-07 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.619990+0000", 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.827e-06 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997896+0000", 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.377906097 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997906+0000", 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5459999999999997e-06 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997967+0000", 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0730999999999998e-05 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.997980+0000", 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3223e-05 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.998093+0000", 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000112853 2024-06-27T21:28:02.855 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:20970 4.3 e301/282 ondisk, result = 0)", 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:01.850376+0000", 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 420.97417668200001, 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.340704061, 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850376+0000", 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850376+0000", 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850378+0000", 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.796e-06 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:01.850386+0000", 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6370000000000005e-06 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190672+0000", 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34028516800000003 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190689+0000", 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7036000000000001e-05 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190815+0000", 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012613199999999999 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.190833+0000", 2024-06-27T21:28:02.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7813e-05 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:02.191080+0000", 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024747899999999999 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.861 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58360 4.8 4:1fb1449d:::benchmark_data_smithi192_88961_object58359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 389.01218908099997, 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78704928399999996, 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58360 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:28:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3799999999999996e-07 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812404+0000", 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9298999999999998e-05 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812413+0000", 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6090000000000007e-06 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812419+0000", 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4029999999999998e-06 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812825+0000", 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040641499999999998 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812925+0000", 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9977999999999997e-05 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.813074+0000", 2024-06-27T21:28:02.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000149226 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.814127+0000", 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010522940000000001 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815268+0000", 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011417160000000001 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599347+0000", 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78407841099999998 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599381+0000", 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4666000000000003e-05 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599413+0000", 2024-06-27T21:28:02.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1429e-05 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:58360 4.8 e314/282 ondisk, result = 0)", 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 389.00941704899998, 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78428261799999999, 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.869 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000002e-07 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815139+0000", 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.354e-06 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599259+0000", 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78411979600000004 2024-06-27T21:28:02.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599265+0000", 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599336+0000", 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0346999999999996e-05 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599343+0000", 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1319999999999996e-06 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599418+0000", 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5303999999999996e-05 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:58365 4.f e314/282 4:fc83d7dd:::benchmark_data_smithi192_88961_object58364:head v 314'3626, mlcod=314'3624)", 2024-06-27T21:28:02.873 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 389.00507210699999, 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29582368799999997, 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9299999999999997e-07 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819529+0000", 2024-06-27T21:28:02.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8269000000000002e-05 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113508+0000", 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29397816399999999 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113516+0000", 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6770000000000001e-06 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113587+0000", 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0828999999999996e-05 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113604+0000", 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6643999999999999e-05 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.115270+0000", 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016662960000000001 2024-06-27T21:28:02.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.115304+0000", 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4116000000000003e-05 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:66868 4.9 4:98080da8:::benchmark_data_smithi192_88961_object66867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:44.949612+0000", 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 377.87494054899997, 2024-06-27T21:28:02.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.419813296, 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 66868 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949612+0000", 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949612+0000", 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949614+0000", 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.525e-06 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949662+0000", 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8232000000000002e-05 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949673+0000", 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1382e-05 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949678+0000", 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2360000000000004e-06 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949724+0000", 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6038000000000002e-05 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949863+0000", 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013807500000000001 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-06-27T21:28:02.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.950013+0000", 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000150247 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952324+0000", 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002310744 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.957959+0000", 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056356289999999996 2024-06-27T21:28:02.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369353+0000", 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41139374899999998 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369385+0000", 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2141999999999998e-05 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369425+0000", 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0296999999999997e-05 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:66868 4.9 e320/282 ondisk, result = 0)", 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:44.952331+0000", 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 377.87222192500002, 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41710191200000002, 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.885 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952331+0000", 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952331+0000", 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952332+0000", 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952338+0000", 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.387e-06 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369292+0000", 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41695419099999997 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369297+0000", 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9479999999999999e-06 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369346+0000", 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8989000000000001e-05 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369350+0000", 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8279999999999996e-06 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369433+0000", 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2727999999999998e-05 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:66878 4.3 4:c506cff3:::benchmark_data_smithi192_88961_object66877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:44.970564+0000", 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 377.85398840400001, 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33568430300000002, 2024-06-27T21:28:02.889 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 66878 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970564+0000", 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970564+0000", 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970566+0000", 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6789999999999999e-06 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970586+0000", 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0545999999999999e-05 2024-06-27T21:28:02.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970605+0000", 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8598999999999999e-05 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970612+0000", 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3860000000000001e-06 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970675+0000", 2024-06-27T21:28:02.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2163999999999996e-05 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970808+0000", 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013358099999999999 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970951+0000", 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014247500000000001 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.973362+0000", 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024116570000000002 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.980807+0000", 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0074446 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.306177+0000", 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.32536981100000001 2024-06-27T21:28:02.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.306218+0000", 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1257000000000002e-05 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.306249+0000", 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0548e-05 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:103647 4.9 4:98ce20a8:::benchmark_data_smithi192_88961_object12908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:26.909872+0000", 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 335.91468042399998, 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30295640299999999, 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 103647 2024-06-27T21:28:02.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909872+0000", 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909872+0000", 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909873+0000", 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.099e-06 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909877+0000", 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0539999999999996e-06 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909888+0000", 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1126e-05 2024-06-27T21:28:02.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909894+0000", 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0270000000000001e-06 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909946+0000", 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1261999999999998e-05 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.910070+0000", 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012383199999999999 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.910186+0000", 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011657 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.912728+0000", 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002542304 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.913604+0000", 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00087590699999999999 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.212665+0000", 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29906064999999998 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.212787+0000", 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012225199999999999 2024-06-27T21:28:02.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.212829+0000", 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1319999999999997e-05 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:130464 4.8 4:198f8843:::benchmark_data_smithi192_88961_object39725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 306.85438079099998, 2024-06-27T21:28:02.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.28377075699999998, 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 130464 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6400000000000002e-07 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970177+0000", 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1219999999999997e-06 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970186+0000", 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6900000000000004e-06 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970191+0000", 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0660000000000003e-06 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970227+0000", 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5488000000000003e-05 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970302+0000", 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4805e-05 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970364+0000", 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2287999999999997e-05 2024-06-27T21:28:02.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.973331+0000", 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029675000000000001 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.973628+0000", 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029691 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.253863+0000", 2024-06-27T21:28:02.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.28023458699999998 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.253898+0000", 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4873000000000003e-05 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.253943+0000", 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4764e-05 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:130521 4.1 4:8d4b026c:::benchmark_data_smithi192_88961_object39782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:56.564874+0000", 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 306.2596782, 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41357545899999998, 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.909 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 130521 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564874+0000", 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564874+0000", 2024-06-27T21:28:02.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564875+0000", 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564878+0000", 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.768e-06 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564886+0000", 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2269999999999993e-06 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564891+0000", 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9429999999999999e-06 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564933+0000", 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2069e-05 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.565024+0000", 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1246e-05 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,6", 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.565094+0000", 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9430000000000004e-05 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.568308+0000", 2024-06-27T21:28:02.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003213905 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.568317+0000", 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1740000000000006e-06 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.978368+0000", 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41005131299999997 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.978408+0000", 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9920999999999999e-05 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.978450+0000", 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1727000000000002e-05 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163716 4.3 4:ce21cfad:::benchmark_data_smithi192_88961_object72977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.970070+0000", 2024-06-27T21:28:02.915 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 258.85448298799997, 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70507940300000005, 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163716 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970070+0000", 2024-06-27T21:28:02.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970070+0000", 2024-06-27T21:28:02.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:02.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/105416220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:02.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:02 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970071+0000", 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5149999999999999e-06 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970078+0000", 2024-06-27T21:28:02.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1400000000000002e-06 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970092+0000", 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3871e-05 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970100+0000", 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8259999999999993e-06 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970157+0000", 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6589999999999999e-05 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970293+0000", 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136756 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970425+0000", 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131297 2024-06-27T21:28:02.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.974073+0000", 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036481590000000002 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.974485+0000", 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041243200000000001 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.675074+0000", 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70058905100000002 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.675115+0000", 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0936999999999998e-05 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.675149+0000", 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3828999999999997e-05 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.923 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152341 5.d 5:bebb6ac0:::benchmark_data_smithi192_104167_object46300:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902303+0000", 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 34.922249307999998, 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35944219999999999, 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:28:02.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152341 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902303+0000", 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902303+0000", 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902305+0000", 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6750000000000001e-06 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902312+0000", 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4719999999999998e-06 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902326+0000", 2024-06-27T21:28:02.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3736e-05 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902338+0000", 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1764e-05 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906260+0000", 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039215999999999999 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906274+0000", 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4262e-05 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.261595+0000", 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35532122599999999 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.261746+0000", 2024-06-27T21:28:02.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015046499999999999 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152348 5.d 5:bda439c2:::benchmark_data_smithi192_104167_object46307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902363+0000", 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 34.922189351999997, 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36148766799999998, 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.929 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152348 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902363+0000", 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902363+0000", 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902364+0000", 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902366+0000", 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0229999999999999e-06 2024-06-27T21:28:02.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902371+0000", 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0520000000000001e-06 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902374+0000", 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4599999999999999e-06 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906311+0000", 2024-06-27T21:28:02.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003936602 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906316+0000", 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5169999999999999e-06 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.263731+0000", 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35741499399999999 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.263851+0000", 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119704 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152349 5.d 5:b479a3b3:::benchmark_data_smithi192_104167_object46308:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 34.922173921999999, 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36166314599999999, 2024-06-27T21:28:02.934 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152349 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.36e-07 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902380+0000", 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.536e-06 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902385+0000", 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.374e-06 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902388+0000", 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9399999999999998e-06 2024-06-27T21:28:02.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906356+0000", 2024-06-27T21:28:02.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039686519999999996 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906364+0000", 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7139999999999993e-06 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.263887+0000", 2024-06-27T21:28:02.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35752254500000002 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264042+0000", 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015514899999999999 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152370 5.d 5:b37fea35:::benchmark_data_smithi192_104167_object46329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:28:02.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 34.922161457000001, 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37976241999999999, 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152370 2024-06-27T21:28:02.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.41e-07 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902393+0000", 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.672e-06 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902396+0000", 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3129999999999999e-06 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902399+0000", 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8679999999999999e-06 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906420+0000", 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040211190000000001 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906429+0000", 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6139999999999992e-06 2024-06-27T21:28:02.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264078+0000", 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35764881300000001 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264194+0000", 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011598499999999999 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,5", 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264288+0000", 2024-06-27T21:28:02.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4084000000000002e-05 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.276652+0000", 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012364557 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.280074+0000", 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00342173 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.280164+0000", 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9519999999999997e-05 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.280207+0000", 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3340999999999997e-05 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:28:02.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.282154+0000", 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019465629999999999 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:28:02.948 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:28:02.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/105416220' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:02.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:02 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:02.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:02.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:03.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:03.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:03.143 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:03.143 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:03.143 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:03.143 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:03.143 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:03.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:03.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:03.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:03.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:03.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:03.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:03.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:03.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:03.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:03.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:03.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:03.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:03.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:03.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:03 smithi151 ceph-mon[28965]: pgmap v1165: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2555/125055 objects misplaced (2.043%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:28:03.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:03 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:03.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:03.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:03 smithi151 ceph-mon[28965]: osdmap e448: 12 total, 12 up, 12 in 2024-06-27T21:28:03.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:03.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:04.001 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:03 smithi192 ceph-mon[32956]: pgmap v1165: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2555/125055 objects misplaced (2.043%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:28:04.001 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:03 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:04.002 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:04.002 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:03 smithi192 ceph-mon[32956]: osdmap e448: 12 total, 12 up, 12 in 2024-06-27T21:28:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:03 smithi190 ceph-mon[32247]: pgmap v1165: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 24 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 2555/125055 objects misplaced (2.043%); 2.5 MiB/s, 39 objects/s recovering 2024-06-27T21:28:04.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:03 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:03 smithi190 ceph-mon[32247]: osdmap e448: 12 total, 12 up, 12 in 2024-06-27T21:28:04.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:28:04.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:04.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:04.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:04.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:04.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:04.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:04.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:04.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:04.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:04.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:04.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:04.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:04.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:04.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:04.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:04.970 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:28:04.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:05.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:05.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:05.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:05.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:05.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:05.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:05.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:05.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:05.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:05.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:05.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:05.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:05.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:05.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:05.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:05 smithi151 ceph-mon[28965]: pgmap v1167: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 510 op/s; 2341/122532 objects misplaced (1.911%); 2.4 MiB/s, 39 objects/s recovering 2024-06-27T21:28:05.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:05.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:06.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:05 smithi190 ceph-mon[32247]: pgmap v1167: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 510 op/s; 2341/122532 objects misplaced (1.911%); 2.4 MiB/s, 39 objects/s recovering 2024-06-27T21:28:06.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:05 smithi192 ceph-mon[32956]: pgmap v1167: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 510 op/s; 2341/122532 objects misplaced (1.911%); 2.4 MiB/s, 39 objects/s recovering 2024-06-27T21:28:06.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:06.068 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:06.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-06-27T21:28:06.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:06.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:06.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:06.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:06.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:06.383 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:06.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:06.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:06.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:06.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:06.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:06.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:06.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:06.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:06.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:06.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:06.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:06.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:28:07.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:07.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:07.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:07.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:07.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:07.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:07.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:07.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:07.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:28:07.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:07.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:07.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:07.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:07.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:07 smithi151 ceph-mon[28965]: pgmap v1168: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 2341/117549 objects misplaced (1.992%); 2.4 MiB/s, 39 objects/s recovering 2024-06-27T21:28:07.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:07.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:07.869 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:07.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:07.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:07.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:08.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:07 smithi190 ceph-mon[32247]: pgmap v1168: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 2341/117549 objects misplaced (1.992%); 2.4 MiB/s, 39 objects/s recovering 2024-06-27T21:28:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:08.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:07 smithi192 ceph-mon[32956]: pgmap v1168: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 25 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 550 op/s; 2341/117549 objects misplaced (1.992%); 2.4 MiB/s, 39 objects/s recovering 2024-06-27T21:28:08.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:08.136 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:08.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:08.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:08.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:08.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:08.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:08.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:08.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:08.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:08.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:08.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:08.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:08.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:08.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:08.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:08.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:08 smithi151 ceph-mon[28965]: osdmap e449: 12 total, 12 up, 12 in 2024-06-27T21:28:09.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:08 smithi190 ceph-mon[32247]: osdmap e449: 12 total, 12 up, 12 in 2024-06-27T21:28:09.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:09.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:08 smithi192 ceph-mon[32956]: osdmap e449: 12 total, 12 up, 12 in 2024-06-27T21:28:09.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:09.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:09.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:09.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:09.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:09.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:09.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:09.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:09.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:09.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:09.695 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:09.696 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:09.696 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:09.696 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:09.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:09.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:09.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:09.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:09 smithi151 ceph-mon[28965]: pgmap v1170: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 2181/113259 objects misplaced (1.926%); 2.9 MiB/s, 46 objects/s recovering 2024-06-27T21:28:09.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:09 smithi151 ceph-mon[28965]: osdmap e450: 12 total, 12 up, 12 in 2024-06-27T21:28:09.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:09.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:09.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:09.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:09.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:10.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:09 smithi190 ceph-mon[32247]: pgmap v1170: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 2181/113259 objects misplaced (1.926%); 2.9 MiB/s, 46 objects/s recovering 2024-06-27T21:28:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:09 smithi190 ceph-mon[32247]: osdmap e450: 12 total, 12 up, 12 in 2024-06-27T21:28:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:09 smithi192 ceph-mon[32956]: pgmap v1170: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 2181/113259 objects misplaced (1.926%); 2.9 MiB/s, 46 objects/s recovering 2024-06-27T21:28:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:09 smithi192 ceph-mon[32956]: osdmap e450: 12 total, 12 up, 12 in 2024-06-27T21:28:10.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:10.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:10.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:10.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:10.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:10.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:10.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:10.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:10.537 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:10.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-06-27T21:28:10.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:10.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:10.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:10.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:10.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:10.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:10.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:10.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:11.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:11.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:11.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:178759 5.1 5:800157c5:::benchmark_data_smithi192_104167_object72718:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e450)", 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:11.217844+0000", 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0024669169999999999, 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002498639, 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:28:11.221 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 178759 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.217844+0000", 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:28:11.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.217844+0000", 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.217845+0000", 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:28:11.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.217850+0000", 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3290000000000001e-06 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.217861+0000", 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0327e-05 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.217867+0000", 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.579e-06 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.218123+0000", 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025566899999999998 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.218219+0000", 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5978999999999997e-05 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 7,9", 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.218363+0000", 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000144155 2024-06-27T21:28:11.225 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(5.1 450/441 [PushReplyOp(5:8d36741f:::benchmark_data_smithi192_104167_object72734:head)])", 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:11.219010+0000", 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0013012449999999999, 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013951429999999999, 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:28:11.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.219010+0000", 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.219010+0000", 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.219011+0000", 2024-06-27T21:28:11.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1200000000000001e-06 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.219017+0000", 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.925e-06 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.219036+0000", 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9431000000000001e-05 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:28:11.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:11.219048+0000", 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1263e-05 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 2 2024-06-27T21:28:11.229 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:28:11.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:11.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:11.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:11.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:11.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:11.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:11.569 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:11.572 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:11.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:11.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:11.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:11.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:11.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:11.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:11.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:11.922 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:11 smithi190 ceph-mon[32247]: pgmap v1172: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 2181/108054 objects misplaced (2.018%); 3.2 MiB/s, 50 objects/s recovering 2024-06-27T21:28:11.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:11 smithi151 ceph-mon[28965]: pgmap v1172: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 2181/108054 objects misplaced (2.018%); 3.2 MiB/s, 50 objects/s recovering 2024-06-27T21:28:12.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:12.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:11 smithi192 ceph-mon[32956]: pgmap v1172: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 2181/108054 objects misplaced (2.018%); 3.2 MiB/s, 50 objects/s recovering 2024-06-27T21:28:12.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:28:12.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:12.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:12.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:12.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:12.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:12.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:12.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:12.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:12.578 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:28:12.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:12.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:12.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:12.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:12.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:12.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:12.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:13.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:13.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:13.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:13.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:13.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:13.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:13.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:13.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:13.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:13.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:13.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:13.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:13.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:13.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:13.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:13 smithi151 ceph-mon[28965]: pgmap v1173: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 2181/103773 objects misplaced (2.102%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:28:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:13 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:13 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:13.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:13.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:13.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:13 smithi190 ceph-mon[32247]: pgmap v1173: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 2181/103773 objects misplaced (2.102%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:28:13.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:13 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:13.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:13 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:14.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:13 smithi192 ceph-mon[32956]: pgmap v1173: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s; 2181/103773 objects misplaced (2.102%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:28:14.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:13 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:13 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:14.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:14.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:14.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:14.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:14.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:14.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:14.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:14.478 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:28:14.478 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:28:14.478 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:28:14.478 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:28:14.478 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:28:14.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:14.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:14.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:14.704 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:28:14.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:14.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:14.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:14.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:14.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:14.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:14 smithi151 ceph-mon[28965]: osdmap e451: 12 total, 12 up, 12 in 2024-06-27T21:28:14.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4115878174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:28:14.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:15.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:15.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:14 smithi190 ceph-mon[32247]: osdmap e451: 12 total, 12 up, 12 in 2024-06-27T21:28:15.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4115878174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:28:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:14 smithi192 ceph-mon[32956]: osdmap e451: 12 total, 12 up, 12 in 2024-06-27T21:28:15.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4115878174' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:28:15.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:15.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:15.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:15.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:28:15.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:15.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:15.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:15.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:15.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:15.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:15.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:28:15.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:28:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:15.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:15.669 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:28:15.670 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:28:15.690 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:28:15.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:28:15.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:28:15.723 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:28:15.723 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:28:15.753 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:28:15.753 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:28:15.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:15.782 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:28:15.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:28:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:15.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:15.811 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:28:15.811 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:28:15.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:28:15.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: pgmap v1175: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 1961/99963 objects misplaced (1.962%); 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4115878174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: osdmap e452: 12 total, 12 up, 12 in 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:28:15.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3790669394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:15.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:28:15.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:28:15.898 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:28:15.899 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:28:15.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:15.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:15.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: pgmap v1175: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 1961/99963 objects misplaced (1.962%); 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4115878174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: osdmap e452: 12 total, 12 up, 12 in 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:28:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3790669394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:16.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: pgmap v1175: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 1961/99963 objects misplaced (1.962%); 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:28:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4115878174' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:28:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: osdmap e452: 12 total, 12 up, 12 in 2024-06-27T21:28:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:28:16.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:28:16.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3790669394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:16.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:16.130 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:28:16.131 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:28:16.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:16.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:16.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:28:16.288 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:28:16.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:16.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:16.359 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:16.359 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:16.359 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:16.360 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:16.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:16.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:16.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:16.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:16.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:16.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:16.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:16.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:16.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:16.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:16.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:16.990 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:16 smithi151 ceph-mon[28965]: 5.5 scrub starts 2024-06-27T21:28:16.990 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:16 smithi151 ceph-mon[28965]: 5.5 scrub ok 2024-06-27T21:28:17.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:16 smithi190 ceph-mon[32247]: 5.5 scrub starts 2024-06-27T21:28:17.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:16 smithi190 ceph-mon[32247]: 5.5 scrub ok 2024-06-27T21:28:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:16 smithi192 ceph-mon[32956]: 5.5 scrub starts 2024-06-27T21:28:17.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:16 smithi192 ceph-mon[32956]: 5.5 scrub ok 2024-06-27T21:28:17.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:17.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:17.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:17.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:17.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:17.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:17.354 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:17.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:17.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:17.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:17.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:17.613 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:17.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-06-27T21:28:17.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:17.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:17.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:17.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:17.899 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: 5.c scrub starts 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: pgmap v1177: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 1961/94092 objects misplaced (2.084%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: 5.c scrub ok 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: osdmap e453: 12 total, 12 up, 12 in 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:28:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:17 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:28:17.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:17.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:17.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: 5.c scrub starts 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: pgmap v1177: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 1961/94092 objects misplaced (2.084%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: 5.c scrub ok 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: osdmap e453: 12 total, 12 up, 12 in 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:28:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:17 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:28:18.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: 5.c scrub starts 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: pgmap v1177: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 1961/94092 objects misplaced (2.084%); 1.9 MiB/s, 29 objects/s recovering 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: 5.c scrub ok 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: osdmap e453: 12 total, 12 up, 12 in 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:28:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:17 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:28:18.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:28:18.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:18.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:18.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:18.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:18.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:18.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:18.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:18.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:18.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:18.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:18.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:18 smithi190 ceph-mon[32247]: 5.d scrub starts 2024-06-27T21:28:18.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:18.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:18.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:18.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:28:19.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:18 smithi151 ceph-mon[28965]: 5.d scrub starts 2024-06-27T21:28:19.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:19.038 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:28:19.038 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:28:19.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:18 smithi192 ceph-mon[32956]: 5.d scrub starts 2024-06-27T21:28:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:19.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:19.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:19.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:19.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:19.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:19.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:19.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:19.459 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:19.465 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:19.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:19.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:19.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:19.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:19.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:19.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:19.912 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:19 smithi151 ceph-mon[28965]: pgmap v1179: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 1720/89577 objects misplaced (1.920%); 4.8 MiB/s, 76 objects/s recovering 2024-06-27T21:28:19.912 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:19 smithi151 ceph-mon[28965]: 5.7 scrub starts 2024-06-27T21:28:19.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:19 smithi151 ceph-mon[28965]: 5.d scrub ok 2024-06-27T21:28:19.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:19 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:28:19.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:19 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:28:19.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:19.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:19.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:19.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:19 smithi190 ceph-mon[32247]: pgmap v1179: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 1720/89577 objects misplaced (1.920%); 4.8 MiB/s, 76 objects/s recovering 2024-06-27T21:28:19.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:19 smithi190 ceph-mon[32247]: 5.7 scrub starts 2024-06-27T21:28:19.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:19 smithi190 ceph-mon[32247]: 5.d scrub ok 2024-06-27T21:28:19.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:19 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:28:19.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:19 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:28:20.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:28:20.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:19 smithi192 ceph-mon[32956]: pgmap v1179: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.8 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 1720/89577 objects misplaced (1.920%); 4.8 MiB/s, 76 objects/s recovering 2024-06-27T21:28:20.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:19 smithi192 ceph-mon[32956]: 5.7 scrub starts 2024-06-27T21:28:20.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:19 smithi192 ceph-mon[32956]: 5.d scrub ok 2024-06-27T21:28:20.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:19 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:28:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:19 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:28:20.072 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:20.072 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:20.072 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:20.072 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:20.072 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:20.073 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:20.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:20.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:20.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:20.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:20.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:20.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:20.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:20.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:20.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:20.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:20.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:20.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:20.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:20.894 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:20.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:20.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:21.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:21.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:20 smithi190 ceph-mon[32247]: 5.7 scrub ok 2024-06-27T21:28:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:20 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:28:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:20 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:28:21.025 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:21.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:20 smithi192 ceph-mon[32956]: 5.7 scrub ok 2024-06-27T21:28:21.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:20 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:28:21.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:20 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:28:21.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:21.140 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:20 smithi151 ceph-mon[28965]: 5.7 scrub ok 2024-06-27T21:28:21.140 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:20 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:28:21.140 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:20 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:28:21.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:21.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:21.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:21.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:21.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:21.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:21.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:21.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:21.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:21.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:21.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:21.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:21.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:21.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:21.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:21 smithi151 ceph-mon[28965]: pgmap v1180: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 1720/84327 objects misplaced (2.040%); 2.4 MiB/s, 37 objects/s recovering 2024-06-27T21:28:22.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:21 smithi190 ceph-mon[32247]: pgmap v1180: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 1720/84327 objects misplaced (2.040%); 2.4 MiB/s, 37 objects/s recovering 2024-06-27T21:28:22.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:22.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:21 smithi192 ceph-mon[32956]: pgmap v1180: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 25 active+clean; 1.7 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 1720/84327 objects misplaced (2.040%); 2.4 MiB/s, 37 objects/s recovering 2024-06-27T21:28:22.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:22.047 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:22.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-06-27T21:28:22.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:22.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:22.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:22.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:22.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:22.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:22.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:22.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:22.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:22.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:22.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:22.795 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:22.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:22.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:22.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:22.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:22 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:22.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:22 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:22 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:22 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:23.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:22 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:23.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:22 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:23.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:23.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:23.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:23.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:23.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:23.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:23.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:23.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:23.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:23.607 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:23.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:23.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:23.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:23.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:23.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:23.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:23.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:24.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:23 smithi190 ceph-mon[32247]: pgmap v1181: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 1720/79983 objects misplaced (2.150%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:28:24.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:23 smithi190 ceph-mon[32247]: osdmap e454: 12 total, 12 up, 12 in 2024-06-27T21:28:24.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:24.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:23 smithi192 ceph-mon[32956]: pgmap v1181: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 1720/79983 objects misplaced (2.150%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:28:24.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:23 smithi192 ceph-mon[32956]: osdmap e454: 12 total, 12 up, 12 in 2024-06-27T21:28:24.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:24.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:23 smithi151 ceph-mon[28965]: pgmap v1181: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 1720/79983 objects misplaced (2.150%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:28:24.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:23 smithi151 ceph-mon[28965]: osdmap e454: 12 total, 12 up, 12 in 2024-06-27T21:28:24.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:24.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:24.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:24.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:24.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:24.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:24.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:24.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:24.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:24.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:24.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:24.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:24.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:24.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:24.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:25.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:25.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:25.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:24 smithi190 ceph-mon[32247]: osdmap e455: 12 total, 12 up, 12 in 2024-06-27T21:28:25.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:25.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:24 smithi192 ceph-mon[32956]: osdmap e455: 12 total, 12 up, 12 in 2024-06-27T21:28:25.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:25.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:25.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:24 smithi151 ceph-mon[28965]: osdmap e455: 12 total, 12 up, 12 in 2024-06-27T21:28:25.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:25.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:25.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:25.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:25.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:25.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:25.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:25.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:25.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:25.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:25.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:25.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:25.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:25.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:25.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:25.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:26.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:25 smithi190 ceph-mon[32247]: pgmap v1184: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 1448/76563 objects misplaced (1.891%); 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:28:26.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:25 smithi192 ceph-mon[32956]: pgmap v1184: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 1448/76563 objects misplaced (1.891%); 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:28:26.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:25 smithi151 ceph-mon[28965]: pgmap v1184: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 1448/76563 objects misplaced (1.891%); 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:28:26.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:26.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:26.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:26.240 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:26.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:26.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:26.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:26.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:26.555 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:28:26.555 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:28:26.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:26.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:26.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:26.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:26.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:26.865 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:28:26.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:28:26.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:28:26.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:28:26.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:28:26.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:26.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:26.908 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:28:26.908 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:28:26.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:27.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:27.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:27.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/55572351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:27.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:27.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:27.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/55572351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:27.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:27.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:27.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:27.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/55572351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:27.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:27.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:27.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:27.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:27.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:27.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:27.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:28:27.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:27.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:27.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:27.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:27.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:27.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:27.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:27.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:27.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:27.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:27 smithi151 ceph-mon[28965]: pgmap v1185: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 777 op/s; 1448/70914 objects misplaced (2.042%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:28:27.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:27 smithi151 ceph-mon[28965]: osdmap e456: 12 total, 12 up, 12 in 2024-06-27T21:28:28.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:27 smithi190 ceph-mon[32247]: pgmap v1185: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 777 op/s; 1448/70914 objects misplaced (2.042%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:28:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:27 smithi190 ceph-mon[32247]: osdmap e456: 12 total, 12 up, 12 in 2024-06-27T21:28:28.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:28.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:27 smithi192 ceph-mon[32956]: pgmap v1185: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 777 op/s; 1448/70914 objects misplaced (2.042%); 2.1 MiB/s, 33 objects/s recovering 2024-06-27T21:28:28.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:27 smithi192 ceph-mon[32956]: osdmap e456: 12 total, 12 up, 12 in 2024-06-27T21:28:28.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:28.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:28.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:28.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:28.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:28.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:28.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:28.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:28.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:28.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:28.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:28.763 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:28:28.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:28.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:28.867 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/700670494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:28.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:28.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:28.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:29.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/700670494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:29.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:29.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:29.157 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/700670494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:29.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:29.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:29.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:29.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:29.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:29.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:29.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:29.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:29.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:29.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:29.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:29.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:29.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:29.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:30.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: pgmap v1187: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 1256/66759 objects misplaced (1.881%); 4.8 MiB/s, 77 objects/s recovering 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/700670494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: osdmap e457: 12 total, 12 up, 12 in 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: 5.8 scrub starts 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:30.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: pgmap v1187: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 1256/66759 objects misplaced (1.881%); 4.8 MiB/s, 77 objects/s recovering 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/700670494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: osdmap e457: 12 total, 12 up, 12 in 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: 5.8 scrub starts 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:30.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:30.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: pgmap v1187: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 1256/66759 objects misplaced (1.881%); 4.8 MiB/s, 77 objects/s recovering 2024-06-27T21:28:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:28:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:28:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/700670494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: osdmap e457: 12 total, 12 up, 12 in 2024-06-27T21:28:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: 5.8 scrub starts 2024-06-27T21:28:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:30.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:30.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:30.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:30.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:30.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:30.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:30.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:30.664 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:30.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:30.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:30.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:28:30.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:30 smithi151 ceph-mon[28965]: 5.8 scrub ok 2024-06-27T21:28:30.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:30 smithi151 ceph-mon[28965]: osdmap e458: 12 total, 12 up, 12 in 2024-06-27T21:28:30.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:30.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:30.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:31.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:31.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:30 smithi192 ceph-mon[32956]: 5.8 scrub ok 2024-06-27T21:28:31.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:30 smithi192 ceph-mon[32956]: osdmap e458: 12 total, 12 up, 12 in 2024-06-27T21:28:31.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:31.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:31.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:31.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:30 smithi190 ceph-mon[32247]: 5.8 scrub ok 2024-06-27T21:28:31.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:30 smithi190 ceph-mon[32247]: osdmap e458: 12 total, 12 up, 12 in 2024-06-27T21:28:31.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:31.280 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:31.301 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:31.301 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:31.301 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:31.301 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:31.301 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:31.301 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:31.302 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:31.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:31.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:31.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:31.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:31.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:31.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:31.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:31.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:31.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:31.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:31.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:31 smithi151 ceph-mon[28965]: pgmap v1190: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 1256/61314 objects misplaced (2.048%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:28:31.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:31.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:32.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:31 smithi192 ceph-mon[32956]: pgmap v1190: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 1256/61314 objects misplaced (2.048%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:28:32.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:32.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:32.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:31 smithi190 ceph-mon[32247]: pgmap v1190: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.2 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 847 op/s; 1256/61314 objects misplaced (2.048%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:28:32.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:32.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:32.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:32.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:32.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.368 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:32.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:32.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:32.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:32.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:32.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:32.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:28:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:32.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:32.789 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:32.806 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:32.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-06-27T21:28:32.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:32.890 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:32.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:32.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:33.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:32 smithi192 ceph-mon[32956]: 5.f scrub starts 2024-06-27T21:28:33.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:32 smithi192 ceph-mon[32956]: 5.f scrub ok 2024-06-27T21:28:33.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:33.128 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:32 smithi190 ceph-mon[32247]: 5.f scrub starts 2024-06-27T21:28:33.128 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:32 smithi190 ceph-mon[32247]: 5.f scrub ok 2024-06-27T21:28:33.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:33.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:32 smithi151 ceph-mon[28965]: 5.f scrub starts 2024-06-27T21:28:33.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:32 smithi151 ceph-mon[28965]: 5.f scrub ok 2024-06-27T21:28:33.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:33.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:33.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:33.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:33.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:33.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:33.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:33.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:33.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:33.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:33.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:33.931 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:33.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:33.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:33 smithi151 ceph-mon[28965]: pgmap v1191: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 1256/56703 objects misplaced (2.215%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:28:34.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:34.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:33 smithi192 ceph-mon[32956]: pgmap v1191: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 1256/56703 objects misplaced (2.215%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:28:34.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:34.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:34.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:33 smithi190 ceph-mon[32247]: pgmap v1191: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 1256/56703 objects misplaced (2.215%); 2.0 MiB/s, 31 objects/s recovering 2024-06-27T21:28:34.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:34.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:34.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:34.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:34.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:34.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:34.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:34.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:34.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:34.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:34.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:34.886 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:34.943 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:34.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:34.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:34.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:35.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:35.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:35.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:35.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:35.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:35.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:35.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:35.512 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:28:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:28:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:28:35.512 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:28:35.512 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:28:35.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:35.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:35.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:35.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:35 smithi151 ceph-mon[28965]: pgmap v1192: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 983/52935 objects misplaced (1.857%); 4.0 MiB/s, 63 objects/s recovering 2024-06-27T21:28:35.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:35 smithi151 ceph-mon[28965]: 5.4 scrub starts 2024-06-27T21:28:35.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:35.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:35.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:35.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:36.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:35 smithi190 ceph-mon[32247]: pgmap v1192: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 983/52935 objects misplaced (1.857%); 4.0 MiB/s, 63 objects/s recovering 2024-06-27T21:28:36.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:35 smithi190 ceph-mon[32247]: 5.4 scrub starts 2024-06-27T21:28:36.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:36.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:35 smithi192 ceph-mon[32956]: pgmap v1192: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 983/52935 objects misplaced (1.857%); 4.0 MiB/s, 63 objects/s recovering 2024-06-27T21:28:36.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:35 smithi192 ceph-mon[32956]: 5.4 scrub starts 2024-06-27T21:28:36.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:36.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:36.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:36.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:36.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:36.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:36.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:36.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:36.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:36.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:36.688 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:36.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:36.823 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:36.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:36.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:36 smithi151 ceph-mon[28965]: 5.4 scrub ok 2024-06-27T21:28:36.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:36.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:37.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:36 smithi190 ceph-mon[32247]: 5.4 scrub ok 2024-06-27T21:28:37.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:36 smithi192 ceph-mon[32956]: 5.4 scrub ok 2024-06-27T21:28:37.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:37.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:37.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:37.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:37.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:37.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:37.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:37.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:37.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:37.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:37.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:37.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:37.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:37 smithi151 ceph-mon[28965]: pgmap v1193: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 25 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 983/47250 objects misplaced (2.080%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:28:37.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:37.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:37.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:37 smithi151 ceph-mon[28965]: osdmap e459: 12 total, 12 up, 12 in 2024-06-27T21:28:37.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:37.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:37.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:37.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:37.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:38.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:37 smithi190 ceph-mon[32247]: pgmap v1193: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 25 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 983/47250 objects misplaced (2.080%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:28:38.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:38.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:38.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:37 smithi190 ceph-mon[32247]: osdmap e459: 12 total, 12 up, 12 in 2024-06-27T21:28:38.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:37 smithi192 ceph-mon[32956]: pgmap v1193: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 25 active+clean; 985 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s; 983/47250 objects misplaced (2.080%); 2.1 MiB/s, 34 objects/s recovering 2024-06-27T21:28:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:37 smithi192 ceph-mon[32956]: osdmap e459: 12 total, 12 up, 12 in 2024-06-27T21:28:38.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:38.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:38.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:38.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:38.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:38.356 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:28:38.356 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:28:38.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:28:38.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:28:38.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:28:38.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:28:38.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:28:38.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:28:38.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:38.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:38.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:38.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:38.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:38.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:38.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:38.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:38.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:38.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:38.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1715524394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:39.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:39.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1715524394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:39.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:39.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1715524394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:39.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:39.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:39.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:39.275 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:39.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:39.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:39.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:39.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:39.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:39.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:39.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:39.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:39.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:39 smithi192 ceph-mon[32956]: pgmap v1195: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 25 active+clean; 909 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 983/43623 objects misplaced (2.253%); 2.1 MiB/s, 32 objects/s recovering 2024-06-27T21:28:39.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:39.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:39.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:39.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:39 smithi151 ceph-mon[28965]: pgmap v1195: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 25 active+clean; 909 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 983/43623 objects misplaced (2.253%); 2.1 MiB/s, 32 objects/s recovering 2024-06-27T21:28:40.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:40.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:39 smithi190 ceph-mon[32247]: pgmap v1195: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 25 active+clean; 909 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 983/43623 objects misplaced (2.253%); 2.1 MiB/s, 32 objects/s recovering 2024-06-27T21:28:40.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:40.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:40.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:40.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:40.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:40.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:40.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:40.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:40.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:40.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:40.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:40.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:40.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:40 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:28:40.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:40.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:40.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:40.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:41.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:40 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:28:41.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:41.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:40 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-06-27T21:28:41.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:41.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:41.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:41.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:41.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:41.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:41.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:41.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:41.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:41.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:41.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:41.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:41.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:41.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:41.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:41 smithi151 ceph-mon[28965]: pgmap v1196: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 780 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 683/37434 objects misplaced (1.825%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:28:41.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:41.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:42.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:41 smithi190 ceph-mon[32247]: pgmap v1196: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 780 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 683/37434 objects misplaced (1.825%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:28:42.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:41 smithi192 ceph-mon[32956]: pgmap v1196: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 780 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s; 683/37434 objects misplaced (1.825%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:28:42.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:42.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:42.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:42.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:42.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:42.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:42.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:42.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:42.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:42.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:42.687 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:28:42.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:42.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:42.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:42.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:42.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:42.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1321568295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:28:42.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:43.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:43.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1321568295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:28:43.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1321568295' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:28:43.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:28:43.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:43.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:43.179 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:43.180 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:43.180 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:43.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:43.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:43.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:43.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:43.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:43.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:43.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:43.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:43.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:43 smithi190 ceph-mon[32247]: pgmap v1197: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 684 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 683/32817 objects misplaced (2.081%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:28:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:43 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1321568295' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:28:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:43 smithi190 ceph-mon[32247]: osdmap e460: 12 total, 12 up, 12 in 2024-06-27T21:28:43.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:43.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:43.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:43.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:43 smithi151 ceph-mon[28965]: pgmap v1197: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 684 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 683/32817 objects misplaced (2.081%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:28:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:43 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1321568295' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:28:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:43 smithi151 ceph-mon[28965]: osdmap e460: 12 total, 12 up, 12 in 2024-06-27T21:28:44.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:44.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:43 smithi192 ceph-mon[32956]: pgmap v1197: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 684 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 683/32817 objects misplaced (2.081%); 3.6 MiB/s, 57 objects/s recovering 2024-06-27T21:28:44.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:43 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1321568295' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:28:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:43 smithi192 ceph-mon[32956]: osdmap e460: 12 total, 12 up, 12 in 2024-06-27T21:28:44.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:44.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:44.160 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:44.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-06-27T21:28:44.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:44.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:44.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:44.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:28:44.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:44.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:44.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:44.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:44.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:44.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:44.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:44.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:44.904 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:44.918 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:44.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:44.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:44.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:45.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:45.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:45.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:45.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:45.322 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:45.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:45.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:45.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:45.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:45.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:45.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:45.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:45.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:45.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:45.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:45.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:45 smithi151 ceph-mon[28965]: pgmap v1199: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 601 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 457/28830 objects misplaced (1.585%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:28:45.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:46.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:46.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:46.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:45 smithi190 ceph-mon[32247]: pgmap v1199: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 601 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 457/28830 objects misplaced (1.585%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:28:46.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:45 smithi192 ceph-mon[32956]: pgmap v1199: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 601 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 457/28830 objects misplaced (1.585%); 4.1 MiB/s, 65 objects/s recovering 2024-06-27T21:28:46.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:46.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:46.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:46.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:46.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:46.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:46.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:46.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:46.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:46.657 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:46.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:46.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:46.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:28:46.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:46.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:46.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:47.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:47.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:47.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:47.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:47.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:47.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:47.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:47.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:47.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:47.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:47.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:47.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:47.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:47.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:47.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:47.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:47 smithi151 ceph-mon[28965]: pgmap v1200: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 470 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 457/22539 objects misplaced (2.028%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:28:47.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:47.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:47 smithi190 ceph-mon[32247]: pgmap v1200: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 470 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 457/22539 objects misplaced (2.028%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:28:48.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:47 smithi192 ceph-mon[32956]: pgmap v1200: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 470 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 796 op/s; 457/22539 objects misplaced (2.028%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:28:48.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:48.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:48.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:48.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:48.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:48.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:48.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:48.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:48.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:48.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:48.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:48.754 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:48.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:48.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:48.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:48.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:49.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:49.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:49.153 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:49.154 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:49.154 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:49.154 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:49.154 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:49.154 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:49.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:49.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:49.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:49.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:49.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:49.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:49.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:49.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:49.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:49.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:49.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:49.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:49 smithi151 ceph-mon[28965]: pgmap v1201: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 391 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 457/18735 objects misplaced (2.439%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:28:49.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:49 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:28:49.894 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:28:49.894 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:28:49.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:49.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:49.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:49 smithi190 ceph-mon[32247]: pgmap v1201: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 391 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 457/18735 objects misplaced (2.439%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:28:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:49 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:28:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:49 smithi192 ceph-mon[32956]: pgmap v1201: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 391 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 457/18735 objects misplaced (2.439%); 3.3 MiB/s, 52 objects/s recovering 2024-06-27T21:28:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:49 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:28:50.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:50.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:50.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:50.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:50.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:50.339 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:28:50.339 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:28:50.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:50.373 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:28:50.373 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:28:50.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:28:50.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:28:50.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:28:50.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:28:50.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:50.465 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:28:50.465 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:28:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.507 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:50.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:50.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:28:50.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:28:50.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:28:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:28:50.569 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:28:50.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:28:50.598 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:28:50.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:28:50.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:50.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:28:50.632 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:28:50.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:50.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:28:50.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:28:50.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:28:50.692 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:28:50.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:50.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:50.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:50.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:50.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:50.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1821665222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:50.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:50 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:50.769 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:50 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:50.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:28:50.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:50.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:50.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1821665222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:50 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:50 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:51.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1821665222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:28:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:50 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:50 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:51.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:51.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:51.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:51.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:51.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:51.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:51.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:28:51.527 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:28:51.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-06-27T21:28:51.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:51.572 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:51.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:51.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:51.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:51.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:51.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:51.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: pgmap v1202: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 263 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 129/12597 objects misplaced (1.024%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: osdmap e461: 12 total, 12 up, 12 in 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:28:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:51 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:52.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:52.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: pgmap v1202: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 263 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 129/12597 objects misplaced (1.024%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:28:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: osdmap e461: 12 total, 12 up, 12 in 2024-06-27T21:28:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:28:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:28:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:28:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:28:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:51 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:52.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: pgmap v1202: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 263 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 129/12597 objects misplaced (1.024%); 3.5 MiB/s, 55 objects/s recovering 2024-06-27T21:28:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: osdmap e461: 12 total, 12 up, 12 in 2024-06-27T21:28:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:28:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:28:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:28:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:28:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:51 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:28:52.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:52.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:52.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:52.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:52.322 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:52.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:52.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:52.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:52.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:52.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:52.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:52.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:52.778 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:52.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:52.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:52.923 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:52.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:53.009 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:52 smithi151 ceph-mon[28965]: osdmap e462: 12 total, 12 up, 12 in 2024-06-27T21:28:53.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:52 smithi190 ceph-mon[32247]: osdmap e462: 12 total, 12 up, 12 in 2024-06-27T21:28:53.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:52 smithi192 ceph-mon[32956]: osdmap e462: 12 total, 12 up, 12 in 2024-06-27T21:28:53.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:53.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:53.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:53.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:53.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:53.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:53.301 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 106038 objects 2024-06-27T21:28:53.301 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :140.833 2024-06-27T21:28:53.311 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-06-27T21:28:53.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-06-27T21:28:53.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:53.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:53.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:53.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:53.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:53.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:53.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:53.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:53.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:53.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:53.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:53.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:53 smithi151 ceph-mon[28965]: pgmap v1205: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 165 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 129/7890 objects misplaced (1.635%); 2.6 MiB/s, 40 objects/s recovering 2024-06-27T21:28:54.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:53 smithi190 ceph-mon[32247]: pgmap v1205: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 165 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 129/7890 objects misplaced (1.635%); 2.6 MiB/s, 40 objects/s recovering 2024-06-27T21:28:54.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:54.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:54.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:53 smithi192 ceph-mon[32956]: pgmap v1205: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 165 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s; 129/7890 objects misplaced (1.635%); 2.6 MiB/s, 40 objects/s recovering 2024-06-27T21:28:54.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:28:54.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:54.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:54.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:54.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:54.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:54.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:54.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:54.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:54.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:54.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:54.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:54.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:54.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:54.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:54.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:55.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:55.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:55.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:55.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:55.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:55.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:55.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:55.408 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:28:55.408 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:28:55.408 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:28:55.408 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:28:55.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:55.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:55.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:55.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:55.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:55.729 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:28:55.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:55.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:55.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:55.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:55.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:55 smithi190 ceph-mon[32247]: pgmap v1206: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 87 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 3.6 MiB/s, 77 objects/s recovering 2024-06-27T21:28:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/186131025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:56.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:56.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:56.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:55 smithi192 ceph-mon[32956]: pgmap v1206: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 87 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 3.6 MiB/s, 77 objects/s recovering 2024-06-27T21:28:56.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/186131025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:56.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:56.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:56.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:55 smithi151 ceph-mon[28965]: pgmap v1206: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 87 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 766 op/s; 3.6 MiB/s, 77 objects/s recovering 2024-06-27T21:28:56.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/186131025' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:28:56.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:56.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:56.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:28:56.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:56.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:56.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:56.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:56.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:56.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:56.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:56.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:56.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:56.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:28:56.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:56.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:28:57.007 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:57.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:56 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/186131025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:56 smithi190 ceph-mon[32247]: osdmap e463: 12 total, 12 up, 12 in 2024-06-27T21:28:57.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:57.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:56 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:57.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/186131025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:57.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:56 smithi192 ceph-mon[32956]: osdmap e463: 12 total, 12 up, 12 in 2024-06-27T21:28:57.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:56 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:28:57.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/186131025' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:28:57.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:56 smithi151 ceph-mon[28965]: osdmap e463: 12 total, 12 up, 12 in 2024-06-27T21:28:57.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:57.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:57.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:57.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:57.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:57.400 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:57.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:28:57.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:28:57.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:57.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.647 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:57.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:28:57.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:28:57.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:28:57.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:57.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:28:57.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:58.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:57 smithi190 ceph-mon[32247]: pgmap v1208: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 14 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 1.3 MiB/s, 48 objects/s recovering 2024-06-27T21:28:58.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:58.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:57 smithi151 ceph-mon[28965]: pgmap v1208: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 14 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 1.3 MiB/s, 48 objects/s recovering 2024-06-27T21:28:58.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:57 smithi192 ceph-mon[32956]: pgmap v1208: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 14 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 1.3 MiB/s, 48 objects/s recovering 2024-06-27T21:28:58.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:58.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:58.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:58.173 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:28:58.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:58.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:58.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:58.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:58.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:28:58.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:28:58.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:58.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:58.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:58.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:58.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:28:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:58.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:28:58.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:28:58.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:59.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:59.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:59.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:59.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:28:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:59.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:28:59.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:28:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:59.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:28:59.400 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:28:59.401 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:28:59.401 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:28:59.401 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:28:59.401 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:28:59.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:28:59.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:28:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:59.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:28:59.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:28:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:28:59.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:28:59.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:28:59.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:28:59.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:28:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:28:59.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:28:59.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:28:59.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:00.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:28:59 smithi190 ceph-mon[32247]: pgmap v1209: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 1.1 MiB/s, 39 objects/s recovering 2024-06-27T21:29:00.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:28:59 smithi151 ceph-mon[28965]: pgmap v1209: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 1.1 MiB/s, 39 objects/s recovering 2024-06-27T21:29:00.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:28:59 smithi192 ceph-mon[32956]: pgmap v1209: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 1.1 MiB/s, 39 objects/s recovering 2024-06-27T21:29:00.066 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:00.067 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:29:00.077 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:29:00.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-06-27T21:29:00.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:00.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:00.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:00.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:00.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:00.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:00.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:00.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:00.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:00.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:00.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:00.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:00.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:00.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:00.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:00.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:00.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:00.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:01.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:01.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:01.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:01.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:01.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:01.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:01.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:01.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:01.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:01.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:01.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:01.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:01.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:01.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:01.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:01 smithi151 ceph-mon[28965]: pgmap v1210: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 315 op/s; 991 KiB/s, 117 objects/s recovering 2024-06-27T21:29:01.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2830428301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:01.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:01.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:01.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:01 smithi190 ceph-mon[32247]: pgmap v1210: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 315 op/s; 991 KiB/s, 117 objects/s recovering 2024-06-27T21:29:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2830428301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:01 smithi192 ceph-mon[32956]: pgmap v1210: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 315 op/s; 991 KiB/s, 117 objects/s recovering 2024-06-27T21:29:02.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2830428301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:02.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:02.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:02.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:02.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:02.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:02.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:02.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:02.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:02.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:02.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:02.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:02.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:02.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:02.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:02.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:29:02.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:29:02.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:29:02.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:29:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:29:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:29:02.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:02.980 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:02.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:03.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:03.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:03.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:03.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:03.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:03.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:03.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:03.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:03.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:03.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:03.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:03.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:03.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:03.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:03.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:03.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:03.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:03.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:03.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:03 smithi190 ceph-mon[32247]: pgmap v1211: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 262 op/s; 825 KiB/s, 97 objects/s recovering 2024-06-27T21:29:04.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:03 smithi151 ceph-mon[28965]: pgmap v1211: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 262 op/s; 825 KiB/s, 97 objects/s recovering 2024-06-27T21:29:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:03 smithi192 ceph-mon[32956]: pgmap v1211: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 262 op/s; 825 KiB/s, 97 objects/s recovering 2024-06-27T21:29:04.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:04.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:04.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.228 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:04.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:04.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:04.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:04.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:04.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:04.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:04.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:04.618 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_3' removed 2024-06-27T21:29:04.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:04.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:04.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:04.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:04.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:04.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/814770073' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:29:04.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/814770073' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:29:04.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:04 smithi151 ceph-mon[28965]: osdmap e464: 12 total, 12 up, 12 in 2024-06-27T21:29:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:04.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:04.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/814770073' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:29:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/814770073' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:29:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:04 smithi190 ceph-mon[32247]: osdmap e464: 12 total, 12 up, 12 in 2024-06-27T21:29:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/814770073' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:29:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/814770073' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:29:05.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:04 smithi192 ceph-mon[32956]: osdmap e464: 12 total, 12 up, 12 in 2024-06-27T21:29:05.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:05.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:05.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:05.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:05.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:05.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:05.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:05.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:05.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:05.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:29:05.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:05.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:05.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:05.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:05.773 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:05 smithi151 ceph-mon[28965]: pgmap v1212: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 0 B/s, 109 objects/s recovering 2024-06-27T21:29:05.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:05.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:05.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:05.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:05.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:05 smithi190 ceph-mon[32247]: pgmap v1212: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 0 B/s, 109 objects/s recovering 2024-06-27T21:29:06.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:05 smithi192 ceph-mon[32956]: pgmap v1212: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 137 op/s; 0 B/s, 109 objects/s recovering 2024-06-27T21:29:06.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:06.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:06.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:06.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:06.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:06.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:06.425 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:29:06.425 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:29:06.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:06.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:06.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:06.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:06.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:06.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:06.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:06.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:06.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:06.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:06.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/314524712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:29:07.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:07.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:07.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:07.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/314524712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:29:07.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:07.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/314524712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:29:07.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:07.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:07.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:07.341 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:29:07.342 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:29:07.343 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-06-27T21:29:07.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-06-27T21:29:07.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:07.445 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:29:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:07.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:07.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:07.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:07.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:07.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:07.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:07.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:07.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:07 smithi151 ceph-mon[28965]: pgmap v1214: 11 pgs: 11 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:07 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-27T21:29:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:07 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/314524712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:29:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:07 smithi151 ceph-mon[28965]: osdmap e465: 12 total, 12 up, 12 in 2024-06-27T21:29:07.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:07.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:07 smithi192 ceph-mon[32956]: pgmap v1214: 11 pgs: 11 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:07 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-27T21:29:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:07 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/314524712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:29:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:07 smithi192 ceph-mon[32956]: osdmap e465: 12 total, 12 up, 12 in 2024-06-27T21:29:08.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:08.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:08.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:08.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:07 smithi190 ceph-mon[32247]: pgmap v1214: 11 pgs: 11 active+clean; 577 KiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:07 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-06-27T21:29:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:07 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/314524712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:29:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:07 smithi190 ceph-mon[32247]: osdmap e465: 12 total, 12 up, 12 in 2024-06-27T21:29:08.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:08.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:08.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:08.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:08.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:08.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:08.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:08.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:08.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:08.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:08.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:08.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:08.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:09.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:09.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:09.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:09.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:09.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:09.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:09.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:09.444 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:09.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:29:09.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:09.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:09.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:09.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:09.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:09.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:09.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:09.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:09.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:09.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:09 smithi151 ceph-mon[28965]: pgmap v1216: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:09.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:10.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:10.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:29:10.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:09 smithi192 ceph-mon[32956]: pgmap v1216: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:10.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:10.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:10.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:10.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:10.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:09 smithi190 ceph-mon[32247]: pgmap v1216: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:10.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:10.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:10.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:10.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:10.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:10.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:29:10.680 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:29:10.681 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:29:10.681 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:29:10.681 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:29:10.681 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:29:10.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:10.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:10.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:10.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:10.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:10.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:11.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:10 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:29:11.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:10 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:29:11.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:11.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:11.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:10 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:29:11.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:10 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:29:11.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:11.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:10 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:29:11.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:10 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:29:11.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:11.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:11.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:11.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:11.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:11.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:11.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:11.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:11.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:11.816 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:11.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:11.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:11.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:11.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:11 smithi151 ceph-mon[28965]: pgmap v1217: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:11.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:11 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:29:11.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:11 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:29:12.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:12.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:12.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:11 smithi192 ceph-mon[32956]: pgmap v1217: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:11 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:29:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:11 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:29:12.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:12.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:11 smithi190 ceph-mon[32247]: pgmap v1217: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:11 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:29:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:11 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:29:12.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:29:12.280 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:29:12.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-06-27T21:29:12.283 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:12.283 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:12.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:12.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:12.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:12.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:12.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:12.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:12.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:12.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:12.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:12.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:12.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:12.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:13.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:13.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:12 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:29:13.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:12 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:29:13.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:12 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:29:13.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:12 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:29:13.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:13.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:13.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:13.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:12 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:29:13.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:12 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:29:13.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:13.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:13.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:13.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:13.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:13.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:13.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:13.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:13.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:13.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:13.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:13.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:13.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:13 smithi151 ceph-mon[28965]: pgmap v1218: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:13 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:29:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:13 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:29:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:13 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:29:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:13 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:29:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2513286126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:13.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:29:13.967 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:29:13.990 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:29:13.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:29:14.012 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:29:14.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:29:14.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:29:14.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:29:14.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:13 smithi192 ceph-mon[32956]: pgmap v1218: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:13 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:29:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:13 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:29:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:13 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:29:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:13 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:29:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2513286126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:14.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:29:14.056 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:29:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:14.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:14.078 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:29:14.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:29:14.100 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:29:14.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:29:14.122 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:29:14.122 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:29:14.144 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:29:14.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:29:14.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:14.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:29:14.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:29:14.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:14.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:29:14.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:29:14.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:29:14.212 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:29:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:13 smithi190 ceph-mon[32247]: pgmap v1218: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:13 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:29:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:13 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:29:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:13 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:29:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:13 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:29:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2513286126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:14.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:14.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:14.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:14.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:14.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:14.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:14.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:14.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:14.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:14.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:14.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:14.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:14.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:15.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:15.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:15.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:15.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:15.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:15.321 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:15.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:15.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:15.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:15.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:15.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:15.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:15.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:15.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:15.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:15.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:15.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:15 smithi151 ceph-mon[28965]: pgmap v1219: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:15.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:15.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:16.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:15 smithi192 ceph-mon[32956]: pgmap v1219: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:16.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:16.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:16.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:16.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:16.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:15 smithi190 ceph-mon[32247]: pgmap v1219: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:16.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:16.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:16.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:16.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:16.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:16.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:16.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:16.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:16.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:16.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:16.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:29:16.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:16.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:16.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:16 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:29:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:16 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:29:17.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:16 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:29:17.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:16 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:29:17.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:17.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:17.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:17.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:17.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:16 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:29:17.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:16 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:29:17.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:17.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:17.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:17.487 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:17.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:17.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:17.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:17.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:17.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:17.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:17 smithi151 ceph-mon[28965]: pgmap v1220: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:17.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:17.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:17.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:18.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:18.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:18.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:18.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:17 smithi190 ceph-mon[32247]: pgmap v1220: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:18.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:17 smithi192 ceph-mon[32956]: pgmap v1220: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:18.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:18.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:18.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:18.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:18.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:18.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:18.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:18.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:18.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:18.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:18.805 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_4' created 2024-06-27T21:29:18.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:18.869 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:9505 4.7 4.8b986b77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e292)", 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:50.518712+0000", 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 508.348308487, 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97395969699999996, 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.870 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9505 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518712+0000", 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518712+0000", 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518713+0000", 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4499999999999997e-07 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518775+0000", 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2514000000000005e-05 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518784+0000", 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7790000000000001e-06 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518795+0000", 2024-06-27T21:29:18.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0967e-05 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518832+0000", 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7075e-05 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.518837+0000", 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0069999999999997e-06 2024-06-27T21:29:18.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:51.492647+0000", 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97380978900000004 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:51.492672+0000", 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4921000000000001e-05 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:9513 4.7 4.c80ceb47 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:50.559514+0000", 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 508.30750589899998, 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93319091799999998, 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.876 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9513 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559514+0000", 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559514+0000", 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559517+0000", 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6079999999999998e-06 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559552+0000", 2024-06-27T21:29:18.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4841000000000001e-05 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559576+0000", 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4510000000000001e-05 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559588+0000", 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1973999999999999e-05 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559681+0000", 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2795999999999997e-05 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.559696+0000", 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4866e-05 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:51.492698+0000", 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93300242099999997 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:51.492705+0000", 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9020000000000002e-06 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:9533 4.7 4.4faae797 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:20:50.734741+0000", 2024-06-27T21:29:18.881 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 508.13227955000002, 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75799372499999995, 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9533 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734741+0000", 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734741+0000", 2024-06-27T21:29:18.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734742+0000", 2024-06-27T21:29:18.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6619999999999999e-06 2024-06-27T21:29:18.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734768+0000", 2024-06-27T21:29:18.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5936e-05 2024-06-27T21:29:18.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734781+0000", 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2520999999999999e-05 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734789+0000", 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4379999999999995e-06 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734873+0000", 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3436000000000005e-05 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:50.734882+0000", 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0520000000000007e-06 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:51.492728+0000", 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75784658999999999 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:20:51.492735+0000", 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0900000000000001e-06 2024-06-27T21:29:18.888 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:20970 4.3 e301/282 4:c5562fc5:::benchmark_data_smithi192_88961_object20969:head v 301'1322, mlcod=300'1318)", 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:01.542069+0000", 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 497.32495156499999, 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30816456199999998, 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.542069+0000", 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.542069+0000", 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.542069+0000", 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8999999999999997e-07 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.542098+0000", 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9094999999999998e-05 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.849211+0000", 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30711249000000002 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.849222+0000", 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1336e-05 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.849321+0000", 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9192999999999997e-05 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.849341+0000", 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9228999999999998e-05 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.850194+0000", 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00085359100000000001 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.850233+0000", 2024-06-27T21:29:18.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9137999999999997e-05 2024-06-27T21:29:18.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.895 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:20971 4.7 e301/299 ondisk, result = 0)", 2024-06-27T21:29:18.895 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:01.544998+0000", 2024-06-27T21:29:18.895 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 497.32202248499999, 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.74332804900000005, 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.544998+0000", 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.544998+0000", 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.544999+0000", 2024-06-27T21:29:18.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4900000000000005e-07 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.545002+0000", 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0089999999999999e-06 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.288055+0000", 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.743053458 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.288072+0000", 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6416e-05 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.288186+0000", 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011466199999999999 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.899 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.288204+0000", 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8216999999999999e-05 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.288326+0000", 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121438 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.900 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:20983 4.3 e301/282 4:c0ede1b3:::benchmark_data_smithi192_88961_object20982:head v 301'1323, mlcod=300'1318)", 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:01.618878+0000", 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 497.24814280999999, 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57424392300000004, 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.618878+0000", 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.618878+0000", 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.618883+0000", 2024-06-27T21:29:18.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9740000000000001e-06 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:01.618929+0000", 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6195000000000001e-05 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.191945+0000", 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57301669 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.191958+0000", 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2262000000000001e-05 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.192034+0000", 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6332e-05 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.192061+0000", 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7107999999999999e-05 2024-06-27T21:29:18.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.193086+0000", 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010253599999999999 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:02.193121+0000", 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5002000000000001e-05 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.906 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:58365 4.f 4:fc83d7dd:::benchmark_data_smithi192_88961_object58364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:29:18.906 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:21:33.818900+0000", 2024-06-27T21:29:18.906 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 465.04812048000002, 2024-06-27T21:29:18.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29665349800000002, 2024-06-27T21:29:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 58365 2024-06-27T21:29:18.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.818900+0000", 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.818900+0000", 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.818902+0000", 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.761e-06 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.818944+0000", 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2191000000000003e-05 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.818958+0000", 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4504999999999999e-05 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.818968+0000", 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9629999999999993e-06 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.819022+0000", 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3229000000000001e-05 2024-06-27T21:29:18.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.819169+0000", 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000147935 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.819328+0000", 2024-06-27T21:29:18.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015842499999999999 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.821273+0000", 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001945497 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:33.821887+0000", 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061374400000000005 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:34.115460+0000", 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29357297900000001 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:34.115516+0000", 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5699999999999999e-05 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:21:34.115553+0000", 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7568999999999999e-05 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:130471 4.7 4:ebbd1fb5:::benchmark_data_smithi192_88961_object39732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:56.011374+0000", 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 382.85564612899998, 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.284091382, 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.915 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 130471 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011374+0000", 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011374+0000", 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011376+0000", 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2950000000000001e-06 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011381+0000", 2024-06-27T21:29:18.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1370000000000001e-06 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011393+0000", 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2201999999999999e-05 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011399+0000", 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.976e-06 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011452+0000", 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2750000000000001e-05 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011588+0000", 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013668 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.011724+0000", 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013573599999999999 2024-06-27T21:29:18.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.014822+0000", 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030982599999999998 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.014932+0000", 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010938300000000001 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.295369+0000", 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.280437569 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.295411+0000", 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2017000000000001e-05 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:56.295466+0000", 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4376999999999998e-05 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.921 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:130535 4.f 4:fe59a0bc:::benchmark_data_smithi192_88961_object39796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:57.044315+0000", 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 381.82270555100001, 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52309249000000002, 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.922 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 130535 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044315+0000", 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044315+0000", 2024-06-27T21:29:18.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044316+0000", 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6700000000000002e-07 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044320+0000", 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1420000000000003e-06 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044330+0000", 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0395e-05 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044335+0000", 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1769999999999997e-06 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044408+0000", 2024-06-27T21:29:18.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2187999999999994e-05 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044521+0000", 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011366600000000001 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.044633+0000", 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011205399999999999 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.047333+0000", 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026997810000000001 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.047828+0000", 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049542 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567300+0000", 2024-06-27T21:29:18.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51947127999999998 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567362+0000", 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1946999999999995e-05 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567407+0000", 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5572999999999999e-05 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:130535 4.f e346/282 ondisk, result = 0)", 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:22:57.047295+0000", 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 381.81972544500002, 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52011967100000001, 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.047295+0000", 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.047295+0000", 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.047296+0000", 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8299999999999995e-07 2024-06-27T21:29:18.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.047300+0000", 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0090000000000001e-06 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567214+0000", 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51991385999999995 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567224+0000", 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0801999999999999e-05 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567281+0000", 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6861999999999999e-05 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567292+0000", 2024-06-27T21:29:18.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0892e-05 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:22:57.567415+0000", 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012236300000000001 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163707 4.f 4:fa8ede5c:::benchmark_data_smithi192_88961_object72968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.945784+0000", 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 334.92123638100003, 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72990378199999995, 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163707 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945784+0000", 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945784+0000", 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945785+0000", 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7699999999999992e-07 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945788+0000", 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4809999999999998e-06 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945799+0000", 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0295000000000001e-05 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945804+0000", 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945854+0000", 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0015999999999997e-05 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945956+0000", 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101816 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.946034+0000", 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8271999999999999e-05 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950082+0000", 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040472379999999999 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950119+0000", 2024-06-27T21:29:18.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7231999999999999e-05 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675577+0000", 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72545799 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675639+0000", 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2314e-05 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675688+0000", 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8671000000000002e-05 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:163707 4.f e362/282 ondisk, result = 0)", 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.949869+0000", 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 334.91715092499999, 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72582532300000002, 2024-06-27T21:29:18.941 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949869+0000", 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949869+0000", 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949870+0000", 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8100000000000002e-07 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949874+0000", 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.551e-06 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675482+0000", 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72560794399999995 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675505+0000", 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3059000000000001e-05 2024-06-27T21:29:18.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675558+0000", 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3541000000000001e-05 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675569+0000", 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1051e-05 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675695+0000", 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012549600000000001 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:163716 4.3 e362/282 4:ce21cfad:::benchmark_data_smithi192_88961_object72977:head v 362'10240, mlcod=362'10238)", 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.970556+0000", 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 334.89646448299999, 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70433514200000003, 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.946 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970556+0000", 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970556+0000", 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970557+0000", 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4169999999999999e-06 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970568+0000", 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0637e-05 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669850+0000", 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.699282395 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669855+0000", 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1599999999999997e-06 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669941+0000", 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5082999999999998e-05 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669964+0000", 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3575999999999999e-05 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:29:18.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.674844+0000", 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048802710000000003 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.674891+0000", 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6603000000000001e-05 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e376,346 4:29d172ee:::benchmark_data_smithi192_88961_object19814:head,337'6947)", 2024-06-27T21:29:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:26.382105+0000", 2024-06-27T21:29:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.484914936, 2024-06-27T21:29:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51754427000000003, 2024-06-27T21:29:18.952 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382105+0000", 2024-06-27T21:29:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382105+0000", 2024-06-27T21:29:18.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382106+0000", 2024-06-27T21:29:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3300000000000002e-07 2024-06-27T21:29:18.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382109+0000", 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0089999999999999e-06 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899025+0000", 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51691604099999999 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899044+0000", 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9266000000000002e-05 2024-06-27T21:29:18.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899208+0000", 2024-06-27T21:29:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016374799999999999 2024-06-27T21:29:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899650+0000", 2024-06-27T21:29:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044167300000000001 2024-06-27T21:29:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.959 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2c9cc615:::benchmark_data_smithi192_88961_object31150:head,343'7618)", 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:33.676427+0000", 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 285.19059322599998, 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36238720299999999, 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.960 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676427+0000", 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676427+0000", 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676428+0000", 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5700000000000002e-07 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676439+0000", 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-06-27T21:29:18.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038278+0000", 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36183875399999998 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038295+0000", 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6702999999999999e-05 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038413+0000", 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118225 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038814+0000", 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040116400000000002 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:24b2f406:::benchmark_data_smithi192_88961_object31387:head,343'7636)", 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:34.126182+0000", 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 284.740838005, 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85182796800000005, 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126182+0000", 2024-06-27T21:29:18.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126182+0000", 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126183+0000", 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3700000000000001e-07 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126186+0000", 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.977601+0000", 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85141412299999997 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.977612+0000", 2024-06-27T21:29:18.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1447000000000001e-05 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.977730+0000", 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011793 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.978010+0000", 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028035499999999998 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2bef463e:::benchmark_data_smithi192_88961_object31885:head,343'7665)", 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:35.105338+0000", 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 283.761682608, 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48611961300000001, 2024-06-27T21:29:18.969 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105338+0000", 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105338+0000", 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105345+0000", 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1339999999999998e-06 2024-06-27T21:29:18.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105355+0000", 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0266e-05 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.590991+0000", 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48563563900000001 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.590999+0000", 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0450000000000001e-06 2024-06-27T21:29:18.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.591087+0000", 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7956999999999998e-05 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.591457+0000", 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000370572 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2c917cec:::benchmark_data_smithi192_88961_object33714:head,343'7766)", 2024-06-27T21:29:18.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:37.211636+0000", 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 281.65538443499997, 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.76569309100000005, 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211636+0000", 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211636+0000", 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211637+0000", 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.457e-06 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211648+0000", 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0370000000000001e-05 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977014+0000", 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.76536581000000004 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977027+0000", 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2965e-05 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977093+0000", 2024-06-27T21:29:18.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6544000000000005e-05 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977329+0000", 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023594499999999999 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:24984d7a:::benchmark_data_smithi192_88961_object34428:head,343'7813)", 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:38.197021+0000", 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 280.66999893600001, 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41096396699999999, 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.982 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197021+0000", 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197021+0000", 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197023+0000", 2024-06-27T21:29:18.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.46e-06 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197032+0000", 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5000000000000005e-06 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607487+0000", 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41045458499999998 2024-06-27T21:29:18.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607505+0000", 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7867e-05 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607701+0000", 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000195818 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607985+0000", 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028473700000000002 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2e7e20f1:::benchmark_data_smithi192_88961_object34450:head,343'7814)", 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:38.611847+0000", 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 280.25517375999999, 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42814653200000002, 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:18.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611847+0000", 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611847+0000", 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611855+0000", 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8689999999999993e-06 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611867+0000", 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.112e-05 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039635+0000", 2024-06-27T21:29:18.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42776847200000001 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039665+0000", 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0187000000000001e-05 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039768+0000", 2024-06-27T21:29:18.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102796 2024-06-27T21:29:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:29:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039993+0000", 2024-06-27T21:29:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000225088 2024-06-27T21:29:18.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:18.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:18.992 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:29:18.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:18.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:19.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:18 smithi151 ceph-mon[28965]: osdmap e466: 12 total, 12 up, 12 in 2024-06-27T21:29:19.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:18 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:29:19.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:18 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:29:19.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/944707819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-27T21:29:19.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:19.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:19.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:19.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:18 smithi190 ceph-mon[32247]: osdmap e466: 12 total, 12 up, 12 in 2024-06-27T21:29:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:18 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:29:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:18 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:29:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/944707819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-27T21:29:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:18 smithi192 ceph-mon[32956]: osdmap e466: 12 total, 12 up, 12 in 2024-06-27T21:29:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:18 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:29:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:18 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:29:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/944707819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-06-27T21:29:19.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:19.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:19.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:19.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:19.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:19.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:19.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:19.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:19.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:19.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:19.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:19.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:19.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-06-27T21:29:20.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:20.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:20.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:20.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:19 smithi151 ceph-mon[28965]: pgmap v1222: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:20.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/944707819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-27T21:29:20.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:19 smithi151 ceph-mon[28965]: osdmap e467: 12 total, 12 up, 12 in 2024-06-27T21:29:20.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:19 smithi190 ceph-mon[32247]: pgmap v1222: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/944707819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-27T21:29:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:19 smithi190 ceph-mon[32247]: osdmap e467: 12 total, 12 up, 12 in 2024-06-27T21:29:20.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:19 smithi192 ceph-mon[32956]: pgmap v1222: 11 pgs: 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/944707819' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-06-27T21:29:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:19 smithi192 ceph-mon[32956]: osdmap e467: 12 total, 12 up, 12 in 2024-06-27T21:29:20.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:20.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:20.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:20.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:20.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:20.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:20.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:20.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:20.800 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-06-27T21:29:20.800 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-06-27T21:29:20.801 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-06-27T21:29:20.819 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:29:20.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:20.910 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:20.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:21.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:21.012 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:20 smithi151 ceph-mon[28965]: osdmap e468: 12 total, 12 up, 12 in 2024-06-27T21:29:21.013 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2190216899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:21.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:21.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:21.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:20 smithi190 ceph-mon[32247]: osdmap e468: 12 total, 12 up, 12 in 2024-06-27T21:29:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2190216899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:21.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:21.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:21.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:20 smithi192 ceph-mon[32956]: osdmap e468: 12 total, 12 up, 12 in 2024-06-27T21:29:21.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2190216899' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:21.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:21.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:21.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:21.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:21.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:21.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:21.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:29:21.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:21.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:21.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:21.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:21.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:22.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:22.053 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:22.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:21 smithi151 ceph-mon[28965]: pgmap v1225: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:22.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:21 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:29:22.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:21 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:29:22.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:21 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:29:22.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2190216899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:29:22.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:21 smithi151 ceph-mon[28965]: osdmap e469: 12 total, 12 up, 12 in 2024-06-27T21:29:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-06-27T21:29:22.147 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-06-27T21:29:22.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:22.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:22.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:21 smithi190 ceph-mon[32247]: pgmap v1225: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:21 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:21 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:21 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2190216899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:29:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:21 smithi190 ceph-mon[32247]: osdmap e469: 12 total, 12 up, 12 in 2024-06-27T21:29:22.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:22.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:21 smithi192 ceph-mon[32956]: pgmap v1225: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:21 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:29:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:21 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:29:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:21 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:29:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2190216899' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:29:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:21 smithi192 ceph-mon[32956]: osdmap e469: 12 total, 12 up, 12 in 2024-06-27T21:29:22.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:22.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:22.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:22.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:22.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:22.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:22.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:22.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:22.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:22.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:22.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:22.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:23.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:23.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:23.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:23.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:23.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:23.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:23.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:23.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:23.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:23.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:23.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:23.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:23.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:23.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:23.857 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:23.896 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:23.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:23.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:23.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:24.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:24.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:24.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:23 smithi151 ceph-mon[28965]: pgmap v1227: 27 pgs: 4 creating+peering, 9 unknown, 14 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:24.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:24.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:23 smithi190 ceph-mon[32247]: pgmap v1227: 27 pgs: 4 creating+peering, 9 unknown, 14 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:23 smithi192 ceph-mon[32956]: pgmap v1227: 27 pgs: 4 creating+peering, 9 unknown, 14 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:24.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:24.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:24.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:24.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:24.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:24.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:24.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:24.736 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:24.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:24.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:24.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:24.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:24.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:25.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:25.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:25.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:25.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:25.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:25.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:25.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:29:25.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:25.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:25.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:25.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:25.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:25.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:25.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:25.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:25.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:25.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:25 smithi151 ceph-mon[28965]: pgmap v1228: 27 pgs: 4 creating+peering, 23 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:25.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:26.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:26.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:26.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:26.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:26.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:25 smithi190 ceph-mon[32247]: pgmap v1228: 27 pgs: 4 creating+peering, 23 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:25 smithi192 ceph-mon[32956]: pgmap v1228: 27 pgs: 4 creating+peering, 23 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:26.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:26.375 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:26.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:26.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:26.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:26.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:26.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:26.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:26.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:26.832 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-06-27T21:29:26.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:26.894 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:26.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:26.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:27.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:27.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:27.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:27.148 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:29:27.148 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:29:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/229093137' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:29:27.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/229093137' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:29:27.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/229093137' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:29:27.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:27.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:27.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:27.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:29:27.439 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:29:27.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:29:27.461 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:29:27.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:29:27.482 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:29:27.502 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:29:27.503 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:29:27.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:27.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:29:27.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:29:27.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:27.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:27.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:29:27.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:29:27.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:29:27.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:29:27.593 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:29:27.593 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:29:27.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:29:27.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:29:27.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:27.639 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:29:27.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:29:27.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:27.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:29:27.662 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:29:27.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:29:27.684 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:29:27.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:27.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:27.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:27.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:27.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:28.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:28.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:28.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:28.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:27 smithi151 ceph-mon[28965]: pgmap v1229: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/229093137' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:29:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:27 smithi151 ceph-mon[28965]: osdmap e470: 12 total, 12 up, 12 in 2024-06-27T21:29:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4278916955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:27 smithi190 ceph-mon[32247]: pgmap v1229: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/229093137' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:27 smithi190 ceph-mon[32247]: osdmap e470: 12 total, 12 up, 12 in 2024-06-27T21:29:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4278916955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:28.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:28.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:27 smithi192 ceph-mon[32956]: pgmap v1229: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/229093137' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:29:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:27 smithi192 ceph-mon[32956]: osdmap e470: 12 total, 12 up, 12 in 2024-06-27T21:29:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4278916955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:28.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:28.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:28.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:28.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:28.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:28.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:28.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:28.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:28.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:29.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:29.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:29.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:28 smithi151 ceph-mon[28965]: pgmap v1231: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:29.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:28 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:29:29.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:28 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:29:29.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:29.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:28 smithi190 ceph-mon[32247]: pgmap v1231: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:28 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:29:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:28 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:29:29.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:28 smithi192 ceph-mon[32956]: pgmap v1231: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:28 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:29:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:28 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:29:29.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:29.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:29.474 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-06-27T21:29:29.475 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-27T21:29:29.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:29:29.479 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:29:29.480 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:29:29.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:29.540 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:29:29.540 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:29:29.540 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_117994 2024-06-27T21:29:29.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:29:29.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:29:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:29.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:29.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:29.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:29.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:29.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:29.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:29.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:29.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:30.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:30.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:30.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:30.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:30.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:30.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:30.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:30.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:30.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 1700 1684 105.244 105.25 0.00421659 0.0094644 2024-06-27T21:29:30.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:30.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:30.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:30.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:30.817 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:29:30.817 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:29:30.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:30.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:30.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:30.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:31.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:31.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:31.123 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:29:31.132 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:31.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:31.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:31.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:31.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:31.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:29:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:31 smithi151 ceph-mon[28965]: pgmap v1232: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4231989583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:29:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:31 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:29:31.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:31.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:31 smithi190 ceph-mon[32247]: pgmap v1232: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:31.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4231989583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:29:31.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:31 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:29:31.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 3350 3334 104.18 103.125 0.00636444 0.00957918 2024-06-27T21:29:31.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:31 smithi192 ceph-mon[32956]: pgmap v1232: 27 pgs: 27 active+clean; 577 KiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:29:31.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4231989583' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:29:31.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:31 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:29:31.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:31.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:31.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:31.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:31.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:31.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:31.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:31.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:29:31.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:32.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:32.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:32.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:32.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:32.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:32.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:32.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:32.398 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:32.407 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:32 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:29:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:32 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:29:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:32 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:32 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:29:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:32 smithi151 ceph-mon[28965]: osdmap e471: 12 total, 12 up, 12 in 2024-06-27T21:29:32.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:32 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:29:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:32 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:29:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:32 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:32 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:29:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:32 smithi190 ceph-mon[32247]: osdmap e471: 12 total, 12 up, 12 in 2024-06-27T21:29:32.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:32.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 16 5488 5472 113.992 133.625 0.00506552 0.00875738 2024-06-27T21:29:32.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:32 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:29:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:32 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:29:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:32 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:32 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:29:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:32 smithi192 ceph-mon[32956]: osdmap e471: 12 total, 12 up, 12 in 2024-06-27T21:29:32.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:32.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:32.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:32.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:32.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:32.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:33.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:33.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:33.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:33.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:33.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:33.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:33.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:33 smithi151 ceph-mon[28965]: pgmap v1234: 27 pgs: 27 active+clean; 29 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 57 op/s 2024-06-27T21:29:33.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:33.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:33 smithi190 ceph-mon[32247]: pgmap v1234: 27 pgs: 27 active+clean; 29 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 57 op/s 2024-06-27T21:29:33.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:33.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:33.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 7060 7044 110.054 98.25 0.00593084 0.00906802 2024-06-27T21:29:33.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:33.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:33 smithi192 ceph-mon[32956]: pgmap v1234: 27 pgs: 27 active+clean; 29 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 57 op/s 2024-06-27T21:29:33.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:33.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:33.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:33.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:33.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:33.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:33.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:34.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:34.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:34.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:34.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:34.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:34.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:34.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:34.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:29:34.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:34.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:34.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:34.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:34.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:34.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 8441 8425 105.305 86.3125 0.038761 0.00946026 2024-06-27T21:29:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:34.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:34.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:34.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:34.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:34.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:34.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:34.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:34.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:35.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:35.055 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:35.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:35.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:35.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:35 smithi190 ceph-mon[32247]: pgmap v1235: 27 pgs: 27 active+clean; 249 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 497 op/s 2024-06-27T21:29:35.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:35 smithi190 ceph-mon[32247]: osdmap e472: 12 total, 12 up, 12 in 2024-06-27T21:29:35.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:35.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:35.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:35.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:35.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:35 smithi151 ceph-mon[28965]: pgmap v1235: 27 pgs: 27 active+clean; 249 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 497 op/s 2024-06-27T21:29:35.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:35 smithi151 ceph-mon[28965]: osdmap e472: 12 total, 12 up, 12 in 2024-06-27T21:29:35.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:35.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 9684 9668 100.701 77.6875 0.0020824 0.00991235 2024-06-27T21:29:35.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:35 smithi192 ceph-mon[32956]: pgmap v1235: 27 pgs: 27 active+clean; 249 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 497 op/s 2024-06-27T21:29:35.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:35 smithi192 ceph-mon[32956]: osdmap e472: 12 total, 12 up, 12 in 2024-06-27T21:29:35.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:35.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:35.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:35.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:35.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:35.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:35.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:35.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:36.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:36.055 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:36.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:36.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:36.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:36.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:36.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:36.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:36.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:29:36.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:29:36.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:29:36.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 10887 10871 97.0551 75.1875 0.00431499 0.0102902 2024-06-27T21:29:36.559 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:29:36.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:29:36.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:36.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:29:36.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:29:36.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:29:36.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:29:36.638 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:29:36.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:29:36.663 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:29:36.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:29:36.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:36.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:36.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:36.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:36.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:36.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:36.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:37.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:37.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:37.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:37.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:37.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:37.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:37.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:37.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:37.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:37 smithi151 ceph-mon[28965]: pgmap v1237: 27 pgs: 27 active+clean; 413 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s 2024-06-27T21:29:37.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/342095237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:37.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:37.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:37.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:37 smithi190 ceph-mon[32247]: pgmap v1237: 27 pgs: 27 active+clean; 413 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s 2024-06-27T21:29:37.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/342095237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:37.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:37.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:37.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 11982 11966 93.4773 68.4375 0.00184748 0.0106145 2024-06-27T21:29:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:37 smithi192 ceph-mon[32956]: pgmap v1237: 27 pgs: 27 active+clean; 413 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s 2024-06-27T21:29:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/342095237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:37.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:37.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:37.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:37.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:37.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:37.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:37.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:37.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:37.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:37.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:37.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:37.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:38.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:38.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:38.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:38.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:38.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:38.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:38.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:38.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:38 smithi151 ceph-mon[28965]: osdmap e473: 12 total, 12 up, 12 in 2024-06-27T21:29:38.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:38 smithi190 ceph-mon[32247]: osdmap e473: 12 total, 12 up, 12 in 2024-06-27T21:29:38.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 13270 13254 92.0347 80.5 0.00457694 0.0108535 2024-06-27T21:29:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:38 smithi192 ceph-mon[32956]: osdmap e473: 12 total, 12 up, 12 in 2024-06-27T21:29:38.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:38.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:38.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:38.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:38.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:38.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:38.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:38.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:38.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:39.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:39.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:39.107 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:29:39.108 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:29:39.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:39.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:39.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:39.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:39.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:39.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:39.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:39 smithi151 ceph-mon[28965]: pgmap v1239: 27 pgs: 27 active+clean; 594 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-27T21:29:39.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:39 smithi151 ceph-mon[28965]: osdmap e474: 12 total, 12 up, 12 in 2024-06-27T21:29:39.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:39 smithi190 ceph-mon[32247]: pgmap v1239: 27 pgs: 27 active+clean; 594 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-27T21:29:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:39 smithi190 ceph-mon[32247]: osdmap e474: 12 total, 12 up, 12 in 2024-06-27T21:29:39.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 14490 14474 90.4557 76.25 0.00161564 0.0110264 2024-06-27T21:29:39.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:39.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:39.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:39 smithi192 ceph-mon[32956]: pgmap v1239: 27 pgs: 27 active+clean; 594 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-06-27T21:29:39.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:39 smithi192 ceph-mon[32956]: osdmap e474: 12 total, 12 up, 12 in 2024-06-27T21:29:39.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:39.587 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:29:39.587 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:29:39.587 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:29:39.587 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:29:39.587 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:29:39.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:39.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:39.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:39.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:39.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:39.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:29:39.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:39.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:39.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:40.041 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1240,"stamp":"2024-06-27T21:29:38.154570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647103008,"num_objects":9867,"num_object_clones":0,"num_object_copies":29601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9865,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1259104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10877,"ondisk_log_size":10877,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5876896,"kb_used_data":1821740,"kb_used_omap":110,"kb_used_meta":4054929,"kb_avail":1119064928,"statfs":{"total":1151940427776,"available":1145922486272,"internally_reserved":0,"allocated":1865461760,"data_stored":1861960242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":4152248279},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616562688,"num_objects":9408,"num_object_clones":0,"num_object_copies":28224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9408,"num_write_kb":602112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.092312"},"pg_stats":[{"pgid":"6.c","version":"472'642","reported_seq":661,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:35.668247+0000","last_change":"2024-06-27T21:29:19.819646+0000","last_active":"2024-06-27T21:29:35.668247+0000","last_peered":"2024-06-27T21:29:35.668247+0000","last_clean":"2024-06-27T21:29:35.668247+0000","last_became_active":"2024-06-27T21:29:19.819315+0000","last_became_peered":"2024-06-27T21:29:19.819315+0000","last_unstale":"2024-06-27T21:29:35.668247+0000","last_undegraded":"2024-06-27T21:29:35.668247+0000","last_fullsized":"2024-06-27T21:29:35.668247+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":642,"log_dups_size":0,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:30.608875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42074112,"num_objects":642,"num_object_clones":0,"num_object_copies":1926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":642,"num_write_kb":41088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,8],"acting":[6,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"473'760","reported_seq":780,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:37.441317+0000","last_change":"2024-06-27T21:29:19.820104+0000","last_active":"2024-06-27T21:29:37.441317+0000","last_peered":"2024-06-27T21:29:37.441317+0000","last_clean":"2024-06-27T21:29:37.441317+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:29:37.441317+0000","last_undegraded":"2024-06-27T21:29:37.441317+0000","last_fullsized":"2024-06-27T21:29:37.441317+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":760,"log_dups_size":0,"ondisk_log_size":760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:36.282749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":760,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":760,"num_write_kb":48640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"473'732","reported_seq":752,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:37.870997+0000","last_change":"2024-06-27T21:29:19.820432+0000","last_active":"2024-06-27T21:29:37.870997+0000","last_peered":"2024-06-27T21:29:37.870997+0000","last_clean":"2024-06-27T21:29:37.870997+0000","last_became_active":"2024-06-27T21:29:19.820126+0000","last_became_peered":"2024-06-27T21:29:19.820126+0000","last_unstale":"2024-06-27T21:29:37.870997+0000","last_undegraded":"2024-06-27T21:29:37.870997+0000","last_fullsized":"2024-06-27T21:29:37.870997+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.126312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47972352,"num_objects":732,"num_object_clones":0,"num_object_copies":2196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":732,"num_write_kb":46848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"472'611","reported_seq":631,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:36.176421+0000","last_change":"2024-06-27T21:29:19.890005+0000","last_active":"2024-06-27T21:29:36.176421+0000","last_peered":"2024-06-27T21:29:36.176421+0000","last_clean":"2024-06-27T21:29:36.176421+0000","last_became_active":"2024-06-27T21:29:19.889662+0000","last_became_peered":"2024-06-27T21:29:19.889662+0000","last_unstale":"2024-06-27T21:29:36.176421+0000","last_undegraded":"2024-06-27T21:29:36.176421+0000","last_fullsized":"2024-06-27T21:29:36.176421+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:48.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40042496,"num_objects":611,"num_object_clones":0,"num_object_copies":1833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":611,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"472'577","reported_seq":597,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.507865+0000","last_change":"2024-06-27T21:29:19.921495+0000","last_active":"2024-06-27T21:29:34.507865+0000","last_peered":"2024-06-27T21:29:34.507865+0000","last_clean":"2024-06-27T21:29:34.507865+0000","last_became_active":"2024-06-27T21:29:19.921198+0000","last_became_peered":"2024-06-27T21:29:19.921198+0000","last_unstale":"2024-06-27T21:29:34.507865+0000","last_undegraded":"2024-06-27T21:29:34.507865+0000","last_fullsized":"2024-06-27T21:29:34.507865+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:39.282908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"473'781","reported_seq":802,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:37.885118+0000","last_change":"2024-06-27T21:29:19.822891+0000","last_active":"2024-06-27T21:29:37.885118+0000","last_peered":"2024-06-27T21:29:37.885118+0000","last_clean":"2024-06-27T21:29:37.885118+0000","last_became_active":"2024-06-27T21:29:19.822526+0000","last_became_peered":"2024-06-27T21:29:19.822526+0000","last_unstale":"2024-06-27T21:29:37.885118+0000","last_undegraded":"2024-06-27T21:29:37.885118+0000","last_fullsized":"2024-06-27T21:29:37.885118+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:46.475678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51183616,"num_objects":781,"num_object_clones":0,"num_object_copies":2343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":781,"num_write_kb":49984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":87,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140280+0000","last_change":"2024-06-27T21:29:15.918767+0000","last_active":"2024-06-27T21:29:34.140280+0000","last_peered":"2024-06-27T21:29:34.140280+0000","last_clean":"2024-06-27T21:29:34.140280+0000","last_became_active":"2024-06-27T21:27:38.938604+0000","last_became_peered":"2024-06-27T21:27:38.938604+0000","last_unstale":"2024-06-27T21:29:34.140280+0000","last_undegraded":"2024-06-27T21:29:34.140280+0000","last_fullsized":"2024-06-27T21:29:34.140280+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:15.918674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:15.918674+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.720776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"471'506","reported_seq":523,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.113434+0000","last_change":"2024-06-27T21:29:19.889785+0000","last_active":"2024-06-27T21:29:34.113434+0000","last_peered":"2024-06-27T21:29:34.113434+0000","last_clean":"2024-06-27T21:29:34.113434+0000","last_became_active":"2024-06-27T21:29:19.889439+0000","last_became_peered":"2024-06-27T21:29:19.889439+0000","last_unstale":"2024-06-27T21:29:34.113434+0000","last_undegraded":"2024-06-27T21:29:34.113434+0000","last_fullsized":"2024-06-27T21:29:34.113434+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:36.151289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":506,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":586,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.141554+0000","last_change":"2024-06-27T21:28:17.076309+0000","last_active":"2024-06-27T21:29:34.141554+0000","last_peered":"2024-06-27T21:29:34.141554+0000","last_clean":"2024-06-27T21:29:34.141554+0000","last_became_active":"2024-06-27T21:23:44.524342+0000","last_became_peered":"2024-06-27T21:23:44.524342+0000","last_unstale":"2024-06-27T21:29:34.141554+0000","last_undegraded":"2024-06-27T21:29:34.141554+0000","last_fullsized":"2024-06-27T21:29:34.141554+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:28:17.076202+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:28:17.076202+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"472'532","reported_seq":551,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.503823+0000","last_change":"2024-06-27T21:29:19.889671+0000","last_active":"2024-06-27T21:29:34.503823+0000","last_peered":"2024-06-27T21:29:34.503823+0000","last_clean":"2024-06-27T21:29:34.503823+0000","last_became_active":"2024-06-27T21:29:19.889320+0000","last_became_peered":"2024-06-27T21:29:19.889320+0000","last_unstale":"2024-06-27T21:29:34.503823+0000","last_undegraded":"2024-06-27T21:29:34.503823+0000","last_fullsized":"2024-06-27T21:29:34.503823+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.902496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":79,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140272+0000","last_change":"2024-06-27T21:29:12.542530+0000","last_active":"2024-06-27T21:29:34.140272+0000","last_peered":"2024-06-27T21:29:34.140272+0000","last_clean":"2024-06-27T21:29:34.140272+0000","last_became_active":"2024-06-27T21:27:38.935538+0000","last_became_peered":"2024-06-27T21:27:38.935538+0000","last_unstale":"2024-06-27T21:29:34.140272+0000","last_undegraded":"2024-06-27T21:29:34.140272+0000","last_fullsized":"2024-06-27T21:29:34.140272+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:12.542433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:12.542433+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:22.271924+0000","scrub_duration":708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"473'812","reported_seq":834,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:38.051814+0000","last_change":"2024-06-27T21:29:19.821949+0000","last_active":"2024-06-27T21:29:38.051814+0000","last_peered":"2024-06-27T21:29:38.051814+0000","last_clean":"2024-06-27T21:29:38.051814+0000","last_became_active":"2024-06-27T21:29:19.821580+0000","last_became_peered":"2024-06-27T21:29:19.821580+0000","last_unstale":"2024-06-27T21:29:38.051814+0000","last_undegraded":"2024-06-27T21:29:38.051814+0000","last_fullsized":"2024-06-27T21:29:38.051814+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":812,"log_dups_size":0,"ondisk_log_size":812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.451785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53215232,"num_objects":812,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":812,"num_write_kb":51968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":398,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140830+0000","last_change":"2024-06-27T21:28:20.439527+0000","last_active":"2024-06-27T21:29:34.140830+0000","last_peered":"2024-06-27T21:29:34.140830+0000","last_clean":"2024-06-27T21:29:34.140830+0000","last_became_active":"2024-06-27T21:21:18.602559+0000","last_became_peered":"2024-06-27T21:21:18.602559+0000","last_unstale":"2024-06-27T21:29:34.140830+0000","last_undegraded":"2024-06-27T21:29:34.140830+0000","last_fullsized":"2024-06-27T21:29:34.140830+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:28:20.439425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:28:20.439425+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:29:45.517944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"472'684","reported_seq":703,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:36.182874+0000","last_change":"2024-06-27T21:29:19.819739+0000","last_active":"2024-06-27T21:29:36.182874+0000","last_peered":"2024-06-27T21:29:36.182874+0000","last_clean":"2024-06-27T21:29:36.182874+0000","last_became_active":"2024-06-27T21:29:19.819457+0000","last_became_peered":"2024-06-27T21:29:19.819457+0000","last_unstale":"2024-06-27T21:29:36.182874+0000","last_undegraded":"2024-06-27T21:29:36.182874+0000","last_fullsized":"2024-06-27T21:29:36.182874+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:22.559387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44826624,"num_objects":684,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":43776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":739,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.141650+0000","last_change":"2024-06-27T21:29:13.047434+0000","last_active":"2024-06-27T21:29:34.141650+0000","last_peered":"2024-06-27T21:29:34.141650+0000","last_clean":"2024-06-27T21:29:34.141650+0000","last_became_active":"2024-06-27T21:27:17.225893+0000","last_became_peered":"2024-06-27T21:27:17.225893+0000","last_unstale":"2024-06-27T21:29:34.141650+0000","last_undegraded":"2024-06-27T21:29:34.141650+0000","last_fullsized":"2024-06-27T21:29:34.141650+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:13.047331+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:29:13.047331+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:21.894277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"473'684","reported_seq":705,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:37.369612+0000","last_change":"2024-06-27T21:29:19.890411+0000","last_active":"2024-06-27T21:29:37.369612+0000","last_peered":"2024-06-27T21:29:37.369612+0000","last_clean":"2024-06-27T21:29:37.369612+0000","last_became_active":"2024-06-27T21:29:19.889910+0000","last_became_peered":"2024-06-27T21:29:19.889910+0000","last_unstale":"2024-06-27T21:29:37.369612+0000","last_undegraded":"2024-06-27T21:29:37.369612+0000","last_fullsized":"2024-06-27T21:29:37.369612+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:32.566921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44826624,"num_objects":684,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":43776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":398,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140464+0000","last_change":"2024-06-27T21:28:15.513998+0000","last_active":"2024-06-27T21:29:34.140464+0000","last_peered":"2024-06-27T21:29:34.140464+0000","last_clean":"2024-06-27T21:29:34.140464+0000","last_became_active":"2024-06-27T21:21:18.602398+0000","last_became_peered":"2024-06-27T21:21:18.602398+0000","last_unstale":"2024-06-27T21:29:34.140464+0000","last_undegraded":"2024-06-27T21:29:34.140464+0000","last_fullsized":"2024-06-27T21:29:34.140464+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:28:15.513900+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:28:15.513900+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:29:34.249508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"471'437","reported_seq":455,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-06-27T21:29:33.400383+0000","last_change":"2024-06-27T21:29:19.921839+0000","last_active":"2024-06-27T21:29:33.400383+0000","last_peered":"2024-06-27T21:29:33.400383+0000","last_clean":"2024-06-27T21:29:33.400383+0000","last_became_active":"2024-06-27T21:29:19.921089+0000","last_became_peered":"2024-06-27T21:29:19.921089+0000","last_unstale":"2024-06-27T21:29:33.400383+0000","last_undegraded":"2024-06-27T21:29:33.400383+0000","last_fullsized":"2024-06-27T21:29:33.400383+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:33.415477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28639232,"num_objects":437,"num_object_clones":0,"num_object_copies":1311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":437,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":394,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140642+0000","last_change":"2024-06-27T21:29:30.721306+0000","last_active":"2024-06-27T21:29:34.140642+0000","last_peered":"2024-06-27T21:29:34.140642+0000","last_clean":"2024-06-27T21:29:34.140642+0000","last_became_active":"2024-06-27T21:21:34.494291+0000","last_became_peered":"2024-06-27T21:21:34.494291+0000","last_unstale":"2024-06-27T21:29:34.140642+0000","last_undegraded":"2024-06-27T21:29:34.140642+0000","last_fullsized":"2024-06-27T21:29:34.140642+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:30.721220+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:29:30.721220+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:53.297813+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"472'496","reported_seq":516,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.489456+0000","last_change":"2024-06-27T21:29:19.921504+0000","last_active":"2024-06-27T21:29:34.489456+0000","last_peered":"2024-06-27T21:29:34.489456+0000","last_clean":"2024-06-27T21:29:34.489456+0000","last_became_active":"2024-06-27T21:29:19.921150+0000","last_became_peered":"2024-06-27T21:29:19.921150+0000","last_unstale":"2024-06-27T21:29:34.489456+0000","last_undegraded":"2024-06-27T21:29:34.489456+0000","last_fullsized":"2024-06-27T21:29:34.489456+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:38.653405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32505856,"num_objects":496,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":31744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":828,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.144652+0000","last_change":"2024-06-27T21:29:17.866834+0000","last_active":"2024-06-27T21:29:34.144652+0000","last_peered":"2024-06-27T21:29:34.144652+0000","last_clean":"2024-06-27T21:29:34.144652+0000","last_became_active":"2024-06-27T21:27:38.935752+0000","last_became_peered":"2024-06-27T21:27:38.935752+0000","last_unstale":"2024-06-27T21:29:34.144652+0000","last_undegraded":"2024-06-27T21:29:34.144652+0000","last_fullsized":"2024-06-27T21:29:34.144652+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:17.866726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:17.866726+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:20.912742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"473'757","reported_seq":778,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:37.876750+0000","last_change":"2024-06-27T21:29:19.862967+0000","last_active":"2024-06-27T21:29:37.876750+0000","last_peered":"2024-06-27T21:29:37.876750+0000","last_clean":"2024-06-27T21:29:37.876750+0000","last_became_active":"2024-06-27T21:29:19.862712+0000","last_became_peered":"2024-06-27T21:29:19.862712+0000","last_unstale":"2024-06-27T21:29:37.876750+0000","last_undegraded":"2024-06-27T21:29:37.876750+0000","last_fullsized":"2024-06-27T21:29:37.876750+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:44.087500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49610752,"num_objects":757,"num_object_clones":0,"num_object_copies":2271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":48448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":705,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140496+0000","last_change":"2024-06-27T21:29:12.778561+0000","last_active":"2024-06-27T21:29:34.140496+0000","last_peered":"2024-06-27T21:29:34.140496+0000","last_clean":"2024-06-27T21:29:34.140496+0000","last_became_active":"2024-06-27T21:23:44.529707+0000","last_became_peered":"2024-06-27T21:23:44.529707+0000","last_unstale":"2024-06-27T21:29:34.140496+0000","last_undegraded":"2024-06-27T21:29:34.140496+0000","last_fullsized":"2024-06-27T21:29:34.140496+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:12.778462+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:29:12.778462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:33.197634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.e","version":"471'423","reported_seq":441,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-06-27T21:29:33.408334+0000","last_change":"2024-06-27T21:29:19.821974+0000","last_active":"2024-06-27T21:29:33.408334+0000","last_peered":"2024-06-27T21:29:33.408334+0000","last_clean":"2024-06-27T21:29:33.408334+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:29:33.408334+0000","last_undegraded":"2024-06-27T21:29:33.408334+0000","last_fullsized":"2024-06-27T21:29:33.408334+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:36.243439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":79,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-06-27T21:29:34.140262+0000","last_change":"2024-06-27T21:29:10.827592+0000","last_active":"2024-06-27T21:29:34.140262+0000","last_peered":"2024-06-27T21:29:34.140262+0000","last_clean":"2024-06-27T21:29:34.140262+0000","last_became_active":"2024-06-27T21:27:38.938524+0000","last_became_peered":"2024-06-27T21:27:38.938524+0000","last_unstale":"2024-06-27T21:29:34.140262+0000","last_undegraded":"2024-06-27T21:29:34.140262+0000","last_fullsized":"2024-06-27T21:29:34.140262+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:10.827466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:10.827466+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.417083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"471'431","reported_seq":448,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-06-27T21:29:33.405305+0000","last_change":"2024-06-27T21:29:19.890876+0000","last_active":"2024-06-27T21:29:33.405305+0000","last_peered":"2024-06-27T21:29:33.405305+0000","last_clean":"2024-06-27T21:29:33.405305+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:29:33.405305+0000","last_undegraded":"2024-06-27T21:29:33.405305+0000","last_fullsized":"2024-06-27T21:29:33.405305+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:47.893415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":431,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":750,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-06-27T21:29:37.172503+0000","last_change":"2024-06-27T21:29:09.230494+0000","last_active":"2024-06-27T21:29:37.172503+0000","last_peered":"2024-06-27T21:29:37.172503+0000","last_clean":"2024-06-27T21:29:37.172503+0000","last_became_active":"2024-06-27T21:27:16.656216+0000","last_became_peered":"2024-06-27T21:27:16.656216+0000","last_unstale":"2024-06-27T21:29:37.172503+0000","last_undegraded":"2024-06-27T21:29:37.172503+0000","last_fullsized":"2024-06-27T21:29:37.172503+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:09.230395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:09.230395+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:25.594711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":646512640,"num_objects":9865,"num_object_clones":0,"num_object_copies":29595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9865,"num_write_kb":631360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1958871040,"data_stored":1958871040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9865,"ondisk_log_size":9865,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1259104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253108,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":525668,"kb_used_data":93276,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93219484,"statfs":{"total":95995035648,"available":95456751616,"internally_reserved":0,"allocated":95514624,"data_stored":95227842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514744,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419892,"kb_used_data":99944,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93325260,"statfs":{"total":95995035648,"available":95565066240,"internally_reserved":0,"allocated":102342656,"data_stored":102051376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743677,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":561712,"kb_used_data":183464,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93183440,"statfs":{"total":95995035648,"available":95419842560,"internally_reserved":0,"allocated":187867136,"data_stored":187575856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972609,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":340344,"kb_used_data":132848,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93404808,"statfs":{"total":95995035648,"available":95646523392,"internally_reserved":0,"allocated":136036352,"data_stored":135741520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266949,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":551204,"kb_used_data":154908,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93193948,"statfs":{"total":95995035648,"available":95430602752,"internally_reserved":0,"allocated":158625792,"data_stored":158339010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":405797322},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528587,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655988,"kb_used_data":187496,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93089164,"statfs":{"total":95995035648,"available":95323303936,"internally_reserved":0,"allocated":191995904,"data_stored":191704624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":479721926},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590408,"kb_used_data":284348,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93154744,"statfs":{"total":95995035648,"available":95390457856,"internally_reserved":0,"allocated":291172352,"data_stored":290873022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161224,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":388800,"kb_used_data":202868,"kb_used_omap":92,"kb_used_meta":185827,"kb_avail":93356352,"statfs":{"total":95995035648,"available":95596904448,"internally_reserved":0,"allocated":207736832,"data_stored":207441054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":190286946},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182681,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472576,"kb_used_data":144308,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93272576,"statfs":{"total":95995035648,"available":95511117824,"internally_reserved":0,"allocated":147771392,"data_stored":147475614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":336132551},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411613,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310892,"kb_used_data":99236,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93434260,"statfs":{"total":95995035648,"available":95676682240,"internally_reserved":0,"allocated":101617664,"data_stored":101327330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575138,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":544228,"kb_used_data":120156,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93200924,"statfs":{"total":95995035648,"available":95437746176,"internally_reserved":0,"allocated":123039744,"data_stored":122752962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738663,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515184,"kb_used_data":118888,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93229968,"statfs":{"total":95995035648,"available":95467487232,"internally_reserved":0,"allocated":121741312,"data_stored":121450032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125566976,"data_stored":125566976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129433600,"data_stored":129433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113704960,"data_stored":113704960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149880832,"data_stored":149880832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":209846272,"data_stored":209846272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":309067776,"data_stored":309067776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":203685888,"data_stored":203685888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":175439872,"data_stored":175439872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":140771328,"data_stored":140771328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":190775296,"data_stored":190775296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102760448,"data_stored":102760448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":107937792,"data_stored":107937792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:29:40.043 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-27T21:29:40.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-06-27T21:29:40.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:40.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:40.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:40.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:40.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:40.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:40.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:40.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:40.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:40.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:40.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:40.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:40 smithi151 ceph-mon[28965]: from='client.16353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:29:40.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:40 smithi151 ceph-mon[28965]: osdmap e475: 12 total, 12 up, 12 in 2024-06-27T21:29:40.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:40 smithi190 ceph-mon[32247]: from='client.16353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:29:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:40 smithi190 ceph-mon[32247]: osdmap e475: 12 total, 12 up, 12 in 2024-06-27T21:29:40.541 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 15565 15549 88.3399 67.1875 0.00442771 0.0112943 2024-06-27T21:29:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:40 smithi192 ceph-mon[32956]: from='client.16353 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:29:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:40 smithi192 ceph-mon[32956]: osdmap e475: 12 total, 12 up, 12 in 2024-06-27T21:29:40.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:40.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:40.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:40.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:40.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:40.836 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:40.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:40.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:40.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:41.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:41.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:41.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:41.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:41.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:29:41.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:41.347 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:41.358 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:41.370 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:41.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:41.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:41.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:41.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 16648 16632 86.6185 67.6875 0.00162668 0.0115352 2024-06-27T21:29:41.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:41.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:41.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:41.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:41.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:41.886 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:41 smithi151 ceph-mon[28965]: pgmap v1242: 27 pgs: 27 active+clean; 789 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-27T21:29:41.894 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:41.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:41.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:42.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:41 smithi190 ceph-mon[32247]: pgmap v1242: 27 pgs: 27 active+clean; 789 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-27T21:29:42.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:42.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:41 smithi192 ceph-mon[32956]: pgmap v1242: 27 pgs: 27 active+clean; 789 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-06-27T21:29:42.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:42.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:42.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:42.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:42.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:42.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:42.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:42.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:42.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 18086 18070 86.8681 89.875 0.091901 0.0114199 2024-06-27T21:29:42.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:42.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:42.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:42.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:42.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:42.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:42.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:43.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:43.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:43.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:43.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:43.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:43.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:43.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:43.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:43.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:43.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:43.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 19218 19202 85.7165 70.75 0.0044521 0.0116556 2024-06-27T21:29:43.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:43.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:43.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:43.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:43.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:43.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:43 smithi151 ceph-mon[28965]: pgmap v1243: 27 pgs: 27 active+clean; 855 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:29:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:43.923 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:43.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:44.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:43 smithi190 ceph-mon[32247]: pgmap v1243: 27 pgs: 27 active+clean; 855 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:29:44.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:43 smithi192 ceph-mon[32956]: pgmap v1243: 27 pgs: 27 active+clean; 855 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:29:44.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:44.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:44.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:44.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:44.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:44.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:44.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:44.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:44.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 20582 20566 85.6849 85.25 0.0202256 0.0116398 2024-06-27T21:29:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:44.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:44.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:44.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:44.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:44.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:44.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:44.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:44.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2470571366,config=2353167246,health=551204407,kv=4185761280}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=1269847603,logm=3773479751}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472600878}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=300418248}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4024290861}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2898681671}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360866222}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545816009}) 2024-06-27T21:29:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4234884099}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=781925627}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3219203263}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1318432564}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664977076}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520298775}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408445580}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672291488}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910929040}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1128383471}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=379648696}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2890948422}) 2024-06-27T21:29:44.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914632881}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2197394728}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311972890}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3259726639}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852020269}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=237316695}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4068406649}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185685666}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530091667}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930107925}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871435228}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1034699907}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3501005462}) 2024-06-27T21:29:44.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3415171296}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047927884}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2306944963}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2325208420}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474750159}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887800341}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2333270133}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2848426790}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362832172}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628157041}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1582312071}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684755371}) 2024-06-27T21:29:44.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1529135979}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2348327283}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=4230942415,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=2600553119}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=886317940,mon_config_key=2103911893}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3579225591}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=93,monmap=5,osd_metadata=2} crc {mon_config_key=3495406727,monmap=3596043196,osd_metadata=427272628}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=10,osd_pg_creating=1,osdmap=89} crc {osd_metadata=2913421361,osd_pg_creating=1635894308,osdmap=2552074700}) 2024-06-27T21:29:44.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=2257549957}) 2024-06-27T21:29:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2470571366,config=2353167246,health=551204407,kv=4185761280}) 2024-06-27T21:29:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=1269847603,logm=3773479751}) 2024-06-27T21:29:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472600878}) 2024-06-27T21:29:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=300418248}) 2024-06-27T21:29:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4024290861}) 2024-06-27T21:29:44.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2898681671}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360866222}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545816009}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4234884099}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=781925627}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3219203263}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1318432564}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664977076}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520298775}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408445580}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672291488}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910929040}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1128383471}) 2024-06-27T21:29:44.997 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=379648696}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2890948422}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914632881}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2197394728}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311972890}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3259726639}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852020269}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=237316695}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4068406649}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185685666}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530091667}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930107925}) 2024-06-27T21:29:44.998 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871435228}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1034699907}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3501005462}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3415171296}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047927884}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2306944963}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2325208420}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474750159}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887800341}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2333270133}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2848426790}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362832172}) 2024-06-27T21:29:44.999 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628157041}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1582312071}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684755371}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1529135979}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2348327283}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=4230942415,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=2600553119}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=886317940,mon_config_key=2103911893}) 2024-06-27T21:29:45.000 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3579225591}) 2024-06-27T21:29:45.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=93,monmap=5,osd_metadata=2} crc {mon_config_key=3495406727,monmap=3596043196,osd_metadata=427272628}) 2024-06-27T21:29:45.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=10,osd_pg_creating=1,osdmap=89} crc {osd_metadata=2913421361,osd_pg_creating=1635894308,osdmap=2552074700}) 2024-06-27T21:29:45.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=2257549957}) 2024-06-27T21:29:45.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2470571366,config=2353167246,health=551204407,kv=4185761280}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=1269847603,logm=3773479751}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1472600878}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=300418248}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4024290861}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2898681671}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3360866222}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545816009}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4234884099}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=781925627}) 2024-06-27T21:29:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3219203263}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1318432564}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=664977076}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1520298775}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408445580}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672291488}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=910929040}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1128383471}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=379648696}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2890948422}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914632881}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2197394728}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311972890}) 2024-06-27T21:29:45.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3259726639}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2852020269}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=237316695}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4068406649}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3185685666}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3530091667}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930107925}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871435228}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1034699907}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3501005462}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3415171296}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047927884}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2306944963}) 2024-06-27T21:29:45.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2325208420}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474750159}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887800341}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2333270133}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2848426790}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362832172}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628157041}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1582312071}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684755371}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1529135979}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2348327283}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=68,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=8} crc {logm=4230942415,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=2600553119}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=886317940,mon_config_key=2103911893}) 2024-06-27T21:29:45.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3579225591}) 2024-06-27T21:29:45.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=93,monmap=5,osd_metadata=2} crc {mon_config_key=3495406727,monmap=3596043196,osd_metadata=427272628}) 2024-06-27T21:29:45.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=10,osd_pg_creating=1,osdmap=89} crc {osd_metadata=2913421361,osd_pg_creating=1635894308,osdmap=2552074700}) 2024-06-27T21:29:45.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=2257549957}) 2024-06-27T21:29:45.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:45.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:45.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:45.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:45.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:45.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:45.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:45.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:45.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 22225 22209 86.7471 102.688 0.0102094 0.0115203 2024-06-27T21:29:45.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:45.569 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:45.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:45.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:45 smithi151 ceph-mon[28965]: pgmap v1244: 27 pgs: 27 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:29:45.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:45.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:45.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:45.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:45.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:45 smithi190 ceph-mon[32247]: pgmap v1244: 27 pgs: 27 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:29:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:45 smithi192 ceph-mon[32956]: pgmap v1244: 27 pgs: 27 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:29:46.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:46.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:46.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:46.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:46.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:46.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:46.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:46.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:46.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:29:46.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:29:46.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:29:46.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:29:46.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 23771 23755 87.3277 96.625 0.0212813 0.0114436 2024-06-27T21:29:46.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:46.561 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:29:46.562 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:29:46.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:46.591 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:29:46.591 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:29:46.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:29:46.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:29:46.646 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:29:46.646 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:29:46.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:29:46.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:29:46.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:46.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:29:46.701 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:29:46.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:46.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:29:46.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:29:46.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:29:46.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:29:46.797 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:29:46.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:29:46.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:46.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:29:46.826 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:29:46.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1418707239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:46.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:46.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:46.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:46.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:47.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1418707239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:47.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1418707239' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:47.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:47.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:47.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:47.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:47.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:47.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:47.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:47.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:47.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 15 25140 25125 87.2327 85.625 0.00186427 0.0114559 2024-06-27T21:29:47.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:47.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:47.645 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:29:47.646 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 21 2024-06-27T21:29:47.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:47.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:47.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:47.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:47.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: pgmap v1245: 27 pgs: 27 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2215929223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/980631822' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-27T21:29:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:47 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-27T21:29:47.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:47.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: pgmap v1245: 27 pgs: 27 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:29:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2215929223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/980631822' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-27T21:29:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:47 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-27T21:29:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: pgmap v1245: 27 pgs: 27 active+clean; 1.2 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:29:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2215929223' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:29:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/980631822' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-27T21:29:48.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:47 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-06-27T21:29:48.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:48.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:48.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:48.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:48.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:48.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:48.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:48.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:48.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 26455 26439 86.9636 82.125 0.0040226 0.0114903 2024-06-27T21:29:48.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:48.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:48.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:48.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:48 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:48 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:29:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:48 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-27T21:29:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:48 smithi151 ceph-mon[28965]: osdmap e476: 12 total, 12 up, 12 in 2024-06-27T21:29:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:48 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:48.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:48 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:48.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:48.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:48.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:48.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:48.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:48 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:48 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:29:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:48 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-27T21:29:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:48 smithi190 ceph-mon[32247]: osdmap e476: 12 total, 12 up, 12 in 2024-06-27T21:29:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:48 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:48 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:49.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:48 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:48 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:29:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:48 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-06-27T21:29:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:48 smithi192 ceph-mon[32956]: osdmap e476: 12 total, 12 up, 12 in 2024-06-27T21:29:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:48 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:48 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:49.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:49.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:49.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:49.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:49.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:49.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:49.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:49.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:49.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:29:49.543183+0000 min lat: 0.00108868 max lat: 0.273193 avg lat: 0.0112414 2024-06-27T21:29:49.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:29:49.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 28461 28445 88.8837 125.375 0.0125419 0.0112414 2024-06-27T21:29:49.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:49.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:49.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:49.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:49.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:49 smithi151 ceph-mon[28965]: pgmap v1247: 27 pgs: 27 active+clean; 1.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-27T21:29:49.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:49 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-27T21:29:49.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:49 smithi151 ceph-mon[28965]: osdmap e477: 12 total, 12 up, 12 in 2024-06-27T21:29:49.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:49.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:49.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:50.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:50.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:49 smithi190 ceph-mon[32247]: pgmap v1247: 27 pgs: 27 active+clean; 1.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-27T21:29:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:49 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-27T21:29:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:49 smithi190 ceph-mon[32247]: osdmap e477: 12 total, 12 up, 12 in 2024-06-27T21:29:50.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:50.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:49 smithi192 ceph-mon[32956]: pgmap v1247: 27 pgs: 27 active+clean; 1.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-06-27T21:29:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:49 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-06-27T21:29:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:49 smithi192 ceph-mon[32956]: osdmap e477: 12 total, 12 up, 12 in 2024-06-27T21:29:50.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:50.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:50.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:50.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:50.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:50.420 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:50.471 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:29:50.471 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:29:50.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:50.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 16 30015 29999 89.2758 97.125 0.0667205 0.0111775 2024-06-27T21:29:50.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:50.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:50.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:50.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:50 smithi151 ceph-mon[28965]: osdmap e478: 12 total, 12 up, 12 in 2024-06-27T21:29:50.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:50 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:29:50.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:50 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:50.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:50 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:50.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:50 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:29:50.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:50.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:50.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:50.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:50.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:50 smithi190 ceph-mon[32247]: osdmap e478: 12 total, 12 up, 12 in 2024-06-27T21:29:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:50 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:29:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:50 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:50 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:50 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:29:51.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:51.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:50 smithi192 ceph-mon[32956]: osdmap e478: 12 total, 12 up, 12 in 2024-06-27T21:29:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:50 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:29:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:50 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:50 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-06-27T21:29:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:50 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:29:51.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:51.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:51.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:51.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:51.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:51.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:51.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:51.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:51.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 31108 31092 88.3227 68.3125 0.00203285 0.0113087 2024-06-27T21:29:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:51.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:51.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:29:51.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:51.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:51.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:51.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:51.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:51.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:51.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: pgmap v1250: 37 pgs: 2 peering, 5 unknown, 30 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: osdmap e479: 12 total, 12 up, 12 in 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:29:51.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:51 smithi151 ceph-mon[28965]: osdmap e480: 12 total, 12 up, 12 in 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: pgmap v1250: 37 pgs: 2 peering, 5 unknown, 30 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: osdmap e479: 12 total, 12 up, 12 in 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:29:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:51 smithi190 ceph-mon[32247]: osdmap e480: 12 total, 12 up, 12 in 2024-06-27T21:29:52.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:29:52.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: pgmap v1250: 37 pgs: 2 peering, 5 unknown, 30 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: osdmap e479: 12 total, 12 up, 12 in 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:29:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:51 smithi192 ceph-mon[32956]: osdmap e480: 12 total, 12 up, 12 in 2024-06-27T21:29:52.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:52.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:52.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:52.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:52.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:29:52.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:52.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:52.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:52.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:52.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 32317 32301 87.7676 75.5625 0.00271017 0.0113869 2024-06-27T21:29:52.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:52.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:52.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:52.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:52.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:52.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:52.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:29:52.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:52 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:29:52.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:52.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:29:53.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:29:53.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:52 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:29:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:52 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:29:53.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:53.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:53.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:53.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:53.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:53.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:53.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:53.429 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:53.430 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:29:53.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:53.542 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 16 33364 33348 86.837 65.4375 0.00231964 0.0115092 2024-06-27T21:29:53.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:53.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:53.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:53.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:53.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:53.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: pgmap v1253: 37 pgs: 5 peering, 4 unknown, 28 active+clean; 1.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:29:53.873 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:29:53.874 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:53 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:29:53.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:53.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:53.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: pgmap v1253: 37 pgs: 5 peering, 4 unknown, 28 active+clean; 1.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:29:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:53 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: pgmap v1253: 37 pgs: 5 peering, 4 unknown, 28 active+clean; 1.7 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:29:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:53 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:29:54.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:54.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:54.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:54.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:54.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:54.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:54.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:54.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:54.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 34875 34859 87.1407 94.4375 0.0108673 0.0114686 2024-06-27T21:29:54.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:54.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:54.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:54.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:54 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:29:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:54 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:29:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:54 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:29:54.783 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:54 smithi151 ceph-mon[28965]: osdmap e481: 12 total, 12 up, 12 in 2024-06-27T21:29:54.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:54.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:54.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:54.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:54.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:54 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:29:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:54 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:29:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:54 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:29:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:54 smithi190 ceph-mon[32247]: osdmap e481: 12 total, 12 up, 12 in 2024-06-27T21:29:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:54 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:29:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:54 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:29:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:54 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:29:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:54 smithi192 ceph-mon[32956]: osdmap e481: 12 total, 12 up, 12 in 2024-06-27T21:29:55.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:55.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:55.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:55.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:55.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:55.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:55.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:29:55.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:29:55.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:55.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:55.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:55.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 36131 36115 86.8082 78.5 0.0103607 0.0115103 2024-06-27T21:29:55.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:55.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:55.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:29:55.612 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:29:55.638 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:29:55.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:29:55.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:29:55.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:29:55.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:29:55.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:29:55.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:55.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:29:55.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:29:55.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:55.742 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:29:55.743 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:29:55.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:55 smithi151 ceph-mon[28965]: pgmap v1254: 37 pgs: 5 peering, 2 unknown, 30 active+clean; 1.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-27T21:29:55.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:55.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:55.743 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2405656100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:55.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:29:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:29:55.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:29:55.798 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:29:55.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:29:55.831 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:29:55.831 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:29:55.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:29:55.861 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:29:55.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:29:55.886 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:29:55.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:29:55.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:55.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:55.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:55 smithi190 ceph-mon[32247]: pgmap v1254: 37 pgs: 5 peering, 2 unknown, 30 active+clean; 1.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-27T21:29:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2405656100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:56.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:55 smithi192 ceph-mon[32956]: pgmap v1254: 37 pgs: 5 peering, 2 unknown, 30 active+clean; 1.9 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-06-27T21:29:56.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:56.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:56.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2405656100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:29:56.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:56.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:56.170 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:29:56.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:29:56.170 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-06-27T21:29:56.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:29:56.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:56.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:56.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:56.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:56.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:56.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:56.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 15 37513 37498 86.7942 86.4375 0.0128401 0.011516 2024-06-27T21:29:56.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:56.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:56.708 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:29:56.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:29:56.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:29:56.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:56.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:56.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:56.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:57.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:56 smithi190 ceph-mon[32247]: osdmap e482: 12 total, 12 up, 12 in 2024-06-27T21:29:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4098946902' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:29:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:56 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:56 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:56 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:56 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:56 smithi192 ceph-mon[32956]: osdmap e482: 12 total, 12 up, 12 in 2024-06-27T21:29:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4098946902' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:29:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:56 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:56 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:56 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:56 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:57.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:57.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:56 smithi151 ceph-mon[28965]: osdmap e482: 12 total, 12 up, 12 in 2024-06-27T21:29:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4098946902' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:29:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:56 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:56 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:56 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:56 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:57.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:57.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:57.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:57.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:57.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:57.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:57.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 38554 38538 86.0156 65 0.00319549 0.0116199 2024-06-27T21:29:57.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:57.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:57.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:29:57.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:29:57.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:57.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:57.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:58.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: pgmap v1257: 37 pgs: 1 active+clean+scrubbing, 2 peering, 34 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4098946902' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: osdmap e483: 12 total, 12 up, 12 in 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:29:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:29:58.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:57 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:29:58.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: pgmap v1257: 37 pgs: 1 active+clean+scrubbing, 2 peering, 34 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4098946902' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: osdmap e483: 12 total, 12 up, 12 in 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:29:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:57 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:29:58.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:58.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: pgmap v1257: 37 pgs: 1 active+clean+scrubbing, 2 peering, 34 active+clean; 2.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:29:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:29:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4098946902' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:29:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: osdmap e483: 12 total, 12 up, 12 in 2024-06-27T21:29:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:29:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:29:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:29:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:29:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:57 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:29:58.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:58.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:58.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:58.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:58.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:29:58.539 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:29:58.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 39707 39691 85.5343 72.0625 0.021902 0.0116826 2024-06-27T21:29:58.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:58.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:58.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:29:58.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:29:58.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:29:58.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:58.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:29:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:29:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:29:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:29:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: osdmap e484: 12 total, 12 up, 12 in 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:29:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:29:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:29:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:58 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:29:59.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:29:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: osdmap e484: 12 total, 12 up, 12 in 2024-06-27T21:29:59.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:29:59.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:29:59.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:29:59.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:29:59.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:29:59.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:58 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:29:59.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:29:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: osdmap e484: 12 total, 12 up, 12 in 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:29:59.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:58 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:29:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:29:59.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:29:59.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:59.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:59.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:29:59.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:29:59.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:29:59.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:59.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:59.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:29:59.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 40854 40838 85.0725 71.6875 0.00228674 0.0117421 2024-06-27T21:29:59.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:29:59.591 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:29:59.591 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:29:59.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:40886 6.b 6:d1123755:::benchmark_data_smithi192_117994_object40885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:29:59.584564+0000", 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0058072760000000001, 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058300859999999999, 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 40886 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:29:59.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584564+0000", 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584564+0000", 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584566+0000", 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.075e-06 2024-06-27T21:29:59.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584607+0000", 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1017999999999999e-05 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584621+0000", 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.467e-05 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:29:59.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584629+0000", 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7559999999999999e-06 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584680+0000", 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1211999999999997e-05 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584782+0000", 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101232 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.584873+0000", 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1749000000000005e-05 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.586154+0000", 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001280661 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:29:59.586571+0000", 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041663900000000001 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:59.596 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:29:59.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:59.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:29:59.597 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:29:59.597 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-06-27T21:29:59.597 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:29:59.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:29:59.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:29:59.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:29:59.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:29:59.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:29:59.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:29:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:29:59.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:29:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:29:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:29:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:29:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:29:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: pgmap v1260: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 80 KiB/s, 0 objects/s recovering 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: osdmap e485: 12 total, 12 up, 12 in 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:29:59.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:29:59 smithi151 ceph-mon[28965]: osdmap e486: 12 total, 12 up, 12 in 2024-06-27T21:30:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:30:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:30:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:30:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: pgmap v1260: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 80 KiB/s, 0 objects/s recovering 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: osdmap e485: 12 total, 12 up, 12 in 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:30:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:30:00.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:29:59 smithi190 ceph-mon[32247]: osdmap e486: 12 total, 12 up, 12 in 2024-06-27T21:30:00.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: pgmap v1260: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 2.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 80 KiB/s, 0 objects/s recovering 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: osdmap e485: 12 total, 12 up, 12 in 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:30:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:30:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:29:59 smithi192 ceph-mon[32956]: osdmap e486: 12 total, 12 up, 12 in 2024-06-27T21:30:00.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:00.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:00.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:00.275 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:00.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:00.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:30:00.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:00.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:00.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:00.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 42425 42409 85.4953 98.1875 0.00282207 0.0116872 2024-06-27T21:30:00.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:00.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:00.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:00.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:00.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:00.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:00.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:00.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:00.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:00 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:30:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:00 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:30:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:00 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:30:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:00 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:30:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:00 smithi190 ceph-mon[32247]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-27T21:30:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:00 smithi190 ceph-mon[32247]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-27T21:30:01.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:00 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:30:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:00 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:30:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:00 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:30:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:00 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:30:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:00 smithi192 ceph-mon[32956]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-27T21:30:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:00 smithi192 ceph-mon[32956]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-27T21:30:01.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:00 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:30:01.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:00 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:30:01.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:00 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:30:01.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:00 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:30:01.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:00 smithi151 ceph-mon[28965]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-06-27T21:30:01.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:00 smithi151 ceph-mon[28965]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-06-27T21:30:01.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:01.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:01.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:30:01.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:01.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:01.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:01.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:01.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:01.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 15 43605 43590 85.1301 73.8125 0.00258472 0.0117407 2024-06-27T21:30:01.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:01.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:01.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:01.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:01.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:01.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:01.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:01.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:01.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:01 smithi190 ceph-mon[32247]: pgmap v1263: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s; 27 KiB/s, 0 objects/s recovering 2024-06-27T21:30:02.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:30:02.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:01 smithi192 ceph-mon[32956]: pgmap v1263: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s; 27 KiB/s, 0 objects/s recovering 2024-06-27T21:30:02.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:01 smithi151 ceph-mon[28965]: pgmap v1263: 37 pgs: 37 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s; 27 KiB/s, 0 objects/s recovering 2024-06-27T21:30:02.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:02.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:02.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:02.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:02.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:02.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:02.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 44926 44910 85.0502 82.5 0.00304872 0.0117525 2024-06-27T21:30:02.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:02.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:02.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:02.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:02.743 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:02.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:02.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:02.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:02.873 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:02.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:03.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:03.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:03.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:03.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:03.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:03.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:03.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:03.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:03.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:03.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 46444 46428 85.3389 94.875 0.00274807 0.011709 2024-06-27T21:30:03.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:03.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:03.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:03.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:03.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:03.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:03.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:03.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:03 smithi151 ceph-mon[28965]: pgmap v1264: 37 pgs: 37 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:30:04.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:03 smithi190 ceph-mon[32247]: pgmap v1264: 37 pgs: 37 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:30:04.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:04.031 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:04.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:03 smithi192 ceph-mon[32956]: pgmap v1264: 37 pgs: 37 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-06-27T21:30:04.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:04.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:04.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:04.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:04.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:04.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:04.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:04.543 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 16 47780 47764 85.2862 83.5 0.012912 0.0117197 2024-06-27T21:30:04.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:04.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:04.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:04.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:04.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:04.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:04.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:04.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:05.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:05.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:05.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:05.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:05.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:05.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:05.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:05.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:05.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:05.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 49091 49075 85.1929 81.9375 0.0249641 0.0117124 2024-06-27T21:30:05.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:05.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:05.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:05.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:05.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:05.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:05.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:05 smithi151 ceph-mon[28965]: pgmap v1265: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:30:05.945 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:05.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:05 smithi190 ceph-mon[32247]: pgmap v1265: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:30:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:05 smithi192 ceph-mon[32956]: pgmap v1265: 37 pgs: 37 active+clean; 2.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:30:06.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:06.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:06.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:06.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:06.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:06.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:06.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:06.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:06.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 50125 50109 84.6368 64.625 0.00318299 0.0118088 2024-06-27T21:30:06.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:06.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:06.626 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:30:06.627 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:30:06.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:06.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:06.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:06.841 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:06.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:06.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:06 smithi151 ceph-mon[28965]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:30:06.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:06.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:07.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:06 smithi190 ceph-mon[32247]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:30:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:06 smithi192 ceph-mon[32956]: from='client.16389 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:30:07.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:07.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:07.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:07.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:07.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:07.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:07.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:07.471 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2024-06-27T21:30:04.068700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2874016288,"num_objects":43847,"num_object_clones":0,"num_object_copies":131541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43845,"num_write_kb":2806080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45777,"ondisk_log_size":45777,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12385988,"kb_used_data":8200296,"kb_used_omap":110,"kb_used_meta":4185425,"kb_avail":1112555836,"statfs":{"total":1151940427776,"available":1139257176064,"internally_reserved":0,"allocated":8397103104,"data_stored":8392871688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":4285876183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":594018304,"num_objects":9064,"num_object_clones":0,"num_object_copies":27192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9064,"num_write_kb":580096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.358267"},"pg_stats":[{"pgid":"1.14","version":"21'92","reported_seq":34,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.637296+0000","last_change":"2024-06-27T21:29:58.633933+0000","last_active":"2024-06-27T21:29:59.637296+0000","last_peered":"2024-06-27T21:29:59.637296+0000","last_clean":"2024-06-27T21:29:59.637296+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:29:59.637296+0000","last_undegraded":"2024-06-27T21:29:59.637296+0000","last_fullsized":"2024-06-27T21:29:59.637296+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.633807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.633807+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:18.467143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":34,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.645514+0000","last_change":"2024-06-27T21:29:58.085712+0000","last_active":"2024-06-27T21:29:59.645514+0000","last_peered":"2024-06-27T21:29:59.645514+0000","last_clean":"2024-06-27T21:29:59.645514+0000","last_became_active":"2024-06-27T21:29:51.673661+0000","last_became_peered":"2024-06-27T21:29:51.673661+0000","last_unstale":"2024-06-27T21:29:59.645514+0000","last_undegraded":"2024-06-27T21:29:59.645514+0000","last_fullsized":"2024-06-27T21:29:59.645514+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.085606+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.085606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:01.550085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":34,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.638633+0000","last_change":"2024-06-27T21:29:57.430062+0000","last_active":"2024-06-27T21:29:59.638633+0000","last_peered":"2024-06-27T21:29:59.638633+0000","last_clean":"2024-06-27T21:29:59.638633+0000","last_became_active":"2024-06-27T21:29:51.676689+0000","last_became_peered":"2024-06-27T21:29:51.676689+0000","last_unstale":"2024-06-27T21:29:59.638633+0000","last_undegraded":"2024-06-27T21:29:59.638633+0000","last_fullsized":"2024-06-27T21:29:59.638633+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.429829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.429829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:13.901732+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":36,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.636011+0000","last_change":"2024-06-27T21:29:57.074980+0000","last_active":"2024-06-27T21:29:59.636011+0000","last_peered":"2024-06-27T21:29:59.636011+0000","last_clean":"2024-06-27T21:29:59.636011+0000","last_became_active":"2024-06-27T21:29:51.676353+0000","last_became_peered":"2024-06-27T21:29:51.676353+0000","last_unstale":"2024-06-27T21:29:59.636011+0000","last_undegraded":"2024-06-27T21:29:59.636011+0000","last_fullsized":"2024-06-27T21:29:59.636011+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.074844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.074844+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:02.630593+0000","scrub_duration":250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":34,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.638592+0000","last_change":"2024-06-27T21:29:59.388470+0000","last_active":"2024-06-27T21:29:59.638592+0000","last_peered":"2024-06-27T21:29:59.638592+0000","last_clean":"2024-06-27T21:29:59.638592+0000","last_became_active":"2024-06-27T21:29:51.677509+0000","last_became_peered":"2024-06-27T21:29:51.677509+0000","last_unstale":"2024-06-27T21:29:59.638592+0000","last_undegraded":"2024-06-27T21:29:59.638592+0000","last_fullsized":"2024-06-27T21:29:59.638592+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:59.388398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:59.388398+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:05.351358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":779,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.639275+0000","last_change":"2024-06-27T21:29:09.230494+0000","last_active":"2024-06-27T21:29:59.639275+0000","last_peered":"2024-06-27T21:29:59.639275+0000","last_clean":"2024-06-27T21:29:59.639275+0000","last_became_active":"2024-06-27T21:27:16.656216+0000","last_became_peered":"2024-06-27T21:27:16.656216+0000","last_unstale":"2024-06-27T21:29:59.639275+0000","last_undegraded":"2024-06-27T21:29:59.639275+0000","last_fullsized":"2024-06-27T21:29:59.639275+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:09.230395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:09.230395+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:11.942837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"486'2886","reported_seq":2934,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:03.388147+0000","last_change":"2024-06-27T21:29:19.890876+0000","last_active":"2024-06-27T21:30:03.388147+0000","last_peered":"2024-06-27T21:30:03.388147+0000","last_clean":"2024-06-27T21:30:03.388147+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:30:03.388147+0000","last_undegraded":"2024-06-27T21:30:03.388147+0000","last_fullsized":"2024-06-27T21:30:03.388147+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2886,"log_dups_size":0,"ondisk_log_size":2886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:47.893415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189136896,"num_objects":2886,"num_object_clones":0,"num_object_copies":8658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":184704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":109,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:58.720278+0000","last_change":"2024-06-27T21:29:10.827592+0000","last_active":"2024-06-27T21:29:58.720278+0000","last_peered":"2024-06-27T21:29:58.720278+0000","last_clean":"2024-06-27T21:29:58.720278+0000","last_became_active":"2024-06-27T21:27:38.938524+0000","last_became_peered":"2024-06-27T21:27:38.938524+0000","last_unstale":"2024-06-27T21:29:58.720278+0000","last_undegraded":"2024-06-27T21:29:58.720278+0000","last_fullsized":"2024-06-27T21:29:58.720278+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:10.827466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:10.827466+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:23.948700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"486'2912","reported_seq":2960,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:03.398690+0000","last_change":"2024-06-27T21:29:19.821974+0000","last_active":"2024-06-27T21:30:03.398690+0000","last_peered":"2024-06-27T21:30:03.398690+0000","last_clean":"2024-06-27T21:30:03.398690+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:30:03.398690+0000","last_undegraded":"2024-06-27T21:30:03.398690+0000","last_fullsized":"2024-06-27T21:30:03.398690+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2912,"log_dups_size":0,"ondisk_log_size":2912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:36.243439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190840832,"num_objects":2912,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":186368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":736,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.641922+0000","last_change":"2024-06-27T21:29:12.778561+0000","last_active":"2024-06-27T21:29:59.641922+0000","last_peered":"2024-06-27T21:29:59.641922+0000","last_clean":"2024-06-27T21:29:59.641922+0000","last_became_active":"2024-06-27T21:23:44.529707+0000","last_became_peered":"2024-06-27T21:23:44.529707+0000","last_unstale":"2024-06-27T21:29:59.641922+0000","last_undegraded":"2024-06-27T21:29:59.641922+0000","last_fullsized":"2024-06-27T21:29:59.641922+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:12.778462+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:29:12.778462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:14.221539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"486'2772","reported_seq":2819,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:02.891577+0000","last_change":"2024-06-27T21:29:19.862967+0000","last_active":"2024-06-27T21:30:02.891577+0000","last_peered":"2024-06-27T21:30:02.891577+0000","last_clean":"2024-06-27T21:30:02.891577+0000","last_became_active":"2024-06-27T21:29:19.862712+0000","last_became_peered":"2024-06-27T21:29:19.862712+0000","last_unstale":"2024-06-27T21:30:02.891577+0000","last_undegraded":"2024-06-27T21:30:02.891577+0000","last_fullsized":"2024-06-27T21:30:02.891577+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2772,"log_dups_size":0,"ondisk_log_size":2772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:44.087500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181665792,"num_objects":2772,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":177408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":878,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.645521+0000","last_change":"2024-06-27T21:29:57.117837+0000","last_active":"2024-06-27T21:29:59.645521+0000","last_peered":"2024-06-27T21:29:59.645521+0000","last_clean":"2024-06-27T21:29:59.645521+0000","last_became_active":"2024-06-27T21:29:49.668296+0000","last_became_peered":"2024-06-27T21:29:49.668296+0000","last_unstale":"2024-06-27T21:29:59.645521+0000","last_undegraded":"2024-06-27T21:29:59.645521+0000","last_fullsized":"2024-06-27T21:29:59.645521+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.117729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.117729+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:02.448597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"485'2629","reported_seq":2675,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.505166+0000","last_change":"2024-06-27T21:29:19.921504+0000","last_active":"2024-06-27T21:29:59.505166+0000","last_peered":"2024-06-27T21:29:59.505166+0000","last_clean":"2024-06-27T21:29:59.505166+0000","last_became_active":"2024-06-27T21:29:19.921150+0000","last_became_peered":"2024-06-27T21:29:19.921150+0000","last_unstale":"2024-06-27T21:29:59.505166+0000","last_undegraded":"2024-06-27T21:29:59.505166+0000","last_fullsized":"2024-06-27T21:29:59.505166+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:38.653405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172294144,"num_objects":2629,"num_object_clones":0,"num_object_copies":7887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":168256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":442,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.641836+0000","last_change":"2024-06-27T21:29:56.965958+0000","last_active":"2024-06-27T21:29:59.641836+0000","last_peered":"2024-06-27T21:29:59.641836+0000","last_clean":"2024-06-27T21:29:59.641836+0000","last_became_active":"2024-06-27T21:29:49.668842+0000","last_became_peered":"2024-06-27T21:29:49.668842+0000","last_unstale":"2024-06-27T21:29:59.641836+0000","last_undegraded":"2024-06-27T21:29:59.641836+0000","last_fullsized":"2024-06-27T21:29:59.641836+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.965862+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.965862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:19.446720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"486'2823","reported_seq":2871,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:03.409422+0000","last_change":"2024-06-27T21:29:19.921839+0000","last_active":"2024-06-27T21:30:03.409422+0000","last_peered":"2024-06-27T21:30:03.409422+0000","last_clean":"2024-06-27T21:30:03.409422+0000","last_became_active":"2024-06-27T21:29:19.921089+0000","last_became_peered":"2024-06-27T21:29:19.921089+0000","last_unstale":"2024-06-27T21:30:03.409422+0000","last_undegraded":"2024-06-27T21:30:03.409422+0000","last_fullsized":"2024-06-27T21:30:03.409422+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2823,"log_dups_size":0,"ondisk_log_size":2823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:33.415477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":180672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":446,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.641908+0000","last_change":"2024-06-27T21:29:57.941498+0000","last_active":"2024-06-27T21:29:59.641908+0000","last_peered":"2024-06-27T21:29:59.641908+0000","last_clean":"2024-06-27T21:29:59.641908+0000","last_became_active":"2024-06-27T21:29:49.667265+0000","last_became_peered":"2024-06-27T21:29:49.667265+0000","last_unstale":"2024-06-27T21:29:59.641908+0000","last_undegraded":"2024-06-27T21:29:59.641908+0000","last_fullsized":"2024-06-27T21:29:59.641908+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.941423+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.941423+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:25.602914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"486'2660","reported_seq":2707,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:02.382621+0000","last_change":"2024-06-27T21:29:19.890411+0000","last_active":"2024-06-27T21:30:02.382621+0000","last_peered":"2024-06-27T21:30:02.382621+0000","last_clean":"2024-06-27T21:30:02.382621+0000","last_became_active":"2024-06-27T21:29:19.889910+0000","last_became_peered":"2024-06-27T21:29:19.889910+0000","last_unstale":"2024-06-27T21:30:02.382621+0000","last_undegraded":"2024-06-27T21:30:02.382621+0000","last_fullsized":"2024-06-27T21:30:02.382621+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2660,"log_dups_size":0,"ondisk_log_size":2660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:32.566921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174325760,"num_objects":2660,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2660,"num_write_kb":170240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":789,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.635523+0000","last_change":"2024-06-27T21:29:58.118216+0000","last_active":"2024-06-27T21:29:59.635523+0000","last_peered":"2024-06-27T21:29:59.635523+0000","last_clean":"2024-06-27T21:29:59.635523+0000","last_became_active":"2024-06-27T21:29:49.668973+0000","last_became_peered":"2024-06-27T21:29:49.668973+0000","last_unstale":"2024-06-27T21:29:59.635523+0000","last_undegraded":"2024-06-27T21:29:59.635523+0000","last_fullsized":"2024-06-27T21:29:59.635523+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.118141+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.118141+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:24.436068+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"486'2738","reported_seq":2786,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:01.195420+0000","last_change":"2024-06-27T21:29:19.819739+0000","last_active":"2024-06-27T21:30:01.195420+0000","last_peered":"2024-06-27T21:30:01.195420+0000","last_clean":"2024-06-27T21:30:01.195420+0000","last_became_active":"2024-06-27T21:29:19.819457+0000","last_became_peered":"2024-06-27T21:29:19.819457+0000","last_unstale":"2024-06-27T21:30:01.195420+0000","last_undegraded":"2024-06-27T21:30:01.195420+0000","last_fullsized":"2024-06-27T21:30:01.195420+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2738,"log_dups_size":0,"ondisk_log_size":2738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:22.559387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179437568,"num_objects":2738,"num_object_clones":0,"num_object_copies":8214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":175232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":449,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.642297+0000","last_change":"2024-06-27T21:29:58.953762+0000","last_active":"2024-06-27T21:29:59.642297+0000","last_peered":"2024-06-27T21:29:59.642297+0000","last_clean":"2024-06-27T21:29:59.642297+0000","last_became_active":"2024-06-27T21:29:49.667338+0000","last_became_peered":"2024-06-27T21:29:49.667338+0000","last_unstale":"2024-06-27T21:29:59.642297+0000","last_undegraded":"2024-06-27T21:29:59.642297+0000","last_fullsized":"2024-06-27T21:29:59.642297+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.953658+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:10.467277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"486'2860","reported_seq":2908,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:03.038285+0000","last_change":"2024-06-27T21:29:19.821949+0000","last_active":"2024-06-27T21:30:03.038285+0000","last_peered":"2024-06-27T21:30:03.038285+0000","last_clean":"2024-06-27T21:30:03.038285+0000","last_became_active":"2024-06-27T21:29:19.821580+0000","last_became_peered":"2024-06-27T21:29:19.821580+0000","last_unstale":"2024-06-27T21:30:03.038285+0000","last_undegraded":"2024-06-27T21:30:03.038285+0000","last_fullsized":"2024-06-27T21:30:03.038285+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2860,"log_dups_size":0,"ondisk_log_size":2860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.451785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187432960,"num_objects":2860,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":183040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":128,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:58.720407+0000","last_change":"2024-06-27T21:29:58.017742+0000","last_active":"2024-06-27T21:29:58.720407+0000","last_peered":"2024-06-27T21:29:58.720407+0000","last_clean":"2024-06-27T21:29:58.720407+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:29:58.720407+0000","last_undegraded":"2024-06-27T21:29:58.720407+0000","last_fullsized":"2024-06-27T21:29:58.720407+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.017623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.017623+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:06.149333+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"485'2590","reported_seq":2636,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.530473+0000","last_change":"2024-06-27T21:29:19.889671+0000","last_active":"2024-06-27T21:29:59.530473+0000","last_peered":"2024-06-27T21:29:59.530473+0000","last_clean":"2024-06-27T21:29:59.530473+0000","last_became_active":"2024-06-27T21:29:19.889320+0000","last_became_peered":"2024-06-27T21:29:19.889320+0000","last_unstale":"2024-06-27T21:29:59.530473+0000","last_undegraded":"2024-06-27T21:29:59.530473+0000","last_fullsized":"2024-06-27T21:29:59.530473+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.902496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":633,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.635407+0000","last_change":"2024-06-27T21:29:57.087735+0000","last_active":"2024-06-27T21:29:59.635407+0000","last_peered":"2024-06-27T21:29:59.635407+0000","last_clean":"2024-06-27T21:29:59.635407+0000","last_became_active":"2024-06-27T21:29:49.669651+0000","last_became_peered":"2024-06-27T21:29:49.669651+0000","last_unstale":"2024-06-27T21:29:59.635407+0000","last_undegraded":"2024-06-27T21:29:59.635407+0000","last_fullsized":"2024-06-27T21:29:59.635407+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.087617+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.087617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:00.566901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"485'2653","reported_seq":2698,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.090320+0000","last_change":"2024-06-27T21:29:19.889785+0000","last_active":"2024-06-27T21:29:59.090320+0000","last_peered":"2024-06-27T21:29:59.090320+0000","last_clean":"2024-06-27T21:29:59.090320+0000","last_became_active":"2024-06-27T21:29:19.889439+0000","last_became_peered":"2024-06-27T21:29:19.889439+0000","last_unstale":"2024-06-27T21:29:59.090320+0000","last_undegraded":"2024-06-27T21:29:59.090320+0000","last_fullsized":"2024-06-27T21:29:59.090320+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2653,"log_dups_size":0,"ondisk_log_size":2653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:36.151289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2653,"num_write_kb":169792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":133,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:58.720376+0000","last_change":"2024-06-27T21:29:56.956056+0000","last_active":"2024-06-27T21:29:58.720376+0000","last_peered":"2024-06-27T21:29:58.720376+0000","last_clean":"2024-06-27T21:29:58.720376+0000","last_became_active":"2024-06-27T21:29:49.671863+0000","last_became_peered":"2024-06-27T21:29:49.671863+0000","last_unstale":"2024-06-27T21:29:58.720376+0000","last_undegraded":"2024-06-27T21:29:58.720376+0000","last_fullsized":"2024-06-27T21:29:58.720376+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.955983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.955983+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:18.807666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"486'2794","reported_seq":2842,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:02.890941+0000","last_change":"2024-06-27T21:29:19.822891+0000","last_active":"2024-06-27T21:30:02.890941+0000","last_peered":"2024-06-27T21:30:02.890941+0000","last_clean":"2024-06-27T21:30:02.890941+0000","last_became_active":"2024-06-27T21:29:19.822526+0000","last_became_peered":"2024-06-27T21:29:19.822526+0000","last_unstale":"2024-06-27T21:30:02.890941+0000","last_undegraded":"2024-06-27T21:30:02.890941+0000","last_fullsized":"2024-06-27T21:30:02.890941+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2794,"log_dups_size":0,"ondisk_log_size":2794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:46.475678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183107584,"num_objects":2794,"num_object_clones":0,"num_object_copies":8382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2794,"num_write_kb":178816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":791,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.635402+0000","last_change":"2024-06-27T21:29:59.071396+0000","last_active":"2024-06-27T21:29:59.635402+0000","last_peered":"2024-06-27T21:29:59.635402+0000","last_clean":"2024-06-27T21:29:59.635402+0000","last_became_active":"2024-06-27T21:29:51.674576+0000","last_became_peered":"2024-06-27T21:29:51.674576+0000","last_unstale":"2024-06-27T21:29:59.635402+0000","last_undegraded":"2024-06-27T21:29:59.635402+0000","last_fullsized":"2024-06-27T21:29:59.635402+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:59.071261+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:29:59.071261+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:22.073181+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"485'2594","reported_seq":2640,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.502367+0000","last_change":"2024-06-27T21:29:19.921495+0000","last_active":"2024-06-27T21:29:59.502367+0000","last_peered":"2024-06-27T21:29:59.502367+0000","last_clean":"2024-06-27T21:29:59.502367+0000","last_became_active":"2024-06-27T21:29:19.921198+0000","last_became_peered":"2024-06-27T21:29:19.921198+0000","last_unstale":"2024-06-27T21:29:59.502367+0000","last_undegraded":"2024-06-27T21:29:59.502367+0000","last_fullsized":"2024-06-27T21:29:59.502367+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2594,"log_dups_size":0,"ondisk_log_size":2594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:39.282908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170000384,"num_objects":2594,"num_object_clones":0,"num_object_copies":7782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2594,"num_write_kb":166016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":34,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.633521+0000","last_change":"2024-06-27T21:29:57.071471+0000","last_active":"2024-06-27T21:29:59.633521+0000","last_peered":"2024-06-27T21:29:59.633521+0000","last_clean":"2024-06-27T21:29:59.633521+0000","last_became_active":"2024-06-27T21:29:51.668654+0000","last_became_peered":"2024-06-27T21:29:51.668654+0000","last_unstale":"2024-06-27T21:29:59.633521+0000","last_undegraded":"2024-06-27T21:29:59.633521+0000","last_fullsized":"2024-06-27T21:29:59.633521+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.071395+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.071395+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:05.563427+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"486'2606","reported_seq":2654,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:01.201100+0000","last_change":"2024-06-27T21:29:19.890005+0000","last_active":"2024-06-27T21:30:01.201100+0000","last_peered":"2024-06-27T21:30:01.201100+0000","last_clean":"2024-06-27T21:30:01.201100+0000","last_became_active":"2024-06-27T21:29:19.889662+0000","last_became_peered":"2024-06-27T21:29:19.889662+0000","last_unstale":"2024-06-27T21:30:01.201100+0000","last_undegraded":"2024-06-27T21:30:01.201100+0000","last_fullsized":"2024-06-27T21:30:01.201100+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":0,"ondisk_log_size":2606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:48.397869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170786816,"num_objects":2606,"num_object_clones":0,"num_object_copies":7818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2606,"num_write_kb":166784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":32,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-06-27T21:29:58.941195+0000","last_change":"2024-06-27T21:29:58.941232+0000","last_active":"2024-06-27T21:29:58.941195+0000","last_peered":"2024-06-27T21:29:58.941195+0000","last_clean":"2024-06-27T21:29:58.941195+0000","last_became_active":"2024-06-27T21:29:51.672132+0000","last_became_peered":"2024-06-27T21:29:51.672132+0000","last_unstale":"2024-06-27T21:29:58.941195+0000","last_undegraded":"2024-06-27T21:29:58.941195+0000","last_fullsized":"2024-06-27T21:29:58.941195+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.941114+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.941114+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:09.275458+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"486'2798","reported_seq":2846,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:02.888359+0000","last_change":"2024-06-27T21:29:19.820432+0000","last_active":"2024-06-27T21:30:02.888359+0000","last_peered":"2024-06-27T21:30:02.888359+0000","last_clean":"2024-06-27T21:30:02.888359+0000","last_became_active":"2024-06-27T21:29:19.820126+0000","last_became_peered":"2024-06-27T21:29:19.820126+0000","last_unstale":"2024-06-27T21:30:02.888359+0000","last_undegraded":"2024-06-27T21:30:02.888359+0000","last_fullsized":"2024-06-27T21:30:02.888359+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2798,"log_dups_size":0,"ondisk_log_size":2798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.126312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":179072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"486'2779","reported_seq":2827,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:02.449698+0000","last_change":"2024-06-27T21:29:19.820104+0000","last_active":"2024-06-27T21:30:02.449698+0000","last_peered":"2024-06-27T21:30:02.449698+0000","last_clean":"2024-06-27T21:30:02.449698+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:30:02.449698+0000","last_undegraded":"2024-06-27T21:30:02.449698+0000","last_fullsized":"2024-06-27T21:30:02.449698+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2779,"log_dups_size":0,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:36.282749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182124544,"num_objects":2779,"num_object_clones":0,"num_object_copies":8337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":177856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":41,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.636147+0000","last_change":"2024-06-27T21:29:58.754806+0000","last_active":"2024-06-27T21:29:59.636147+0000","last_peered":"2024-06-27T21:29:59.636147+0000","last_clean":"2024-06-27T21:29:59.636147+0000","last_became_active":"2024-06-27T21:29:51.676349+0000","last_became_peered":"2024-06-27T21:29:51.676349+0000","last_unstale":"2024-06-27T21:29:59.636147+0000","last_undegraded":"2024-06-27T21:29:59.636147+0000","last_fullsized":"2024-06-27T21:29:59.636147+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.754689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.754689+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:20.863142+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"486'2751","reported_seq":2799,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:30:00.682772+0000","last_change":"2024-06-27T21:29:19.819646+0000","last_active":"2024-06-27T21:30:00.682772+0000","last_peered":"2024-06-27T21:30:00.682772+0000","last_clean":"2024-06-27T21:30:00.682772+0000","last_became_active":"2024-06-27T21:29:19.819315+0000","last_became_peered":"2024-06-27T21:29:19.819315+0000","last_unstale":"2024-06-27T21:30:00.682772+0000","last_undegraded":"2024-06-27T21:30:00.682772+0000","last_fullsized":"2024-06-27T21:30:00.682772+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":2751,"log_dups_size":0,"ondisk_log_size":2751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:30.608875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180289536,"num_objects":2751,"num_object_clones":0,"num_object_copies":8253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":176064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,8],"acting":[6,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":34,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-06-27T21:29:59.645519+0000","last_change":"2024-06-27T21:29:59.044100+0000","last_active":"2024-06-27T21:29:59.645519+0000","last_peered":"2024-06-27T21:29:59.645519+0000","last_clean":"2024-06-27T21:29:59.645519+0000","last_became_active":"2024-06-27T21:29:51.675328+0000","last_became_peered":"2024-06-27T21:29:51.675328+0000","last_unstale":"2024-06-27T21:29:59.645519+0000","last_undegraded":"2024-06-27T21:29:59.645519+0000","last_fullsized":"2024-06-27T21:29:59.645519+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:59.044030+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:29:59.044030+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:10.321618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2873425920,"num_objects":43845,"num_object_clones":0,"num_object_copies":131535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43845,"num_write_kb":2806080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8500281344,"data_stored":8500281344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43845,"ondisk_log_size":43845,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253113,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950032,"kb_used_data":489860,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92795120,"statfs":{"total":95995035648,"available":95022202880,"internally_reserved":0,"allocated":501616640,"data_stored":501267942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514749,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837268,"kb_used_data":472900,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92907884,"statfs":{"total":95995035648,"available":95137673216,"internally_reserved":0,"allocated":484249600,"data_stored":483900902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":373094848},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743682,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1224352,"kb_used_data":837776,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":92520800,"statfs":{"total":95995035648,"available":94741299200,"internally_reserved":0,"allocated":857882624,"data_stored":857529378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":395835840},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972614,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733732,"kb_used_data":517912,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93011420,"statfs":{"total":95995035648,"available":95243694080,"internally_reserved":0,"allocated":530341888,"data_stored":529985090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266954,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1225560,"kb_used_data":819524,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92519592,"statfs":{"total":95995035648,"available":94740062208,"internally_reserved":0,"allocated":839192576,"data_stored":838843878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528592,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1303068,"kb_used_data":842896,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92442084,"statfs":{"total":95995035648,"available":94660694016,"internally_reserved":0,"allocated":863125504,"data_stored":862772258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757526,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1401760,"kb_used_data":1077656,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92343392,"statfs":{"total":95995035648,"available":94559633408,"internally_reserved":0,"allocated":1103519744,"data_stored":1103162946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161229,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912024,"kb_used_data":726096,"kb_used_omap":92,"kb_used_meta":185827,"kb_avail":92833128,"statfs":{"total":95995035648,"available":95061123072,"internally_reserved":0,"allocated":743522304,"data_stored":743169058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":190286946},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182686,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870556,"kb_used_data":532560,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92874596,"statfs":{"total":95995035648,"available":95103586304,"internally_reserved":0,"allocated":545341440,"data_stored":544988194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":346094023},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411619,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940896,"kb_used_data":720920,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92804256,"statfs":{"total":95995035648,"available":95031558144,"internally_reserved":0,"allocated":738222080,"data_stored":737865282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575143,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077336,"kb_used_data":653252,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92667816,"statfs":{"total":95995035648,"available":94891843584,"internally_reserved":0,"allocated":668930048,"data_stored":668581350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909404,"kb_used_data":508944,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92835748,"statfs":{"total":95995035648,"available":95063805952,"internally_reserved":0,"allocated":521158656,"data_stored":520805410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524746752,"data_stored":524746752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":672727040,"data_stored":672727040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":757596160,"data_stored":757596160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":549584896,"data_stored":549584896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":743768064,"data_stored":743768064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1120862208,"data_stored":1120862208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":885391360,"data_stored":885391360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":849870848,"data_stored":849870848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":539557888,"data_stored":539557888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":857407488,"data_stored":857407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":491520000,"data_stored":491520000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":507248640,"data_stored":507248640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:30:07.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-06-27T21:30:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:07.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:07.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 51278 51262 84.3057 72.0625 0.0691666 0.0118032 2024-06-27T21:30:07.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:07.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:07.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:07.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:07.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:07.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:07.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:07.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:07.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:07 smithi190 ceph-mon[32247]: pgmap v1266: 37 pgs: 37 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:30:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:07 smithi192 ceph-mon[32956]: pgmap v1266: 37 pgs: 37 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:30:08.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:07 smithi151 ceph-mon[28965]: pgmap v1266: 37 pgs: 37 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:30:08.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:08.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:08.195 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:30:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:30:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:30:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:30:08.195 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:30:08.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:08.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:08.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:08.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:08.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:08.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:08.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 15 51960 51945 83.2384 42.6875 0.00303815 0.0120089 2024-06-27T21:30:08.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:08.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:08.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:08.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:08.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:08.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:08.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:08.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:08.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:09.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:09.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:09.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:09.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:09.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:09.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:09.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:09.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:09.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:30:09.544875+0000 min lat: 0.00108868 max lat: 0.636781 avg lat: 0.0120159 2024-06-27T21:30:09.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:30:09.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 53260 53244 83.187 81.1875 0.0459477 0.0120159 2024-06-27T21:30:09.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:09.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:09.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:09.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:09.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:09.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:09.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:09.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:10.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:09 smithi190 ceph-mon[32247]: pgmap v1267: 37 pgs: 37 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-06-27T21:30:10.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:09 smithi192 ceph-mon[32956]: pgmap v1267: 37 pgs: 37 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-06-27T21:30:10.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:10.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:09 smithi151 ceph-mon[28965]: pgmap v1267: 37 pgs: 37 active+clean; 3.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-06-27T21:30:10.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:10.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:10.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:10.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:30:10.330 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:10.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:10.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:10.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:10.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:10.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 16 54490 54474 83.0329 76.875 0.012357 0.0120332 2024-06-27T21:30:10.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:10.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:10.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:10.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:10.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:10.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:10.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:10.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:11.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:11.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:11.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:11.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:11.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:11.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:11.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:11.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:11.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 55832 55816 83.0528 83.875 0.0149869 0.0120344 2024-06-27T21:30:11.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:11.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:11.732 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:30:11.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:11.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:11.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:11.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:11.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:12.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:11 smithi190 ceph-mon[32247]: pgmap v1268: 37 pgs: 37 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:30:12.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3084713924' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:12.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:11 smithi192 ceph-mon[32956]: pgmap v1268: 37 pgs: 37 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:30:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3084713924' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:12.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:12.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:12.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:11 smithi151 ceph-mon[28965]: pgmap v1268: 37 pgs: 37 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:30:12.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3084713924' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:12.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:12.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:12.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:12.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:12.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:12.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:12.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:12.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 57365 57349 83.3493 95.8125 0.00345527 0.0119872 2024-06-27T21:30:12.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:12.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:12.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:12.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:12.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:12.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:12.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:12.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:12.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:12 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:30:12.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:12 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:30:12.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3084713924' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:12.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:12 smithi151 ceph-mon[28965]: osdmap e487: 12 total, 12 up, 12 in 2024-06-27T21:30:13.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:12 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:30:13.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:12 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:30:13.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3084713924' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:13.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:12 smithi190 ceph-mon[32247]: osdmap e487: 12 total, 12 up, 12 in 2024-06-27T21:30:13.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:12 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:30:13.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:12 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:30:13.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3084713924' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:13.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:12 smithi192 ceph-mon[32956]: osdmap e487: 12 total, 12 up, 12 in 2024-06-27T21:30:13.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:13.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:13.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:13.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:13.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:13.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:13.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:13.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:13.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:13.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 58421 58405 82.9549 66 0.00254265 0.012042 2024-06-27T21:30:13.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:13.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:13.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:13.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:13.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:13.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:13.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:13.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:13.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:14.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:13 smithi190 ceph-mon[32247]: pgmap v1270: 37 pgs: 37 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:30:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:13 smithi192 ceph-mon[32956]: pgmap v1270: 37 pgs: 37 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:30:14.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:14.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:14.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:13 smithi151 ceph-mon[28965]: pgmap v1270: 37 pgs: 37 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:30:14.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:14.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:14.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:14.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:14.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:14.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:14.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 59177 59161 82.1614 47.25 0.0412744 0.012165 2024-06-27T21:30:14.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:14.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:14.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:14.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:14.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:14.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:14.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:14.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:14.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:15.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:14 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:30:15.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:14 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:30:15.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:14 smithi190 ceph-mon[32247]: osdmap e488: 12 total, 12 up, 12 in 2024-06-27T21:30:15.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:14 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:30:15.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:14 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:30:15.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:14 smithi192 ceph-mon[32956]: osdmap e488: 12 total, 12 up, 12 in 2024-06-27T21:30:15.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:15.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:15.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:14 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:30:15.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:14 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:30:15.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:14 smithi151 ceph-mon[28965]: osdmap e488: 12 total, 12 up, 12 in 2024-06-27T21:30:15.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:15.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:15.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:15.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:15.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:15.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:15.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:30:15.544 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 60481 60465 82.1469 81.5 0.0285431 0.0121676 2024-06-27T21:30:15.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:15.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:15.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:15.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:15.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:15.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:15 smithi151 ceph-mon[28965]: pgmap v1271: 37 pgs: 37 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:15.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:15 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:30:15.821 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:15 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:30:15.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:15.879 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:15.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:15.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:15 smithi190 ceph-mon[32247]: pgmap v1271: 37 pgs: 37 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:15 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:30:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:15 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:30:16.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:15 smithi192 ceph-mon[32956]: pgmap v1271: 37 pgs: 37 active+clean; 3.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:16.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:15 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:30:16.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:15 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:30:16.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:16.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:16.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:16.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:16.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:16.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:16.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:30:16.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:30:16.435 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:30:16.435 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:30:16.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:30:16.462 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:30:16.492 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:30:16.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:30:16.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:16.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:16.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:30:16.521 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:30:16.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 61651 61635 81.9547 73.125 0.00561627 0.0121947 2024-06-27T21:30:16.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:30:16.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:30:16.572 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:30:16.573 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:30:16.600 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:30:16.601 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:30:16.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:16.630 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:30:16.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:30:16.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:16.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:30:16.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:30:16.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:30:16.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:30:16.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:30:16.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:30:16.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:16.764 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 6 pgp_num to 16 2024-06-27T21:30:16.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:16.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:16.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:16.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:16.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1786534501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:16.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3657624300' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:16.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:16 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:17.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1786534501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3657624300' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:17.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:16 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:17.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1786534501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3657624300' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:17.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:16 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:17.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:17.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:17.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:17.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:17.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:17.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:17.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:17.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:17.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 62835 62819 81.7888 74 0.00313951 0.0122209 2024-06-27T21:30:17.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:17.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:17.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:17.791 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:17.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:17.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:17.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:17 smithi151 ceph-mon[28965]: pgmap v1273: 37 pgs: 37 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:30:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:17 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:30:17.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:17 smithi151 ceph-mon[28965]: osdmap e489: 12 total, 12 up, 12 in 2024-06-27T21:30:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:17 smithi190 ceph-mon[32247]: pgmap v1273: 37 pgs: 37 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:30:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:17 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:30:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:17 smithi190 ceph-mon[32247]: osdmap e489: 12 total, 12 up, 12 in 2024-06-27T21:30:18.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:17 smithi192 ceph-mon[32956]: pgmap v1273: 37 pgs: 37 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:30:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:17 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:30:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:17 smithi192 ceph-mon[32956]: osdmap e489: 12 total, 12 up, 12 in 2024-06-27T21:30:18.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:18.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:18.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:18.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:18.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:18.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:18.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:18.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:18.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:18.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 15 64233 64218 81.904 87.4375 0.03964 0.0121994 2024-06-27T21:30:18.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:18.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:18.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:18.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:18.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:18.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:18.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:18.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:19.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:19.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:19.093 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:19.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:19.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:19.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:19.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:19.370 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:19.370 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:19.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:19.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:19.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 65265 65249 81.5545 64.4375 0.0119229 0.0122525 2024-06-27T21:30:19.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:19.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:19.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:19.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:19.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:19.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:19.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:19 smithi151 ceph-mon[28965]: pgmap v1275: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:30:19.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:19 smithi151 ceph-mon[28965]: osdmap e490: 12 total, 12 up, 12 in 2024-06-27T21:30:19.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:19.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:19.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:19 smithi192 ceph-mon[32956]: pgmap v1275: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:30:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:19 smithi192 ceph-mon[32956]: osdmap e490: 12 total, 12 up, 12 in 2024-06-27T21:30:20.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:20.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:20.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:20.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:20.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:30:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:19 smithi190 ceph-mon[32247]: pgmap v1275: 37 pgs: 37 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:30:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:19 smithi190 ceph-mon[32247]: osdmap e490: 12 total, 12 up, 12 in 2024-06-27T21:30:20.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:20.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:20.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:20.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:20.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 66447 66431 81.4038 73.875 0.00415039 0.0122792 2024-06-27T21:30:20.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:20.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:20.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:20.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:20.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:20.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:20.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:20.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:20.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:21.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:21.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:21.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:21.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:21.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:21.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:21.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:21.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:21.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 67575 67559 81.194 70.5 0.00351257 0.0123114 2024-06-27T21:30:21.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:21.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:21.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:21.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:21.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:21.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:21.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:21.908 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:21 smithi151 ceph-mon[28965]: pgmap v1277: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:21.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:30:22.004 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:22.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:22.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:22.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:21 smithi192 ceph-mon[32956]: pgmap v1277: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:22.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:22.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:21 smithi190 ceph-mon[32247]: pgmap v1277: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:22.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:22.305 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:22.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:22.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:22.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:22.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 68633 68617 80.9096 66.125 0.00849114 0.0123527 2024-06-27T21:30:22.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:22.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:22.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:22.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:22.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:22.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:22.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:22.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:23.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:23.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:23.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:23.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:23.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:23.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:23.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:23.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:23.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:23.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:23.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 69776 69760 80.7341 71.4375 0.00670544 0.0123805 2024-06-27T21:30:23.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:23.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:23.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:23.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:23.776 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:30:23.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:23 smithi151 ceph-mon[28965]: pgmap v1278: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 934 op/s 2024-06-27T21:30:23.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4052726877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:30:23.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:23.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:23.839 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:30:23.839 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:30:23.839 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-27T21:30:23.839 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.089000 2024-06-27T21:30:23.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 6 0.08899959744411123 2024-06-27T21:30:23.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:23.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:23.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:24.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:23 smithi192 ceph-mon[32956]: pgmap v1278: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 934 op/s 2024-06-27T21:30:24.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4052726877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:30:24.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:24.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:23 smithi190 ceph-mon[32247]: pgmap v1278: 37 pgs: 37 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 934 op/s 2024-06-27T21:30:24.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4052726877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:30:24.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:24.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:24.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:24.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:24.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:24.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:24.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:24.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 70793 70777 80.4218 63.5625 0.0040053 0.0124276 2024-06-27T21:30:24.554 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:24.554 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:24.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:24.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:24.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:24.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:24.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:24.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:24.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:24.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:25.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:24 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4052726877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:30:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:24 smithi192 ceph-mon[32956]: osdmap e491: 12 total, 12 up, 12 in 2024-06-27T21:30:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:24 smithi192 ceph-mon[32956]: osdmap e492: 12 total, 12 up, 12 in 2024-06-27T21:30:25.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:25.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:24 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:25.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4052726877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:30:25.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:24 smithi151 ceph-mon[28965]: osdmap e491: 12 total, 12 up, 12 in 2024-06-27T21:30:25.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:24 smithi151 ceph-mon[28965]: osdmap e492: 12 total, 12 up, 12 in 2024-06-27T21:30:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:25.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:25.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:24 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:25.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4052726877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:30:25.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:24 smithi190 ceph-mon[32247]: osdmap e491: 12 total, 12 up, 12 in 2024-06-27T21:30:25.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:24 smithi190 ceph-mon[32247]: osdmap e492: 12 total, 12 up, 12 in 2024-06-27T21:30:25.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:25.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:30:25.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:25.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:25.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:30:25.426 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:30:25.431 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:25.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:30:25.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:30:25.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:25.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:25.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 72052 72036 80.3907 78.6875 0.0204098 0.0124326 2024-06-27T21:30:25.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:25.665 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:25.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:25.674 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:30:25.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:30:25.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:30:25.701 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:30:25.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:30:25.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:30:25.754 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:30:25.754 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:30:25.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:25.781 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:30:25.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:30:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:25.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:25.810 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:30:25.810 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:30:25.841 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:30:25.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:30:25.875 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:30:25.875 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:30:25.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:25.902 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:30:25.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:30:25.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:25.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:25.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:30:25.933 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:30:25.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:25 smithi151 ceph-mon[28965]: pgmap v1280: 37 pgs: 37 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:25.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3941166905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:26.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:26.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:25 smithi192 ceph-mon[32956]: pgmap v1280: 37 pgs: 37 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:26.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3941166905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:26.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:26.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:26.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:25 smithi190 ceph-mon[32247]: pgmap v1280: 37 pgs: 37 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3941166905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:26.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:26.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:26.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:26.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:26.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:26.545 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 73105 73089 80.1349 65.8125 0.028974 0.0124738 2024-06-27T21:30:26.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:26.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:26.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:26.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:26.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:26.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:30:26.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:26.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:26.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:27.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:27.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:27.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:27.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:27.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:27.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:27.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:27.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:27.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 74170 74154 79.9008 66.5625 0.00753946 0.0125097 2024-06-27T21:30:27.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:27.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:27.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:27.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:27.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:27 smithi190 ceph-mon[32247]: pgmap v1282: 37 pgs: 37 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-06-27T21:30:27.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:27.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:27.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:27.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:27 smithi151 ceph-mon[28965]: pgmap v1282: 37 pgs: 37 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-06-27T21:30:27.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:27.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:28.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:27 smithi192 ceph-mon[32956]: pgmap v1282: 37 pgs: 37 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-06-27T21:30:28.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:28.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:28.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:28.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:28.254 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:30:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:30:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:30:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:30:28.255 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:30:28.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:28.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:28.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:28.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:28.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 75456 75440 79.9087 80.375 0.0073297 0.01251 2024-06-27T21:30:28.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:28.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:28.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:28.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:28.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:28.880 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:28.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:28.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:28.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:29.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:29.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:29.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:29.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:29.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:29.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:29.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:29.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:29.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:30:29.546535+0000 min lat: 0.00108868 max lat: 0.636781 avg lat: 0.0125518 2024-06-27T21:30:29.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:30:29.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 76469 76453 79.632 63.3125 0.00575862 0.0125518 2024-06-27T21:30:29.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:29.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:29.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:29.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:29.790 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.6 primary-affinity to 0.0889996 (858322) 2024-06-27T21:30:29.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:29.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:29.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:29.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:29 smithi151 ceph-mon[28965]: pgmap v1283: 37 pgs: 37 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:30:29.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1639996360' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]: dispatch 2024-06-27T21:30:29.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:29 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]: dispatch 2024-06-27T21:30:30.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:30.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:29 smithi192 ceph-mon[32956]: pgmap v1283: 37 pgs: 37 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:30:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1639996360' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]: dispatch 2024-06-27T21:30:30.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:29 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]: dispatch 2024-06-27T21:30:30.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:30.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:29 smithi190 ceph-mon[32247]: pgmap v1283: 37 pgs: 37 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:30:30.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1639996360' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]: dispatch 2024-06-27T21:30:30.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:29 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]: dispatch 2024-06-27T21:30:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:30.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:30.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:30.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:30.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:30.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:30:30.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:30.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:30.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 77778 77762 79.6677 81.8125 0.0223196 0.0125475 2024-06-27T21:30:30.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:30:30.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:30.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:30.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:30.801 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:30.892 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:30.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:30 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]': finished 2024-06-27T21:30:30.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:30 smithi151 ceph-mon[28965]: osdmap e493: 12 total, 12 up, 12 in 2024-06-27T21:30:30.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:30.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:30.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:31.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:31.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:30 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]': finished 2024-06-27T21:30:31.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:30 smithi192 ceph-mon[32956]: osdmap e493: 12 total, 12 up, 12 in 2024-06-27T21:30:31.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:31.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:30 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.08899959744411123}]': finished 2024-06-27T21:30:31.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:30 smithi190 ceph-mon[32247]: osdmap e493: 12 total, 12 up, 12 in 2024-06-27T21:30:31.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:31.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:31.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:31.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:31.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:31.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:31.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 79048 79032 79.6628 79.375 0.0050006 0.0125479 2024-06-27T21:30:31.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:31.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:31.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:31.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:31.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:31 smithi151 ceph-mon[28965]: pgmap v1285: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:30:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:31 smithi151 ceph-mon[28965]: osdmap e494: 12 total, 12 up, 12 in 2024-06-27T21:30:31.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:31 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:30:31.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:31.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:32.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:32.101 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:31 smithi190 ceph-mon[32247]: pgmap v1285: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:30:32.102 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:31 smithi190 ceph-mon[32247]: osdmap e494: 12 total, 12 up, 12 in 2024-06-27T21:30:32.102 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:31 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:30:32.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:32.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:32.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:32.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:31 smithi192 ceph-mon[32956]: pgmap v1285: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:30:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:31 smithi192 ceph-mon[32956]: osdmap e494: 12 total, 12 up, 12 in 2024-06-27T21:30:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:31 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:30:32.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:32.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:32.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:32.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:32.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:32.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 80231 80215 79.5719 73.9375 0.0298024 0.0125614 2024-06-27T21:30:32.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:32.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:32.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:32.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:32.812 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:30:32.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:32.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:32.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:32.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:32 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:30:32.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:32 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:30:32.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3929300867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:32.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:32 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:32.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:32 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:30:32.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:32 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:30:32.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3929300867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:32.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:32 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:32.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:33.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:33.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:33.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:33.182 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:30:33.182 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:693 4.1 e288/282 4:851bfe3a:::benchmark_data_smithi192_88961_object692:head v 288'41, mlcod=288'38)", 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:20:43.821644+0000", 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 589.35841684000002, 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22977736500000001, 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.183 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821644+0000", 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821644+0000", 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821645+0000", 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.155e-06 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:43.821746+0000", 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010046400000000001 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.050812+0000", 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22906600199999999 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.050827+0000", 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5605999999999999e-05 2024-06-27T21:30:33.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.050956+0000", 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012841999999999999 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.051002+0000", 2024-06-27T21:30:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6372000000000003e-05 2024-06-27T21:30:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.051385+0000", 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038305300000000001 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:20:44.051422+0000", 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6293e-05 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:121157 4.2 e342/282 4:4e90fc28:::benchmark_data_smithi192_88961_object30418:head v 342'7610, mlcod=342'7607)", 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:45.407230+0000", 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 467.77283120999999, 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22298259000000001, 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407230+0000", 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407230+0000", 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407231+0000", 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9899999999999999e-07 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.407237+0000", 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0820000000000004e-06 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627012+0000", 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21977543199999999 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627018+0000", 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4079999999999997e-06 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627070+0000", 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2119e-05 2024-06-27T21:30:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.627084+0000", 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4353e-05 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.630177+0000", 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030928629999999999 2024-06-27T21:30:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:45.630212+0000", 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5434e-05 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:122582 4.1 e343/282 4:800d8a34:::benchmark_data_smithi192_88961_object31843:head v 343'7629, mlcod=343'7624)", 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:47.131786+0000", 2024-06-27T21:30:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 466.04827499100003, 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.233553917, 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131786+0000", 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131786+0000", 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131787+0000", 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.131792+0000", 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0919999999999997e-06 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361794+0000", 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23000235199999999 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361800+0000", 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1489999999999999e-06 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361863+0000", 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2535999999999999e-05 2024-06-27T21:30:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.361887+0000", 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3864999999999999e-05 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.365300+0000", 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00341274 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:47.365340+0000", 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0342000000000001e-05 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:130521 4.1 e346/282 4:8d4b026c:::benchmark_data_smithi192_88961_object39782:head v 346'8090, mlcod=345'8088)", 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:56.565130+0000", 2024-06-27T21:30:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 456.61493097200002, 2024-06-27T21:30:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41281596300000001, 2024-06-27T21:30:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565130+0000", 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565130+0000", 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565131+0000", 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.82e-07 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565137+0000", 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2809999999999997e-06 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974674+0000", 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40953738000000001 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974679+0000", 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2530000000000004e-06 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974759+0000", 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9676000000000001e-05 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.974778+0000", 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8512000000000002e-05 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.977918+0000", 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031405159999999999 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.977946+0000", 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7863000000000001e-05 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:34779 5.f 5:f92bf575:::benchmark_data_smithi192_104167_object34778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:27.159406+0000", 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 306.02065524699998, 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22271158399999999, 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 34779 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159406+0000", 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159406+0000", 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159406+0000", 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1999999999999999e-07 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159468+0000", 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1725999999999998e-05 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159476+0000", 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5440000000000001e-06 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159480+0000", 2024-06-27T21:30:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.634e-06 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159513+0000", 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2434999999999997e-05 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159592+0000", 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9123999999999997e-05 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,11", 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.159675+0000", 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3450999999999998e-05 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.160796+0000", 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011211400000000001 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.171824+0000", 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011027202999999999 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.382020+0000", 2024-06-27T21:30:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21019611399999999 2024-06-27T21:30:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.382081+0000", 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1107999999999998e-05 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.382117+0000", 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6485000000000001e-05 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25615.0:34779 5.f e394/382 ondisk, result = 0)", 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:27.160825+0000", 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 306.01923586999999, 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.221299097, 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.160825+0000", 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.160825+0000", 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.160826+0000", 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0900000000000001e-07 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.160829+0000", 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3840000000000001e-06 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.381942+0000", 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22111244199999999 2024-06-27T21:30:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.381947+0000", 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5219999999999998e-06 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.382004+0000", 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6886999999999999e-05 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.382013+0000", 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9959999999999999e-06 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.382124+0000", 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000111257 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:72474 5.f 5:f65b48d2:::benchmark_data_smithi192_104167_object72473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:57.282166+0000", 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 275.89789469099998, 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.227114117, 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 72474 2024-06-27T21:30:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282166+0000", 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282166+0000", 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282168+0000", 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1909999999999999e-06 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282232+0000", 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4814999999999995e-05 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282245+0000", 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.225e-05 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282250+0000", 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.507e-06 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282292+0000", 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1686e-05 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282387+0000", 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5416999999999998e-05 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,11", 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282475+0000", 2024-06-27T21:30:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7666999999999999e-05 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.284856+0000", 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023813570000000002 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.284874+0000", 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7413000000000001e-05 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.509177+0000", 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.224303007 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.509236+0000", 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9753999999999997e-05 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.509280+0000", 2024-06-27T21:30:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4053e-05 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:95424 5.f 5:f11b0162:::benchmark_data_smithi192_104167_object95423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:26:19.337358+0000", 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 253.84270334000001, 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22122109200000001, 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 95424 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337358+0000", 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337358+0000", 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337369+0000", 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0902e-05 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337403+0000", 2024-06-27T21:30:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3949999999999999e-05 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337415+0000", 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2187999999999999e-05 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337421+0000", 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7560000000000002e-06 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337473+0000", 2024-06-27T21:30:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1345999999999999e-05 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337601+0000", 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012841099999999999 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,11", 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.337727+0000", 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012615099999999999 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.343717+0000", 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059899740000000003 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.343728+0000", 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0817e-05 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.558511+0000", 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21478277900000001 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.558548+0000", 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6908e-05 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:19.558579+0000", 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0910000000000001e-05 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:115234 5.7 5:e15d9162:::benchmark_data_smithi192_104167_object9193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:26:42.403321+0000", 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 230.77674000499999, 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22121579, 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 115234 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403321+0000", 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403321+0000", 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403322+0000", 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8999999999999996e-07 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403325+0000", 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.455e-06 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403333+0000", 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2849999999999995e-06 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403338+0000", 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8999999999999997e-06 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403383+0000", 2024-06-27T21:30:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5148000000000002e-05 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403499+0000", 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011534799999999999 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,9", 2024-06-27T21:30:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.403608+0000", 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109578 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.406884+0000", 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003275951 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.452691+0000", 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.045807052000000001 2024-06-27T21:30:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.624467+0000", 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.17177516600000001 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.624505+0000", 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8050999999999999e-05 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.624537+0000", 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2165999999999997e-05 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:152380 5.7 5:e168ed80:::benchmark_data_smithi192_104167_object46339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:25.832175+0000", 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 187.347885932, 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30182810599999998, 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 152380 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832175+0000", 2024-06-27T21:30:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832175+0000", 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832176+0000", 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2449999999999999e-06 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832182+0000", 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9900000000000002e-06 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832197+0000", 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4783999999999999e-05 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832203+0000", 2024-06-27T21:30:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3110000000000002e-06 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832268+0000", 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4398000000000002e-05 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832414+0000", 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000146284 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,9", 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.832549+0000", 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013464099999999999 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.836203+0000", 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036539760000000002 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.836255+0000", 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2398999999999998e-05 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.133906+0000", 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29765045400000001 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.133958+0000", 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2658999999999998e-05 2024-06-27T21:30:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.134003+0000", 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4965000000000001e-05 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25615.0:152380 5.7 e437/382 ondisk, result = 0)", 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:25.836090+0000", 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 187.34397078200001, 2024-06-27T21:30:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29791958600000001, 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.836090+0000", 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.836090+0000", 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.836091+0000", 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7400000000000002e-07 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:25.836095+0000", 2024-06-27T21:30:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3050000000000003e-06 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.133844+0000", 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.297748293 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.133848+0000", 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7509999999999999e-06 2024-06-27T21:30:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.133891+0000", 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3078000000000003e-05 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.133899+0000", 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5240000000000003e-06 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.134010+0000", 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011086099999999999 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:152392 5.b e437/382 5:d937ed31:::benchmark_data_smithi192_104167_object46351:head v 437'9564, mlcod=437'9562)", 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:26.184484+0000", 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 186.99557694800001, 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25858250599999999, 2024-06-27T21:30:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.184484+0000", 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.184484+0000", 2024-06-27T21:30:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.184486+0000", 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5400000000000001e-06 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.184497+0000", 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1459e-05 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.439411+0000", 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25491370000000002 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.439415+0000", 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2810000000000002e-06 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.439495+0000", 2024-06-27T21:30:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0408000000000007e-05 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.439509+0000", 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3723000000000001e-05 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.443031+0000", 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003521803 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:26.443067+0000", 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5592000000000001e-05 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:195448 5.7 5:ecafd372:::benchmark_data_smithi192_104167_object89407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:32.400056+0000", 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 120.780004664, 2024-06-27T21:30:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22313045200000001, 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 195448 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.400056+0000", 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.400056+0000", 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.400058+0000", 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.252e-06 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.400065+0000", 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4279999999999999e-06 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.400080+0000", 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5209e-05 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.400087+0000", 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5889999999999999e-06 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.401617+0000", 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015299669999999999 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.401730+0000", 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011271499999999999 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,9", 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.401851+0000", 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121827 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.407138+0000", 2024-06-27T21:30:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052864590000000003 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.415646+0000", 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085077450000000006 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.623071+0000", 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20742538499999999 2024-06-27T21:30:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.623136+0000", 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5450999999999994e-05 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:32.623187+0000", 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0424999999999999e-05 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:212032 5.7 5:e0ee91a0:::benchmark_data_smithi192_104167_object105991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.472922+0000", 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 100.70713938599999, 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.602999701, 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 212032 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.472922+0000", 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.472922+0000", 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.472922+0000", 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7e-07 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.472925+0000", 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.802e-06 2024-06-27T21:30:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.472934+0000", 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8689999999999993e-06 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.472939+0000", 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1220000000000003e-06 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.487064+0000", 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014125453 2024-06-27T21:30:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.487176+0000", 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112141 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,9", 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.487291+0000", 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000114263 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.495728+0000", 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084376220000000005 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.539927+0000", 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.044198223000000002 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075827+0000", 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.53590022599999998 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075875+0000", 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7978e-05 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075921+0000", 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6532000000000001e-05 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25615.0:212032 5.7 e462/382 ondisk, result = 0)", 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.492967+0000", 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 100.687093553, 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58295890900000003, 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.492967+0000", 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.492967+0000", 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.492968+0000", 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7000000000000004e-07 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.492972+0000", 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0999999999999997e-06 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075714+0000", 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58274184100000004 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075727+0000", 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2782999999999999e-05 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075814+0000", 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7651999999999994e-05 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075823+0000", 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2840000000000006e-06 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.075926+0000", 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010357900000000001 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:212069 5.4 e462/382 5:2da33f59:::benchmark_data_smithi192_104167_object106028:head v 462'13294, mlcod=462'13289)", 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.524755+0000", 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 100.655306078, 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31549386200000001, 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.524755+0000", 2024-06-27T21:30:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.524755+0000", 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.524755+0000", 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3600000000000004e-07 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.524761+0000", 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8189999999999997e-06 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.836383+0000", 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31162185399999998 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.836398+0000", 2024-06-27T21:30:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5265999999999999e-05 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.836537+0000", 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000138307 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.836593+0000", 2024-06-27T21:30:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6657999999999999e-05 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.840217+0000", 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003623902 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.840249+0000", 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1520000000000003e-05 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:37943 6.a 6:50dd083e:::benchmark_data_smithi192_117994_object37942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e483)", 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:29:56.834721+0000", 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 36.345339961000001, 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22927323299999999, 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 37943 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834721+0000", 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834721+0000", 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834722+0000", 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.44e-06 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834778+0000", 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5204999999999997e-05 2024-06-27T21:30:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834788+0000", 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0158e-05 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834793+0000", 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5090000000000001e-06 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834842+0000", 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8662000000000003e-05 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.834946+0000", 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010402 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.835088+0000", 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014164900000000001 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.836456+0000", 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013682309999999999 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:56.840616+0000", 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041599719999999996 2024-06-27T21:30:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:57.063934+0000", 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22331773299999999 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:57.063968+0000", 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4650000000000002e-05 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:57.063994+0000", 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6004e-05 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:51260 6.a 6:5a7f44e6:::benchmark_data_smithi192_117994_object51259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:07.359204+0000", 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.820857426, 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.216528743, 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 51260 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359204+0000", 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359204+0000", 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359205+0000", 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.531e-06 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359255+0000", 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0362000000000003e-05 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359264+0000", 2024-06-27T21:30:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3850000000000002e-06 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359269+0000", 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6829999999999999e-06 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359308+0000", 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9718000000000002e-05 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359394+0000", 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5246000000000003e-05 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.359479+0000", 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5791000000000006e-05 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.361481+0000", 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020019980000000001 2024-06-27T21:30:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.373827+0000", 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012345589000000001 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.575661+0000", 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20183375100000001 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.575700+0000", 2024-06-27T21:30:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9480999999999997e-05 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.575732+0000", 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2208000000000001e-05 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:76269 6.d 6:b11857a4:::benchmark_data_smithi192_117994_object76268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:29.257851+0000", 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.9222098270000001, 2024-06-27T21:30:33.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.158562753, 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 76269 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257851+0000", 2024-06-27T21:30:33.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257851+0000", 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257853+0000", 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.387e-06 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257885+0000", 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2344000000000001e-05 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257899+0000", 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4457999999999999e-05 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257906+0000", 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8090000000000004e-06 2024-06-27T21:30:33.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.257960+0000", 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4216999999999999e-05 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.258057+0000", 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6387999999999999e-05 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,10", 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.258154+0000", 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7101000000000004e-05 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.260118+0000", 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019643320000000001 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.260683+0000", 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00056433299999999998 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.416315+0000", 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.155632782 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.416369+0000", 2024-06-27T21:30:33.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3976e-05 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.416414+0000", 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4626000000000003e-05 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:76279 6.a 6:5b643c5c:::benchmark_data_smithi192_117994_object76278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:29.285660+0000", 2024-06-27T21:30:33.307 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.8944014519999999, 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.12586186599999999, 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 76279 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:33.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285660+0000", 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285660+0000", 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285661+0000", 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3030000000000001e-06 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285726+0000", 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4752999999999995e-05 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285735+0000", 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7229999999999999e-06 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285740+0000", 2024-06-27T21:30:33.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7029999999999997e-06 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285780+0000", 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0253000000000002e-05 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285882+0000", 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010131700000000001 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,6", 2024-06-27T21:30:33.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.285975+0000", 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3418999999999994e-05 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.288000+0000", 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002025284 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.288414+0000", 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000413867 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.411446+0000", 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.123031443 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.411490+0000", 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4425999999999998e-05 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:29.411521+0000", 2024-06-27T21:30:33.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1374999999999997e-05 2024-06-27T21:30:33.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.314 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:33.314 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:33.314 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:30:33.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:32 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:30:33.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:32 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:30:33.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3929300867' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:33.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:32 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:33.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:33.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:33.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:33.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 15 81361 81346 79.433 70.6875 0.0204287 0.0125853 2024-06-27T21:30:33.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:33.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:33.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:33.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:33.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:33.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:33.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:33.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:33.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:33.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:34.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:34.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:34.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:33 smithi151 ceph-mon[28965]: pgmap v1287: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s 2024-06-27T21:30:34.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:33 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:34.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:33 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:34.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:33 smithi151 ceph-mon[28965]: osdmap e495: 12 total, 12 up, 12 in 2024-06-27T21:30:34.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:34.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:33 smithi190 ceph-mon[32247]: pgmap v1287: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s 2024-06-27T21:30:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:33 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:33 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:33 smithi190 ceph-mon[32247]: osdmap e495: 12 total, 12 up, 12 in 2024-06-27T21:30:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:33 smithi192 ceph-mon[32956]: pgmap v1287: 37 pgs: 37 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s 2024-06-27T21:30:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:33 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:33 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:33 smithi192 ceph-mon[32956]: osdmap e495: 12 total, 12 up, 12 in 2024-06-27T21:30:34.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:34.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:34.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:34.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:34.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 82376 82360 79.1858 63.375 0.0220539 0.0126227 2024-06-27T21:30:34.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:30:34.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:34.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:34.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:34.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:34.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:34.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:34.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:34.923 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:34.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:35.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:35.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:35.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:35.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:35.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:35.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:35.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:35.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:35.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:35.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 83415 83399 78.9699 64.9375 0.00895553 0.0126582 2024-06-27T21:30:35.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:35.628 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:30:35.628 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:30:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:35.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:30:35.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:30:35.690 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:30:35.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:30:35.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:30:35.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:30:35.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:30:35.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:30:35.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:35.777 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:30:35.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:30:35.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:35.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:35.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:30:35.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:30:35.808 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:30:35.843 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:30:35.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:30:35.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:30:35.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:30:35.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:35.907 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:30:35.907 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:30:35.908 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:35 smithi151 ceph-mon[28965]: pgmap v1289: 37 pgs: 37 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:35.908 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:35 smithi151 ceph-mon[28965]: osdmap e496: 12 total, 12 up, 12 in 2024-06-27T21:30:35.908 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1976554321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:35.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:35.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:35.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:30:35.938 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:30:35.974 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:30:35.974 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:30:36.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:35 smithi190 ceph-mon[32247]: pgmap v1289: 37 pgs: 37 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:36.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:35 smithi190 ceph-mon[32247]: osdmap e496: 12 total, 12 up, 12 in 2024-06-27T21:30:36.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1976554321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:36.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:36.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:36.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:35 smithi192 ceph-mon[32956]: pgmap v1289: 37 pgs: 37 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-06-27T21:30:36.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:35 smithi192 ceph-mon[32956]: osdmap e496: 12 total, 12 up, 12 in 2024-06-27T21:30:36.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1976554321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:36.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:36.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:36.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:36.292 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:30:36.293 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:30:36.293 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-06-27T21:30:36.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:30:36.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:36.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:36.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:36.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:36.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 16 84629 84613 78.9236 75.875 0.0233768 0.0126546 2024-06-27T21:30:36.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:36.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:36.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:36.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:36.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:36.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:36.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:36.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:37.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:37.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:37.127 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:37.127 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:37.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:37.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:37.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:37.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:37.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:37.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:37.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:37.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 16 85732 85716 78.7767 68.9375 0.00633573 0.0126894 2024-06-27T21:30:37.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:37.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:37.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:37.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:37 smithi190 ceph-mon[32247]: pgmap v1291: 37 pgs: 37 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:30:37.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:37.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:37.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:37.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:37 smithi151 ceph-mon[28965]: pgmap v1291: 37 pgs: 37 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:30:37.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:37.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:37 smithi192 ceph-mon[32956]: pgmap v1291: 37 pgs: 37 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:30:38.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:38.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:38.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:30:38.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:38.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:38.293 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:38.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:38.370 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:38.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:38.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:38.506 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:38.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 86888 86872 78.682 72.25 0.0115159 0.012704 2024-06-27T21:30:38.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:38.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:38.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:38.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:38.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:38.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:38 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:30:38.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:38 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:30:38.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:38 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:30:38.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:38.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:38.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:38 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:30:38.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:38 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:30:38.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:38 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:30:38.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:39.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:39.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:38 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:30:39.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:38 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:30:39.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:38 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:30:39.061 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:39.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:39.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:39.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:39.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:39.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:39.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:39.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:39.546 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 88051 88035 78.5963 72.6875 0.0170847 0.0127188 2024-06-27T21:30:39.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:39.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:39.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:39.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:39.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:39.900 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:39 smithi151 ceph-mon[28965]: pgmap v1292: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-27T21:30:39.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:39.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:39.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:39 smithi190 ceph-mon[32247]: pgmap v1292: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-27T21:30:40.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:40.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:39 smithi192 ceph-mon[32956]: pgmap v1292: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-27T21:30:40.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:40.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:40.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:40.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:40.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:30:40.314 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:40.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:40.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:40.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 89205 89189 78.5051 72.125 0.0110524 0.0127332 2024-06-27T21:30:40.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:40.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:40.618 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:40.618 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:40.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:40.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:40.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:40.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:40.943 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:40.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:40.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:40.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:41.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:41.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:41.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:41.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:41.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:41.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:41.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:41.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:30:41.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:41.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:41.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:41.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:41.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:41.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 90394 90378 78.4468 74.3125 0.03248 0.0127408 2024-06-27T21:30:41.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:41.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:41.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:41.704 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:41.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:41.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:41.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:41.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:42.199 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:41 smithi192 ceph-mon[32956]: pgmap v1293: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-06-27T21:30:42.199 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:41 smithi190 ceph-mon[32247]: pgmap v1293: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-06-27T21:30:42.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:41 smithi151 ceph-mon[28965]: pgmap v1293: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-06-27T21:30:42.201 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:30:42.202 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:30:42.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:42.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:42.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:42.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:42.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:42.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:42.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:42.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:42.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 91399 91383 78.2326 62.8125 0.00679603 0.0127753 2024-06-27T21:30:42.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:42.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:42.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:42.667 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:30:42.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:42.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:42.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:42.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:30:42.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:30:42.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2885442498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:30:42.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:42.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:42.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:42.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:43.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:43.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:30:43.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:30:43.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2885442498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:30:43.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:30:43.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:30:43.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2885442498' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:30:43.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:43.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:43.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:43.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:43.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:43.403 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:43.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:43.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 92505 92489 78.1094 69.125 0.00538857 0.0127974 2024-06-27T21:30:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:43.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:43.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.703 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:43.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:43.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:43.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:43.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: pgmap v1294: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-27T21:30:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:30:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: from='client.16443 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:30:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2885442498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:30:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: osdmap e497: 12 total, 12 up, 12 in 2024-06-27T21:30:43.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:30:43.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:30:43.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:43.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: pgmap v1294: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: from='client.16443 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2885442498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: osdmap e497: 12 total, 12 up, 12 in 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:30:43.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:30:44.007 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:44.007 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: pgmap v1294: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: from='client.16443 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2885442498' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: osdmap e497: 12 total, 12 up, 12 in 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:30:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:30:44.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:44.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:44.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:44.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:44.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:44.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:44.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:44.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:44.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 93420 93404 77.8304 57.1875 0.00368339 0.0128436 2024-06-27T21:30:44.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:44.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:44.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:44.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:44.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:30:44.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:44.865 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1294,"stamp":"2024-06-27T21:30:42.081867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5725749792,"num_objects":87361,"num_object_clones":0,"num_object_copies":262083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87359,"num_write_kb":5590976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89291,"ondisk_log_size":89291,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":21142872,"kb_used_data":16711480,"kb_used_omap":110,"kb_used_meta":4431185,"kb_avail":1103798952,"statfs":{"total":1151940427776,"available":1130290126848,"internally_reserved":0,"allocated":17112555520,"data_stored":17107764616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":4537534423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":703791104,"num_objects":10739,"num_object_clones":0,"num_object_copies":32217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10739,"num_write_kb":687296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.275902"},"pg_stats":[{"pgid":"1.14","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.648722+0000","last_change":"2024-06-27T21:29:58.633933+0000","last_active":"2024-06-27T21:30:34.648722+0000","last_peered":"2024-06-27T21:30:34.648722+0000","last_clean":"2024-06-27T21:30:34.648722+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:30:34.648722+0000","last_undegraded":"2024-06-27T21:30:34.648722+0000","last_fullsized":"2024-06-27T21:30:34.648722+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.633807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.633807+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:18.467143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.653058+0000","last_change":"2024-06-27T21:29:58.085712+0000","last_active":"2024-06-27T21:30:34.653058+0000","last_peered":"2024-06-27T21:30:34.653058+0000","last_clean":"2024-06-27T21:30:34.653058+0000","last_became_active":"2024-06-27T21:29:51.673661+0000","last_became_peered":"2024-06-27T21:29:51.673661+0000","last_unstale":"2024-06-27T21:30:34.653058+0000","last_undegraded":"2024-06-27T21:30:34.653058+0000","last_fullsized":"2024-06-27T21:30:34.653058+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.085606+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.085606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:01.550085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649498+0000","last_change":"2024-06-27T21:29:57.430062+0000","last_active":"2024-06-27T21:30:34.649498+0000","last_peered":"2024-06-27T21:30:34.649498+0000","last_clean":"2024-06-27T21:30:34.649498+0000","last_became_active":"2024-06-27T21:29:51.676689+0000","last_became_peered":"2024-06-27T21:29:51.676689+0000","last_unstale":"2024-06-27T21:30:34.649498+0000","last_undegraded":"2024-06-27T21:30:34.649498+0000","last_fullsized":"2024-06-27T21:30:34.649498+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.429829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.429829+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:13.901732+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":56,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.644115+0000","last_change":"2024-06-27T21:29:57.074980+0000","last_active":"2024-06-27T21:30:34.644115+0000","last_peered":"2024-06-27T21:30:34.644115+0000","last_clean":"2024-06-27T21:30:34.644115+0000","last_became_active":"2024-06-27T21:29:51.676353+0000","last_became_peered":"2024-06-27T21:29:51.676353+0000","last_unstale":"2024-06-27T21:30:34.644115+0000","last_undegraded":"2024-06-27T21:30:34.644115+0000","last_fullsized":"2024-06-27T21:30:34.644115+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.074844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.074844+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:02.630593+0000","scrub_duration":250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649202+0000","last_change":"2024-06-27T21:29:59.388470+0000","last_active":"2024-06-27T21:30:34.649202+0000","last_peered":"2024-06-27T21:30:34.649202+0000","last_clean":"2024-06-27T21:30:34.649202+0000","last_became_active":"2024-06-27T21:29:51.677509+0000","last_became_peered":"2024-06-27T21:29:51.677509+0000","last_unstale":"2024-06-27T21:30:34.649202+0000","last_undegraded":"2024-06-27T21:30:34.649202+0000","last_fullsized":"2024-06-27T21:30:34.649202+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:59.388398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:59.388398+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:05.351358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":808,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649374+0000","last_change":"2024-06-27T21:30:13.565744+0000","last_active":"2024-06-27T21:30:34.649374+0000","last_peered":"2024-06-27T21:30:34.649374+0000","last_clean":"2024-06-27T21:30:34.649374+0000","last_became_active":"2024-06-27T21:27:16.656216+0000","last_became_peered":"2024-06-27T21:27:16.656216+0000","last_unstale":"2024-06-27T21:30:34.649374+0000","last_undegraded":"2024-06-27T21:30:34.649374+0000","last_fullsized":"2024-06-27T21:30:34.649374+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:13.565668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:30:13.565668+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:21.911022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"496'5498","reported_seq":5567,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:38.367938+0000","last_change":"2024-06-27T21:29:19.890876+0000","last_active":"2024-06-27T21:30:38.367938+0000","last_peered":"2024-06-27T21:30:38.367938+0000","last_clean":"2024-06-27T21:30:38.367938+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:30:38.367938+0000","last_undegraded":"2024-06-27T21:30:38.367938+0000","last_fullsized":"2024-06-27T21:30:38.367938+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.713540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":351872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":131,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.648808+0000","last_change":"2024-06-27T21:29:10.827592+0000","last_active":"2024-06-27T21:30:34.648808+0000","last_peered":"2024-06-27T21:30:34.648808+0000","last_clean":"2024-06-27T21:30:34.648808+0000","last_became_active":"2024-06-27T21:27:38.938524+0000","last_became_peered":"2024-06-27T21:27:38.938524+0000","last_unstale":"2024-06-27T21:30:34.648808+0000","last_undegraded":"2024-06-27T21:30:34.648808+0000","last_fullsized":"2024-06-27T21:30:34.648808+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:10.827466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:10.827466+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:35.172191+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"496'5476","reported_seq":5544,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:38.388941+0000","last_change":"2024-06-27T21:29:19.821974+0000","last_active":"2024-06-27T21:30:38.388941+0000","last_peered":"2024-06-27T21:30:38.388941+0000","last_clean":"2024-06-27T21:30:38.388941+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:30:38.388941+0000","last_undegraded":"2024-06-27T21:30:38.388941+0000","last_fullsized":"2024-06-27T21:30:38.388941+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358875136,"num_objects":5476,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5476,"num_write_kb":350464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":765,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.650398+0000","last_change":"2024-06-27T21:30:15.295538+0000","last_active":"2024-06-27T21:30:34.650398+0000","last_peered":"2024-06-27T21:30:34.650398+0000","last_clean":"2024-06-27T21:30:34.650398+0000","last_became_active":"2024-06-27T21:23:44.529707+0000","last_became_peered":"2024-06-27T21:23:44.529707+0000","last_unstale":"2024-06-27T21:30:34.650398+0000","last_undegraded":"2024-06-27T21:30:34.650398+0000","last_fullsized":"2024-06-27T21:30:34.650398+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:15.295432+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:30:15.295432+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:27.648757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"496'5401","reported_seq":5469,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:37.871484+0000","last_change":"2024-06-27T21:29:19.862967+0000","last_active":"2024-06-27T21:30:37.871484+0000","last_peered":"2024-06-27T21:30:37.871484+0000","last_clean":"2024-06-27T21:30:37.871484+0000","last_became_active":"2024-06-27T21:29:19.862712+0000","last_became_peered":"2024-06-27T21:29:19.862712+0000","last_unstale":"2024-06-27T21:30:37.871484+0000","last_undegraded":"2024-06-27T21:30:37.871484+0000","last_fullsized":"2024-06-27T21:30:37.871484+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5401,"log_dups_size":0,"ondisk_log_size":5401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:38.489764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353959936,"num_objects":5401,"num_object_clones":0,"num_object_copies":16203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5401,"num_write_kb":345664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":898,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.653050+0000","last_change":"2024-06-27T21:29:57.117837+0000","last_active":"2024-06-27T21:30:34.653050+0000","last_peered":"2024-06-27T21:30:34.653050+0000","last_clean":"2024-06-27T21:30:34.653050+0000","last_became_active":"2024-06-27T21:29:49.668296+0000","last_became_peered":"2024-06-27T21:29:49.668296+0000","last_unstale":"2024-06-27T21:30:34.653050+0000","last_undegraded":"2024-06-27T21:30:34.653050+0000","last_fullsized":"2024-06-27T21:30:34.653050+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.117729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.117729+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:02.448597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"496'5568","reported_seq":5637,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:39.529467+0000","last_change":"2024-06-27T21:29:19.921504+0000","last_active":"2024-06-27T21:30:39.529467+0000","last_peered":"2024-06-27T21:30:39.529467+0000","last_clean":"2024-06-27T21:30:39.529467+0000","last_became_active":"2024-06-27T21:29:19.921150+0000","last_became_peered":"2024-06-27T21:29:19.921150+0000","last_unstale":"2024-06-27T21:30:39.529467+0000","last_undegraded":"2024-06-27T21:30:39.529467+0000","last_fullsized":"2024-06-27T21:30:39.529467+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364904448,"num_objects":5568,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":356352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":462,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649938+0000","last_change":"2024-06-27T21:29:56.965958+0000","last_active":"2024-06-27T21:30:34.649938+0000","last_peered":"2024-06-27T21:30:34.649938+0000","last_clean":"2024-06-27T21:30:34.649938+0000","last_became_active":"2024-06-27T21:29:49.668842+0000","last_became_peered":"2024-06-27T21:29:49.668842+0000","last_unstale":"2024-06-27T21:30:34.649938+0000","last_undegraded":"2024-06-27T21:30:34.649938+0000","last_fullsized":"2024-06-27T21:30:34.649938+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.965862+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.965862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:19.446720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"496'5352","reported_seq":5421,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:38.364117+0000","last_change":"2024-06-27T21:29:19.921839+0000","last_active":"2024-06-27T21:30:38.364117+0000","last_peered":"2024-06-27T21:30:38.364117+0000","last_clean":"2024-06-27T21:30:38.364117+0000","last_became_active":"2024-06-27T21:29:19.921089+0000","last_became_peered":"2024-06-27T21:29:19.921089+0000","last_unstale":"2024-06-27T21:30:38.364117+0000","last_undegraded":"2024-06-27T21:30:38.364117+0000","last_fullsized":"2024-06-27T21:30:38.364117+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:46.227116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350748672,"num_objects":5352,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":342528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":466,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649924+0000","last_change":"2024-06-27T21:29:57.941498+0000","last_active":"2024-06-27T21:30:34.649924+0000","last_peered":"2024-06-27T21:30:34.649924+0000","last_clean":"2024-06-27T21:30:34.649924+0000","last_became_active":"2024-06-27T21:29:49.667265+0000","last_became_peered":"2024-06-27T21:29:49.667265+0000","last_unstale":"2024-06-27T21:30:34.649924+0000","last_undegraded":"2024-06-27T21:30:34.649924+0000","last_fullsized":"2024-06-27T21:30:34.649924+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.941423+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.941423+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:25.602914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"496'5160","reported_seq":5229,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:37.388496+0000","last_change":"2024-06-27T21:29:19.890411+0000","last_active":"2024-06-27T21:30:37.388496+0000","last_peered":"2024-06-27T21:30:37.388496+0000","last_clean":"2024-06-27T21:30:37.388496+0000","last_became_active":"2024-06-27T21:29:19.889910+0000","last_became_peered":"2024-06-27T21:29:19.889910+0000","last_unstale":"2024-06-27T21:30:37.388496+0000","last_undegraded":"2024-06-27T21:30:37.388496+0000","last_fullsized":"2024-06-27T21:30:37.388496+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5160,"log_dups_size":0,"ondisk_log_size":5160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.417356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165760,"num_objects":5160,"num_object_clones":0,"num_object_copies":15480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":330240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":376,"reported_epoch":496,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:30:37.090201+0000","last_change":"2024-06-27T21:30:37.089269+0000","last_active":"2024-06-27T21:30:37.090201+0000","last_peered":"2024-06-27T21:30:37.090201+0000","last_clean":"2024-06-27T21:30:37.090201+0000","last_became_active":"2024-06-27T21:30:30.809436+0000","last_became_peered":"2024-06-27T21:30:30.809436+0000","last_unstale":"2024-06-27T21:30:37.090201+0000","last_undegraded":"2024-06-27T21:30:37.090201+0000","last_fullsized":"2024-06-27T21:30:37.090201+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.118141+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.118141+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,5],"acting":[10,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"496'5618","reported_seq":5686,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:41.196342+0000","last_change":"2024-06-27T21:29:19.819739+0000","last_active":"2024-06-27T21:30:41.196342+0000","last_peered":"2024-06-27T21:30:41.196342+0000","last_clean":"2024-06-27T21:30:41.196342+0000","last_became_active":"2024-06-27T21:29:19.819457+0000","last_became_peered":"2024-06-27T21:29:19.819457+0000","last_unstale":"2024-06-27T21:30:41.196342+0000","last_undegraded":"2024-06-27T21:30:41.196342+0000","last_fullsized":"2024-06-27T21:30:41.196342+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5618,"log_dups_size":0,"ondisk_log_size":5618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:43.852077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368181248,"num_objects":5618,"num_object_clones":0,"num_object_copies":16854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5618,"num_write_kb":359552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":469,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649924+0000","last_change":"2024-06-27T21:29:58.953762+0000","last_active":"2024-06-27T21:30:34.649924+0000","last_peered":"2024-06-27T21:30:34.649924+0000","last_clean":"2024-06-27T21:30:34.649924+0000","last_became_active":"2024-06-27T21:29:49.667338+0000","last_became_peered":"2024-06-27T21:29:49.667338+0000","last_unstale":"2024-06-27T21:30:34.649924+0000","last_undegraded":"2024-06-27T21:30:34.649924+0000","last_fullsized":"2024-06-27T21:30:34.649924+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.953658+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:10.467277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"496'5370","reported_seq":5438,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:38.073207+0000","last_change":"2024-06-27T21:29:19.821949+0000","last_active":"2024-06-27T21:30:38.073207+0000","last_peered":"2024-06-27T21:30:38.073207+0000","last_clean":"2024-06-27T21:30:38.073207+0000","last_became_active":"2024-06-27T21:29:19.821580+0000","last_became_peered":"2024-06-27T21:29:19.821580+0000","last_unstale":"2024-06-27T21:30:38.073207+0000","last_undegraded":"2024-06-27T21:30:38.073207+0000","last_fullsized":"2024-06-27T21:30:38.073207+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.919881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351928320,"num_objects":5370,"num_object_clones":0,"num_object_copies":16110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5370,"num_write_kb":343680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":150,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.648925+0000","last_change":"2024-06-27T21:29:58.017742+0000","last_active":"2024-06-27T21:30:34.648925+0000","last_peered":"2024-06-27T21:30:34.648925+0000","last_clean":"2024-06-27T21:30:34.648925+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:30:34.648925+0000","last_undegraded":"2024-06-27T21:30:34.648925+0000","last_fullsized":"2024-06-27T21:30:34.648925+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.017623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.017623+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:06.149333+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"496'5453","reported_seq":5522,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:39.528311+0000","last_change":"2024-06-27T21:29:19.889671+0000","last_active":"2024-06-27T21:30:39.528311+0000","last_peered":"2024-06-27T21:30:39.528311+0000","last_clean":"2024-06-27T21:30:39.528311+0000","last_became_active":"2024-06-27T21:29:19.889320+0000","last_became_peered":"2024-06-27T21:29:19.889320+0000","last_unstale":"2024-06-27T21:30:39.528311+0000","last_undegraded":"2024-06-27T21:30:39.528311+0000","last_fullsized":"2024-06-27T21:30:39.528311+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5453,"log_dups_size":0,"ondisk_log_size":5453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.122955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357367808,"num_objects":5453,"num_object_clones":0,"num_object_copies":16359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":348992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":23,"reported_epoch":496,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:30:37.232330+0000","last_change":"2024-06-27T21:30:37.231574+0000","last_active":"2024-06-27T21:30:37.232330+0000","last_peered":"2024-06-27T21:30:37.232330+0000","last_clean":"2024-06-27T21:30:37.232330+0000","last_became_active":"2024-06-27T21:30:30.805587+0000","last_became_peered":"2024-06-27T21:30:30.805587+0000","last_unstale":"2024-06-27T21:30:37.232330+0000","last_undegraded":"2024-06-27T21:30:37.232330+0000","last_fullsized":"2024-06-27T21:30:37.232330+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.087617+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.087617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"496'5625","reported_seq":5694,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:39.110908+0000","last_change":"2024-06-27T21:29:19.889785+0000","last_active":"2024-06-27T21:30:39.110908+0000","last_peered":"2024-06-27T21:30:39.110908+0000","last_clean":"2024-06-27T21:30:39.110908+0000","last_became_active":"2024-06-27T21:29:19.889439+0000","last_became_peered":"2024-06-27T21:29:19.889439+0000","last_unstale":"2024-06-27T21:30:39.110908+0000","last_undegraded":"2024-06-27T21:30:39.110908+0000","last_fullsized":"2024-06-27T21:30:39.110908+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.921191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368640000,"num_objects":5625,"num_object_clones":0,"num_object_copies":16875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":360000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":155,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.648734+0000","last_change":"2024-06-27T21:29:56.956056+0000","last_active":"2024-06-27T21:30:34.648734+0000","last_peered":"2024-06-27T21:30:34.648734+0000","last_clean":"2024-06-27T21:30:34.648734+0000","last_became_active":"2024-06-27T21:29:49.671863+0000","last_became_peered":"2024-06-27T21:29:49.671863+0000","last_unstale":"2024-06-27T21:30:34.648734+0000","last_undegraded":"2024-06-27T21:30:34.648734+0000","last_fullsized":"2024-06-27T21:30:34.648734+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.955983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.955983+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:18.807666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"496'5314","reported_seq":5383,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:37.889585+0000","last_change":"2024-06-27T21:29:19.822891+0000","last_active":"2024-06-27T21:30:37.889585+0000","last_peered":"2024-06-27T21:30:37.889585+0000","last_clean":"2024-06-27T21:30:37.889585+0000","last_became_active":"2024-06-27T21:29:19.822526+0000","last_became_peered":"2024-06-27T21:29:19.822526+0000","last_unstale":"2024-06-27T21:30:37.889585+0000","last_undegraded":"2024-06-27T21:30:37.889585+0000","last_fullsized":"2024-06-27T21:30:37.889585+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:38.535433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348258304,"num_objects":5314,"num_object_clones":0,"num_object_copies":15942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5314,"num_write_kb":340096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":655,"reported_epoch":496,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:30:37.480708+0000","last_change":"2024-06-27T21:30:37.479900+0000","last_active":"2024-06-27T21:30:37.480708+0000","last_peered":"2024-06-27T21:30:37.480708+0000","last_clean":"2024-06-27T21:30:37.480708+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:30:37.480708+0000","last_undegraded":"2024-06-27T21:30:37.480708+0000","last_fullsized":"2024-06-27T21:30:37.480708+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:59.071261+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:29:59.071261+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"496'5567","reported_seq":5634,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:39.546535+0000","last_change":"2024-06-27T21:29:19.921495+0000","last_active":"2024-06-27T21:30:39.546535+0000","last_peered":"2024-06-27T21:30:39.546535+0000","last_clean":"2024-06-27T21:30:39.546535+0000","last_became_active":"2024-06-27T21:29:19.921198+0000","last_became_peered":"2024-06-27T21:29:19.921198+0000","last_unstale":"2024-06-27T21:30:39.546535+0000","last_undegraded":"2024-06-27T21:30:39.546535+0000","last_fullsized":"2024-06-27T21:30:39.546535+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:41.088638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364838912,"num_objects":5567,"num_object_clones":0,"num_object_copies":16701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":356288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.647534+0000","last_change":"2024-06-27T21:29:57.071471+0000","last_active":"2024-06-27T21:30:34.647534+0000","last_peered":"2024-06-27T21:30:34.647534+0000","last_clean":"2024-06-27T21:30:34.647534+0000","last_became_active":"2024-06-27T21:29:51.668654+0000","last_became_peered":"2024-06-27T21:29:51.668654+0000","last_unstale":"2024-06-27T21:30:34.647534+0000","last_undegraded":"2024-06-27T21:30:34.647534+0000","last_fullsized":"2024-06-27T21:30:34.647534+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.071395+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.071395+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:05.563427+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"496'5633","reported_seq":5702,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:41.181941+0000","last_change":"2024-06-27T21:29:19.890005+0000","last_active":"2024-06-27T21:30:41.181941+0000","last_peered":"2024-06-27T21:30:41.181941+0000","last_clean":"2024-06-27T21:30:41.181941+0000","last_became_active":"2024-06-27T21:29:19.889662+0000","last_became_peered":"2024-06-27T21:29:19.889662+0000","last_unstale":"2024-06-27T21:30:41.181941+0000","last_undegraded":"2024-06-27T21:30:41.181941+0000","last_fullsized":"2024-06-27T21:30:41.181941+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5633,"log_dups_size":0,"ondisk_log_size":5633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:44.852720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369164288,"num_objects":5633,"num_object_clones":0,"num_object_copies":16899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5633,"num_write_kb":360512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.649169+0000","last_change":"2024-06-27T21:29:58.941232+0000","last_active":"2024-06-27T21:30:34.649169+0000","last_peered":"2024-06-27T21:30:34.649169+0000","last_clean":"2024-06-27T21:30:34.649169+0000","last_became_active":"2024-06-27T21:29:51.672132+0000","last_became_peered":"2024-06-27T21:29:51.672132+0000","last_unstale":"2024-06-27T21:30:34.649169+0000","last_undegraded":"2024-06-27T21:30:34.649169+0000","last_fullsized":"2024-06-27T21:30:34.649169+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.941114+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.941114+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:09.275458+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"496'5370","reported_seq":5438,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:37.902199+0000","last_change":"2024-06-27T21:29:19.820432+0000","last_active":"2024-06-27T21:30:37.902199+0000","last_peered":"2024-06-27T21:30:37.902199+0000","last_clean":"2024-06-27T21:30:37.902199+0000","last_became_active":"2024-06-27T21:29:19.820126+0000","last_became_peered":"2024-06-27T21:29:19.820126+0000","last_unstale":"2024-06-27T21:30:37.902199+0000","last_undegraded":"2024-06-27T21:30:37.902199+0000","last_fullsized":"2024-06-27T21:30:37.902199+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:40.391505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351928320,"num_objects":5370,"num_object_clones":0,"num_object_copies":16110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5370,"num_write_kb":343680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"496'5194","reported_seq":5263,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:37.420352+0000","last_change":"2024-06-27T21:29:19.820104+0000","last_active":"2024-06-27T21:30:37.420352+0000","last_peered":"2024-06-27T21:30:37.420352+0000","last_clean":"2024-06-27T21:30:37.420352+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:30:37.420352+0000","last_undegraded":"2024-06-27T21:30:37.420352+0000","last_fullsized":"2024-06-27T21:30:37.420352+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:30:42.066610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340393984,"num_objects":5194,"num_object_clones":0,"num_object_copies":15582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5194,"num_write_kb":332416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":61,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.644262+0000","last_change":"2024-06-27T21:29:58.754806+0000","last_active":"2024-06-27T21:30:34.644262+0000","last_peered":"2024-06-27T21:30:34.644262+0000","last_clean":"2024-06-27T21:30:34.644262+0000","last_became_active":"2024-06-27T21:29:51.676349+0000","last_became_peered":"2024-06-27T21:29:51.676349+0000","last_unstale":"2024-06-27T21:30:34.644262+0000","last_undegraded":"2024-06-27T21:30:34.644262+0000","last_fullsized":"2024-06-27T21:30:34.644262+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.754689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.754689+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:20.863142+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"496'5760","reported_seq":5829,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:40.694738+0000","last_change":"2024-06-27T21:29:19.819646+0000","last_active":"2024-06-27T21:30:40.694738+0000","last_peered":"2024-06-27T21:30:40.694738+0000","last_clean":"2024-06-27T21:30:40.694738+0000","last_became_active":"2024-06-27T21:29:19.819315+0000","last_became_peered":"2024-06-27T21:29:19.819315+0000","last_unstale":"2024-06-27T21:30:40.694738+0000","last_undegraded":"2024-06-27T21:30:40.694738+0000","last_fullsized":"2024-06-27T21:30:40.694738+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:29:18.797481+0000","objects_scrubbed":0,"log_size":5760,"log_dups_size":0,"ondisk_log_size":5760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377487360,"num_objects":5760,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5760,"num_write_kb":368640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,8],"acting":[6,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":54,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-06-27T21:30:34.653045+0000","last_change":"2024-06-27T21:29:59.044100+0000","last_active":"2024-06-27T21:30:34.653045+0000","last_peered":"2024-06-27T21:30:34.653045+0000","last_clean":"2024-06-27T21:30:34.653045+0000","last_became_active":"2024-06-27T21:29:51.675328+0000","last_became_peered":"2024-06-27T21:29:51.675328+0000","last_unstale":"2024-06-27T21:30:34.653045+0000","last_undegraded":"2024-06-27T21:30:34.653045+0000","last_fullsized":"2024-06-27T21:30:34.653045+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:59.044030+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:29:59.044030+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:31:10.321618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5725159424,"num_objects":87359,"num_object_clones":0,"num_object_copies":262077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87359,"num_write_kb":5590976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17229021184,"data_stored":17229021184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87359,"ondisk_log_size":87359,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253121,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1517776,"kb_used_data":1053448,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92227376,"statfs":{"total":95995035648,"available":94440833024,"internally_reserved":0,"allocated":1078730752,"data_stored":1078332398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514757,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1397264,"kb_used_data":1045384,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92347888,"statfs":{"total":95995035648,"available":94564237312,"internally_reserved":0,"allocated":1070473216,"data_stored":1070074862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":360315328},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743690,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2173776,"kb_used_data":1759432,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91571376,"statfs":{"total":95995035648,"available":93769089024,"internally_reserved":0,"allocated":1801658368,"data_stored":1801260014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972621,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1261400,"kb_used_data":1027536,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92483752,"statfs":{"total":95995035648,"available":94703362048,"internally_reserved":0,"allocated":1052196864,"data_stored":1051794958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266962,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2174160,"kb_used_data":1741768,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91570992,"statfs":{"total":95995035648,"available":93768695808,"internally_reserved":0,"allocated":1783570432,"data_stored":1783172078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":442759626},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528600,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267024,"kb_used_data":1780488,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91478128,"statfs":{"total":95995035648,"available":93673603072,"internally_reserved":0,"allocated":1823219712,"data_stored":1822821358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":498203078},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2407960,"kb_used_data":2074128,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91337192,"statfs":{"total":95995035648,"available":93529284608,"internally_reserved":0,"allocated":2123907072,"data_stored":2123505166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161236,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1556432,"kb_used_data":1360776,"kb_used_omap":92,"kb_used_meta":195555,"kb_avail":92188720,"statfs":{"total":95995035648,"available":94401249280,"internally_reserved":0,"allocated":1393434624,"data_stored":1393036270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":200248418},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182693,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346960,"kb_used_data":1013128,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92398192,"statfs":{"total":95995035648,"available":94615748608,"internally_reserved":0,"allocated":1037443072,"data_stored":1037044718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411626,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663448,"kb_used_data":1379600,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92081704,"statfs":{"total":95995035648,"available":94291664896,"internally_reserved":0,"allocated":1412710400,"data_stored":1412308494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575151,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1895888,"kb_used_data":1401032,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":91849264,"statfs":{"total":95995035648,"available":94053646336,"internally_reserved":0,"allocated":1434656768,"data_stored":1434258414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1480784,"kb_used_data":1074760,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92264368,"statfs":{"total":95995035648,"available":94478712832,"internally_reserved":0,"allocated":1100554240,"data_stored":1100155886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1104019456,"data_stored":1104019456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1439039488,"data_stored":1439039488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1425735680,"data_stored":1425735680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1040711680,"data_stored":1040711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1408368640,"data_stored":1408368640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2135490560,"data_stored":2135490560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1838612480,"data_stored":1838612480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1795948544,"data_stored":1795948544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1051656192,"data_stored":1051656192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1823014912,"data_stored":1823014912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1078329344,"data_stored":1078329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1088094208,"data_stored":1088094208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:30:44.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-06-27T21:30:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:44.871 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:44.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:44.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:45.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:45.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:30:45.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:30:45.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:30:45.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:30:45.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:30:45.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: osdmap e498: 12 total, 12 up, 12 in 2024-06-27T21:30:45.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:44 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:30:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:30:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:30:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:30:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:30:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:30:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: osdmap e498: 12 total, 12 up, 12 in 2024-06-27T21:30:45.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:44 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:30:45.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:45.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:45.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:30:45.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:30:45.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:30:45.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:30:45.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:30:45.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: osdmap e498: 12 total, 12 up, 12 in 2024-06-27T21:30:45.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:44 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:30:45.177 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:45.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:45.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:45.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:45.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:45.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:45.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:45.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 94396 94380 77.6089 61 0.0153768 0.0128797 2024-06-27T21:30:45.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:45.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:45.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:30:45.667 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:30:45.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:30:45.692 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:30:45.709 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:45.722 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:30:45.723 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:30:45.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:45.750 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:30:45.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:30:45.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:45.784 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:30:45.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:30:45.809 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:30:45.809 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:30:45.841 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:30:45.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:30:45.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:45.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:30:45.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:30:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:45.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:45.906 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:30:45.907 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:30:45.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:30:45.933 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:30:45.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:45 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:30:45.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:45 smithi151 ceph-mon[28965]: pgmap v1297: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:30:45.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:45 smithi151 ceph-mon[28965]: 6.b scrub ok 2024-06-27T21:30:45.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:45 smithi151 ceph-mon[28965]: 6.8 scrub ok 2024-06-27T21:30:45.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3702020932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:45.961 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:30:45.961 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:30:45.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:30:45.988 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:30:46.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:45 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:30:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:45 smithi190 ceph-mon[32247]: pgmap v1297: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:30:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:45 smithi190 ceph-mon[32247]: 6.b scrub ok 2024-06-27T21:30:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:45 smithi190 ceph-mon[32247]: 6.8 scrub ok 2024-06-27T21:30:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3702020932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:46.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:46.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:45 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:30:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:45 smithi192 ceph-mon[32956]: pgmap v1297: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:30:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:45 smithi192 ceph-mon[32956]: 6.b scrub ok 2024-06-27T21:30:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:45 smithi192 ceph-mon[32956]: 6.8 scrub ok 2024-06-27T21:30:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3702020932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:46.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:46.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:46.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:46.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:46.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:46.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:46.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:30:46.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:46.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:46.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 95313 95297 77.3452 57.3125 0.00570344 0.0129232 2024-06-27T21:30:46.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:46.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:46.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:46.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:46.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:46 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:30:46.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:46 smithi190 ceph-mon[32247]: 6.0 scrub ok 2024-06-27T21:30:46.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:46 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:30:46.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:46 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:30:46.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:46.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:46.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:95703 6.8 6:1c46399b:::benchmark_data_smithi192_117994_object95702:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:46.985764+0000", 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.006334503, 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063658569999999999, 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:46.993 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 95703 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.985764+0000", 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.985764+0000", 2024-06-27T21:30:46.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.985765+0000", 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.71e-07 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.985971+0000", 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020605600000000001 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:46.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.985980+0000", 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6969999999999999e-06 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.985986+0000", 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4790000000000001e-06 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.986036+0000", 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0107e-05 2024-06-27T21:30:46.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.986128+0000", 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1928000000000005e-05 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.986276+0000", 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014778099999999999 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:46.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.989977+0000", 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037006080000000002 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:95708 6.3 6:c1fc7685:::benchmark_data_smithi192_117994_object95707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:46.990265+0000", 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0018337360000000001, 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019380619999999999, 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:46.998 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 95708 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990265+0000", 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:46.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990265+0000", 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990266+0000", 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.119e-06 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990343+0000", 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7075000000000003e-05 2024-06-27T21:30:47.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990354+0000", 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1323e-05 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990360+0000", 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5130000000000004e-06 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:47.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990413+0000", 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2833000000000001e-05 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990567+0000", 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000154501 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:46.990720+0000", 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015236199999999999 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:47.002 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:47.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:47.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:47.003 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-06-27T21:30:47.003 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 2 2024-06-27T21:30:47.003 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:30:47.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:47.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:46 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:30:47.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:46 smithi151 ceph-mon[28965]: 6.0 scrub ok 2024-06-27T21:30:47.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:46 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:30:47.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:46 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:30:47.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:46 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:30:47.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:46 smithi192 ceph-mon[32956]: 6.0 scrub ok 2024-06-27T21:30:47.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:46 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:30:47.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:46 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:30:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:47.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:47.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:47.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:47.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:47.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:47.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:47.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:47.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 96218 96202 77.0787 56.5625 0.0159983 0.012968 2024-06-27T21:30:47.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:47.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:47.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:47.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:30:47.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:47.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:47.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:47.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:47 smithi151 ceph-mon[28965]: pgmap v1298: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:30:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:47 smithi151 ceph-mon[28965]: 6.6 scrub ok 2024-06-27T21:30:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:47 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:30:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:47 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:30:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:47 smithi151 ceph-mon[28965]: 6.5 scrub starts 2024-06-27T21:30:47.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:47.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:47 smithi190 ceph-mon[32247]: pgmap v1298: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:30:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:47 smithi190 ceph-mon[32247]: 6.6 scrub ok 2024-06-27T21:30:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:47 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:30:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:47 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:30:48.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:47 smithi190 ceph-mon[32247]: 6.5 scrub starts 2024-06-27T21:30:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:47 smithi192 ceph-mon[32956]: pgmap v1298: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:30:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:47 smithi192 ceph-mon[32956]: 6.6 scrub ok 2024-06-27T21:30:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:47 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:30:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:47 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:30:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:47 smithi192 ceph-mon[32956]: 6.5 scrub starts 2024-06-27T21:30:48.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:48.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:48.143 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:48.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:48.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:48.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:48.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:48.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:48.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:48.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:48.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 97165 97149 76.8522 59.1875 0.00914599 0.0130076 2024-06-27T21:30:48.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:48.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:48.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:48.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:48.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:48.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:48.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:48.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:48.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:30:48.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:48 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:30:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:48 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:30:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:48 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:30:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:48 smithi190 ceph-mon[32247]: 6.e scrub ok 2024-06-27T21:30:49.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:48 smithi190 ceph-mon[32247]: 6.5 scrub ok 2024-06-27T21:30:49.020 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:49.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:48 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:30:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:48 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:30:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:48 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:30:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:48 smithi192 ceph-mon[32956]: 6.e scrub ok 2024-06-27T21:30:49.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:48 smithi192 ceph-mon[32956]: 6.5 scrub ok 2024-06-27T21:30:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:48 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:30:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:48 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:30:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:48 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:30:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:48 smithi151 ceph-mon[28965]: 6.e scrub ok 2024-06-27T21:30:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:48 smithi151 ceph-mon[28965]: 6.5 scrub ok 2024-06-27T21:30:49.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:49.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:49.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:49.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:49.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:49.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:49.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:49.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:30:49.548058+0000 min lat: 0.00108868 max lat: 0.636781 avg lat: 0.0130299 2024-06-27T21:30:49.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:30:49.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 15 98210 98195 76.7087 65.375 0.00418694 0.0130299 2024-06-27T21:30:49.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:49.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:49.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:49.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:49.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:49.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:49.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:49.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:30:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:30:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: pgmap v1299: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 996 op/s 2024-06-27T21:30:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:30:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: 6.4 scrub ok 2024-06-27T21:30:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: osdmap e499: 12 total, 12 up, 12 in 2024-06-27T21:30:50.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:49 smithi190 ceph-mon[32247]: osdmap e500: 12 total, 12 up, 12 in 2024-06-27T21:30:50.020 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:30:50.020 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:30:50.020 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:30:50.020 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:30:50.020 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: pgmap v1299: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 996 op/s 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: 6.4 scrub ok 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: osdmap e499: 12 total, 12 up, 12 in 2024-06-27T21:30:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:49 smithi192 ceph-mon[32956]: osdmap e500: 12 total, 12 up, 12 in 2024-06-27T21:30:50.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: pgmap v1299: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 5.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 996 op/s 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: 6.4 scrub ok 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: osdmap e499: 12 total, 12 up, 12 in 2024-06-27T21:30:50.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:49 smithi151 ceph-mon[28965]: osdmap e500: 12 total, 12 up, 12 in 2024-06-27T21:30:50.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:50.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:50.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:50.270 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:50.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:50.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:50.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:50.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:50.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 99267 99251 76.5764 66 0.0070399 0.0130539 2024-06-27T21:30:50.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:50.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:50.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:50.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:50.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:30:50.767 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 6 pgp_num to 16 2024-06-27T21:30:50.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:50.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:50.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:50.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:50.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:50 smithi190 ceph-mon[32247]: 6.7 scrub ok 2024-06-27T21:30:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2336477244' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:50 smithi190 ceph-mon[32247]: 6.1 scrub ok 2024-06-27T21:30:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:50 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:30:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:50 smithi192 ceph-mon[32956]: 6.7 scrub ok 2024-06-27T21:30:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2336477244' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:50 smithi192 ceph-mon[32956]: 6.1 scrub ok 2024-06-27T21:30:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:50 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:30:51.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:51.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:51.119 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:50 smithi151 ceph-mon[28965]: 6.7 scrub ok 2024-06-27T21:30:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2336477244' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-06-27T21:30:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:50 smithi151 ceph-mon[28965]: 6.1 scrub ok 2024-06-27T21:30:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:50 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:30:51.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:51.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:51.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:51.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:51.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:51.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:51.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 100181 100165 76.3391 57.125 0.00427003 0.0130936 2024-06-27T21:30:51.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:51.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:51.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:51.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:51.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:51.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:51.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:51.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:52.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:51 smithi190 ceph-mon[32247]: pgmap v1302: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-06-27T21:30:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2336477244' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:30:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:51 smithi190 ceph-mon[32247]: osdmap e501: 12 total, 12 up, 12 in 2024-06-27T21:30:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:51 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:30:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:51 smithi192 ceph-mon[32956]: pgmap v1302: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-06-27T21:30:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2336477244' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:30:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:51 smithi192 ceph-mon[32956]: osdmap e501: 12 total, 12 up, 12 in 2024-06-27T21:30:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:51 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:30:52.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:52.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:52.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:51 smithi151 ceph-mon[28965]: pgmap v1302: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 5.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-06-27T21:30:52.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2336477244' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-06-27T21:30:52.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:51 smithi151 ceph-mon[28965]: osdmap e501: 12 total, 12 up, 12 in 2024-06-27T21:30:52.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:51 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:30:52.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:52.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:52.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:52.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:52.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:52.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:52.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 100949 100933 75.9976 48 0.00998657 0.0131534 2024-06-27T21:30:52.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:52.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:52.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:52.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: 6.9 scrub starts 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: 6.3 scrub ok 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: osdmap e502: 12 total, 12 up, 12 in 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: 6.f scrub ok 2024-06-27T21:30:52.876 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:52 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:30:52.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:52.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: 6.9 scrub starts 2024-06-27T21:30:52.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:30:52.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:30:52.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: 6.3 scrub ok 2024-06-27T21:30:52.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:30:52.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: osdmap e502: 12 total, 12 up, 12 in 2024-06-27T21:30:52.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: 6.f scrub ok 2024-06-27T21:30:52.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:52 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:30:52.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:52.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:53.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: 6.9 scrub starts 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: 6.3 scrub ok 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: osdmap e502: 12 total, 12 up, 12 in 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: 6.f scrub ok 2024-06-27T21:30:53.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:52 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:30:53.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:53.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:53.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:53.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:53.313 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:30:53.313 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:30:53.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:30:53.313 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:66878 4.3 e320/282 4:c506cff3:::benchmark_data_smithi192_88961_object66877:head v 320'4172, mlcod=320'4170)", 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:21:44.971075+0000", 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 548.33858961400006, 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33489006599999999, 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971075+0000", 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971075+0000", 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971076+0000", 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5300000000000003e-07 2024-06-27T21:30:53.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:44.971165+0000", 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9655999999999997e-05 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.303997+0000", 2024-06-27T21:30:53.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33283127899999998 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.304004+0000", 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9970000000000001e-06 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.304059+0000", 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5937999999999999e-05 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.304077+0000", 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7231000000000002e-05 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.305933+0000", 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001856555 2024-06-27T21:30:53.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:21:45.305965+0000", 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1557000000000003e-05 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130343 4.4 4.aee468b4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.588913+0000", 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.72075121699999, 2024-06-27T21:30:53.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97634029499999997, 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130343 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588913+0000", 2024-06-27T21:30:53.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588913+0000", 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588914+0000", 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1519999999999999e-06 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588918+0000", 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1679999999999997e-06 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588926+0000", 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.34e-06 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588931+0000", 2024-06-27T21:30:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8879999999999996e-06 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588989+0000", 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8211999999999998e-05 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.588997+0000", 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2830000000000001e-06 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565243+0000", 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97624575099999999 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565253+0000", 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0501e-05 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130350 4.4 4.db94d154 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.592713+0000", 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.716951419, 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97257341200000003, 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.326 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130350 2024-06-27T21:30:53.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.327 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592713+0000", 2024-06-27T21:30:53.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592713+0000", 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592714+0000", 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.288e-06 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592723+0000", 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2110000000000004e-06 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592737+0000", 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4038e-05 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592745+0000", 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2189999999999998e-06 2024-06-27T21:30:53.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592788+0000", 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.367e-05 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.592794+0000", 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5029999999999997e-06 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565279+0000", 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97248522199999998 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565286+0000", 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2609999999999996e-06 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.332 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130380 4.4 4.f594c0b4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.633122+0000", 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.676542267, 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.93219419800000003, 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130380 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633122+0000", 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633122+0000", 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633123+0000", 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.141e-06 2024-06-27T21:30:53.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633127+0000", 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7249999999999999e-06 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633139+0000", 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1657e-05 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633144+0000", 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4500000000000003e-06 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633192+0000", 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8196999999999999e-05 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.633199+0000", 2024-06-27T21:30:53.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5409999999999997e-06 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565309+0000", 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.93211017699999998 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565316+0000", 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3100000000000003e-06 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.337 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130386 4.4 4.b7dc4824 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.641448+0000", 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.66821647099999, 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.92389724399999995, 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.338 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130386 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641448+0000", 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641448+0000", 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641449+0000", 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0869999999999999e-06 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641454+0000", 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7199999999999997e-06 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641466+0000", 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2486999999999999e-05 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641474+0000", 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1300000000000001e-06 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641529+0000", 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4432000000000003e-05 2024-06-27T21:30:53.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.641535+0000", 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5030000000000002e-06 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565338+0000", 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.923803072 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565345+0000", 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8129999999999998e-06 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130390 4.4 4.72e9fe74 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.647900+0000", 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.66176400699999, 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91747367300000005, 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130390 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647900+0000", 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647900+0000", 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647901+0000", 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1500000000000004e-07 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647905+0000", 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6969999999999999e-06 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647915+0000", 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0122000000000001e-05 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647922+0000", 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0090000000000001e-06 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647985+0000", 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3336000000000005e-05 2024-06-27T21:30:53.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.647990+0000", 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5680000000000001e-06 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565367+0000", 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.917377263 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565374+0000", 2024-06-27T21:30:53.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9630000000000001e-06 2024-06-27T21:30:53.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130428 4.4 4.d1ff1df4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.739569+0000", 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.57009494499999, 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82583467799999999, 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130428 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739569+0000", 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739569+0000", 2024-06-27T21:30:53.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739571+0000", 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2559999999999999e-06 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739575+0000", 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4560000000000002e-06 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739586+0000", 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0891e-05 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739591+0000", 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4820000000000002e-06 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739629+0000", 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7082000000000002e-05 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.739634+0000", 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3709999999999999e-06 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565397+0000", 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82576353800000002 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565404+0000", 2024-06-27T21:30:53.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6019999999999996e-06 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130448 4.4 4.cedc1aa4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.821951+0000", 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.48771374900002, 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.74348186000000005, 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.355 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130448 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821951+0000", 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821951+0000", 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821952+0000", 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5230000000000001e-06 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821958+0000", 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1670000000000004e-06 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821974+0000", 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5716e-05 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.821997+0000", 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3374999999999999e-05 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.822057+0000", 2024-06-27T21:30:53.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9338e-05 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.822063+0000", 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2879999999999998e-06 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565426+0000", 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.74336267600000006 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565432+0000", 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7769999999999997e-06 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130449 4.4 4.c6957ab4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.833860+0000", 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.47580398999997, 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.731599997, 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.360 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130449 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833860+0000", 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833860+0000", 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833862+0000", 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833872+0000", 2024-06-27T21:30:53.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6770000000000007e-06 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833899+0000", 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7248999999999999e-05 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833914+0000", 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.521e-05 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.833999+0000", 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4529000000000003e-05 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.834005+0000", 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0920000000000003e-06 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565453+0000", 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.731448443 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565460+0000", 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8639999999999998e-06 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130454 4.4 4.c3c86de4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.871420+0000", 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.438244607, 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.694069504, 2024-06-27T21:30:53.365 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130454 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.366 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871420+0000", 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871420+0000", 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871421+0000", 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.33e-06 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871425+0000", 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1189999999999999e-06 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871436+0000", 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0814e-05 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871458+0000", 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2430999999999999e-05 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871507+0000", 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8266000000000002e-05 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.871512+0000", 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3469999999999998e-06 2024-06-27T21:30:53.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565482+0000", 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69397007799999999 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565489+0000", 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1189999999999999e-06 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.370 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130455 4.4 4.76b76934 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.896512+0000", 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.41315239099998, 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.66900683900000002, 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.371 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130455 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896512+0000", 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896512+0000", 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896514+0000", 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9130000000000001e-06 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896523+0000", 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1370000000000008e-06 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896550+0000", 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7118999999999999e-05 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896564+0000", 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3423e-05 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896698+0000", 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000134177 2024-06-27T21:30:53.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.896703+0000", 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.733e-06 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565512+0000", 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.66880835599999999 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565519+0000", 2024-06-27T21:30:53.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9809999999999997e-06 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25312.0:130464 4.8 e345/282 4:198f8843:::benchmark_data_smithi192_88961_object39725:head v 345'8149, mlcod=345'8147)", 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.33922260899999, 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28325987400000002, 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:53.376 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970442+0000", 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8800000000000002e-07 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:55.970447+0000", 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3109999999999999e-06 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250043+0000", 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.279596446 2024-06-27T21:30:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250057+0000", 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4209000000000001e-05 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250125+0000", 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7310000000000004e-05 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.250142+0000", 2024-06-27T21:30:53.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7373000000000001e-05 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.253653+0000", 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035112680000000001 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.253702+0000", 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8368999999999997e-05 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.380 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130472 4.4 4.6db037f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:56.012165+0000", 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.29749917599997, 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.55339389400000005, 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.381 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130472 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012165+0000", 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012165+0000", 2024-06-27T21:30:53.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012167+0000", 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.801e-06 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012173+0000", 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0889999999999996e-06 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012188+0000", 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5378000000000001e-05 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012212+0000", 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3447000000000001e-05 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012265+0000", 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3480999999999998e-05 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.012273+0000", 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8029999999999997e-06 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565548+0000", 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.55327492599999994 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565559+0000", 2024-06-27T21:30:53.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0969e-05 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25312.0:130474 4.4 4.bfd3c974 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:22:56.018145+0000", 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.291519607, 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54743817800000005, 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.386 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25312", 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130474 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018145+0000", 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018145+0000", 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018147+0000", 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9269999999999999e-06 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018156+0000", 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6660000000000003e-06 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018193+0000", 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6958000000000001e-05 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018198+0000", 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9949999999999996e-06 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018258+0000", 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0081999999999998e-05 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.018263+0000", 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4959999999999998e-06 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565576+0000", 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54731361000000001 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:22:56.565583+0000", 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4439999999999996e-06 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:28d715ef:::benchmark_data_smithi192_88961_object27315:head,341'7381)", 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:24:30.681611+0000", 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 382.62805351499998, 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83259944799999996, 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.391 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681611+0000", 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681611+0000", 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681618+0000", 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6900000000000003e-06 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:30.681628+0000", 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0316000000000001e-05 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.513986+0000", 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83235856100000005 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514007+0000", 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0356e-05 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514135+0000", 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012872599999999999 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514143+0000", 2024-06-27T21:30:53.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5970000000000003e-06 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:31.514210+0000", 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7201999999999998e-05 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:72417 5.9 5:9fcfca4f:::benchmark_data_smithi192_104167_object72416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:57.166396+0000", 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 296.14326840799998, 2024-06-27T21:30:53.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.287432464, 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 72417 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166396+0000", 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166396+0000", 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166397+0000", 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.114e-06 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166449+0000", 2024-06-27T21:30:53.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1529000000000001e-05 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166460+0000", 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1270999999999999e-05 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166466+0000", 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9410000000000004e-06 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166509+0000", 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3612999999999997e-05 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166602+0000", 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2732000000000006e-05 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.166692+0000", 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0198999999999995e-05 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.168629+0000", 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019362610000000001 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.169390+0000", 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00076179500000000001 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.453695+0000", 2024-06-27T21:30:53.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28430495 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.453774+0000", 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8935000000000002e-05 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.453828+0000", 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4123999999999999e-05 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:212078 5.9 5:9e102c65:::benchmark_data_smithi192_104167_object106037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.529548+0000", 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 120.78011682, 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36362435199999998, 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:30:53.402 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 212078 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529548+0000", 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529548+0000", 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529549+0000", 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7e-06 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529556+0000", 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2459999999999997e-06 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529570+0000", 2024-06-27T21:30:53.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3801000000000001e-05 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529578+0000", 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3019999999999997e-06 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529637+0000", 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9497000000000003e-05 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529761+0000", 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012365600000000001 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.529885+0000", 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012398300000000001 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.534705+0000", 2024-06-27T21:30:53.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004820324 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.534887+0000", 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000182015 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.893096+0000", 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35820920899999997 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.893137+0000", 2024-06-27T21:30:53.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1177999999999998e-05 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.893172+0000", 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4440999999999998e-05 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25615.0:212078 5.9 e462/444 ondisk, result = 0)", 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.534728+0000", 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 120.774935997, 2024-06-27T21:30:53.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35844839699999997, 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.534728+0000", 2024-06-27T21:30:53.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:53.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:53.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.534728+0000", 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.534729+0000", 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7800000000000001e-07 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.534734+0000", 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6550000000000003e-06 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.892983+0000", 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35824915800000001 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.892992+0000", 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0110000000000006e-06 2024-06-27T21:30:53.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.893084+0000", 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2448999999999995e-05 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.893092+0000", 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7239999999999992e-06 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.893177+0000", 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4722000000000006e-05 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:51270 6.8 6:136c287d:::benchmark_data_smithi192_117994_object51269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:07.365633+0000", 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.944031123000002, 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37307769800000001, 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.414 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 51270 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365633+0000", 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365633+0000", 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365634+0000", 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1200000000000001e-06 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365705+0000", 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0711999999999997e-05 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365716+0000", 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1267000000000001e-05 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365722+0000", 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9750000000000004e-06 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365864+0000", 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000141619 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.365953+0000", 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9259999999999996e-05 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:30:53.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.366041+0000", 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8257000000000006e-05 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.367469+0000", 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001427174 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.367878+0000", 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040922499999999999 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.738591+0000", 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37071307399999998 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.738667+0000", 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5628000000000002e-05 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:07.738711+0000", 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4387e-05 2024-06-27T21:30:53.420 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:100682 6.b e502/467 6:d71a4924:::benchmark_data_smithi192_117994_object100681:head v 502'6167, mlcod=502'6164)", 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:52.098454+0000", 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.211210527, 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15666544399999999, 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:30:53.421 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.098454+0000", 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.098454+0000", 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:30:53.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.098455+0000", 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.161e-06 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.098505+0000", 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9620000000000003e-05 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.252260+0000", 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15375556800000001 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.252269+0000", 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1880000000000008e-06 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.252337+0000", 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7436999999999998e-05 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.252359+0000", 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2357e-05 2024-06-27T21:30:53.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.255070+0000", 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002711022 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:52.255119+0000", 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9091000000000001e-05 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:30:53.425 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:30:53.426 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:30:53.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:53.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:53.547 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 101957 101941 75.8429 63 0.0310572 0.0131801 2024-06-27T21:30:53.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:53.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:53.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:53.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:53.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:53.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:54.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:53 smithi192 ceph-mon[32956]: pgmap v1305: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 810 op/s 2024-06-27T21:30:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:53 smithi192 ceph-mon[32956]: 6.9 scrub ok 2024-06-27T21:30:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:53 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:30:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:53 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:54.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:53 smithi151 ceph-mon[28965]: pgmap v1305: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 810 op/s 2024-06-27T21:30:54.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:53 smithi151 ceph-mon[28965]: 6.9 scrub ok 2024-06-27T21:30:54.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:53 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:30:54.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:53 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:54.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:54.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:54.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:54.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:53 smithi190 ceph-mon[32247]: pgmap v1305: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 810 op/s 2024-06-27T21:30:54.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:53 smithi190 ceph-mon[32247]: 6.9 scrub ok 2024-06-27T21:30:54.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:53 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:30:54.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:53 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:54.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:54.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:54.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:54.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:54.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 103000 102984 75.7174 65.1875 0.00998559 0.0132017 2024-06-27T21:30:54.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:54.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:54.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:54.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:54.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:30:54.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:54.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:54.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:54.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:54.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:54 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:54 smithi190 ceph-mon[32247]: 6.2 scrub ok 2024-06-27T21:30:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:30:55.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:30:55.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:54 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:55.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/957651559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:54 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:54 smithi192 ceph-mon[32956]: 6.2 scrub ok 2024-06-27T21:30:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:30:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:30:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:54 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/957651559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:55.052 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:30:55.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:55.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:55.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:54 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:55.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:54 smithi151 ceph-mon[28965]: 6.2 scrub ok 2024-06-27T21:30:55.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:30:55.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:30:55.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:54 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:30:55.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/957651559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:30:55.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:55.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:55.256 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:30:55.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:55.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:55.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:55.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:55.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 103781 103765 75.4045 48.8125 0.0274068 0.0132492 2024-06-27T21:30:55.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:55.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:55.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:55.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:55.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:55.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: pgmap v1306: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 908 op/s 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: 6.a scrub ok 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: 6.c scrub starts 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:30:55.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/957651559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:55.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:55 smithi151 ceph-mon[28965]: osdmap e503: 12 total, 12 up, 12 in 2024-06-27T21:30:55.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:55.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:55.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:56.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:56.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:56.210 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: pgmap v1306: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 908 op/s 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: 6.a scrub ok 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: 6.c scrub starts 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/957651559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:56.211 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:55 smithi192 ceph-mon[32956]: osdmap e503: 12 total, 12 up, 12 in 2024-06-27T21:30:56.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:56.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: pgmap v1306: 37 pgs: 3 active+clean+scrubbing, 34 active+clean; 6.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 908 op/s 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: 6.a scrub ok 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: 6.c scrub starts 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:30:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/957651559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:30:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:55 smithi190 ceph-mon[32247]: osdmap e503: 12 total, 12 up, 12 in 2024-06-27T21:30:56.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:56.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:56.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:56.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:30:56.482 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:30:56.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:56.506 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:30:56.507 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:30:56.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:30:56.537 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:30:56.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 104298 104282 74.9092 32.3125 0.00580029 0.0133454 2024-06-27T21:30:56.572 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:30:56.572 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:30:56.602 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:30:56.602 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:30:56.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:56.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:56.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:30:56.637 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:30:56.676 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:30:56.677 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:30:56.708 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:30:56.708 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:30:56.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:56.740 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:30:56.740 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:30:56.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:56.779 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:30:56.779 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:30:56.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:30:56.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:30:56.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:30:56.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:30:56.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:56.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:56.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:56.948 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:56.949 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:56.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:30:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:57.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:30:57.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:57.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:57.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/168284193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:57.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:30:57.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/168284193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:57.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:30:57.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/168284193' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:30:57.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:57.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:57.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:57.535 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:57.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 105426 105410 74.859 70.5 0.0122281 0.013354 2024-06-27T21:30:57.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:57.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:57.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:30:57.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:57.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:57.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:30:57.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:57.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:30:57.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:30:57.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:57 smithi151 ceph-mon[28965]: pgmap v1308: 37 pgs: 4 active+clean+scrubbing, 33 active+clean; 6.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 859 op/s 2024-06-27T21:30:57.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:57 smithi151 ceph-mon[28965]: 6.c scrub ok 2024-06-27T21:30:57.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:57 smithi151 ceph-mon[28965]: 6.d scrub ok 2024-06-27T21:30:58.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:57 smithi192 ceph-mon[32956]: pgmap v1308: 37 pgs: 4 active+clean+scrubbing, 33 active+clean; 6.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 859 op/s 2024-06-27T21:30:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:57 smithi192 ceph-mon[32956]: 6.c scrub ok 2024-06-27T21:30:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:57 smithi192 ceph-mon[32956]: 6.d scrub ok 2024-06-27T21:30:58.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:58.073 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:30:58.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:58.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:58.263 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:30:58.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:57 smithi190 ceph-mon[32247]: pgmap v1308: 37 pgs: 4 active+clean+scrubbing, 33 active+clean; 6.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 859 op/s 2024-06-27T21:30:58.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:57 smithi190 ceph-mon[32247]: 6.c scrub ok 2024-06-27T21:30:58.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:57 smithi190 ceph-mon[32247]: 6.d scrub ok 2024-06-27T21:30:58.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:30:58.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:30:58.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:58.461 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:58.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 16 106486 106470 74.7622 66.25 0.00695591 0.0133672 2024-06-27T21:30:58.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:58.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:58.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:58.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:58.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:30:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:58.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:30:58.954 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:30:58.954 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:30:58.954 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-06-27T21:30:58.955 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-06-27T21:30:58.955 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-06-27T21:30:58.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-06-27T21:30:58.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:58.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:58.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:59.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:30:59.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:30:59.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:30:59.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:30:59.337 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:30:59.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:30:59.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:30:59.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:30:59.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:30:59.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 6 107591 107585 74.7057 69.6875 0.0103849 0.0133813 2024-06-27T21:30:59.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:30:59.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:30:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:30:59.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:30:59.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:30:59.757 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:30:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:30:59.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:30:59.868 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:30:59.868 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:30:59.869 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:59 smithi192 ceph-mon[32956]: pgmap v1309: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 923 op/s 2024-06-27T21:30:59.869 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:30:59.869 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:30:59.869 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:30:59 smithi192 ceph-mon[32956]: osdmap e504: 12 total, 12 up, 12 in 2024-06-27T21:30:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:30:59.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:30:59.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:30:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:59 smithi151 ceph-mon[28965]: pgmap v1309: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 923 op/s 2024-06-27T21:30:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:30:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:30:59.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:30:59 smithi151 ceph-mon[28965]: osdmap e504: 12 total, 12 up, 12 in 2024-06-27T21:30:59.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:00.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:00.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:00.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:00.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:00.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:00.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:59 smithi190 ceph-mon[32247]: pgmap v1309: 37 pgs: 2 active+clean+scrubbing, 35 active+clean; 6.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 923 op/s 2024-06-27T21:31:00.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:00.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:00.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:30:59 smithi190 ceph-mon[32247]: osdmap e504: 12 total, 12 up, 12 in 2024-06-27T21:31:00.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:00.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:00.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:00.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:00.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.7928 2024-06-27T21:31:00.548 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 107591 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 74.0635 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 15.79 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 133.625 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 32.3125 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1185 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 252.641 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 2138 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 517 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0133993 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0201195 2024-06-27T21:31:00.549 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 0.808115 2024-06-27T21:31:00.550 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00108868 2024-06-27T21:31:00.554 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:31:00.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:00.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:00.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:31:00.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:00.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:00.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:00.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:00.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:00.995 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:01.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:00 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:00 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:00 smithi190 ceph-mon[32247]: osdmap e505: 12 total, 12 up, 12 in 2024-06-27T21:31:01.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:00 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:00 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:00 smithi192 ceph-mon[32956]: osdmap e505: 12 total, 12 up, 12 in 2024-06-27T21:31:01.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:00 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:01.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:00 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:01.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:00 smithi151 ceph-mon[28965]: osdmap e505: 12 total, 12 up, 12 in 2024-06-27T21:31:01.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:01.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:01.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:01.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:01.379 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:01.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:01.404 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:01.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:01.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:01.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:01.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:01.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:01.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:01.841 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:31:01.882 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:01.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:01.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:01.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:01 smithi151 ceph-mon[28965]: pgmap v1311: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:31:01.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:01.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:01.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/436691279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:31:02.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:02.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:02.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:02.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:02.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:02.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:01 smithi192 ceph-mon[32956]: pgmap v1311: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:31:02.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:02.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:02.184 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/436691279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:31:02.239 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:01 smithi190 ceph-mon[32247]: pgmap v1311: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:31:02.239 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:02.240 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:02.240 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/436691279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:31:02.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:02.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:02.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:02.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:02.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:02.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:02.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:02.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:02.741 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:02.741 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:02.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:02.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:02.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:02.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:02.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:03.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:03.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:03.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:02 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:03.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/436691279' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:31:03.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:02 smithi192 ceph-mon[32956]: osdmap e506: 12 total, 12 up, 12 in 2024-06-27T21:31:03.101 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:02 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:03.102 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/436691279' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:31:03.102 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:02 smithi190 ceph-mon[32247]: osdmap e506: 12 total, 12 up, 12 in 2024-06-27T21:31:03.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:03.163 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:31:03.163 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-06-27T21:31:03.163 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:31:03.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:02 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/436691279' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:31:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:02 smithi151 ceph-mon[28965]: osdmap e506: 12 total, 12 up, 12 in 2024-06-27T21:31:03.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:03.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:03.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:03.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:31:03.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:03.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:03.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:03.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:03.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:03.710 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:03.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:03.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:03.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:03.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:03.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:04.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:04.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:04.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:04.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:03 smithi151 ceph-mon[28965]: pgmap v1314: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 830 op/s 2024-06-27T21:31:04.107 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:03 smithi192 ceph-mon[32956]: pgmap v1314: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 830 op/s 2024-06-27T21:31:04.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:04.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:04.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:03 smithi190 ceph-mon[32247]: pgmap v1314: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 830 op/s 2024-06-27T21:31:04.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:04.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:04.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:31:04.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:31:04.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:31:04.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:31:04.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:31:04.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:31:04.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:31:04.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:31:04.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:31:04.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:31:04.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:31:04.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:31:04.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:31:04.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:31:04.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:04.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:04.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:04.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:04.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:04.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:04.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:04.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:04.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:31:04.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:04.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4227703291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:04.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:04.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:04.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:04.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4227703291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:05.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:05.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:05.179 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:05.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:05.213 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4227703291' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:05.213 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:05.213 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:05.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:05.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:05.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:05.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:111581 6.1 e507/467 6:8adbb0b8:::benchmark_data_smithi192_117994_object3987:head v 507'6769, mlcod=507'6767)", 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:31:05.471574+0000", 2024-06-27T21:31:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0017561899999999999, 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001776106, 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471574+0000", 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471574+0000", 2024-06-27T21:31:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471575+0000", 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1319999999999999e-06 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471582+0000", 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2089999999999999e-06 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471606+0000", 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.385e-05 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471615+0000", 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8480000000000007e-06 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471704+0000", 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8795000000000006e-05 2024-06-27T21:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:05.471722+0000", 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8374999999999999e-05 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-06-27T21:31:05.478 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:31:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:05.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:05.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:05.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:05.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:05.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:05.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:05.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:05.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:05.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:06.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:06.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:06.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:06.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:05 smithi192 ceph-mon[32956]: pgmap v1315: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 40 MiB/s wr, 843 op/s 2024-06-27T21:31:06.159 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:05 smithi192 ceph-mon[32956]: osdmap e507: 12 total, 12 up, 12 in 2024-06-27T21:31:06.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:31:06.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:05 smithi151 ceph-mon[28965]: pgmap v1315: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 40 MiB/s wr, 843 op/s 2024-06-27T21:31:06.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:05 smithi151 ceph-mon[28965]: osdmap e507: 12 total, 12 up, 12 in 2024-06-27T21:31:06.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:06.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:06.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:06.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:05 smithi190 ceph-mon[32247]: pgmap v1315: 37 pgs: 37 active+clean; 6.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 40 MiB/s wr, 843 op/s 2024-06-27T21:31:06.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:05 smithi190 ceph-mon[32247]: osdmap e507: 12 total, 12 up, 12 in 2024-06-27T21:31:06.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:06.377 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:06.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:06.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:06.518 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:06.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:06.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:06.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:06.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:06.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:06.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:06.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:06.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:07.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:07.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:31:07.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:07.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:07.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:07.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:07.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:07.393 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:07.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:07.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:07.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:07.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:07.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:07.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:07.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:07.876 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:31:07.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:07.923 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:07.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:08.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:07 smithi192 ceph-mon[32956]: pgmap v1317: 37 pgs: 37 active+clean; 6.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 17 MiB/s wr, 658 op/s 2024-06-27T21:31:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4029763922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:08.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:08.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:08.179 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:08.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:07 smithi151 ceph-mon[28965]: pgmap v1317: 37 pgs: 37 active+clean; 6.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 17 MiB/s wr, 658 op/s 2024-06-27T21:31:08.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4029763922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:07 smithi190 ceph-mon[32247]: pgmap v1317: 37 pgs: 37 active+clean; 6.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 17 MiB/s wr, 658 op/s 2024-06-27T21:31:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4029763922' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:08.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:08.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:08.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:08.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:08.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:08.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:08.643 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:31:08.643 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:31:08.643 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:31:08.643 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:31:08.643 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:31:08.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:08.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:08.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:08.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:08.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:31:08.867 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:31:08.868 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-06-27T21:31:08.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-06-27T21:31:08.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:08.944 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:08 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:08.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4029763922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:08.945 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:08 smithi192 ceph-mon[32956]: osdmap e508: 12 total, 12 up, 12 in 2024-06-27T21:31:08.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:08.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:09.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:09.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:09.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:09.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:08 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:09.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4029763922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:09.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:08 smithi151 ceph-mon[28965]: osdmap e508: 12 total, 12 up, 12 in 2024-06-27T21:31:09.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:08 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4029763922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:08 smithi190 ceph-mon[32247]: osdmap e508: 12 total, 12 up, 12 in 2024-06-27T21:31:09.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:31:09.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:09.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:09.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:09.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:09.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:09.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:09.647 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:09.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:09.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:09.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:09.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:09.946 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:09 smithi192 ceph-mon[32956]: pgmap v1319: 37 pgs: 37 active+clean; 6.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s rd, 9.8 MiB/s wr, 800 op/s 2024-06-27T21:31:09.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:09.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:09.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:09.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:10.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:10.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:10.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:09 smithi151 ceph-mon[28965]: pgmap v1319: 37 pgs: 37 active+clean; 6.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s rd, 9.8 MiB/s wr, 800 op/s 2024-06-27T21:31:10.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:10.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:10.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:09 smithi190 ceph-mon[32247]: pgmap v1319: 37 pgs: 37 active+clean; 6.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 164 B/s rd, 9.8 MiB/s wr, 800 op/s 2024-06-27T21:31:10.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:10.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:10.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:10.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:10.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:10.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:10.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:10.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:10.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:10.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:10.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:10.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:11.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:11.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:11.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:11.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:11.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:11.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:11.381 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:31:11.381 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:31:11.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58291 4.6 4.30c53896 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 577.84789164899996, 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95344075399999995, 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58291 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530166+0000", 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530167+0000", 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0860000000000001e-06 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530189+0000", 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1943999999999999e-05 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530201+0000", 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1959e-05 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530208+0000", 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6880000000000002e-06 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530259+0000", 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0924000000000002e-05 2024-06-27T21:31:11.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.530267+0000", 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5790000000000004e-06 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483575+0000", 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.95330775800000001 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483607+0000", 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1816000000000002e-05 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58315 4.6 4.d0954b56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 577.816144044, 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92179534699999999, 2024-06-27T21:31:11.387 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58315 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:31:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561914+0000", 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561915+0000", 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.141e-06 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561933+0000", 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8337000000000001e-05 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561941+0000", 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7729999999999999e-06 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.561946+0000", 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9230000000000001e-06 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.562002+0000", 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6212999999999999e-05 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.562008+0000", 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5849999999999996e-06 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483698+0000", 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.92168988600000001 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483709+0000", 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0489000000000001e-05 2024-06-27T21:31:11.392 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58321 4.6 4.898f9226 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 577.81007345, 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91576660700000001, 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.393 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58321 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567984+0000", 2024-06-27T21:31:11.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.567986+0000", 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.516e-06 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568008+0000", 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2464e-05 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568022+0000", 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3726999999999999e-05 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568028+0000", 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1530000000000002e-06 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568097+0000", 2024-06-27T21:31:11.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8694000000000003e-05 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.568110+0000", 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3739e-05 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483740+0000", 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91563002900000001 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483751+0000", 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0285000000000001e-05 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58331 4.6 4.e4097d56 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 577.69498530999999, 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80079522000000003, 2024-06-27T21:31:11.398 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58331 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683072+0000", 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683074+0000", 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5370000000000001e-06 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683096+0000", 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1880999999999999e-05 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683109+0000", 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2858e-05 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683114+0000", 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7069999999999997e-06 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683168+0000", 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3949999999999997e-05 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.683177+0000", 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4479999999999994e-06 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483855+0000", 2024-06-27T21:31:11.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80067863299999997 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483867+0000", 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2206e-05 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58364 4.6 4.a2603346 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 577.55971296899997, 2024-06-27T21:31:11.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66557216799999996, 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58364 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818345+0000", 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818346+0000", 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.539e-06 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818373+0000", 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7081e-05 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818392+0000", 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8879999999999999e-05 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818402+0000", 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7419999999999992e-06 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818457+0000", 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5664000000000001e-05 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.818466+0000", 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9029999999999993e-06 2024-06-27T21:31:11.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483904+0000", 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66543803300000004 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483917+0000", 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2326e-05 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58368 4.6 4.bdd01c66 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 577.44656187999999, 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55245678099999995, 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58368 2024-06-27T21:31:11.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931496+0000", 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931497+0000", 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3260000000000001e-06 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931517+0000", 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0378e-05 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931528+0000", 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.078e-05 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931534+0000", 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9429999999999997e-06 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931583+0000", 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9036000000000003e-05 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.931592+0000", 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8230000000000001e-06 2024-06-27T21:31:11.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483946+0000", 2024-06-27T21:31:11.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55235438299999995 2024-06-27T21:31:11.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.483952+0000", 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1120000000000001e-06 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163683 4.d 4.883d3e0d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 447.46773902500001, 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.612742381, 2024-06-27T21:31:11.415 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163683 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910319+0000", 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910320+0000", 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.08e-06 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910324+0000", 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7899999999999999e-06 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910336+0000", 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1676e-05 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910341+0000", 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2360000000000004e-06 2024-06-27T21:31:11.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910386+0000", 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4273000000000001e-05 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.910393+0000", 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2790000000000001e-06 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523031+0000", 2024-06-27T21:31:11.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.612637722 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523061+0000", 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0324999999999998e-05 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163697 4.d 4.ba6d2ad (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:31:11.420 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 447.44532752800001, 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59065456900000002, 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163697 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932730+0000", 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932731+0000", 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932735+0000", 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3970000000000004e-06 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932745+0000", 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7929999999999992e-06 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932750+0000", 2024-06-27T21:31:11.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2100000000000001e-06 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932794+0000", 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3946000000000002e-05 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.932802+0000", 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5909999999999998e-06 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523377+0000", 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.59057547099999996 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523385+0000", 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2579999999999998e-06 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.425 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163710 4.d 4.4da514d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 447.43195483099998, 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57733542599999998, 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163710 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946103+0000", 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946104+0000", 2024-06-27T21:31:11.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.277e-06 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946109+0000", 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3809999999999998e-06 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946122+0000", 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3040000000000001e-05 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946130+0000", 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8469999999999996e-06 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946187+0000", 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7229999999999999e-05 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.946199+0000", 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-05 2024-06-27T21:31:11.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523432+0000", 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57723327000000002 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.523438+0000", 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8309999999999997e-06 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.430 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163683 4.d 4:b07cbc11:::benchmark_data_smithi192_88961_object72944:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 445.84714843, 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46728570899999999, 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.431 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163683 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530909+0000", 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530911+0000", 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2199999999999999e-06 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530921+0000", 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1949999999999992e-06 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530939+0000", 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8541000000000001e-05 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530951+0000", 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.202e-05 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531024+0000", 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2605e-05 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531031+0000", 2024-06-27T21:31:11.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6669999999999996e-06 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.980931+0000", 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44989959299999999 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.981077+0000", 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014631999999999999 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.981221+0000", 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000143874 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.987926+0000", 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006704738 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991110+0000", 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031838470000000001 2024-06-27T21:31:11.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.997913+0000", 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006803478 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.997982+0000", 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9077000000000003e-05 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.998195+0000", 2024-06-27T21:31:11.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021253400000000001 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163697 4.d 4:b54b65d0:::benchmark_data_smithi192_88961_object72958:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 445.84706884500002, 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47026454699999998, 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:11.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163697 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530989+0000", 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.530993+0000", 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5439999999999999e-06 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531002+0000", 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0939999999999997e-06 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531007+0000", 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4369999999999998e-06 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531076+0000", 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8229e-05 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531084+0000", 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9260000000000006e-06 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984774+0000", 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45368975900000003 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984852+0000", 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7356999999999998e-05 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.984929+0000", 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7101999999999995e-05 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991124+0000", 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0061952420000000001 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.995794+0000", 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004670319 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001042+0000", 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0052480169999999998 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001078+0000", 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5355000000000003e-05 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.001253+0000", 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017562099999999999 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163710 4.d 4:b28a5b20:::benchmark_data_smithi192_88961_object72971:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 445.84704078800002, 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47328603800000002, 2024-06-27T21:31:11.445 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163710 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531017+0000", 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4299999999999999e-07 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531019+0000", 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3039999999999999e-06 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531025+0000", 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.874e-06 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531030+0000", 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2359999999999998e-06 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531108+0000", 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8033000000000003e-05 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.531115+0000", 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1980000000000004e-06 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988002+0000", 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45688679300000001 2024-06-27T21:31:11.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988079+0000", 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7922000000000004e-05 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.988158+0000", 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.991184+0000", 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030261609999999999 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:45.995891+0000", 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004706422 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004153+0000", 2024-06-27T21:31:11.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082619319999999996 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004184+0000", 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1028999999999997e-05 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:46.004303+0000", 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119159 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.452 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e376,346 4:26f4a754:::benchmark_data_smithi192_88961_object19828:head,337'6948)", 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:26.386183+0000", 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 404.99187425399998, 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51000209500000004, 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386183+0000", 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386183+0000", 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386184+0000", 2024-06-27T21:31:11.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1699999999999997e-07 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.386188+0000", 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9559999999999999e-06 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896011+0000", 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50982296199999999 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896020+0000", 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4910000000000005e-06 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896110+0000", 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0766000000000005e-05 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896122+0000", 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2082000000000001e-05 2024-06-27T21:31:11.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:26.896185+0000", 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2921000000000003e-05 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:2b2a0b1c:::benchmark_data_smithi192_88961_object26504:head,341'7333)", 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:29.727966+0000", 2024-06-27T21:31:11.457 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 401.65009126299998, 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67969687300000003, 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.727966+0000", 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.727966+0000", 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.727968+0000", 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5030000000000001e-06 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:29.728000+0000", 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2143e-05 2024-06-27T21:31:11.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407501+0000", 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67950098000000003 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407512+0000", 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.065e-05 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407586+0000", 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4127000000000004e-05 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407598+0000", 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1833e-05 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:30.407663+0000", 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5636999999999995e-05 2024-06-27T21:31:11.461 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:27ab21ae:::benchmark_data_smithi192_88961_object28689:head,342'7457)", 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:32.115635+0000", 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 399.26242287000002, 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61824737100000005, 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.462 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115635+0000", 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115635+0000", 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115636+0000", 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.483e-06 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.115651+0000", 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4844000000000001e-05 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733722+0000", 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.618070546 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733731+0000", 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3570000000000004e-06 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733813+0000", 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1669999999999996e-05 2024-06-27T21:31:11.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733819+0000", 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8970000000000002e-06 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:32.733882+0000", 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2574000000000001e-05 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(4.4 e377,346 4:2acc291c:::benchmark_data_smithi192_88961_object33232:head,343'7743)", 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:24:36.282894+0000", 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 395.09516366499997, 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63189966099999995, 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282894+0000", 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282894+0000", 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282895+0000", 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7599999999999997e-07 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.282899+0000", 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.729e-06 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914615+0000", 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63171538800000004 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914642+0000", 2024-06-27T21:31:11.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7322e-05 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914707+0000", 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4821999999999997e-05 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914717+0000", 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0548e-05 2024-06-27T21:31:11.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:24:36.914794+0000", 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6175999999999997e-05 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:163152 5.e 5:71dd832e:::benchmark_data_smithi192_104167_object57111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:45.864530+0000", 2024-06-27T21:31:11.471 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 205.51352750199999, 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50746658499999997, 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163152 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864530+0000", 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864530+0000", 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864531+0000", 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.342e-06 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864536+0000", 2024-06-27T21:31:11.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0889999999999999e-06 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864549+0000", 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2221e-05 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864554+0000", 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.242e-06 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864614+0000", 2024-06-27T21:31:11.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:11.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:11.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9647e-05 2024-06-27T21:31:11.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864719+0000", 2024-06-27T21:31:11.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105387 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,8", 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864797+0000", 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7749000000000004e-05 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868345+0000", 2024-06-27T21:31:11.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035478649999999999 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868453+0000", 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010888899999999999 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371912+0000", 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50345891799999998 2024-06-27T21:31:11.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371953+0000", 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0469000000000002e-05 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371997+0000", 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3767000000000003e-05 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.479 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25615.0:163152 5.e e444/382 ondisk, result = 0)", 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 205.50924996000001, 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50319430700000001, 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:31:11.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3200000000000005e-07 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868815+0000", 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3790000000000003e-06 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371852+0000", 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50303705799999998 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371859+0000", 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9489999999999999e-06 2024-06-27T21:31:11.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371903+0000", 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4549000000000003e-05 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371909+0000", 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4020000000000001e-06 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.372002+0000", 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3338000000000007e-05 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(5.1 e463,441 5:8240b269:::benchmark_data_smithi192_104167_object9873:head,419'7188)", 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:28:56.068059+0000", 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 135.30999857399999, 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47462944000000001, 2024-06-27T21:31:11.484 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068059+0000", 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068059+0000", 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068060+0000", 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-07 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068064+0000", 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9899999999999999e-06 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542461+0000", 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47439695399999998 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542475+0000", 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4139e-05 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542597+0000", 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012245100000000001 2024-06-27T21:31:11.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542615+0000", 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7456999999999999e-05 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542688+0000", 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3833999999999998e-05 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(5.1 e463,441 5:8c81c6a2:::benchmark_data_smithi192_104167_object16628:head,420'7624)", 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:29:00.278266+0000", 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 131.099791525, 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51104331599999997, 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278266+0000", 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278266+0000", 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278267+0000", 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.466e-06 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:11.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278277+0000", 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7200000000000001e-06 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789046+0000", 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51076909400000003 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789062+0000", 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5509999999999999e-05 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789183+0000", 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120825 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789200+0000", 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7036000000000001e-05 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789309+0000", 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109665 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.492 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.493 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.493 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:11.493 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:11.493 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:31:11.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:11.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:11.675 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:31:11.675 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:31:11.700 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:11.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:11.707 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:31:11.707 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:31:11.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:11.737 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:31:11.737 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:31:11.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:31:11.763 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:31:11.788 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:31:11.788 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:31:11.814 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:31:11.815 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:31:11.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:11.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:31:11.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:31:11.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:11.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:31:11.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:31:11.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:31:11.910 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:31:11.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:11 smithi151 ceph-mon[28965]: pgmap v1320: 37 pgs: 37 active+clean; 6.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 7.6 MiB/s wr, 924 op/s 2024-06-27T21:31:11.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4246019778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:11.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:31:11.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:31:11.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:11.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:31:11.970 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:31:11.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:11.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:12.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:31:12.002 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:31:12.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:12.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:12.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:12.212 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:11 smithi192 ceph-mon[32956]: pgmap v1320: 37 pgs: 37 active+clean; 6.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 7.6 MiB/s wr, 924 op/s 2024-06-27T21:31:12.212 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4246019778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:12.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:12.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:31:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:12.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:12.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:11 smithi190 ceph-mon[32247]: pgmap v1320: 37 pgs: 37 active+clean; 6.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 7.6 MiB/s wr, 924 op/s 2024-06-27T21:31:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4246019778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:12.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:12.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:12.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:12.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:12.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:12.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:12.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:12.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:12.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:12.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:12.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:12.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:12.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:13.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:13.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:13.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:31:13.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:13.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:13.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:13.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:13.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:13.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:13.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:13.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:13.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:13.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:13.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:13.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:13.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:13 smithi151 ceph-mon[28965]: pgmap v1321: 37 pgs: 37 active+clean; 6.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s wr, 786 op/s 2024-06-27T21:31:13.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:13.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:13.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:13 smithi192 ceph-mon[32956]: pgmap v1321: 37 pgs: 37 active+clean; 6.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s wr, 786 op/s 2024-06-27T21:31:14.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:14.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:14.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:14.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:13 smithi190 ceph-mon[32247]: pgmap v1321: 37 pgs: 37 active+clean; 6.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s wr, 786 op/s 2024-06-27T21:31:14.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:14.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:14.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:14.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:14.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:14.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:14.733 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:14.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:14.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:14.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:14.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:14.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:moved 10 / 111 (9.00901%) 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:avg 9.25 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.68095 -> 2.04634 (expected baseline 2.9119) 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 4 -> 4 pgs (0.432432 -> 0.432432 * mean) 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.6 with 13 -> 9 pgs (1.40541 -> 0.972973 * mean) 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-06-27T21:31:14.971 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0209 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0230 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 weight 1.0000 -> 0.7262 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 weight 1.0000 -> 0.7849 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 weight 1.0000 -> 0.8077 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.2276 -> 0.2982 2024-06-27T21:31:14.972 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-06-27T21:31:14.983 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:14.983 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:14.983 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:14 smithi151 ceph-mon[28965]: osdmap e509: 12 total, 12 up, 12 in 2024-06-27T21:31:14.984 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:31:14 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:31:14.969+0000 7fc312b3e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-27T21:31:15.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:15.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:15.112 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:15.112 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:15.112 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:14 smithi192 ceph-mon[32956]: osdmap e509: 12 total, 12 up, 12 in 2024-06-27T21:31:15.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:15.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:15.233 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:15.233 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:15.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:14 smithi190 ceph-mon[32247]: osdmap e509: 12 total, 12 up, 12 in 2024-06-27T21:31:15.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:15.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:15.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:15.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:15.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:15.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:15.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:15.760 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-27T21:31:15.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-06-27T21:31:15.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:15.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:15.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:15.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:15.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:16.020 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:15 smithi151 ceph-mon[28965]: pgmap v1323: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s 2024-06-27T21:31:16.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:16.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:16.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:31:16.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:16.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:15 smithi192 ceph-mon[32956]: pgmap v1323: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s 2024-06-27T21:31:16.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:16.253 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:31:16.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:15 smithi190 ceph-mon[32247]: pgmap v1323: 37 pgs: 37 active+clean; 6.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s 2024-06-27T21:31:16.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:16.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:16.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:16.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:16.560 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:16.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:16.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:16.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:16.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:16.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:16.877 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:16.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:16.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:16.980 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:16 smithi192 ceph-mon[32956]: from='client.26173 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:17.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:17.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:17.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:17.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:17.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:16 smithi151 ceph-mon[28965]: from='client.26173 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:17.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:17.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:16 smithi190 ceph-mon[32247]: from='client.26173 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:17.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:17.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:17.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:17.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:17.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:17.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:17.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:17.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:17.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:17.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:17.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:17.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:18.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:17 smithi192 ceph-mon[32956]: pgmap v1324: 37 pgs: 37 active+clean; 5.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-06-27T21:31:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:17 smithi192 ceph-mon[32956]: osdmap e510: 12 total, 12 up, 12 in 2024-06-27T21:31:18.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:18.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:17 smithi151 ceph-mon[28965]: pgmap v1324: 37 pgs: 37 active+clean; 5.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-06-27T21:31:18.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:18.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:18.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:17 smithi151 ceph-mon[28965]: osdmap e510: 12 total, 12 up, 12 in 2024-06-27T21:31:18.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:18.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:17 smithi190 ceph-mon[32247]: pgmap v1324: 37 pgs: 37 active+clean; 5.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-06-27T21:31:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:17 smithi190 ceph-mon[32247]: osdmap e510: 12 total, 12 up, 12 in 2024-06-27T21:31:18.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:18.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:18.332 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:18.332 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:18.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:18.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:18.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:18.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:18.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:18.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:18.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:18.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:18.912 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:31:18.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:18.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:18.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:19.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:19.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:19.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:31:19.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:19.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:18 smithi151 ceph-mon[28965]: pgmap v1326: 37 pgs: 37 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-06-27T21:31:19.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:18 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:19.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:18 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:19.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4150736682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:31:19.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:18 smithi190 ceph-mon[32247]: pgmap v1326: 37 pgs: 37 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-06-27T21:31:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:18 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:18 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4150736682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:31:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:18 smithi192 ceph-mon[32956]: pgmap v1326: 37 pgs: 37 active+clean; 5.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s 2024-06-27T21:31:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:18 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:18 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4150736682' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:31:19.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:19.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:19.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:19.445 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:19.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:19.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:19.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:19.668 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:19.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:19.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:19.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:19.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4150736682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: osdmap e511: 12 total, 12 up, 12 in 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:31:19.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:19 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:31:19.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:19.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:19.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:20.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:20.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:20.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4150736682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: osdmap e511: 12 total, 12 up, 12 in 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:31:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:19 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:31:20.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:20.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4150736682' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:31:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: osdmap e511: 12 total, 12 up, 12 in 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:31:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:19 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:31:20.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:20.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:20.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:20.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:20.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:20.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:20.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:20.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:20.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:20.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:20.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:20.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:20.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:21.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:21.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:21.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:20 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:31:21.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:20 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:31:21.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:20 smithi151 ceph-mon[28965]: pgmap v1328: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s 2024-06-27T21:31:21.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:20 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:31:21.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:20 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:31:21.204 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:21.204 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:21.204 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:20 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:31:21.204 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:20 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:31:21.204 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:20 smithi192 ceph-mon[32956]: pgmap v1328: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s 2024-06-27T21:31:21.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:20 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:31:21.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:20 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:31:21.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:21.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:20 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:31:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:20 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:31:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:20 smithi190 ceph-mon[32247]: pgmap v1328: 37 pgs: 37 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 940 op/s 2024-06-27T21:31:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:20 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:31:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:20 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:31:21.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:21.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:21.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:21.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:21.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:21.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:31:21.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:21.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:21.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:21.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:31:21.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:21.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:21.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:moved 9 / 111 (8.10811%) 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:avg 9.25 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.68095 -> 2.38485 (expected baseline 2.9119) 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 4 -> 4 pgs (0.432432 -> 0.432432 * mean) 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.6 with 13 -> 8 pgs (1.40541 -> 0.864865 * mean) 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-06-27T21:31:21.928 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0199 2024-06-27T21:31:21.929 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0219 2024-06-27T21:31:21.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 weight 1.0000 -> 0.7207 2024-06-27T21:31:21.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 weight 1.0000 -> 0.7771 2024-06-27T21:31:21.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 1.0000 -> 0.8012 2024-06-27T21:31:21.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.2276 -> 0.2944 2024-06-27T21:31:22.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:22.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:22.106 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:22.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:22.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:31:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:31:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:31:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:31:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: from='client.16515 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:22.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]: dispatch 2024-06-27T21:31:22.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:21 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]: dispatch 2024-06-27T21:31:22.228 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:31:22.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:31:22.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:31:22.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:31:22.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: from='client.16515 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:22.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]: dispatch 2024-06-27T21:31:22.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:21 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]: dispatch 2024-06-27T21:31:22.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: from='client.16515 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]: dispatch 2024-06-27T21:31:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:21 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]: dispatch 2024-06-27T21:31:22.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:22.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:22.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:22.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:22.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:22.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:22.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:22.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:22.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:22.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:22.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:22.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:23.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:23.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:23.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:23.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]': finished 2024-06-27T21:31:23.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: osdmap e512: 12 total, 12 up, 12 in 2024-06-27T21:31:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: pgmap v1330: 37 pgs: 37 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s 2024-06-27T21:31:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:31:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:31:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:31:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:22 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:31:23.231 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-06-27T21:31:23.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-06-27T21:31:23.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:23.240 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]': finished 2024-06-27T21:31:23.240 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: osdmap e512: 12 total, 12 up, 12 in 2024-06-27T21:31:23.240 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: pgmap v1330: 37 pgs: 37 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s 2024-06-27T21:31:23.240 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:31:23.240 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:31:23.240 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:31:23.241 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:22 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:31:23.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'19297','5':'47230','6':'50928','9':'52508'}"}]': finished 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: osdmap e512: 12 total, 12 up, 12 in 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: pgmap v1330: 37 pgs: 37 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:31:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:22 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:31:23.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:23.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:23.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:23.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:23.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:23.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:23.657 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:23.658 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:23.682 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:31:23.682 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:31:23.707 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:31:23.707 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:31:23.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:31:23.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:31:23.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:23.744 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:23.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:23.765 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:23.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:31:23.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:31:23.800 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:31:23.801 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:31:23.822 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:31:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:31:23.844 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:31:23.845 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:31:23.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:23.868 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:31:23.868 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:31:23.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:23.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:23.889 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:31:23.889 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:31:23.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:31:23.911 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:31:23.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:31:23.933 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:31:23.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:31:23.967 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:31:23.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:24.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:24.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:24.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:24.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:23 smithi151 ceph-mon[28965]: osdmap e513: 12 total, 12 up, 12 in 2024-06-27T21:31:24.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:23 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:31:24.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:23 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:31:24.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:23 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:31:24.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:23 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:31:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2432937723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:24.235 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:23 smithi192 ceph-mon[32956]: osdmap e513: 12 total, 12 up, 12 in 2024-06-27T21:31:24.235 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:23 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:31:24.235 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:23 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:31:24.235 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:23 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:31:24.236 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:23 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:31:24.236 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2432937723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:24.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:23 smithi190 ceph-mon[32247]: osdmap e513: 12 total, 12 up, 12 in 2024-06-27T21:31:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:23 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:31:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:23 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:31:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:23 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:31:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:23 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:31:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2432937723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:24.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:31:24.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:24.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:24.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:24.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:24.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:24.584 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:24.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:24.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:24.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:24.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:24.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:24.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:24.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:25.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:24 smithi192 ceph-mon[32956]: osdmap e514: 12 total, 12 up, 12 in 2024-06-27T21:31:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:24 smithi192 ceph-mon[32956]: pgmap v1333: 37 pgs: 2 remapped+peering, 35 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 38 KiB/s, 0 objects/s recovering 2024-06-27T21:31:25.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:25.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:25.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:25.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:24 smithi151 ceph-mon[28965]: osdmap e514: 12 total, 12 up, 12 in 2024-06-27T21:31:25.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:24 smithi151 ceph-mon[28965]: pgmap v1333: 37 pgs: 2 remapped+peering, 35 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 38 KiB/s, 0 objects/s recovering 2024-06-27T21:31:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:24 smithi190 ceph-mon[32247]: osdmap e514: 12 total, 12 up, 12 in 2024-06-27T21:31:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:24 smithi190 ceph-mon[32247]: pgmap v1333: 37 pgs: 2 remapped+peering, 35 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 38 KiB/s, 0 objects/s recovering 2024-06-27T21:31:25.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:25.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:25.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:25.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:25.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:25.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:25.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:25.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:25.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:25.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:25.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:25.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:25.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:25.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:26.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:26.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:26.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:26.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:26.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:26.279 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:31:26.279 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:31:26.279 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:31:26.279 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:31:26.279 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:31:26.279 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:26 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 17/264234 objects degraded (0.006%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:31:26.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:26.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:26.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:26.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:26 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 17/264234 objects degraded (0.006%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:31:26.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:26.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:26 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 17/264234 objects degraded (0.006%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:31:26.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:26.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:26.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:26.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:26.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:26.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:26.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:26.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:26.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:26.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:31:27.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:31:27.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:27.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:27.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:27.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:27 smithi192 ceph-mon[32956]: pgmap v1334: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 34 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 17/264234 objects degraded (0.006%); 5349/264234 objects misplaced (2.024%); 139 KiB/s, 1 objects/s recovering 2024-06-27T21:31:27.313 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:27 smithi151 ceph-mon[28965]: pgmap v1334: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 34 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 17/264234 objects degraded (0.006%); 5349/264234 objects misplaced (2.024%); 139 KiB/s, 1 objects/s recovering 2024-06-27T21:31:27.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:27.358 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:31:27.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:27.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:27.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:27.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:27 smithi190 ceph-mon[32247]: pgmap v1334: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 34 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 17/264234 objects degraded (0.006%); 5349/264234 objects misplaced (2.024%); 139 KiB/s, 1 objects/s recovering 2024-06-27T21:31:27.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:27.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:27.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:27.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:27.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:27.890 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:27.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:27.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:28.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:28.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:28.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:28.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:28.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:28.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:28.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:28.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:28.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:28.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:28.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:28.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:28.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:28.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:28.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:29.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:29.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:58360 4.8 4:1fb1449d:::benchmark_data_smithi192_88961_object58359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-06-27T21:31:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 595.31159659699995, 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78704928399999996, 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 58360 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812364+0000", 2024-06-27T21:31:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3799999999999996e-07 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812404+0000", 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9298999999999998e-05 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812413+0000", 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6090000000000007e-06 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812419+0000", 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4029999999999998e-06 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812825+0000", 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040641499999999998 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.812925+0000", 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9977999999999997e-05 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.813074+0000", 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000149226 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.814127+0000", 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010522940000000001 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815268+0000", 2024-06-27T21:31:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011417160000000001 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599347+0000", 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78407841099999998 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599381+0000", 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4666000000000003e-05 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599413+0000", 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1429e-05 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:58360 4.8 e314/282 ondisk, result = 0)", 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 595.30882456500001, 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78428261799999999, 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815136+0000", 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000002e-07 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.815139+0000", 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.354e-06 2024-06-27T21:31:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599259+0000", 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78411979600000004 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599265+0000", 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599336+0000", 2024-06-27T21:31:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0346999999999996e-05 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599343+0000", 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1319999999999996e-06 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.599418+0000", 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5303999999999996e-05 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.25312.0:58365 4.f e314/282 4:fc83d7dd:::benchmark_data_smithi192_88961_object58364:head v 314'3626, mlcod=314'3624)", 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 595.30447962300002, 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29582368799999997, 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:31:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819481+0000", 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9299999999999997e-07 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:33.819529+0000", 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8269000000000002e-05 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113508+0000", 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29397816399999999 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113516+0000", 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6770000000000001e-06 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113587+0000", 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0828999999999996e-05 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.113604+0000", 2024-06-27T21:31:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6643999999999999e-05 2024-06-27T21:31:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.115270+0000", 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016662960000000001 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:34.115304+0000", 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4116000000000003e-05 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:66868 4.9 4:98080da8:::benchmark_data_smithi192_88961_object66867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:44.949612+0000", 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 584.174348065, 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.419813296, 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 66868 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949612+0000", 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949612+0000", 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949614+0000", 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.525e-06 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949662+0000", 2024-06-27T21:31:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8232000000000002e-05 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949673+0000", 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1382e-05 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949678+0000", 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2360000000000004e-06 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949724+0000", 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6038000000000002e-05 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.949863+0000", 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013807500000000001 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.950013+0000", 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000150247 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952324+0000", 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002310744 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.957959+0000", 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056356289999999996 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369353+0000", 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41139374899999998 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369385+0000", 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2141999999999998e-05 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369425+0000", 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0296999999999997e-05 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25312.0:66868 4.9 e320/282 ondisk, result = 0)", 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:44.952331+0000", 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 584.17162944100005, 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41710191200000002, 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952331+0000", 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952331+0000", 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952332+0000", 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.952338+0000", 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.387e-06 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369292+0000", 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41695419099999997 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369297+0000", 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9479999999999999e-06 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369346+0000", 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8989000000000001e-05 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369350+0000", 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8279999999999996e-06 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.369433+0000", 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2727999999999998e-05 2024-06-27T21:31:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:66878 4.3 4:c506cff3:::benchmark_data_smithi192_88961_object66877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:21:44.970564+0000", 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 584.15339591999998, 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33568430300000002, 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 66878 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970564+0000", 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970564+0000", 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970566+0000", 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6789999999999999e-06 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970586+0000", 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0545999999999999e-05 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970605+0000", 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8598999999999999e-05 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970612+0000", 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3860000000000001e-06 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970675+0000", 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2163999999999996e-05 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970808+0000", 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013358099999999999 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.970951+0000", 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014247500000000001 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.973362+0000", 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024116570000000002 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:44.980807+0000", 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0074446 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.306177+0000", 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.32536981100000001 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.306218+0000", 2024-06-27T21:31:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1257000000000002e-05 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:21:45.306249+0000", 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0548e-05 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:103647 4.9 4:98ce20a8:::benchmark_data_smithi192_88961_object12908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-06-27T21:31:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:26.909872+0000", 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 542.21408794000001, 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30295640299999999, 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 103647 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909872+0000", 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909872+0000", 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909873+0000", 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.099e-06 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909877+0000", 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0539999999999996e-06 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909888+0000", 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1126e-05 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909894+0000", 2024-06-27T21:31:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0270000000000001e-06 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.909946+0000", 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1261999999999998e-05 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.910070+0000", 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012383199999999999 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,3", 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.910186+0000", 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011657 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.912728+0000", 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002542304 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:26.913604+0000", 2024-06-27T21:31:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00087590699999999999 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.212665+0000", 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.29906064999999998 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.212787+0000", 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012225199999999999 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:27.212829+0000", 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1319999999999997e-05 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:130464 4.8 4:198f8843:::benchmark_data_smithi192_88961_object39725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 513.15378830700001, 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.28377075699999998, 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 130464 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970172+0000", 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6400000000000002e-07 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970177+0000", 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1219999999999997e-06 2024-06-27T21:31:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970186+0000", 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6900000000000004e-06 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970191+0000", 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0660000000000003e-06 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970227+0000", 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5488000000000003e-05 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970302+0000", 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4805e-05 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.970364+0000", 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2287999999999997e-05 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.973331+0000", 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029675000000000001 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:55.973628+0000", 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029691 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.253863+0000", 2024-06-27T21:31:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.28023458699999998 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.253898+0000", 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4873000000000003e-05 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.253943+0000", 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4764e-05 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:130521 4.1 4:8d4b026c:::benchmark_data_smithi192_88961_object39782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:22:56.564874+0000", 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 512.55908571600003, 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41357545899999998, 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 130521 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564874+0000", 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564874+0000", 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564875+0000", 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3600000000000003e-07 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564878+0000", 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.768e-06 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564886+0000", 2024-06-27T21:31:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2269999999999993e-06 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564891+0000", 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9429999999999999e-06 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.564933+0000", 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2069e-05 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.565024+0000", 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1246e-05 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,6", 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.565094+0000", 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9430000000000004e-05 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.568308+0000", 2024-06-27T21:31:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003213905 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.568317+0000", 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1740000000000006e-06 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.978368+0000", 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41005131299999997 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.978408+0000", 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9920999999999999e-05 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:22:56.978450+0000", 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1727000000000002e-05 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25312.0:163716 4.3 4:ce21cfad:::benchmark_data_smithi192_88961_object72977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:23:43.970070+0000", 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 465.153890504, 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70507940300000005, 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25312", 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163716 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970070+0000", 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970070+0000", 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970071+0000", 2024-06-27T21:31:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5149999999999999e-06 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970078+0000", 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1400000000000002e-06 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970092+0000", 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3871e-05 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970100+0000", 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8259999999999993e-06 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970157+0000", 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6589999999999999e-05 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970293+0000", 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136756 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.970425+0000", 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131297 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.974073+0000", 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036481590000000002 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:43.974485+0000", 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041243200000000001 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.675074+0000", 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70058905100000002 2024-06-27T21:31:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.675115+0000", 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0936999999999998e-05 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:23:44.675149+0000", 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3828999999999997e-05 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152341 5.d 5:bebb6ac0:::benchmark_data_smithi192_104167_object46300:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902303+0000", 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 241.22165682400001, 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35944219999999999, 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152341 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902303+0000", 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902303+0000", 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902305+0000", 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6750000000000001e-06 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902312+0000", 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4719999999999998e-06 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902326+0000", 2024-06-27T21:31:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3736e-05 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902338+0000", 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1764e-05 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906260+0000", 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039215999999999999 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906274+0000", 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4262e-05 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.261595+0000", 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35532122599999999 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.261746+0000", 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015046499999999999 2024-06-27T21:31:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152348 5.d 5:bda439c2:::benchmark_data_smithi192_104167_object46307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902363+0000", 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 241.22159686800001, 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36148766799999998, 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152348 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902363+0000", 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902363+0000", 2024-06-27T21:31:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902364+0000", 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902366+0000", 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0229999999999999e-06 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902371+0000", 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0520000000000001e-06 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902374+0000", 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4599999999999999e-06 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906311+0000", 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003936602 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906316+0000", 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5169999999999999e-06 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.263731+0000", 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35741499399999999 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.263851+0000", 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119704 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.199 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152349 5.d 5:b479a3b3:::benchmark_data_smithi192_104167_object46308:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 241.22158143799999, 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36166314599999999, 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:29.200 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152349 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902379+0000", 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.36e-07 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902380+0000", 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.536e-06 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902385+0000", 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.374e-06 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902388+0000", 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9399999999999998e-06 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906356+0000", 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039686519999999996 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906364+0000", 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7139999999999993e-06 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.263887+0000", 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35752254500000002 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264042+0000", 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015514899999999999 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.204 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:152370 5.d 5:b37fea35:::benchmark_data_smithi192_104167_object46329:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e439)", 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 241.22156897299999, 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37976241999999999, 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:29.205 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 152370 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902391+0000", 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.41e-07 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902393+0000", 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.672e-06 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902396+0000", 2024-06-27T21:31:29.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3129999999999999e-06 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.902399+0000", 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8679999999999999e-06 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906420+0000", 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040211190000000001 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:27.906429+0000", 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6139999999999992e-06 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264078+0000", 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35764881300000001 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264194+0000", 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011598499999999999 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,5", 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.264288+0000", 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4084000000000002e-05 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.276652+0000", 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012364557 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.280074+0000", 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00342173 2024-06-27T21:31:29.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.280164+0000", 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9519999999999997e-05 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.280207+0000", 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3340999999999997e-05 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:28.282154+0000", 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019465629999999999 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:98544 6.2 e500/467 6:4b1cd652:::benchmark_data_smithi192_117994_object98543:head v 500'6088, mlcod=500'6085)", 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:30:49.885245+0000", 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 39.238714952999999, 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.13188586999999999, 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.212 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:49.885245+0000", 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:49.885245+0000", 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:49.885246+0000", 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8400000000000004e-07 2024-06-27T21:31:29.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:49.885297+0000", 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0855e-05 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:50.014276+0000", 2024-06-27T21:31:29.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12897952800000001 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:50.014283+0000", 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0500000000000003e-06 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:50.014334+0000", 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0361000000000001e-05 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:50.014349+0000", 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5252000000000001e-05 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:50.017079+0000", 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027301959999999998 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:50.017131+0000", 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1944000000000003e-05 2024-06-27T21:31:29.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:99896 6.2 e501/467 6:4594c44a:::benchmark_data_smithi192_117994_object99895:head v 501'6162, mlcod=501'6159)", 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:30:51.222090+0000", 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.901870293000002, 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.083164887000000007, 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.220 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.222090+0000", 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.222090+0000", 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.222091+0000", 2024-06-27T21:31:29.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.105e-06 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.222125+0000", 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4159999999999998e-05 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.302530+0000", 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.080405074000000007 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.302538+0000", 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9780000000000005e-06 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.302595+0000", 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6925000000000001e-05 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.302616+0000", 2024-06-27T21:31:29.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0800000000000001e-05 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.305213+0000", 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025973049999999998 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:30:51.305255+0000", 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1539999999999999e-05 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.224 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:129431 6.2 e514/467 6:4675a041:::benchmark_data_smithi192_117994_object21837:head v 514'7990, mlcod=514'7987)", 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:31:27.824587+0000", 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.2993732179999999, 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.104937848, 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.824587+0000", 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.824587+0000", 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.824588+0000", 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8899999999999999e-07 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.824592+0000", 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8520000000000003e-06 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.926331+0000", 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.101738282 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.926340+0000", 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6490000000000003e-06 2024-06-27T21:31:29.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.926410+0000", 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9847999999999999e-05 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.926429+0000", 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8355999999999998e-05 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.929492+0000", 2024-06-27T21:31:29.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030629960000000001 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.929525+0000", 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3176000000000002e-05 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:129528 6.2 e514/467 6:40236f4b:::benchmark_data_smithi192_117994_object21934:head v 514'7991, mlcod=514'7989)", 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:31:27.977105+0000", 2024-06-27T21:31:29.229 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.146855197, 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.111679004, 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.977105+0000", 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.977105+0000", 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.977106+0000", 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2900000000000002e-07 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:27.977111+0000", 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4199999999999998e-06 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.085631+0000", 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.10851962800000001 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.085638+0000", 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5730000000000002e-06 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.085693+0000", 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4197000000000003e-05 2024-06-27T21:31:29.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.085716+0000", 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2963e-05 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.088746+0000", 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003030376 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.088784+0000", 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7917999999999998e-05 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:129726 6.2 e514/467 6:41a52def:::benchmark_data_smithi192_117994_object22132:head v 514'8000, mlcod=514'7997)", 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:31:28.256864+0000", 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.86709625700000004, 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12566274599999999, 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.234 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.256864+0000", 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.256864+0000", 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.256865+0000", 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3900000000000003e-07 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.256870+0000", 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6690000000000003e-06 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378917+0000", 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.122046354 2024-06-27T21:31:29.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378922+0000", 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0170000000000004e-06 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378973+0000", 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0688e-05 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.379000+0000", 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7475e-05 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.382492+0000", 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034922310000000002 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.382527+0000", 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4372999999999998e-05 2024-06-27T21:31:29.238 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:129801 6.2 e514/467 6:4e43f6d6:::benchmark_data_smithi192_117994_object22207:head v 514'8001, mlcod=514'7998)", 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:31:28.378896+0000", 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.74506427900000005, 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.126568456, 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:31:29.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378896+0000", 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378896+0000", 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378897+0000", 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0500000000000002e-07 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.378902+0000", 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3399999999999997e-06 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.501708+0000", 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.122805884 2024-06-27T21:31:29.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.501716+0000", 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9689999999999994e-06 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.501773+0000", 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7469000000000002e-05 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:31:29.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.501793+0000", 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9873000000000001e-05 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.505419+0000", 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036259500000000002 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:28.505464+0000", 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5166000000000001e-05 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.243 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:31:29.244 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:31:29.244 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:31:29.244 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:29.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:29.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:29.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:29.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:29 smithi151 ceph-mon[28965]: pgmap v1335: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 33 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 112/261087 objects degraded (0.043%); 16230/261087 objects misplaced (6.216%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:31:29.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:29.512 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:29 smithi192 ceph-mon[32956]: pgmap v1335: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 33 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 112/261087 objects degraded (0.043%); 16230/261087 objects misplaced (6.216%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:31:29.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:29 smithi190 ceph-mon[32247]: pgmap v1335: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 33 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 112/261087 objects degraded (0.043%); 16230/261087 objects misplaced (6.216%); 1.9 MiB/s, 30 objects/s recovering 2024-06-27T21:31:29.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:29.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:29.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:29.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:29.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:29.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:29.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:29.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:30.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:30.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:31:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:30.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:30.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:31:30.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:30 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:31:30.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:30.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:31:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:30 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:31:30.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:30.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:30.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:30.495 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:31:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:31:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:31:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:31:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:31:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:31:30.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:31:30.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:30 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:31:30.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:30.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:30.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:30.667 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:30.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:30.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:30.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:30.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:30.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:31.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:31.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:31.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:31.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:31.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:31.298 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:31:31.302 INFO:teuthology.orchestra.run.smithi151.stdout:moved 8 / 111 (7.20721%) 2024-06-27T21:31:31.302 INFO:teuthology.orchestra.run.smithi151.stdout:avg 9.25 2024-06-27T21:31:31.302 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.38485 -> 2.91905 (expected baseline 2.9119) 2024-06-27T21:31:31.302 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 4 -> 4 pgs (0.432432 -> 0.432432 * mean) 2024-06-27T21:31:31.302 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 14 -> 14 pgs (1.51351 -> 1.51351 * mean) 2024-06-27T21:31:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:31:31.303 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0188 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0207 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 weight 0.7207 -> 0.5223 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 weight 0.7771 -> 0.6059 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.2944 -> 0.3846 2024-06-27T21:31:31.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 0.8012 -> 0.6502 2024-06-27T21:31:31.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:31.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:31.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:31.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:31.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:31.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:31 smithi192 ceph-mon[32956]: pgmap v1336: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 230/256695 objects degraded (0.090%); 21322/256695 objects misplaced (8.306%); 3.4 MiB/s, 54 objects/s recovering 2024-06-27T21:31:31.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:31 smithi192 ceph-mon[32956]: from='client.26120 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:31.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]: dispatch 2024-06-27T21:31:31.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:31 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]: dispatch 2024-06-27T21:31:31.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/152912256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:31.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:31.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:31.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:31.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:31 smithi151 ceph-mon[28965]: pgmap v1336: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 230/256695 objects degraded (0.090%); 21322/256695 objects misplaced (8.306%); 3.4 MiB/s, 54 objects/s recovering 2024-06-27T21:31:31.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:31 smithi151 ceph-mon[28965]: from='client.26120 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:31.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]: dispatch 2024-06-27T21:31:31.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:31 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]: dispatch 2024-06-27T21:31:31.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/152912256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:31.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:31 smithi190 ceph-mon[32247]: pgmap v1336: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 667 op/s; 230/256695 objects degraded (0.090%); 21322/256695 objects misplaced (8.306%); 3.4 MiB/s, 54 objects/s recovering 2024-06-27T21:31:31.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:31 smithi190 ceph-mon[32247]: from='client.26120 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:31.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]: dispatch 2024-06-27T21:31:31.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:31 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]: dispatch 2024-06-27T21:31:31.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/152912256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:31.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:31.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:31.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:31.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:31.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:32.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:32.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:32.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:32.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:32.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:32.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:32.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:32.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:32 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:31:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:32 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]': finished 2024-06-27T21:31:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/152912256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:32 smithi151 ceph-mon[28965]: osdmap e515: 12 total, 12 up, 12 in 2024-06-27T21:31:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:32 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:31:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:32 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]': finished 2024-06-27T21:31:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/152912256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:32 smithi190 ceph-mon[32247]: osdmap e515: 12 total, 12 up, 12 in 2024-06-27T21:31:32.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:32 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:31:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:32 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'25205','5':'34227','6':'39709','9':'42611'}"}]': finished 2024-06-27T21:31:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/152912256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:32.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:32 smithi192 ceph-mon[32956]: osdmap e515: 12 total, 12 up, 12 in 2024-06-27T21:31:32.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:32.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:32.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:32.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:32.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:32.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:32.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:32.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:33.026 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:31:33.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:31:33.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:33.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:31:33.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:31:33.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:33.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:31:33.071 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:31:33.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:31:33.094 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:31:33.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:31:33.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:31:33.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:31:33.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:31:33.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:33.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:31:33.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:31:33.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:33.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:31:33.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:31:33.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:31:33.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:31:33.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:31:33.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:31:33.259 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:31:33.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:31:33.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:33.283 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:31:33.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:31:33.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:33.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:33.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:33.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:33 smithi151 ceph-mon[28965]: pgmap v1338: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 439 op/s; 271/255405 objects degraded (0.106%); 21141/255405 objects misplaced (8.277%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:31:33.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:33 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 271/255405 objects degraded (0.106%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:33.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:33 smithi151 ceph-mon[28965]: osdmap e516: 12 total, 12 up, 12 in 2024-06-27T21:31:33.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/284516890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:33.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:33.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:33 smithi192 ceph-mon[32956]: pgmap v1338: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 439 op/s; 271/255405 objects degraded (0.106%); 21141/255405 objects misplaced (8.277%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:31:33.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:33 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 271/255405 objects degraded (0.106%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:33.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:33 smithi192 ceph-mon[32956]: osdmap e516: 12 total, 12 up, 12 in 2024-06-27T21:31:33.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/284516890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:33.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:33.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:33.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:33.721 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-06-27T21:31:33.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-06-27T21:31:33.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:33 smithi190 ceph-mon[32247]: pgmap v1338: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 439 op/s; 271/255405 objects degraded (0.106%); 21141/255405 objects misplaced (8.277%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:31:33.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:33 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 271/255405 objects degraded (0.106%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:33.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:33 smithi190 ceph-mon[32247]: osdmap e516: 12 total, 12 up, 12 in 2024-06-27T21:31:33.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/284516890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:33.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:33.820 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:33.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:33.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:33.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:34.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:34.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:34.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:34.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:34.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:34.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:34.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:34 smithi190 ceph-mon[32247]: 1.10 deep-scrub starts 2024-06-27T21:31:34.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:34 smithi151 ceph-mon[28965]: 1.10 deep-scrub starts 2024-06-27T21:31:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:34 smithi151 ceph-mon[28965]: 1.10 deep-scrub ok 2024-06-27T21:31:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:34 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:31:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:34 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:31:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:34 smithi151 ceph-mon[28965]: osdmap e517: 12 total, 12 up, 12 in 2024-06-27T21:31:34.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:34.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:34.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:34.612 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:34.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:31:34.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:34 smithi190 ceph-mon[32247]: 1.10 deep-scrub ok 2024-06-27T21:31:34.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:34 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:31:34.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:34 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:31:34.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:34 smithi190 ceph-mon[32247]: osdmap e517: 12 total, 12 up, 12 in 2024-06-27T21:31:34.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:34.730 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:34.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:34 smithi192 ceph-mon[32956]: 1.10 deep-scrub starts 2024-06-27T21:31:34.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:34 smithi192 ceph-mon[32956]: 1.10 deep-scrub ok 2024-06-27T21:31:34.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:34 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:31:34.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:34 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:31:34.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:34 smithi192 ceph-mon[32956]: osdmap e517: 12 total, 12 up, 12 in 2024-06-27T21:31:34.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:34.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:34.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:34.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:34.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:35.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:35.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:35.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:35.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:35.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:35.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:35.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:35.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:35.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:35 smithi190 ceph-mon[32247]: pgmap v1341: 37 pgs: 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 490 op/s; 116/252261 objects degraded (0.046%); 15483/252261 objects misplaced (6.138%); 6.2 MiB/s, 99 objects/s recovering 2024-06-27T21:31:35.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:35.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:35.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:35 smithi151 ceph-mon[28965]: pgmap v1341: 37 pgs: 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 490 op/s; 116/252261 objects degraded (0.046%); 15483/252261 objects misplaced (6.138%); 6.2 MiB/s, 99 objects/s recovering 2024-06-27T21:31:35.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:35.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:35.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:35 smithi192 ceph-mon[32956]: pgmap v1341: 37 pgs: 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 490 op/s; 116/252261 objects degraded (0.046%); 15483/252261 objects misplaced (6.138%); 6.2 MiB/s, 99 objects/s recovering 2024-06-27T21:31:35.847 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:35.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:35.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:35.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:36.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:36.025 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:36.025 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:36.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:36.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.179 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:36.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:36.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:36.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:36.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:36.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:36.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:31:36.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:36.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:36.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:36.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:36.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:31:36.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:36.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:36.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:37.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:37.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:37.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:37.150 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:31:37.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:37.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:37.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:37.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:37.434 INFO:teuthology.orchestra.run.smithi151.stdout:moved 4 / 111 (3.6036%) 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:avg 9.25 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.91905 -> 3.05846 (expected baseline 2.9119) 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 4 -> 5 pgs (0.432432 -> 0.540541 * mean) 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 14 -> 14 pgs (1.51351 -> 1.51351 * mean) 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0185 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0203 2024-06-27T21:31:37.435 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 weight 0.5223 -> 0.4723 2024-06-27T21:31:37.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 weight 0.6059 -> 0.5559 2024-06-27T21:31:37.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.3846 -> 0.4346 2024-06-27T21:31:37.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 weight 1.0000 -> 0.9500 2024-06-27T21:31:37.436 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-06-27T21:31:37.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:37.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:37.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:37 smithi190 ceph-mon[32247]: pgmap v1342: 37 pgs: 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 349 op/s; 178/250410 objects degraded (0.071%); 25819/250410 objects misplaced (10.311%); 5.6 MiB/s, 90 objects/s recovering 2024-06-27T21:31:37.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:37.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:37.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:37 smithi151 ceph-mon[28965]: pgmap v1342: 37 pgs: 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 349 op/s; 178/250410 objects degraded (0.071%); 25819/250410 objects misplaced (10.311%); 5.6 MiB/s, 90 objects/s recovering 2024-06-27T21:31:37.679 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:31:37 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:31:37.433+0000 7fc312b3e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-06-27T21:31:37.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:37.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:37 smithi192 ceph-mon[32956]: pgmap v1342: 37 pgs: 1 remapped+peering, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 5.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 349 op/s; 178/250410 objects degraded (0.071%); 25819/250410 objects misplaced (10.311%); 5.6 MiB/s, 90 objects/s recovering 2024-06-27T21:31:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:37.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:37.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:37.905 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:37.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:38.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:38.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:38.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:38.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:38.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:38.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:38.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:38.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:38.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:38 smithi190 ceph-mon[32247]: from='client.16542 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:38.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:38.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:38.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:38 smithi151 ceph-mon[28965]: from='client.16542 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:38.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:38.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:38.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:38 smithi192 ceph-mon[32956]: from='client.16542 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:38.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:38.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:38.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:38.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:38.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:39.084 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:31:39.085 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:31:39.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:39.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:39.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:39.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:39.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:39.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:39.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:39 smithi151 ceph-mon[28965]: pgmap v1343: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 185/247623 objects degraded (0.075%); 41268/247623 objects misplaced (16.666%); 7.2 MiB/s, 116 objects/s recovering 2024-06-27T21:31:39.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 185/247623 objects degraded (0.075%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:39.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:39.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:39 smithi190 ceph-mon[32247]: pgmap v1343: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 185/247623 objects degraded (0.075%); 41268/247623 objects misplaced (16.666%); 7.2 MiB/s, 116 objects/s recovering 2024-06-27T21:31:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 185/247623 objects degraded (0.075%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:39.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:39.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:39.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:39.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:39.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:39 smithi192 ceph-mon[32956]: pgmap v1343: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 185/247623 objects degraded (0.075%); 41268/247623 objects misplaced (16.666%); 7.2 MiB/s, 116 objects/s recovering 2024-06-27T21:31:39.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 185/247623 objects degraded (0.075%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:39.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:39.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:39.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:40.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:40.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:40.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:40.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:31:40.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:40.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:40.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:40.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:40.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:40.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:40.508 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:31:40.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:40.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:40.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:40.677 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:31:40.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2466516327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:31:40.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:40.715 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-06-27T21:31:40.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-06-27T21:31:40.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:40.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:40.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2466516327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:31:40.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:40.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:40.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:41.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2466516327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:31:41.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:41.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:41.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:41.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:41.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:41.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:41.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:41.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:41.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:41.639 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:41.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:41.771 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:41 smithi151 ceph-mon[28965]: pgmap v1344: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 252/243288 objects degraded (0.104%); 40608/243288 objects misplaced (16.691%); 11 MiB/s, 172 objects/s recovering 2024-06-27T21:31:41.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:41 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:41.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2466516327' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:31:41.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:41 smithi151 ceph-mon[28965]: osdmap e518: 12 total, 12 up, 12 in 2024-06-27T21:31:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:41.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:41.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:41.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:41.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:42.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:41 smithi190 ceph-mon[32247]: pgmap v1344: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 252/243288 objects degraded (0.104%); 40608/243288 objects misplaced (16.691%); 11 MiB/s, 172 objects/s recovering 2024-06-27T21:31:42.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:41 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:42.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2466516327' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:31:42.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:41 smithi190 ceph-mon[32247]: osdmap e518: 12 total, 12 up, 12 in 2024-06-27T21:31:42.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:42.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:41 smithi192 ceph-mon[32956]: pgmap v1344: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 252/243288 objects degraded (0.104%); 40608/243288 objects misplaced (16.691%); 11 MiB/s, 172 objects/s recovering 2024-06-27T21:31:42.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:41 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:42.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2466516327' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:31:42.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:41 smithi192 ceph-mon[32956]: osdmap e518: 12 total, 12 up, 12 in 2024-06-27T21:31:42.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:42.107 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:42.122 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:31:42.122 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:31:42.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:42.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:31:42.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:31:42.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:42.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:31:42.204 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:31:42.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:31:42.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:31:42.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:31:42.269 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:31:42.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:42.303 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:31:42.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:31:42.308 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(4.b e374,363 4:da29f04d:::benchmark_data_smithi192_88961_object70657:head,359'10177)", 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:24:21.917596+0000", 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 440.387197157, 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25539912100000001, 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:21.917596+0000", 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:21.917596+0000", 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:21.917596+0000", 2024-06-27T21:31:42.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3999999999999997e-07 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:21.917598+0000", 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.165e-06 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.172855+0000", 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25525671799999999 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.172864+0000", 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0310000000000004e-06 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.172943+0000", 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9354e-05 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.172950+0000", 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8970000000000002e-06 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.172995+0000", 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4616000000000001e-05 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(4.b e374,363 4:d12a0ed6:::benchmark_data_smithi192_88961_object71231:head,360'10213)", 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:24:22.275800+0000", 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 440.02899228699999, 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.448263103, 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.275800+0000", 2024-06-27T21:31:42.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.275800+0000", 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.275801+0000", 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.82e-07 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.275804+0000", 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.79e-06 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.723910+0000", 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44810608000000002 2024-06-27T21:31:42.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.723918+0000", 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8599999999999993e-06 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.723999+0000", 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1123000000000002e-05 2024-06-27T21:31:42.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.724010+0000", 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1311000000000001e-05 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:24:22.724064+0000", 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3457e-05 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.318 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:34766 5.0 e394/382 5:0834d305:::benchmark_data_smithi192_104167_object34765:head v 394'2195, mlcod=394'2192)", 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:27.149530+0000", 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 375.155263026, 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.226177568, 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.149530+0000", 2024-06-27T21:31:42.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.149530+0000", 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.149530+0000", 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3200000000000005e-07 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.149617+0000", 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7097999999999999e-05 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.374621+0000", 2024-06-27T21:31:42.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.225003909 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.374629+0000", 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5270000000000001e-06 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.374677+0000", 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8698e-05 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.374693+0000", 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5324000000000001e-05 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.375675+0000", 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00098240299999999997 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:27.375707+0000", 2024-06-27T21:31:42.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2076999999999998e-05 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:72474 5.f e406/382 5:f65b48d2:::benchmark_data_smithi192_104167_object72473:head v 406'4638, mlcod=406'4634)", 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:25:57.282535+0000", 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 345.02225740599999, 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22617189800000001, 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282535+0000", 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282535+0000", 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282536+0000", 2024-06-27T21:31:42.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7800000000000001e-07 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.282559+0000", 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3323e-05 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.506929+0000", 2024-06-27T21:31:42.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22436982499999999 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.506936+0000", 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6259999999999997e-06 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.506988+0000", 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2290000000000002e-05 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.507009+0000", 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0718000000000001e-05 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.508669+0000", 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016607309999999999 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:25:57.508707+0000", 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7806999999999999e-05 2024-06-27T21:31:42.328 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:115222 5.1 5:83bceee2:::benchmark_data_smithi192_104167_object9181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:26:42.390735+0000", 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 299.91405758600001, 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26140819799999998, 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:31:42.329 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 115222 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390735+0000", 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390735+0000", 2024-06-27T21:31:42.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390736+0000", 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7000000000000003e-07 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390741+0000", 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4950000000000002e-06 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390754+0000", 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3763e-05 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390762+0000", 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.807e-06 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390812+0000", 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9684000000000001e-05 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.390934+0000", 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012212099999999999 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,7", 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.391012+0000", 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8410000000000003e-05 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.393416+0000", 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024039920000000002 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.394992+0000", 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001575536 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652038+0000", 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25704618800000001 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652093+0000", 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.52e-05 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652143+0000", 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0031999999999998e-05 2024-06-27T21:31:42.335 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25615.0:115222 5.1 e419/382 ondisk, result = 0)", 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:26:42.393988+0000", 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 299.91080509599999, 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25816085999999999, 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.336 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.393988+0000", 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.393988+0000", 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.393988+0000", 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7899999999999997e-07 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.393993+0000", 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3270000000000002e-06 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.651975+0000", 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25798232300000001 2024-06-27T21:31:42.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.651981+0000", 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7790000000000001e-06 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652023+0000", 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2289000000000001e-05 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652031+0000", 2024-06-27T21:31:42.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1049999999999995e-06 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652149+0000", 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011725799999999999 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:115253 5.1 5:8ecdb464:::benchmark_data_smithi192_104167_object9212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:26:42.427390+0000", 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 299.87740279500002, 2024-06-27T21:31:42.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22490109699999999, 2024-06-27T21:31:42.342 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.342 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 115253 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427390+0000", 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427390+0000", 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427391+0000", 2024-06-27T21:31:42.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0839999999999999e-06 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427395+0000", 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3109999999999999e-06 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427407+0000", 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1681e-05 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427413+0000", 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8980000000000001e-06 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427460+0000", 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7348000000000001e-05 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427547+0000", 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6932e-05 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,7", 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.427620+0000", 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3245999999999996e-05 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.430071+0000", 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024505600000000001 2024-06-27T21:31:42.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.430860+0000", 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00078927999999999995 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652227+0000", 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22136621000000001 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652271+0000", 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4617999999999999e-05 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652291+0000", 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9928999999999998e-05 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.25615.0:115253 5.1 e419/382 ondisk, result = 0)", 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:26:42.430599+0000", 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 299.874193945, 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22169614600000001, 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.430599+0000", 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.430599+0000", 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.430600+0000", 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.048e-06 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.430605+0000", 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7509999999999999e-06 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652130+0000", 2024-06-27T21:31:42.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22152529600000001 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652135+0000", 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9269999999999996e-06 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652216+0000", 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0690000000000002e-05 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652222+0000", 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1460000000000001e-06 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:26:42.652295+0000", 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3287999999999994e-05 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.354 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:163151 5.c e444/382 5:3b58818c:::benchmark_data_smithi192_104167_object57110:head v 444'14197, mlcod=444'14194)", 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:45.859743+0000", 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 236.445049987, 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.478047059, 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859743+0000", 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859743+0000", 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859744+0000", 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859752+0000", 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6149999999999997e-06 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333734+0000", 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.473982142 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333749+0000", 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.505e-05 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333923+0000", 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017347200000000001 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333988+0000", 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4951999999999997e-05 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.337749+0000", 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037617269999999999 2024-06-27T21:31:42.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.337790+0000", 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0340999999999999e-05 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(5.1 444/441 [PushReplyOp(5:8b0d3622:::benchmark_data_smithi192_104167_object57550:head)])", 2024-06-27T21:31:42.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:46.041690+0000", 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 236.26310305800001, 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33981578699999998, 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.041690+0000", 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.041690+0000", 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.041691+0000", 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.736e-06 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.041702+0000", 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0679999999999999e-05 2024-06-27T21:31:42.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.378345+0000", 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33664313400000001 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.378368+0000", 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2334999999999999e-05 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.378431+0000", 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3694000000000004e-05 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.381505+0000", 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030742080000000001 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(5.1 444/441 [PushReplyOp(5:85261b28:::benchmark_data_smithi192_104167_object57561:head)])", 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:46.390835+0000", 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 235.913957369, 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23793767699999999, 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.390835+0000", 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.390835+0000", 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.390836+0000", 2024-06-27T21:31:42.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5199999999999995e-07 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.390842+0000", 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2009999999999996e-06 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.625263+0000", 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23442069400000001 2024-06-27T21:31:42.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.625271+0000", 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0430000000000008e-06 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.625391+0000", 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011976699999999999 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.628773+0000", 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033821200000000002 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25615.0:171585 5.1 5:8b4a2c97:::benchmark_data_smithi192_104167_object65544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:01.704819+0000", 2024-06-27T21:31:42.368 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.599973968, 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.235891609, 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25615", 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171585 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.704819+0000", 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.704819+0000", 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.704819+0000", 2024-06-27T21:31:42.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4300000000000003e-07 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.704823+0000", 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.32e-06 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.704835+0000", 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2332e-05 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.704841+0000", 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6869999999999999e-06 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.709347+0000", 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045059849999999997 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.709502+0000", 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000155721 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 7,9", 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.710155+0000", 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065235399999999995 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.721103+0000", 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010947701000000001 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.721116+0000", 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3492999999999999e-05 2024-06-27T21:31:42.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.938493+0000", 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21737726099999999 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.938559+0000", 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5287e-05 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:01.940710+0000", 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002151825 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(5.1 e462,441 5:8856e036:::benchmark_data_smithi192_104167_object2374:head,418'6731)", 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.695520+0000", 2024-06-27T21:31:42.375 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 169.60927233500001, 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.824091307, 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695520+0000", 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695520+0000", 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695521+0000", 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0200000000000001e-07 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695527+0000", 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5729999999999998e-06 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519007+0000", 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82348054000000004 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519017+0000", 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4229999999999994e-06 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519104+0000", 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7374999999999996e-05 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519612+0000", 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050769400000000005 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(5.1 e463,441 5:81f76185:::benchmark_data_smithi192_104167_object9863:head,419'7187)", 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 166.23482795800001, 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.469556784, 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4600000000000005e-07 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069969+0000", 2024-06-27T21:31:42.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5659999999999998e-06 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539033+0000", 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46906430100000002 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539041+0000", 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.858e-06 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539101+0000", 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9648000000000002e-05 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539522+0000", 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042086499999999998 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.383 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(5.1 e463,441 5:83820c48:::benchmark_data_smithi192_104167_object14200:head,419'7457)", 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:58.249046+0000", 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 164.05574703400001, 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.656992821, 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249046+0000", 2024-06-27T21:31:42.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249046+0000", 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249052+0000", 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.46e-06 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249063+0000", 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.044e-05 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.905492+0000", 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.65642939600000005 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.905501+0000", 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0850000000000003e-06 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.905598+0000", 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7083000000000005e-05 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.906039+0000", 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000440357 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.387 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:15623 6.6 6:63925ace:::benchmark_data_smithi192_117994_object15622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:29:40.687350+0000", 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 121.617442506, 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23098422900000001, 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:31:42.388 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 15623 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687350+0000", 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687350+0000", 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687352+0000", 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4339999999999999e-06 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687386+0000", 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4545000000000002e-05 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687397+0000", 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1243999999999999e-05 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687403+0000", 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5969999999999999e-06 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687458+0000", 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5427999999999999e-05 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687581+0000", 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012223900000000001 2024-06-27T21:31:42.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.687728+0000", 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014763600000000001 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.688532+0000", 2024-06-27T21:31:42.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080381500000000004 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.688936+0000", 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000403476 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918233+0000", 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.229297792 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918294+0000", 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0238000000000002e-05 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918334+0000", 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0785000000000003e-05 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.26020.0:15623 6.6 e475/467 ondisk, result = 0)", 2024-06-27T21:31:42.394 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:29:40.688693+0000", 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 121.61609983699999, 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22964943900000001, 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.688693+0000", 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.688693+0000", 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.688694+0000", 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.61e-07 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.688698+0000", 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5639999999999998e-06 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918160+0000", 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.229461728 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918165+0000", 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0300000000000001e-06 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918218+0000", 2024-06-27T21:31:42.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3038000000000003e-05 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918227+0000", 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1919999999999994e-06 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:29:40.918342+0000", 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115126 2024-06-27T21:31:42.398 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:49051 6.6 6:63421cbb:::benchmark_data_smithi192_117994_object49050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:05.431836+0000", 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 96.872956259000006, 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23828271100000001, 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:31:42.399 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 49051 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.431836+0000", 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.431836+0000", 2024-06-27T21:31:42.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.431838+0000", 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0109999999999999e-06 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.431947+0000", 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109598 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.431956+0000", 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0450000000000006e-06 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.431961+0000", 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6029999999999998e-06 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.432010+0000", 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8878999999999997e-05 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.432119+0000", 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010927199999999999 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.432228+0000", 2024-06-27T21:31:42.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010932700000000001 2024-06-27T21:31:42.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:31:42.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.433843+0000", 2024-06-27T21:31:42.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001615067 2024-06-27T21:31:42.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.434244+0000", 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040031300000000001 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.670011+0000", 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.235767538 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.670064+0000", 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2701000000000002e-05 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:05.670119+0000", 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5356999999999999e-05 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.406 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:103715 6.6 6:657c8769:::benchmark_data_smithi192_117994_object103714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:55.451659+0000", 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 46.853133948999997, 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37733334299999999, 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.407 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 103715 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451659+0000", 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451659+0000", 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451660+0000", 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.378e-06 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451702+0000", 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1346e-05 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451718+0000", 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6872999999999999e-05 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451726+0000", 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5989999999999996e-06 2024-06-27T21:31:42.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451950+0000", 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022395299999999999 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.452040+0000", 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0006999999999994e-05 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.452167+0000", 2024-06-27T21:31:42.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000127155 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.455909+0000", 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037418730000000002 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.455944+0000", 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5447000000000001e-05 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828876+0000", 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37293180799999998 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828944+0000", 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7915000000000003e-05 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828992+0000", 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7988999999999997e-05 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.26020.0:103715 6.6 e503/467 ondisk, result = 0)", 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:55.456069+0000", 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 46.848723956000001, 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37293120099999999, 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:31:42.414 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456069+0000", 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456069+0000", 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456070+0000", 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1600000000000001e-07 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456074+0000", 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1239999999999998e-06 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828792+0000", 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37271848800000001 2024-06-27T21:31:42.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828800+0000", 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5769999999999996e-06 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828857+0000", 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7281999999999998e-05 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828868+0000", 2024-06-27T21:31:42.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1195e-05 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.829000+0000", 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013181900000000001 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:31:42.418 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:31:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:42.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:42.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:31:42.420 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:31:42.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:31:42.420 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:31:42.421 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:31:42.421 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:31:42.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:31:42.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:31:42.463 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:31:42.463 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:31:42.498 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:31:42.498 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:31:42.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:42.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:42.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:42.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:42.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:42.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:42.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:42.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:42.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:42.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4054488206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:43.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4054488206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:43.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4054488206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:31:43.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:43.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:43.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:43.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:43.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:43.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:43.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:43.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:31:43.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:43.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:43.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:31:43.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:43.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:43.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:43.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:43.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:43.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:43.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:43.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:43.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:43.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:43 smithi151 ceph-mon[28965]: pgmap v1346: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 439 op/s; 261/241716 objects degraded (0.108%); 40294/241716 objects misplaced (16.670%); 11 MiB/s, 168 objects/s recovering 2024-06-27T21:31:43.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:44.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:43 smithi190 ceph-mon[32247]: pgmap v1346: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 439 op/s; 261/241716 objects degraded (0.108%); 40294/241716 objects misplaced (16.670%); 11 MiB/s, 168 objects/s recovering 2024-06-27T21:31:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.022 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:44.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:43 smithi192 ceph-mon[32956]: pgmap v1346: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 439 op/s; 261/241716 objects degraded (0.108%); 40294/241716 objects misplaced (16.670%); 11 MiB/s, 168 objects/s recovering 2024-06-27T21:31:44.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:44.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:44.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:44.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:44.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:44.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:44.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:44.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:44.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:44.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:44.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:44.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:44.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:44.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:44.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:44.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 261/241716 objects degraded (0.108%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:45.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 261/241716 objects degraded (0.108%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:45.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 261/241716 objects degraded (0.108%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:45.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:45.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:45.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:45.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:45.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:45.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:45.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:45.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:45.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:45.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:45.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:45 smithi151 ceph-mon[28965]: pgmap v1347: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 479 op/s; 261/237858 objects degraded (0.110%); 39791/237858 objects misplaced (16.729%); 11 MiB/s, 181 objects/s recovering 2024-06-27T21:31:45.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:45.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:45.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:45.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:45.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:45.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:46.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:45 smithi190 ceph-mon[32247]: pgmap v1347: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 479 op/s; 261/237858 objects degraded (0.110%); 39791/237858 objects misplaced (16.729%); 11 MiB/s, 181 objects/s recovering 2024-06-27T21:31:46.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:45 smithi192 ceph-mon[32956]: pgmap v1347: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 479 op/s; 261/237858 objects degraded (0.110%); 39791/237858 objects misplaced (16.729%); 11 MiB/s, 181 objects/s recovering 2024-06-27T21:31:46.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:46.133 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:46.133 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:46.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:46.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:46.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:46.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:46.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout:moved 1 / 111 (0.900901%) 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout:avg 9.25 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.91905 -> 2.74241 (expected baseline 2.9119) 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 4 -> 5 pgs (0.432432 -> 0.540541 * mean) 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 14 -> 14 pgs (1.51351 -> 1.51351 * mean) 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:31:46.447 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-06-27T21:31:46.448 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-06-27T21:31:46.448 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-06-27T21:31:46.448 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0178 2024-06-27T21:31:46.448 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.1782 2024-06-27T21:31:46.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.3846 -> 0.5043 2024-06-27T21:31:46.448 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-06-27T21:31:46.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:46.534 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:31:46 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:31:46.445+0000 7fc312b3e640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-06-27T21:31:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.548 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:46.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:46.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:46.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:46.777 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:46.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:46.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:46.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:46.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:46.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:47.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:47.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:47.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:47.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:47.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:47.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:47.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:31:47.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:47.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:47.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:47.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:47.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:47.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:47.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:47.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:47.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:47 smithi151 ceph-mon[28965]: pgmap v1348: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 291/235275 objects degraded (0.124%); 39292/235275 objects misplaced (16.700%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:31:47.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:47 smithi151 ceph-mon[28965]: from='client.16557 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:47.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:48.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:47 smithi190 ceph-mon[32247]: pgmap v1348: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 291/235275 objects degraded (0.124%); 39292/235275 objects misplaced (16.700%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:31:48.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:47 smithi190 ceph-mon[32247]: from='client.16557 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:47 smithi192 ceph-mon[32956]: pgmap v1348: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 291/235275 objects degraded (0.124%); 39292/235275 objects misplaced (16.700%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:31:48.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:47 smithi192 ceph-mon[32956]: from='client.16557 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:31:48.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:48.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:48.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:48.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:48.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:48.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:48.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:48.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:48.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:48.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:48.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:48.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:48.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:48.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:48.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:49.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.036 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:49.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:49.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:49.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:49.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:49.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:49.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:49.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:49.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:49.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:49.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:49.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:49.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:49.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:49.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:49.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:49.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:49 smithi190 ceph-mon[32247]: pgmap v1349: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 305/231915 objects degraded (0.132%); 38648/231915 objects misplaced (16.665%); 14 MiB/s, 216 objects/s recovering 2024-06-27T21:31:50.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 305/231915 objects degraded (0.132%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:50.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:49 smithi151 ceph-mon[28965]: pgmap v1349: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 305/231915 objects degraded (0.132%); 38648/231915 objects misplaced (16.665%); 14 MiB/s, 216 objects/s recovering 2024-06-27T21:31:50.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 305/231915 objects degraded (0.132%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:49 smithi192 ceph-mon[32956]: pgmap v1349: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 305/231915 objects degraded (0.132%); 38648/231915 objects misplaced (16.665%); 14 MiB/s, 216 objects/s recovering 2024-06-27T21:31:50.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 305/231915 objects degraded (0.132%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:50.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:50.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:50.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:50.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:50.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:50.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:50.398 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:31:50.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:50.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:50.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:50.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:50.716 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:31:50.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:50.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:50.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:50.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:50.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/791516644' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:51.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:51.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/791516644' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:51.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:51.051 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:51.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:51.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/791516644' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:31:51.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:51.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:51.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:51.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:51.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:51.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:51.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:51.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:51.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:51.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:51.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:51.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:51.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:51 smithi190 ceph-mon[32247]: pgmap v1350: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 353/227619 objects degraded (0.155%); 37917/227619 objects misplaced (16.658%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:31:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:51 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:52.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:52.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:51 smithi190 ceph-mon[32247]: osdmap e519: 12 total, 12 up, 12 in 2024-06-27T21:31:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:51 smithi192 ceph-mon[32956]: pgmap v1350: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 353/227619 objects degraded (0.155%); 37917/227619 objects misplaced (16.658%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:31:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:51 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:52.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:51 smithi192 ceph-mon[32956]: osdmap e519: 12 total, 12 up, 12 in 2024-06-27T21:31:52.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:52.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:52.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:51 smithi151 ceph-mon[28965]: pgmap v1350: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 353/227619 objects degraded (0.155%); 37917/227619 objects misplaced (16.658%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:31:52.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:51 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:31:52.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:31:52.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:51 smithi151 ceph-mon[28965]: osdmap e519: 12 total, 12 up, 12 in 2024-06-27T21:31:52.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:52.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:52.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:52.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:52.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:52.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:52.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:52.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:52.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:52.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:52.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:52.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:52.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:52.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:52.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:53.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:53.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:53.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:53.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:53.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:53.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:53.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:53.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:53.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:53.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:53.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:53.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:53.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:53.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:53.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:53.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:53.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:53 smithi151 ceph-mon[28965]: pgmap v1352: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 392/226155 objects degraded (0.173%); 37584/226155 objects misplaced (16.619%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:54.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:53 smithi190 ceph-mon[32247]: pgmap v1352: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 392/226155 objects degraded (0.173%); 37584/226155 objects misplaced (16.619%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:54.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:54.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:53 smithi192 ceph-mon[32956]: pgmap v1352: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 518 op/s; 392/226155 objects degraded (0.173%); 37584/226155 objects misplaced (16.619%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:54.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:54.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:54.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:54.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:54.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:54.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:54.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:54.443 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:31:54.443 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:31:54.443 INFO:tasks.thrashosds.thrasher:Removing osd 8, in_osds are: [0, 1, 3, 4, 5, 6, 8, 9, 11, 10, 2, 7] 2024-06-27T21:31:54.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 8 2024-06-27T21:31:54.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:54.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:54.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:54.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:54.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:54.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:54.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:54.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 392/226155 objects degraded (0.173%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:54.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:54.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:55.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 392/226155 objects degraded (0.173%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:55.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 392/226155 objects degraded (0.173%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:31:55.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:55.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:55.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:55.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:55.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:55.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:55.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:55.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:55.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:55.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:55.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:55.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:55.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:55.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:55.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:55.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:55.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:55 smithi190 ceph-mon[32247]: pgmap v1353: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 392/222276 objects degraded (0.176%); 37082/222276 objects misplaced (16.683%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:56.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:55 smithi192 ceph-mon[32956]: pgmap v1353: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 392/222276 objects degraded (0.176%); 37082/222276 objects misplaced (16.683%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:56.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:56.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:56.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:55 smithi151 ceph-mon[28965]: pgmap v1353: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 392/222276 objects degraded (0.176%); 37082/222276 objects misplaced (16.683%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:56.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:56.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:56.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:56.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:56.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:56.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:56.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:56.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:56.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:56.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:56.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:31:56.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:56.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:31:57.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:31:57.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:31:57.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:31:57.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:31:57.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:57.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:57.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:57.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:57.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:31:57.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:31:57.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:57.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:57.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:31:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:57.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:31:57.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:57.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:57.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:57 smithi190 ceph-mon[32247]: pgmap v1354: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 405/219597 objects degraded (0.184%); 36597/219597 objects misplaced (16.666%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:58.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:57 smithi192 ceph-mon[32956]: pgmap v1354: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 405/219597 objects degraded (0.184%); 36597/219597 objects misplaced (16.666%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:58.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:58.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:58.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:57 smithi151 ceph-mon[28965]: pgmap v1354: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 405/219597 objects degraded (0.184%); 36597/219597 objects misplaced (16.666%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:31:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:58.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:58.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:58.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:58.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:31:58.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:31:58.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:58.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:58.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:31:58.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:31:58.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:31:58.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:58.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:31:58.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:59.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:59.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:31:59.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:31:59.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:59.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:59.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:59.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:59.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:31:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:31:59.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:31:59.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:59.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:31:59.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:31:59.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:31:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:31:59.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:31:59.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:31:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:31:59.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:00.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:59 smithi190 ceph-mon[32247]: pgmap v1355: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 431/216066 objects degraded (0.199%); 35940/216066 objects misplaced (16.634%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:32:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:31:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 431/216066 objects degraded (0.199%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:00.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:59 smithi192 ceph-mon[32956]: pgmap v1355: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 431/216066 objects degraded (0.199%); 35940/216066 objects misplaced (16.634%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:32:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:31:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 431/216066 objects degraded (0.199%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:00.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:00.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:00.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:59 smithi151 ceph-mon[28965]: pgmap v1355: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 431/216066 objects degraded (0.199%); 35940/216066 objects misplaced (16.634%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:32:00.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:31:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 431/216066 objects degraded (0.199%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:00.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:00.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:00.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:00.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:00.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:00.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:00.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:00.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:00.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:00.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:00.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:00.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:00.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:01.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:01.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:01.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:01.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:01.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:01.343 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:01.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:01.475 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:01.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:01.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:01.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:01.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:01.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:01.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:01.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:01.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:01.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:01 smithi151 ceph-mon[28965]: pgmap v1356: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 474/211533 objects degraded (0.224%); 35230/211533 objects misplaced (16.655%); 14 MiB/s, 223 objects/s recovering 2024-06-27T21:32:02.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:01 smithi190 ceph-mon[32247]: pgmap v1356: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 474/211533 objects degraded (0.224%); 35230/211533 objects misplaced (16.655%); 14 MiB/s, 223 objects/s recovering 2024-06-27T21:32:02.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:01 smithi192 ceph-mon[32956]: pgmap v1356: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 474/211533 objects degraded (0.224%); 35230/211533 objects misplaced (16.655%); 14 MiB/s, 223 objects/s recovering 2024-06-27T21:32:02.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:02.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:02.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:02.272 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:02.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:02.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:02.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:02.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:02.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:02.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:02.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:02.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:02.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:02.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:02.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:03.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:03.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:03.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:03.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:03.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:03.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:03.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:03.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:03.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:03.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:03.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:03.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:03.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:03.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:03.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:32:03.897 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:32:03.897 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:32:03.897 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:32:03.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:32:03.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:32:03.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:03 smithi151 ceph-mon[28965]: pgmap v1357: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 518/210099 objects degraded (0.247%); 34862/210099 objects misplaced (16.593%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:32:03.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:03.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:03.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:03 smithi151 ceph-mon[28965]: osdmap e520: 12 total, 12 up, 12 in 2024-06-27T21:32:03.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:03.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:32:03.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:32:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:03.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:03.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:03.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:32:03.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:32:03.997 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:32:03.997 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:32:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:03 smithi190 ceph-mon[32247]: pgmap v1357: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 518/210099 objects degraded (0.247%); 34862/210099 objects misplaced (16.593%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:32:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:03 smithi190 ceph-mon[32247]: osdmap e520: 12 total, 12 up, 12 in 2024-06-27T21:32:04.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:32:04.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:32:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:03 smithi192 ceph-mon[32956]: pgmap v1357: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 512 op/s; 518/210099 objects degraded (0.247%); 34862/210099 objects misplaced (16.593%); 14 MiB/s, 224 objects/s recovering 2024-06-27T21:32:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:03 smithi192 ceph-mon[32956]: osdmap e520: 12 total, 12 up, 12 in 2024-06-27T21:32:04.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:32:04.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:32:04.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:04.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:32:04.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:32:04.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:04.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:32:04.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:32:04.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:32:04.152 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:32:04.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:04.196 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:32:04.197 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:32:04.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:04.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:04.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:04.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:32:04.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:32:04.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:04.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:04.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:04.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:04.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:04.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:04.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:32:04.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:04.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:04.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:04.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:05.001 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:04 smithi190 ceph-mon[32247]: osdmap e521: 12 total, 12 up, 12 in 2024-06-27T21:32:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2114032822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 518/210099 objects degraded (0.247%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:05.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:04 smithi192 ceph-mon[32956]: osdmap e521: 12 total, 12 up, 12 in 2024-06-27T21:32:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2114032822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 518/210099 objects degraded (0.247%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:05.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:04 smithi151 ceph-mon[28965]: osdmap e521: 12 total, 12 up, 12 in 2024-06-27T21:32:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2114032822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:05.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 518/210099 objects degraded (0.247%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:05.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:05.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:05.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:32:05.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:05.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:05.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:05.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:05.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:05.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:05.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:05.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:05.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:05.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:05.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:05.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:05 smithi190 ceph-mon[32247]: pgmap v1360: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 518/206349 objects degraded (0.251%); 34345/206349 objects misplaced (16.644%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:32:06.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:05 smithi190 ceph-mon[32247]: osdmap e522: 12 total, 12 up, 12 in 2024-06-27T21:32:06.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:06.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:06.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:05 smithi192 ceph-mon[32956]: pgmap v1360: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 518/206349 objects degraded (0.251%); 34345/206349 objects misplaced (16.644%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:32:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:05 smithi192 ceph-mon[32956]: osdmap e522: 12 total, 12 up, 12 in 2024-06-27T21:32:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:06.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:06.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:06.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:06.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:05 smithi151 ceph-mon[28965]: pgmap v1360: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 518/206349 objects degraded (0.251%); 34345/206349 objects misplaced (16.644%); 14 MiB/s, 225 objects/s recovering 2024-06-27T21:32:06.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:05 smithi151 ceph-mon[28965]: osdmap e522: 12 total, 12 up, 12 in 2024-06-27T21:32:06.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:06.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:06.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:06.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:06.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:06.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:06.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:06.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:06.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:06.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:06.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:06.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:06.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:06.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:06.951 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:06.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:06.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:06 smithi192 ceph-mon[32956]: osdmap e523: 12 total, 12 up, 12 in 2024-06-27T21:32:07.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:07.113 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:07.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:06 smithi151 ceph-mon[28965]: osdmap e523: 12 total, 12 up, 12 in 2024-06-27T21:32:07.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:07.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:07.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:06 smithi190 ceph-mon[32247]: osdmap e523: 12 total, 12 up, 12 in 2024-06-27T21:32:07.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:07.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:07.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:07.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:07.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:07.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:07.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:07.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:07.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:07.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:07.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:07.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:08.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:07 smithi192 ceph-mon[32956]: pgmap v1363: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 521/203928 objects degraded (0.255%); 33850/203928 objects misplaced (16.599%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:32:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:08.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:32:08.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:08.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:07 smithi151 ceph-mon[28965]: pgmap v1363: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 521/203928 objects degraded (0.255%); 33850/203928 objects misplaced (16.599%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:32:08.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:08.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:08.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:08.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:07 smithi190 ceph-mon[32247]: pgmap v1363: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 521/203928 objects degraded (0.255%); 33850/203928 objects misplaced (16.599%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:32:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:08.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:08.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:08.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:08.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:08.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:08.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:08.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:08.749 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:08.752 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:08.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:08.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:08.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:08.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:08 smithi151 ceph-mon[28965]: osdmap e524: 12 total, 12 up, 12 in 2024-06-27T21:32:08.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:08.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:09.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:08 smithi192 ceph-mon[32956]: osdmap e524: 12 total, 12 up, 12 in 2024-06-27T21:32:09.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:09.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:09.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:09.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:08 smithi190 ceph-mon[32247]: osdmap e524: 12 total, 12 up, 12 in 2024-06-27T21:32:09.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:09.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:09.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:09.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:09.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:09.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:09.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:09.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:09.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:09.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:09.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:09.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:10.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:10.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:09 smithi192 ceph-mon[32956]: pgmap v1365: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 564/200418 objects degraded (0.281%); 33132/200418 objects misplaced (16.531%); 14 MiB/s, 226 objects/s recovering 2024-06-27T21:32:10.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 564/200418 objects degraded (0.281%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:10.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:10.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:10.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:09 smithi151 ceph-mon[28965]: pgmap v1365: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 564/200418 objects degraded (0.281%); 33132/200418 objects misplaced (16.531%); 14 MiB/s, 226 objects/s recovering 2024-06-27T21:32:10.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 564/200418 objects degraded (0.281%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:10.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:10.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:09 smithi190 ceph-mon[32247]: pgmap v1365: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 564/200418 objects degraded (0.281%); 33132/200418 objects misplaced (16.531%); 14 MiB/s, 226 objects/s recovering 2024-06-27T21:32:10.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 564/200418 objects degraded (0.281%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:10.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:10.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:10.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:10.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:10.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:10.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:10.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:10.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:10.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:10.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:10.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:10.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:11.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:11.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:11.096 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:11.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:11.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:11.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:11.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:11.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:11.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:11.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:11.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:11.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:11.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:11.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:11.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:11.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:11.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:12.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:11 smithi192 ceph-mon[32956]: pgmap v1366: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 592/195819 objects degraded (0.302%); 32408/195819 objects misplaced (16.550%); 17 MiB/s, 279 objects/s recovering 2024-06-27T21:32:12.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:12.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:12.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:11 smithi151 ceph-mon[28965]: pgmap v1366: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 592/195819 objects degraded (0.302%); 32408/195819 objects misplaced (16.550%); 17 MiB/s, 279 objects/s recovering 2024-06-27T21:32:12.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:12.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:11 smithi190 ceph-mon[32247]: pgmap v1366: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 592/195819 objects degraded (0.302%); 32408/195819 objects misplaced (16.550%); 17 MiB/s, 279 objects/s recovering 2024-06-27T21:32:12.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:12.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:12.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:12.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:12.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:12.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:12.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:12 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:32:12.371+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e524 definitely_dead 0 2024-06-27T21:32:12.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:12.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:12.806 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.8. 2024-06-27T21:32:12.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:12.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:12.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:13.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:13.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:13.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:13.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:13.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/764722619' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-06-27T21:32:13.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:12 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-06-27T21:32:13.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:12 smithi151 ceph-mon[28965]: Client client.admin marked osd.8 out, while it was still marked up 2024-06-27T21:32:13.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:13.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/764722619' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-06-27T21:32:13.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:12 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-06-27T21:32:13.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:12 smithi190 ceph-mon[32247]: Client client.admin marked osd.8 out, while it was still marked up 2024-06-27T21:32:13.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/764722619' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-06-27T21:32:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:12 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["8"]}]: dispatch 2024-06-27T21:32:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:12 smithi192 ceph-mon[32956]: Client client.admin marked osd.8 out, while it was still marked up 2024-06-27T21:32:13.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:13.402 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:13.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:13.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:13.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:13.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:13.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:13.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:13.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:13.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:13.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:14.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:14.022 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:14.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:14.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:14.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:13 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:14.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:13 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:13 smithi151 ceph-mon[28965]: pgmap v1367: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 620/194355 objects degraded (0.319%); 32073/194355 objects misplaced (16.502%); 14 MiB/s, 222 objects/s recovering 2024-06-27T21:32:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:13 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-06-27T21:32:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:13 smithi151 ceph-mon[28965]: osdmap e525: 12 total, 12 up, 11 in 2024-06-27T21:32:14.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:13 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:13 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:13 smithi190 ceph-mon[32247]: pgmap v1367: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 620/194355 objects degraded (0.319%); 32073/194355 objects misplaced (16.502%); 14 MiB/s, 222 objects/s recovering 2024-06-27T21:32:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:13 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-06-27T21:32:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:13 smithi190 ceph-mon[32247]: osdmap e525: 12 total, 12 up, 11 in 2024-06-27T21:32:14.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:13 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:13 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:13 smithi192 ceph-mon[32956]: pgmap v1367: 37 pgs: 7 active+recovering+undersized+degraded+remapped, 30 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 620/194355 objects degraded (0.319%); 32073/194355 objects misplaced (16.502%); 14 MiB/s, 222 objects/s recovering 2024-06-27T21:32:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:13 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["8"]}]': finished 2024-06-27T21:32:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:13 smithi192 ceph-mon[32956]: osdmap e525: 12 total, 12 up, 11 in 2024-06-27T21:32:14.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:14.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:14.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:14.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:14.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:14.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:14.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:14.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:14.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:14.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:14.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:14.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:14 smithi151 ceph-mon[28965]: osdmap e526: 12 total, 12 up, 11 in 2024-06-27T21:32:14.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 620/194355 objects degraded (0.319%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:14.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:14 smithi151 ceph-mon[28965]: osdmap e527: 12 total, 12 up, 11 in 2024-06-27T21:32:15.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:15.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:15.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:15.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:14 smithi190 ceph-mon[32247]: osdmap e526: 12 total, 12 up, 11 in 2024-06-27T21:32:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 620/194355 objects degraded (0.319%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:14 smithi190 ceph-mon[32247]: osdmap e527: 12 total, 12 up, 11 in 2024-06-27T21:32:15.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:15.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:15.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:14 smithi192 ceph-mon[32956]: osdmap e526: 12 total, 12 up, 11 in 2024-06-27T21:32:15.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 620/194355 objects degraded (0.319%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:15.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:14 smithi192 ceph-mon[32956]: osdmap e527: 12 total, 12 up, 11 in 2024-06-27T21:32:15.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:15.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:15.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:15.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:15.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:15.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:15.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:15.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:15.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:15.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:15.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:16.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:15 smithi190 ceph-mon[32247]: pgmap v1370: 37 pgs: 2 peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 620/190377 objects degraded (0.326%); 31533/190377 objects misplaced (16.563%); 13 MiB/s, 215 objects/s recovering 2024-06-27T21:32:16.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:16.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:16.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:16.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:16.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:16.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:15 smithi151 ceph-mon[28965]: pgmap v1370: 37 pgs: 2 peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 620/190377 objects degraded (0.326%); 31533/190377 objects misplaced (16.563%); 13 MiB/s, 215 objects/s recovering 2024-06-27T21:32:16.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:16.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:15 smithi192 ceph-mon[32956]: pgmap v1370: 37 pgs: 2 peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 620/190377 objects degraded (0.326%); 31533/190377 objects misplaced (16.563%); 13 MiB/s, 215 objects/s recovering 2024-06-27T21:32:16.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:16.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:16.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:16.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:16.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:16.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:16.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:16.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:16.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:16.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:16.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:16.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:17.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:17.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:17.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:17.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:17.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:17.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:17.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:17.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:17.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:17.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:17.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:17.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:17.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:17.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:17.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:17.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:18.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:18.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:18.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:17 smithi151 ceph-mon[28965]: pgmap v1372: 37 pgs: 1 remapped+peering, 2 peering, 7 active+recovering+undersized+degraded+remapped, 27 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 395 op/s; 580/188694 objects degraded (0.307%); 26776/188694 objects misplaced (14.190%); 13 MiB/s, 205 objects/s recovering 2024-06-27T21:32:18.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:18.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:17 smithi190 ceph-mon[32247]: pgmap v1372: 37 pgs: 1 remapped+peering, 2 peering, 7 active+recovering+undersized+degraded+remapped, 27 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 395 op/s; 580/188694 objects degraded (0.307%); 26776/188694 objects misplaced (14.190%); 13 MiB/s, 205 objects/s recovering 2024-06-27T21:32:18.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:18.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:17 smithi192 ceph-mon[32956]: pgmap v1372: 37 pgs: 1 remapped+peering, 2 peering, 7 active+recovering+undersized+degraded+remapped, 27 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 395 op/s; 580/188694 objects degraded (0.307%); 26776/188694 objects misplaced (14.190%); 13 MiB/s, 205 objects/s recovering 2024-06-27T21:32:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.312 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:18.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:18.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:18.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:18.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:18.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:18.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:18.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:18.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:18.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:18.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:18.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:18.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:19.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:19.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:19.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:19.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:19.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:19.314 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:19.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:19.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:19.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:19.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:19.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:19.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:19.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:19.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:19.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:19.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:19.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:19.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:20.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:20.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:20.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:20.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:20.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:19 smithi151 ceph-mon[28965]: pgmap v1373: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 2 peering, 7 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 354 op/s; 7325/187980 objects degraded (3.897%); 26337/187980 objects misplaced (14.011%); 14 MiB/s, 228 objects/s recovering 2024-06-27T21:32:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7325/187980 objects degraded (3.897%), 8 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:19 smithi190 ceph-mon[32247]: pgmap v1373: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 2 peering, 7 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 354 op/s; 7325/187980 objects degraded (3.897%); 26337/187980 objects misplaced (14.011%); 14 MiB/s, 228 objects/s recovering 2024-06-27T21:32:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7325/187980 objects degraded (3.897%), 8 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:19 smithi192 ceph-mon[32956]: pgmap v1373: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 2 peering, 7 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 354 op/s; 7325/187980 objects degraded (3.897%); 26337/187980 objects misplaced (14.011%); 14 MiB/s, 228 objects/s recovering 2024-06-27T21:32:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7325/187980 objects degraded (3.897%), 8 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:20.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:20.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:20.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:20.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:20.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:20.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:20.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:20.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:20.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:20.825 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:32:20.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:20.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:20.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:20 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:20.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:20.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3398746957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:32:20.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:20.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:21.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:21.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:21.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:21.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:20 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3398746957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:32:21.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:20 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:21.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:21.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3398746957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:32:21.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:21.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:21.420 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:32:21.420 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:32:21.421 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-06-27T21:32:21.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:32:21.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:21.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:21.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:21.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:21.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:21.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:21.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:21.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:21.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:21.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:21.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:22.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:22.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:22.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:21 smithi151 ceph-mon[28965]: pgmap v1374: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 231 op/s; 7325/187746 objects degraded (3.902%); 25836/187746 objects misplaced (13.761%); 14 MiB/s, 231 objects/s recovering 2024-06-27T21:32:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:21 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:32:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3398746957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:32:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:21 smithi151 ceph-mon[28965]: osdmap e528: 12 total, 12 up, 11 in 2024-06-27T21:32:22.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:21 smithi190 ceph-mon[32247]: pgmap v1374: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 231 op/s; 7325/187746 objects degraded (3.902%); 25836/187746 objects misplaced (13.761%); 14 MiB/s, 231 objects/s recovering 2024-06-27T21:32:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:21 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:32:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3398746957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:32:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:21 smithi190 ceph-mon[32247]: osdmap e528: 12 total, 12 up, 11 in 2024-06-27T21:32:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:21 smithi192 ceph-mon[32956]: pgmap v1374: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 231 op/s; 7325/187746 objects degraded (3.902%); 25836/187746 objects misplaced (13.761%); 14 MiB/s, 231 objects/s recovering 2024-06-27T21:32:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:21 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:32:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3398746957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:32:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:21 smithi192 ceph-mon[32956]: osdmap e528: 12 total, 12 up, 11 in 2024-06-27T21:32:22.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:22.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:22.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:22.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:22.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:22.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:22.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:22.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:22.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:22.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:22.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:22.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:22.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:22.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:23.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:23.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:23.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:23.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:23.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:23.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:23.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:23.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:23.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:23.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:23.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:23.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:23.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:23.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:23.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:23.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:23 smithi151 ceph-mon[28965]: pgmap v1376: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 110 op/s; 7325/187731 objects degraded (3.902%); 25057/187731 objects misplaced (13.347%); 18 MiB/s, 285 objects/s recovering 2024-06-27T21:32:23.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:23.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:24.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:24.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:24.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:24.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:23 smithi190 ceph-mon[32247]: pgmap v1376: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 110 op/s; 7325/187731 objects degraded (3.902%); 25057/187731 objects misplaced (13.347%); 18 MiB/s, 285 objects/s recovering 2024-06-27T21:32:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:23 smithi192 ceph-mon[32956]: pgmap v1376: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 110 op/s; 7325/187731 objects degraded (3.902%); 25057/187731 objects misplaced (13.347%); 18 MiB/s, 285 objects/s recovering 2024-06-27T21:32:24.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:24.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:24.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:24.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:24.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:24.602 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:24.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:24.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:24.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:24.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:24.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:24.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:24.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:25.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:25.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:25.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:25.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:25.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7325/187731 objects degraded (3.902%), 8 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:25.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:25.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7325/187731 objects degraded (3.902%), 8 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7325/187731 objects degraded (3.902%), 8 pgs degraded (PG_DEGRADED) 2024-06-27T21:32:25.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:25.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:25.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:25.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:25.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:25.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:25.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:25.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:25.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:25.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:25.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:25.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:25.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:26.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:26.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:26.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:25 smithi151 ceph-mon[28965]: pgmap v1377: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 51 op/s; 25220/187617 objects degraded (13.442%); 23990/187617 objects misplaced (12.787%); 20 MiB/s, 318 objects/s recovering 2024-06-27T21:32:26.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:25 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:26.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:25 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:26.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:25 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:32:26.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:25 smithi151 ceph-mon[28965]: osdmap e529: 12 total, 12 up, 11 in 2024-06-27T21:32:26.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:26.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:25 smithi190 ceph-mon[32247]: pgmap v1377: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 51 op/s; 25220/187617 objects degraded (13.442%); 23990/187617 objects misplaced (12.787%); 20 MiB/s, 318 objects/s recovering 2024-06-27T21:32:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:25 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:25 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:25 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:32:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:25 smithi190 ceph-mon[32247]: osdmap e529: 12 total, 12 up, 11 in 2024-06-27T21:32:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:25 smithi192 ceph-mon[32956]: pgmap v1377: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 51 op/s; 25220/187617 objects degraded (13.442%); 23990/187617 objects misplaced (12.787%); 20 MiB/s, 318 objects/s recovering 2024-06-27T21:32:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:25 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:25 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:25 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:32:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:25 smithi192 ceph-mon[32956]: osdmap e529: 12 total, 12 up, 11 in 2024-06-27T21:32:26.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:26.377 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:26.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:26.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:26.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:26.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:26.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:26.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:26.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:26.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:26.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:26.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:26.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:26 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:32:26.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:26.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:26.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:26 smithi151 ceph-mon[28965]: osdmap e530: 12 total, 12 up, 11 in 2024-06-27T21:32:26.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:26 smithi151 ceph-mon[28965]: 6.b scrub ok 2024-06-27T21:32:27.004 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:27.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:27.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:27.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:26 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:32:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:26 smithi192 ceph-mon[32956]: osdmap e530: 12 total, 12 up, 11 in 2024-06-27T21:32:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:26 smithi192 ceph-mon[32956]: 6.b scrub ok 2024-06-27T21:32:27.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:27.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:26 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:32:27.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:27.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:27.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:26 smithi190 ceph-mon[32247]: osdmap e530: 12 total, 12 up, 11 in 2024-06-27T21:32:27.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:26 smithi190 ceph-mon[32247]: 6.b scrub ok 2024-06-27T21:32:27.267 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:32:27.267 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:32:27.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:27.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:27.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:27.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:32:27.502 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:32:27.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:27.527 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:32:27.527 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:32:27.555 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:32:27.555 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:32:27.581 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:32:27.581 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:32:27.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:32:27.605 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:32:27.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:27.630 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:32:27.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:32:27.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:27.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:32:27.654 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:32:27.679 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:32:27.679 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:32:27.706 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:32:27.707 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:32:27.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:32:27.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:32:27.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:27.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:27.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:32:27.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:32:27.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:32:27.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:32:27.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:27.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:27.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:27.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:28.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:28.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:28.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:27 smithi151 ceph-mon[28965]: 6.7 scrub ok 2024-06-27T21:32:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:27 smithi151 ceph-mon[28965]: pgmap v1380: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 25220/187617 objects degraded (13.442%); 23380/187617 objects misplaced (12.462%); 26 MiB/s, 409 objects/s recovering 2024-06-27T21:32:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:27 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:32:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:27 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:32:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1941022611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:28.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:28.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:27 smithi190 ceph-mon[32247]: 6.7 scrub ok 2024-06-27T21:32:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:27 smithi190 ceph-mon[32247]: pgmap v1380: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 25220/187617 objects degraded (13.442%); 23380/187617 objects misplaced (12.462%); 26 MiB/s, 409 objects/s recovering 2024-06-27T21:32:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:27 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:32:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:27 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:32:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1941022611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:27 smithi192 ceph-mon[32956]: 6.7 scrub ok 2024-06-27T21:32:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:27 smithi192 ceph-mon[32956]: pgmap v1380: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 25220/187617 objects degraded (13.442%); 23380/187617 objects misplaced (12.462%); 26 MiB/s, 409 objects/s recovering 2024-06-27T21:32:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:27 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:32:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:27 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:32:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1941022611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:28.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:28.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:28.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:32:28.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:28.507 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:28.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:28.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:28.694 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:28.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:28.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:28.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:32:28.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:28.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:28.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:28.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:29.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:29.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:29.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:29.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:29.249 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:29.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:29.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:29.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:29.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:29.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:29.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:29.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:29.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:29.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:29.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:29.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:29.903 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:29.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:30.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:30.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:30.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:29 smithi151 ceph-mon[28965]: pgmap v1381: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 5 op/s; 25220/187617 objects degraded (13.442%); 22760/187617 objects misplaced (12.131%); 25 MiB/s, 392 objects/s recovering 2024-06-27T21:32:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:29 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:32:30.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 25220/187617 objects degraded (13.442%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:30.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:30.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:29 smithi190 ceph-mon[32247]: pgmap v1381: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 5 op/s; 25220/187617 objects degraded (13.442%); 22760/187617 objects misplaced (12.131%); 25 MiB/s, 392 objects/s recovering 2024-06-27T21:32:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:29 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:32:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 25220/187617 objects degraded (13.442%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:29 smithi192 ceph-mon[32956]: pgmap v1381: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 5 op/s; 25220/187617 objects degraded (13.442%); 22760/187617 objects misplaced (12.131%); 25 MiB/s, 392 objects/s recovering 2024-06-27T21:32:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:29 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:32:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 25220/187617 objects degraded (13.442%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:30.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:30.374 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:30.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:30.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:30.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:30.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:30.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:30.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:30.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:30.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:30.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:31.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:31.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:31.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:31.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:31.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:31.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:31.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:31.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:31.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:31.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:31.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:31.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:31.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:31.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:31.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:31.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:32:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:31.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:31.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:32.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:32.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:32.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:32.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:31 smithi151 ceph-mon[28965]: pgmap v1382: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 4 op/s; 25220/187617 objects degraded (13.442%); 22180/187617 objects misplaced (11.822%); 22 MiB/s, 359 objects/s recovering 2024-06-27T21:32:32.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:32.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:31 smithi190 ceph-mon[32247]: pgmap v1382: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 4 op/s; 25220/187617 objects degraded (13.442%); 22180/187617 objects misplaced (11.822%); 22 MiB/s, 359 objects/s recovering 2024-06-27T21:32:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:31 smithi192 ceph-mon[32956]: pgmap v1382: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 4 op/s; 25220/187617 objects degraded (13.442%); 22180/187617 objects misplaced (11.822%); 22 MiB/s, 359 objects/s recovering 2024-06-27T21:32:32.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:32.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:32.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:32.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:32.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:32.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:32.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:32.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:32.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:32.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:32.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:32.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:32.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:33.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:33.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:33.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:33.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:33.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:33.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:33.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:33.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:33.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:33.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:33.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:33.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:33.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:33.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:33.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:33.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:33.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:33.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:34.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:34.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:34.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:34.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:33 smithi151 ceph-mon[28965]: pgmap v1383: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 21446/187617 objects misplaced (11.431%); 20 MiB/s, 317 objects/s recovering 2024-06-27T21:32:34.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:32:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:32:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:33 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:32:34.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:33 smithi190 ceph-mon[32247]: pgmap v1383: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 21446/187617 objects misplaced (11.431%); 20 MiB/s, 317 objects/s recovering 2024-06-27T21:32:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:32:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:32:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:33 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:32:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:33 smithi192 ceph-mon[32956]: pgmap v1383: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 21446/187617 objects misplaced (11.431%); 20 MiB/s, 317 objects/s recovering 2024-06-27T21:32:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:32:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:32:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:33 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:32:34.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:34.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:34.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:34.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:34.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:34.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:34.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:34.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:34.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:34.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:34.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:34.955 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:34.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:35.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:35.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:35.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:35.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:35.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:35.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:35.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:35.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:35.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:35.333 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:35.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:35.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:35.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:35.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:35.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:35.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:35.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:35.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:35.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:35.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:35.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:36.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:36.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:36.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:36.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:35 smithi151 ceph-mon[28965]: pgmap v1384: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 21172/187617 objects misplaced (11.285%); 19 MiB/s, 303 objects/s recovering 2024-06-27T21:32:36.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:35 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 25220/187617 objects degraded (13.442%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:36.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:35 smithi151 ceph-mon[28965]: osdmap e531: 12 total, 12 up, 11 in 2024-06-27T21:32:36.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:36.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:35 smithi190 ceph-mon[32247]: pgmap v1384: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 21172/187617 objects misplaced (11.285%); 19 MiB/s, 303 objects/s recovering 2024-06-27T21:32:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:35 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 25220/187617 objects degraded (13.442%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:35 smithi190 ceph-mon[32247]: osdmap e531: 12 total, 12 up, 11 in 2024-06-27T21:32:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:35 smithi192 ceph-mon[32956]: pgmap v1384: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 21172/187617 objects misplaced (11.285%); 19 MiB/s, 303 objects/s recovering 2024-06-27T21:32:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:35 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 25220/187617 objects degraded (13.442%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:35 smithi192 ceph-mon[32956]: osdmap e531: 12 total, 12 up, 11 in 2024-06-27T21:32:36.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:36.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:36.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:36.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:36.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:36.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:36.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:36.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:36.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:36.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:36.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:36.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:36.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:37.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:37.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:37.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:37.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:37.312 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:32:37.313 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:32:37.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:37.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:37.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:37.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:37.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:37.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:37.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:37.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:37.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:37.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:37.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:37.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:38.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:38.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:38.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:38.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:37 smithi151 ceph-mon[28965]: pgmap v1386: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 20618/187617 objects misplaced (10.989%); 17 MiB/s, 276 objects/s recovering 2024-06-27T21:32:38.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:37 smithi151 ceph-mon[28965]: from='client.16590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:32:38.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:38.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:37 smithi190 ceph-mon[32247]: pgmap v1386: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 20618/187617 objects misplaced (10.989%); 17 MiB/s, 276 objects/s recovering 2024-06-27T21:32:38.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:37 smithi190 ceph-mon[32247]: from='client.16590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:32:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:37 smithi192 ceph-mon[32956]: pgmap v1386: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 25220/187617 objects degraded (13.442%); 20618/187617 objects misplaced (10.989%); 17 MiB/s, 276 objects/s recovering 2024-06-27T21:32:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:37 smithi192 ceph-mon[32956]: from='client.16590 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:32:38.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:38.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:38.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:38.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:38.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:38.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:38.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:38.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:38.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:38.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:38.837 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:38.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:38.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:38.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:39.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:39.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:39.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:39.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:38 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:32:39.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:38 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:32:39.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:39.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:38 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:32:39.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:38 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:32:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:38 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:32:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:38 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:32:39.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:39.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:39.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:39.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:39.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:39.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:39.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:39.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:39.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:39.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:39.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:39.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:40.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:40.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:40.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:40.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:39 smithi151 ceph-mon[28965]: pgmap v1387: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 25220/187605 objects degraded (13.443%); 18449/187605 objects misplaced (9.834%); 27 MiB/s, 431 objects/s recovering 2024-06-27T21:32:40.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:39 smithi151 ceph-mon[28965]: osdmap e532: 12 total, 12 up, 11 in 2024-06-27T21:32:40.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:40.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:39 smithi190 ceph-mon[32247]: pgmap v1387: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 25220/187605 objects degraded (13.443%); 18449/187605 objects misplaced (9.834%); 27 MiB/s, 431 objects/s recovering 2024-06-27T21:32:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:39 smithi190 ceph-mon[32247]: osdmap e532: 12 total, 12 up, 11 in 2024-06-27T21:32:40.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:39 smithi192 ceph-mon[32956]: pgmap v1387: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 25220/187605 objects degraded (13.443%); 18449/187605 objects misplaced (9.834%); 27 MiB/s, 431 objects/s recovering 2024-06-27T21:32:40.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:39 smithi192 ceph-mon[32956]: osdmap e532: 12 total, 12 up, 11 in 2024-06-27T21:32:40.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:40.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:40.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:40.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:40.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:40.612 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1386,"stamp":"2024-06-27T21:32:36.109182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4098949688,"num_objects":62539,"num_object_clones":0,"num_object_copies":187617,"num_objects_missing_on_primary":5404,"num_objects_missing":5404,"num_objects_degraded":25220,"num_objects_misplaced":20618,"num_objects_unfound":0,"num_objects_dirty":62539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152647,"num_write_kb":6885825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16203,"num_bytes_recovered":1061247648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154579,"ondisk_log_size":154579,"up":111,"acting":102,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17024944,"kb_used_data":12453460,"kb_used_omap":108,"kb_used_meta":4570771,"kb_avail":1014171728,"statfs":{"total":1055945392128,"available":1038511849472,"internally_reserved":0,"allocated":12752343040,"data_stored":12749471251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111086,"internal_metadata":4680470034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2762,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2762,"num_bytes_recovered":181010432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002201"},"pg_stats":[{"pgid":"1.14","version":"21'92","reported_seq":131,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.858170+0000","last_change":"2024-06-27T21:31:19.010738+0000","last_active":"2024-06-27T21:32:25.858170+0000","last_peered":"2024-06-27T21:32:25.858170+0000","last_clean":"2024-06-27T21:32:25.858170+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:32:25.858170+0000","last_undegraded":"2024-06-27T21:32:25.858170+0000","last_fullsized":"2024-06-27T21:32:25.858170+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.010614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.010614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:36.942024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":131,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857310+0000","last_change":"2024-06-27T21:31:19.025719+0000","last_active":"2024-06-27T21:32:25.857310+0000","last_peered":"2024-06-27T21:32:25.857310+0000","last_clean":"2024-06-27T21:32:25.857310+0000","last_became_active":"2024-06-27T21:29:51.673661+0000","last_became_peered":"2024-06-27T21:29:51.673661+0000","last_unstale":"2024-06-27T21:32:25.857310+0000","last_undegraded":"2024-06-27T21:32:25.857310+0000","last_fullsized":"2024-06-27T21:32:25.857310+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:39.707895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":131,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.855798+0000","last_change":"2024-06-27T21:31:22.784342+0000","last_active":"2024-06-27T21:32:25.855798+0000","last_peered":"2024-06-27T21:32:25.855798+0000","last_clean":"2024-06-27T21:32:25.855798+0000","last_became_active":"2024-06-27T21:29:51.676689+0000","last_became_peered":"2024-06-27T21:29:51.676689+0000","last_unstale":"2024-06-27T21:32:25.855798+0000","last_undegraded":"2024-06-27T21:32:25.855798+0000","last_fullsized":"2024-06-27T21:32:25.855798+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:22.784251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:22.784251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:33.888249+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":139,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857515+0000","last_change":"2024-06-27T21:32:13.808607+0000","last_active":"2024-06-27T21:32:25.857515+0000","last_peered":"2024-06-27T21:32:25.857515+0000","last_clean":"2024-06-27T21:32:25.857515+0000","last_became_active":"2024-06-27T21:32:13.808038+0000","last_became_peered":"2024-06-27T21:32:13.808038+0000","last_unstale":"2024-06-27T21:32:25.857515+0000","last_undegraded":"2024-06-27T21:32:25.857515+0000","last_fullsized":"2024-06-27T21:32:25.857515+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1],"acting":[0,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":156,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-06-27T21:32:20.827107+0000","last_change":"2024-06-27T21:32:13.808094+0000","last_active":"2024-06-27T21:32:20.827107+0000","last_peered":"2024-06-27T21:32:20.827107+0000","last_clean":"2024-06-27T21:32:20.827107+0000","last_became_active":"2024-06-27T21:32:13.807596+0000","last_became_peered":"2024-06-27T21:32:13.807596+0000","last_unstale":"2024-06-27T21:32:20.827107+0000","last_undegraded":"2024-06-27T21:32:20.827107+0000","last_fullsized":"2024-06-27T21:32:20.827107+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:56.251515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":420,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.858013+0000","last_change":"2024-06-27T21:32:13.813079+0000","last_active":"2024-06-27T21:32:25.858013+0000","last_peered":"2024-06-27T21:32:25.858013+0000","last_clean":"2024-06-27T21:32:25.858013+0000","last_became_active":"2024-06-27T21:32:13.809702+0000","last_became_peered":"2024-06-27T21:32:13.809702+0000","last_unstale":"2024-06-27T21:32:25.858013+0000","last_undegraded":"2024-06-27T21:32:25.858013+0000","last_fullsized":"2024-06-27T21:32:25.858013+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:13.565668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:30:13.565668+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1006240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"527'9592","reported_seq":9802,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.858388+0000","last_change":"2024-06-27T21:30:52.123816+0000","last_active":"2024-06-27T21:32:25.858388+0000","last_peered":"2024-06-27T21:32:25.858388+0000","last_clean":"2024-06-27T21:32:25.858388+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:32:25.858388+0000","last_undegraded":"2024-06-27T21:32:25.858388+0000","last_fullsized":"2024-06-27T21:32:25.858388+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6360","last_scrub_stamp":"2024-06-27T21:30:52.123515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.123515+0000","objects_scrubbed":6329,"log_size":9592,"log_dups_size":0,"ondisk_log_size":9592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260833280,"num_objects":3980,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":204,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-06-27T21:32:20.827367+0000","last_change":"2024-06-27T21:30:46.150422+0000","last_active":"2024-06-27T21:32:20.827367+0000","last_peered":"2024-06-27T21:32:20.827367+0000","last_clean":"2024-06-27T21:32:20.827367+0000","last_became_active":"2024-06-27T21:27:38.938524+0000","last_became_peered":"2024-06-27T21:27:38.938524+0000","last_unstale":"2024-06-27T21:32:20.827367+0000","last_undegraded":"2024-06-27T21:32:20.827367+0000","last_fullsized":"2024-06-27T21:32:20.827367+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:46.150318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:30:46.150318+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"527'9626","reported_seq":9833,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.858096+0000","last_change":"2024-06-27T21:30:48.343185+0000","last_active":"2024-06-27T21:32:25.858096+0000","last_peered":"2024-06-27T21:32:25.858096+0000","last_clean":"2024-06-27T21:32:25.858096+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:32:25.858096+0000","last_undegraded":"2024-06-27T21:32:25.858096+0000","last_fullsized":"2024-06-27T21:32:25.858096+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6100","last_scrub_stamp":"2024-06-27T21:30:48.342993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.342993+0000","objects_scrubbed":6058,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257556480,"num_objects":3930,"num_object_clones":0,"num_object_copies":11790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9626,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":50,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.842990+0000","last_change":"2024-06-27T21:31:33.228927+0000","last_active":"2024-06-27T21:32:25.842990+0000","last_peered":"2024-06-27T21:32:25.842990+0000","last_clean":"2024-06-27T21:32:25.842990+0000","last_became_active":"2024-06-27T21:31:32.310522+0000","last_became_peered":"2024-06-27T21:31:32.310522+0000","last_unstale":"2024-06-27T21:32:25.842990+0000","last_undegraded":"2024-06-27T21:32:25.842990+0000","last_fullsized":"2024-06-27T21:32:25.842990+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:33.228822+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:31:33.228822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:01.396646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"527'9506","reported_seq":14493,"reported_epoch":530,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:34.885494+0000","last_change":"2024-06-27T21:31:33.570353+0000","last_active":"2024-06-27T21:32:34.885494+0000","last_peered":"2024-06-27T21:32:34.885494+0000","last_clean":"2024-06-27T21:31:31.204488+0000","last_became_active":"2024-06-27T21:31:33.525500+0000","last_became_peered":"2024-06-27T21:31:33.525500+0000","last_unstale":"2024-06-27T21:32:34.885494+0000","last_undegraded":"2024-06-27T21:31:33.354229+0000","last_fullsized":"2024-06-27T21:31:33.314507+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":9506,"log_dups_size":0,"ondisk_log_size":9506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261947416,"num_objects":3998,"num_object_clones":0,"num_object_copies":11994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":210,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9506,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":156106752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1],"avail_no_missing":["10","1","5"],"object_location_counts":[{"shards":"1,5,10","objects":2705},{"shards":"1,10","objects":1083}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":981,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857251+0000","last_change":"2024-06-27T21:32:13.809592+0000","last_active":"2024-06-27T21:32:25.857251+0000","last_peered":"2024-06-27T21:32:25.857251+0000","last_clean":"2024-06-27T21:32:25.857251+0000","last_became_active":"2024-06-27T21:32:13.809197+0000","last_became_peered":"2024-06-27T21:32:13.809197+0000","last_unstale":"2024-06-27T21:32:25.857251+0000","last_undegraded":"2024-06-27T21:32:25.857251+0000","last_fullsized":"2024-06-27T21:32:25.857251+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:41.800734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"527'9719","reported_seq":13403,"reported_epoch":528,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:22.370464+0000","last_change":"2024-06-27T21:32:14.121194+0000","last_active":"2024-06-27T21:32:22.370464+0000","last_peered":"2024-06-27T21:32:22.370464+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:32:14.047570+0000","last_became_peered":"2024-06-27T21:32:14.047570+0000","last_unstale":"2024-06-27T21:32:22.370464+0000","last_undegraded":"2024-06-27T21:32:13.843384+0000","last_fullsized":"2024-06-27T21:32:13.802097+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":9719,"log_dups_size":0,"ondisk_log_size":9719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":3519,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":113573888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"6,7,11","objects":3519},{"shards":"7,11","objects":386}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":545,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.858840+0000","last_change":"2024-06-27T21:31:22.939385+0000","last_active":"2024-06-27T21:32:25.858840+0000","last_peered":"2024-06-27T21:32:25.858840+0000","last_clean":"2024-06-27T21:32:25.858840+0000","last_became_active":"2024-06-27T21:31:22.938995+0000","last_became_peered":"2024-06-27T21:31:22.938995+0000","last_unstale":"2024-06-27T21:32:25.858840+0000","last_undegraded":"2024-06-27T21:32:25.858840+0000","last_fullsized":"2024-06-27T21:32:25.858840+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:20.148961+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:31:20.148961+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:36.027802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"527'9380","reported_seq":9591,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.859135+0000","last_change":"2024-06-27T21:30:53.801878+0000","last_active":"2024-06-27T21:32:25.859135+0000","last_peered":"2024-06-27T21:32:25.859135+0000","last_clean":"2024-06-27T21:32:25.859135+0000","last_became_active":"2024-06-27T21:29:19.921089+0000","last_became_peered":"2024-06-27T21:29:19.921089+0000","last_unstale":"2024-06-27T21:32:25.859135+0000","last_undegraded":"2024-06-27T21:32:25.859135+0000","last_fullsized":"2024-06-27T21:32:25.859135+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6306","last_scrub_stamp":"2024-06-27T21:30:53.801648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:53.801648+0000","objects_scrubbed":6238,"log_size":9380,"log_dups_size":0,"ondisk_log_size":9380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253493248,"num_objects":3868,"num_object_clones":0,"num_object_copies":11604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":423936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":541,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857657+0000","last_change":"2024-06-27T21:32:13.826072+0000","last_active":"2024-06-27T21:32:25.857657+0000","last_peered":"2024-06-27T21:32:25.857657+0000","last_clean":"2024-06-27T21:32:25.857657+0000","last_became_active":"2024-06-27T21:32:13.825703+0000","last_became_peered":"2024-06-27T21:32:13.825703+0000","last_unstale":"2024-06-27T21:32:25.857657+0000","last_undegraded":"2024-06-27T21:32:25.857657+0000","last_fullsized":"2024-06-27T21:32:25.857657+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:57.941423+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:29:57.941423+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"527'9218","reported_seq":11482,"reported_epoch":531,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:35.385148+0000","last_change":"2024-06-27T21:32:15.169057+0000","last_active":"2024-06-27T21:32:35.385148+0000","last_peered":"2024-06-27T21:32:35.385148+0000","last_clean":"2024-06-27T21:32:12.748576+0000","last_became_active":"2024-06-27T21:32:15.075574+0000","last_became_peered":"2024-06-27T21:32:15.075574+0000","last_unstale":"2024-06-27T21:32:35.385148+0000","last_undegraded":"2024-06-27T21:32:14.833604+0000","last_fullsized":"2024-06-27T21:32:14.802569+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'5961","last_scrub_stamp":"2024-06-27T21:30:49.972634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:49.972634+0000","objects_scrubbed":5918,"log_size":9218,"log_dups_size":0,"ondisk_log_size":9218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252182528,"num_objects":3848,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":2830,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1026,"num_bytes_recovered":67239936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,7],"acting":[9,10],"avail_no_missing":["9","8","10"],"object_location_counts":[{"shards":"8,9,10","objects":2830},{"shards":"9,10","objects":1012}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":461,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857729+0000","last_change":"2024-06-27T21:31:32.310553+0000","last_active":"2024-06-27T21:32:25.857729+0000","last_peered":"2024-06-27T21:32:25.857729+0000","last_clean":"2024-06-27T21:32:25.857729+0000","last_became_active":"2024-06-27T21:31:32.310137+0000","last_became_peered":"2024-06-27T21:31:32.310137+0000","last_unstale":"2024-06-27T21:32:25.857729+0000","last_undegraded":"2024-06-27T21:32:25.857729+0000","last_fullsized":"2024-06-27T21:32:25.857729+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:48.058828+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.058828+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"527'9578","reported_seq":14605,"reported_epoch":530,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:31.510727+0000","last_change":"2024-06-27T21:31:24.150930+0000","last_active":"2024-06-27T21:32:31.510727+0000","last_peered":"2024-06-27T21:32:31.510727+0000","last_clean":"2024-06-27T21:31:21.922862+0000","last_became_active":"2024-06-27T21:31:24.117632+0000","last_became_peered":"2024-06-27T21:31:24.117632+0000","last_unstale":"2024-06-27T21:32:31.510727+0000","last_undegraded":"2024-06-27T21:31:23.977903+0000","last_fullsized":"2024-06-27T21:31:23.937741+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'5888","last_scrub_stamp":"2024-06-27T21:30:45.481539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.481539+0000","objects_scrubbed":5828,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1590,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252444672,"num_objects":3852,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":191,"num_objects_misplaced":2890,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2409,"num_bytes_recovered":157745152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2","objects":771},{"shards":"0,2,5","objects":2890}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":552,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857556+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:32:25.857556+0000","last_peered":"2024-06-27T21:32:25.857556+0000","last_clean":"2024-06-27T21:32:25.857556+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:32:25.857556+0000","last_undegraded":"2024-06-27T21:32:25.857556+0000","last_fullsized":"2024-06-27T21:32:25.857556+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"527'9557","reported_seq":15120,"reported_epoch":530,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:33.223293+0000","last_change":"2024-06-27T21:31:32.527777+0000","last_active":"2024-06-27T21:32:33.223293+0000","last_peered":"2024-06-27T21:32:33.223293+0000","last_clean":"2024-06-27T21:31:21.899317+0000","last_became_active":"2024-06-27T21:31:32.491754+0000","last_became_peered":"2024-06-27T21:31:32.491754+0000","last_unstale":"2024-06-27T21:32:33.223293+0000","last_undegraded":"2024-06-27T21:31:32.338579+0000","last_fullsized":"2024-06-27T21:31:32.307033+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'5932","last_scrub_stamp":"2024-06-27T21:30:46.556700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:46.556700+0000","objects_scrubbed":5888,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254869504,"num_objects":3889,"num_object_clones":0,"num_object_copies":11667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2664,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":430272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2675,"num_bytes_recovered":175046656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":2664},{"shards":"4,7","objects":1223}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":223,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-06-27T21:32:20.827417+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:32:20.827417+0000","last_peered":"2024-06-27T21:32:20.827417+0000","last_clean":"2024-06-27T21:32:20.827417+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:32:20.827417+0000","last_undegraded":"2024-06-27T21:32:20.827417+0000","last_fullsized":"2024-06-27T21:32:20.827417+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:36.452075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"527'9475","reported_seq":14051,"reported_epoch":528,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:22.775092+0000","last_change":"2024-06-27T21:31:24.180605+0000","last_active":"2024-06-27T21:32:22.775092+0000","last_peered":"2024-06-27T21:32:22.775092+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:31:24.125120+0000","last_became_peered":"2024-06-27T21:31:24.125120+0000","last_unstale":"2024-06-27T21:32:22.775092+0000","last_undegraded":"2024-06-27T21:31:23.981588+0000","last_fullsized":"2024-06-27T21:31:23.941280+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":9475,"log_dups_size":0,"ondisk_log_size":9475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246480896,"num_objects":3761,"num_object_clones":0,"num_object_copies":11283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":3092,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9475,"num_write_kb":423552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2185,"num_bytes_recovered":143065088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"6,7,9","objects":3092},{"shards":"7,9","objects":569}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":111,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.842960+0000","last_change":"2024-06-27T21:32:13.812610+0000","last_active":"2024-06-27T21:32:25.842960+0000","last_peered":"2024-06-27T21:32:25.842960+0000","last_clean":"2024-06-27T21:32:25.842960+0000","last_became_active":"2024-06-27T21:32:13.812217+0000","last_became_peered":"2024-06-27T21:32:13.812217+0000","last_unstale":"2024-06-27T21:32:25.842960+0000","last_undegraded":"2024-06-27T21:32:25.842960+0000","last_fullsized":"2024-06-27T21:32:25.842960+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,10],"acting":[4,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"527'9821","reported_seq":10029,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.845981+0000","last_change":"2024-06-27T21:30:48.475928+0000","last_active":"2024-06-27T21:32:25.845981+0000","last_peered":"2024-06-27T21:32:25.845981+0000","last_clean":"2024-06-27T21:32:25.845981+0000","last_became_active":"2024-06-27T21:29:19.889439+0000","last_became_peered":"2024-06-27T21:29:19.889439+0000","last_unstale":"2024-06-27T21:32:25.845981+0000","last_undegraded":"2024-06-27T21:32:25.845981+0000","last_fullsized":"2024-06-27T21:32:25.845981+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260374528,"num_objects":3973,"num_object_clones":0,"num_object_copies":11919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":441408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":220,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-06-27T21:32:20.827310+0000","last_change":"2024-06-27T21:29:56.956056+0000","last_active":"2024-06-27T21:32:20.827310+0000","last_peered":"2024-06-27T21:32:20.827310+0000","last_clean":"2024-06-27T21:32:20.827310+0000","last_became_active":"2024-06-27T21:29:49.671863+0000","last_became_peered":"2024-06-27T21:29:49.671863+0000","last_unstale":"2024-06-27T21:32:20.827310+0000","last_undegraded":"2024-06-27T21:32:20.827310+0000","last_fullsized":"2024-06-27T21:32:20.827310+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.955983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.955983+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"527'9415","reported_seq":9667,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.997290+0000","last_change":"2024-06-27T21:32:25.997427+0000","last_active":"2024-06-27T21:32:25.997290+0000","last_peered":"2024-06-27T21:32:25.997290+0000","last_clean":"2024-06-27T21:32:25.997290+0000","last_became_active":"2024-06-27T21:29:19.822526+0000","last_became_peered":"2024-06-27T21:29:19.822526+0000","last_unstale":"2024-06-27T21:32:25.997290+0000","last_undegraded":"2024-06-27T21:32:25.997290+0000","last_fullsized":"2024-06-27T21:32:25.997290+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'9415","last_scrub_stamp":"2024-06-27T21:32:25.997194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:25.997194+0000","objects_scrubbed":3847,"log_size":9415,"log_dups_size":0,"ondisk_log_size":9415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:52.701904+0000","scrub_duration":1334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252116992,"num_objects":3847,"num_object_clones":0,"num_object_copies":11541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9415,"num_write_kb":424384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":739,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:26.712752+0000","last_change":"2024-06-27T21:32:26.712793+0000","last_active":"2024-06-27T21:32:26.712752+0000","last_peered":"2024-06-27T21:32:26.712752+0000","last_clean":"2024-06-27T21:32:26.712752+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:32:26.712752+0000","last_undegraded":"2024-06-27T21:32:26.712752+0000","last_fullsized":"2024-06-27T21:32:26.712752+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:26.712657+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:32:26.712657+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:37.728190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"527'9623","reported_seq":9826,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-06-27T21:32:20.827215+0000","last_change":"2024-06-27T21:30:45.505788+0000","last_active":"2024-06-27T21:32:20.827215+0000","last_peered":"2024-06-27T21:32:20.827215+0000","last_clean":"2024-06-27T21:32:20.827215+0000","last_became_active":"2024-06-27T21:29:19.921198+0000","last_became_peered":"2024-06-27T21:29:19.921198+0000","last_unstale":"2024-06-27T21:32:20.827215+0000","last_undegraded":"2024-06-27T21:32:20.827215+0000","last_fullsized":"2024-06-27T21:32:20.827215+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'5955","last_scrub_stamp":"2024-06-27T21:30:45.505543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.505543+0000","objects_scrubbed":5927,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256704512,"num_objects":3917,"num_object_clones":0,"num_object_copies":11751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9623,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":131,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.842991+0000","last_change":"2024-06-27T21:31:19.371088+0000","last_active":"2024-06-27T21:32:25.842991+0000","last_peered":"2024-06-27T21:32:25.842991+0000","last_clean":"2024-06-27T21:32:25.842991+0000","last_became_active":"2024-06-27T21:29:51.668654+0000","last_became_peered":"2024-06-27T21:29:51.668654+0000","last_unstale":"2024-06-27T21:32:25.842991+0000","last_undegraded":"2024-06-27T21:32:25.842991+0000","last_fullsized":"2024-06-27T21:32:25.842991+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.370974+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.370974+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:49.073740+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"527'9473","reported_seq":9688,"reported_epoch":530,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:25.857641+0000","last_change":"2024-06-27T21:32:15.062048+0000","last_active":"2024-06-27T21:32:25.857641+0000","last_peered":"2024-06-27T21:32:25.857641+0000","last_clean":"2024-06-27T21:32:12.787863+0000","last_became_active":"2024-06-27T21:32:15.039338+0000","last_became_peered":"2024-06-27T21:32:15.039338+0000","last_unstale":"2024-06-27T21:32:25.857641+0000","last_undegraded":"2024-06-27T21:32:14.836078+0000","last_fullsized":"2024-06-27T21:32:14.801629+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6309","last_scrub_stamp":"2024-06-27T21:30:52.261532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.261532+0000","objects_scrubbed":6289,"log_size":9473,"log_dups_size":0,"ondisk_log_size":9473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9473,"num_write_kb":431808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9],"avail_no_missing":["0","8","9"],"object_location_counts":[{"shards":"0,8,9","objects":4021},{"shards":"0,9","objects":-13}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":119,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-06-27T21:32:20.827170+0000","last_change":"2024-06-27T21:29:58.941232+0000","last_active":"2024-06-27T21:32:20.827170+0000","last_peered":"2024-06-27T21:32:20.827170+0000","last_clean":"2024-06-27T21:32:20.827170+0000","last_became_active":"2024-06-27T21:29:51.672132+0000","last_became_peered":"2024-06-27T21:29:51.672132+0000","last_unstale":"2024-06-27T21:32:20.827170+0000","last_undegraded":"2024-06-27T21:32:20.827170+0000","last_fullsized":"2024-06-27T21:32:20.827170+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:58.941114+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:29:58.941114+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"527'9481","reported_seq":14242,"reported_epoch":530,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:31.521456+0000","last_change":"2024-06-27T21:31:33.655323+0000","last_active":"2024-06-27T21:32:31.521456+0000","last_peered":"2024-06-27T21:32:31.521456+0000","last_clean":"2024-06-27T21:31:31.288940+0000","last_became_active":"2024-06-27T21:31:33.577869+0000","last_became_peered":"2024-06-27T21:31:33.577869+0000","last_unstale":"2024-06-27T21:32:31.521456+0000","last_undegraded":"2024-06-27T21:31:33.336543+0000","last_fullsized":"2024-06-27T21:31:33.308088+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":9481,"log_dups_size":0,"ondisk_log_size":9481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256704512,"num_objects":3917,"num_object_clones":0,"num_object_copies":11751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":2918,"num_objects_unfound":0,"num_objects_dirty":3917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9481,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2273,"num_bytes_recovered":148963328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":2918},{"shards":"0,6","objects":933}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"527'9389","reported_seq":9638,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:26.025047+0000","last_change":"2024-06-27T21:32:26.025196+0000","last_active":"2024-06-27T21:32:26.025047+0000","last_peered":"2024-06-27T21:32:26.025047+0000","last_clean":"2024-06-27T21:32:26.025047+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:32:26.025047+0000","last_undegraded":"2024-06-27T21:32:26.025047+0000","last_fullsized":"2024-06-27T21:32:26.025047+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'9389","last_scrub_stamp":"2024-06-27T21:32:26.024974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:26.024974+0000","objects_scrubbed":3795,"log_size":9389,"log_dups_size":0,"ondisk_log_size":9389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:31.641075+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248709120,"num_objects":3795,"num_object_clones":0,"num_object_copies":11385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":138,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857585+0000","last_change":"2024-06-27T21:31:23.211897+0000","last_active":"2024-06-27T21:32:25.857585+0000","last_peered":"2024-06-27T21:32:25.857585+0000","last_clean":"2024-06-27T21:32:25.857585+0000","last_became_active":"2024-06-27T21:29:51.676349+0000","last_became_peered":"2024-06-27T21:29:51.676349+0000","last_unstale":"2024-06-27T21:32:25.857585+0000","last_undegraded":"2024-06-27T21:32:25.857585+0000","last_fullsized":"2024-06-27T21:32:25.857585+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.211775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.211775+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:36.362115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"524'9794","reported_seq":14356,"reported_epoch":528,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-06-27T21:32:22.720204+0000","last_change":"2024-06-27T21:32:22.720204+0000","last_active":"2024-06-27T21:32:12.818032+0000","last_peered":"2024-06-27T21:32:12.790357+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:31:33.589674+0000","last_became_peered":"2024-06-27T21:31:33.589674+0000","last_unstale":"2024-06-27T21:32:22.720204+0000","last_undegraded":"2024-06-27T21:32:22.691354+0000","last_fullsized":"2024-06-27T21:32:22.682941+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264110080,"num_objects":4030,"num_object_clones":0,"num_object_copies":12090,"num_objects_missing_on_primary":5404,"num_objects_missing":5404,"num_objects_degraded":17895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9794,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1507,"num_bytes_recovered":98500608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10],"avail_no_missing":["5","6","8"],"object_location_counts":[{"shards":"7,8,10","objects":3},{"shards":"8","objects":4027}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":137,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-06-27T21:32:25.857205+0000","last_change":"2024-06-27T21:31:22.939947+0000","last_active":"2024-06-27T21:32:25.857205+0000","last_peered":"2024-06-27T21:32:25.857205+0000","last_clean":"2024-06-27T21:32:25.857205+0000","last_became_active":"2024-06-27T21:31:22.939550+0000","last_became_peered":"2024-06-27T21:31:22.939550+0000","last_unstale":"2024-06-27T21:32:25.857205+0000","last_undegraded":"2024-06-27T21:32:25.857205+0000","last_fullsized":"2024-06-27T21:32:25.857205+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.017115+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.017115+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:32:39.655374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4098359320,"num_objects":62537,"num_object_clones":0,"num_object_copies":187611,"num_objects_missing_on_primary":5404,"num_objects_missing":5404,"num_objects_degraded":25220,"num_objects_misplaced":20618,"num_objects_unfound":0,"num_objects_dirty":62537,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152647,"num_write_kb":6885825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16198,"num_bytes_recovered":1060241408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13543419904,"data_stored":13543407688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152647,"ondisk_log_size":152647,"up":48,"acting":39,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1006240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253144,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385628,"kb_used_data":831060,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92359524,"statfs":{"total":95995035648,"available":94576152576,"internally_reserved":0,"allocated":851005440,"data_stored":850749271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514780,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226464,"kb_used_data":830168,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92518688,"statfs":{"total":95995035648,"available":94739136512,"internally_reserved":0,"allocated":850092032,"data_stored":849831791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743713,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1682528,"kb_used_data":1254292,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92062624,"statfs":{"total":95995035648,"available":94272126976,"internally_reserved":0,"allocated":1284395008,"data_stored":1284138839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":266287972644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266982,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712560,"kb_used_data":1248232,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92032592,"statfs":{"total":95995035648,"available":94241374208,"internally_reserved":0,"allocated":1278189568,"data_stored":1277902341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528623,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142172,"kb_used_data":1645780,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91602980,"statfs":{"total":95995035648,"available":93801451520,"internally_reserved":0,"allocated":1685278720,"data_stored":1685022551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757556,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2239584,"kb_used_data":1871064,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":91505568,"statfs":{"total":95995035648,"available":93701701632,"internally_reserved":0,"allocated":1915969536,"data_stored":1915709295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161260,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208868,"kb_used_data":995036,"kb_used_omap":92,"kb_used_meta":213603,"kb_avail":92536284,"statfs":{"total":95995035648,"available":94757154816,"internally_reserved":0,"allocated":1018916864,"data_stored":1018657143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":218729570},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182716,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1096604,"kb_used_data":740436,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92648548,"statfs":{"total":95995035648,"available":94872113152,"internally_reserved":0,"allocated":758206464,"data_stored":757950295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411649,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1477092,"kb_used_data":1157148,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92268060,"statfs":{"total":95995035648,"available":94482493440,"internally_reserved":0,"allocated":1184919552,"data_stored":1184659831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1518176,"kb_used_data":1005144,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92226976,"statfs":{"total":95995035648,"available":94440423424,"internally_reserved":0,"allocated":1029267456,"data_stored":1029007215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":525203909},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335268,"kb_used_data":875100,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92409884,"statfs":{"total":95995035648,"available":94627721216,"internally_reserved":0,"allocated":896102400,"data_stored":895842679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":895287296,"data_stored":895287296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1028591616,"data_stored":1028587544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1188560896,"data_stored":1188560896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":757530624,"data_stored":757530624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1018036224,"data_stored":1018036224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1915293696,"data_stored":1915289624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1684275200,"data_stored":1684275200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1283260416,"data_stored":1283260416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":784138240,"data_stored":784138240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1283391488,"data_stored":1283391488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":852561920,"data_stored":852557848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":852492288,"data_stored":852492288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:32:40.614 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-27T21:32:40.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-06-27T21:32:40.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:40.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:40.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:40.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:40.864 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:40.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:40.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:40.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:41.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:41.113 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:41.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:40 smithi151 ceph-mon[28965]: pgmap v1389: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 9 op/s; 25220/187392 objects degraded (13.458%); 17813/187392 objects misplaced (9.506%); 28 MiB/s, 454 objects/s recovering 2024-06-27T21:32:41.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:40 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 25220/187392 objects degraded (13.458%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:41.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163695 4.b 4.43f0c96b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.927129+0000", 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 537.28587585100001, 2024-06-27T21:32:41.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.59741082000000001, 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163695 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927129+0000", 2024-06-27T21:32:41.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927129+0000", 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927131+0000", 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6700000000000001e-06 2024-06-27T21:32:41.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927138+0000", 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4800000000000004e-06 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927153+0000", 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4185000000000001e-05 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927161+0000", 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4670000000000004e-06 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927269+0000", 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108291 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.927281+0000", 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2072000000000001e-05 2024-06-27T21:32:41.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.524502+0000", 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.59722061599999998 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.524540+0000", 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8039e-05 2024-06-27T21:32:41.221 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163706 4.b 4.8d253dcb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.945650+0000", 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 537.26735470699998, 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57931229299999998, 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163706 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945650+0000", 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945650+0000", 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945652+0000", 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8589999999999999e-06 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945663+0000", 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0889e-05 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945680+0000", 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6364000000000001e-05 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945687+0000", 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0389999999999999e-06 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945764+0000", 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7813999999999997e-05 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945779+0000", 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4993e-05 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.524954+0000", 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57917454400000001 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.524963+0000", 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7909999999999993e-06 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163711 4.b 4.4a1a48eb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.948565+0000", 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 537.26444034799999, 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57642689999999996, 2024-06-27T21:32:41.227 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163711 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948565+0000", 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948565+0000", 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948566+0000", 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.584e-06 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948574+0000", 2024-06-27T21:32:41.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3540000000000002e-06 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948590+0000", 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6325999999999999e-05 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948598+0000", 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9680000000000006e-06 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948669+0000", 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0508999999999999e-05 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.948681+0000", 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2078e-05 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.524987+0000", 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57630615399999996 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.524992+0000", 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9269999999999996e-06 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:163712 4.0 e362/282 4:0319ddeb:::benchmark_data_smithi192_88961_object72973:head v 362'10416, mlcod=361'10413)", 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.950524+0000", 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 537.26248074299997, 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56522372700000001, 2024-06-27T21:32:41.232 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950524+0000", 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950524+0000", 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950525+0000", 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6400000000000001e-07 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950533+0000", 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6260000000000003e-06 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.511949+0000", 2024-06-27T21:32:41.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.56141659300000002 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.511957+0000", 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7649999999999993e-06 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.512005+0000", 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7935000000000001e-05 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.512023+0000", 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7472000000000001e-05 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.515708+0000", 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036855 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.515748+0000", 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0071999999999999e-05 2024-06-27T21:32:41.236 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163695 4.b 4:d6930fc2:::benchmark_data_smithi192_88961_object72956:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.638280+0000", 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 535.57472528400001, 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37816459299999999, 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.237 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163695 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638280+0000", 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638280+0000", 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638282+0000", 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.798e-06 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638288+0000", 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.567e-06 2024-06-27T21:32:41.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638300+0000", 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1313e-05 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638307+0000", 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3300000000000001e-06 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638355+0000", 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8362000000000002e-05 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638365+0000", 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0087e-05 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.987185+0000", 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34882000400000002 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.987330+0000", 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144468 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.987480+0000", 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150598 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001546+0000", 2024-06-27T21:32:41.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014065424 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001610+0000", 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3856999999999995e-05 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016094+0000", 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014484245 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016166+0000", 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2123e-05 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016444+0000", 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027841699999999999 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163706 4.b 4:d3bca4b1:::benchmark_data_smithi192_88961_object72967:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:32:41.244 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.638325+0000", 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 535.57467973400003, 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.378436997, 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163706 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638325+0000", 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638325+0000", 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638326+0000", 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5000000000000003e-07 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638329+0000", 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-06 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638335+0000", 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5100000000000004e-06 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638341+0000", 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5509999999999999e-06 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638405+0000", 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4529999999999994e-05 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638414+0000", 2024-06-27T21:32:41.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7509999999999997e-06 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.992491+0000", 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35407698999999998 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.992623+0000", 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013168699999999999 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.992761+0000", 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000137761 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001714+0000", 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089537839999999994 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001752+0000", 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7950000000000001e-05 2024-06-27T21:32:41.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016515+0000", 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014762160999999999 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016599+0000", 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4841999999999998e-05 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016762+0000", 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016302499999999999 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163711 4.b 4:d7125852:::benchmark_data_smithi192_88961_object72972:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.638348+0000", 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 535.57465666999997, 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37869991200000003, 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.253 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163711 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638348+0000", 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638348+0000", 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638349+0000", 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8500000000000002e-07 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638351+0000", 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.278e-06 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638359+0000", 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5909999999999998e-06 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638364+0000", 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.587e-06 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638481+0000", 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116495 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638493+0000", 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2612999999999999e-05 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.997498+0000", 2024-06-27T21:32:41.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35900418200000001 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.997595+0000", 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6952000000000003e-05 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.997682+0000", 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7017999999999999e-05 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001759+0000", 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040777699999999997 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.006154+0000", 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004394617 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016850+0000", 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010696199 2024-06-27T21:32:41.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.016906+0000", 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5544000000000002e-05 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.017048+0000", 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142681 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163717 4.b 4:de8e949f:::benchmark_data_smithi192_88961_object72978:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:45.638388+0000", 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 535.574616827, 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37891473399999998, 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163717 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638388+0000", 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638388+0000", 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638389+0000", 2024-06-27T21:32:41.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0100000000000005e-07 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638392+0000", 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7489999999999999e-06 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638406+0000", 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4848000000000001e-05 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638412+0000", 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6350000000000003e-06 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638512+0000", 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9803999999999995e-05 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:45.638517+0000", 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6439999999999997e-06 2024-06-27T21:32:41.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001797+0000", 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36327949100000001 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.001940+0000", 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014286799999999999 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:32:41.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.002080+0000", 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000139779 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.006286+0000", 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004206624 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.006560+0000", 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027340099999999999 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.017099+0000", 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010538838 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.017175+0000", 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6625999999999994e-05 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:46.017303+0000", 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012792599999999999 2024-06-27T21:32:41.267 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(4.b,GRANT,e:376,from:4,reservation_nonce:0)", 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:27.820500+0000", 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 493.39250506500002, 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30962536800000001, 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:27.820500+0000", 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:27.820500+0000", 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:27.820500+0000", 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8000000000000001e-07 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:27.820502+0000", 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.561e-06 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:28.129939+0000", 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30943664700000001 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:28.129947+0000", 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7609999999999996e-06 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:28.130000+0000", 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2899999999999998e-05 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:28.130016+0000", 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5676e-05 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:28.130125+0000", 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109443 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:163151 5.c 5:3b58818c:::benchmark_data_smithi192_104167_object57110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-06-27T21:32:41.272 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:27:45.859443+0000", 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 295.35356164699999, 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47873714099999998, 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163151 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859443+0000", 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859443+0000", 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859445+0000", 2024-06-27T21:32:41.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.314e-06 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859449+0000", 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.668e-06 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859461+0000", 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1090999999999999e-05 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859468+0000", 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.712e-06 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859510+0000", 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1700999999999999e-05 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859603+0000", 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2633999999999994e-05 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.859681+0000", 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8784999999999999e-05 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.864293+0000", 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046114770000000001 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:45.864593+0000", 2024-06-27T21:32:41.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030020300000000003 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:46.338099+0000", 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47350551400000002 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:46.338141+0000", 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2369e-05 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:27:46.338181+0000", 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9672999999999998e-05 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25615.0:212063 5.5 5:a82762d3:::benchmark_data_smithi192_104167_object106022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:28:52.511466+0000", 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 228.701538666, 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79036286200000005, 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.279 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25615", 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 212063 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511466+0000", 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511466+0000", 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511468+0000", 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4139999999999999e-06 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511472+0000", 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2950000000000004e-06 2024-06-27T21:32:41.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511484+0000", 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.172e-05 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511489+0000", 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4809999999999997e-06 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511534+0000", 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4718000000000001e-05 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511614+0000", 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0304999999999998e-05 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.511681+0000", 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.669e-05 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.515622+0000", 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039411109999999997 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.516030+0000", 2024-06-27T21:32:41.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040790500000000002 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301752+0000", 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78572220699999995 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301793+0000", 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0846999999999997e-05 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301829+0000", 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6168999999999999e-05 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25615.0:212063 5.5 e462/382 ondisk, result = 0)", 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:28:52.516447+0000", 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 228.69655862100001, 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78538869700000002, 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.516447+0000", 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.516447+0000", 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.516447+0000", 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5499999999999998e-07 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:52.516449+0000", 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.334e-06 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301630+0000", 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.78518026299999999 2024-06-27T21:32:41.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301644+0000", 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.437e-05 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301740+0000", 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5864000000000002e-05 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301748+0000", 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0129999999999993e-06 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:28:53.301835+0000", 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7298000000000003e-05 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.a 530/516 [PushReplyOp(6:5f3a3f53:::benchmark_data_smithi192_117994_object59178:head)])", 2024-06-27T21:32:41.290 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:28.170868+0000", 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.042137045, 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71445639100000002, 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.170868+0000", 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.170868+0000", 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.170869+0000", 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9800000000000003e-07 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.170875+0000", 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3500000000000002e-06 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.881459+0000", 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.71058392000000004 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.881468+0000", 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4120000000000001e-06 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.881525+0000", 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7615000000000003e-05 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.885324+0000", 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037992960000000002 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.a 530/516 [PushReplyOp(6:53c4a9c7:::benchmark_data_smithi192_117994_object59231:head)])", 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:28.909458+0000", 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.303547633999999, 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39294016900000001, 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.294 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.909458+0000", 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.909458+0000", 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.909459+0000", 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0920000000000001e-06 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:28.909464+0000", 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.558e-06 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:29.298938+0000", 2024-06-27T21:32:41.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38947366500000002 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:29.298945+0000", 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1060000000000002e-06 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:29.299020+0000", 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5512000000000005e-05 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:29.302398+0000", 2024-06-27T21:32:41.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003377236 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.a 530/516 [PushReplyOp(6:5806ca8a:::benchmark_data_smithi192_117994_object60563:head)])", 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:30.340194+0000", 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.872810683000001, 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83808573500000005, 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.298 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:30.340194+0000", 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:30.340194+0000", 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:30.340195+0000", 2024-06-27T21:32:41.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5999999999999991e-07 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:30.340202+0000", 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0380000000000002e-06 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:31.175303+0000", 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83510076600000005 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:31.175308+0000", 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1830000000000002e-06 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:31.175357+0000", 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8588000000000003e-05 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:31.178280+0000", 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029231999999999999 2024-06-27T21:32:41.301 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.a 530/516 [PushReplyOp(6:528313a8:::benchmark_data_smithi192_117994_object61397:head)])", 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:32.035014+0000", 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.1779909530000001, 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.680305414, 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.035014+0000", 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.035014+0000", 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.035015+0000", 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.09e-07 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.035020+0000", 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.527e-06 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.711636+0000", 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67661591899999995 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.711652+0000", 2024-06-27T21:32:41.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6537000000000001e-05 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.711774+0000", 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121882 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:32.715320+0000", 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035456400000000001 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:152692 6.a 6:587d9b67:::benchmark_data_smithi192_117994_object45098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:37.260421+0000", 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.952584399, 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.373052628, 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152692 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.306 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:40 smithi190 ceph-mon[32247]: pgmap v1389: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 9 op/s; 25220/187392 objects degraded (13.458%); 17813/187392 objects misplaced (9.506%); 28 MiB/s, 454 objects/s recovering 2024-06-27T21:32:41.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:40 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 25220/187392 objects degraded (13.458%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:41.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260421+0000", 2024-06-27T21:32:41.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260421+0000", 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260423+0000", 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.818e-06 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260430+0000", 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9109999999999993e-06 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260446+0000", 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5515e-05 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260455+0000", 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7539999999999995e-06 2024-06-27T21:32:41.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260515+0000", 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0343000000000001e-05 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260832+0000", 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000316507 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,6", 2024-06-27T21:32:41.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.260914+0000", 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2805000000000004e-05 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.264544+0000", 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036292249999999998 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.266051+0000", 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00150724 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.631698+0000", 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36564708600000001 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.631739+0000", 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0970000000000002e-05 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.633473+0000", 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017344540000000001 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.a 531/516 [PushReplyOp(6:5db6e1e7:::benchmark_data_smithi192_117994_object67097:head)])", 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:38.457077+0000", 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 2.7559278090000001, 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37403090700000002, 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.314 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.457077+0000", 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.457077+0000", 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.457078+0000", 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1310000000000001e-06 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.457085+0000", 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3029999999999996e-06 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.828263+0000", 2024-06-27T21:32:41.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37117842400000001 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.828287+0000", 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3407000000000001e-05 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.828339+0000", 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2512999999999997e-05 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:38.831108+0000", 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027691289999999999 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:152934 6.9 6:9f3ccd01:::benchmark_data_smithi192_117994_object45340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:40.164387+0000", 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.048618209, 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48721907199999998, 2024-06-27T21:32:41.318 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152934 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164387+0000", 2024-06-27T21:32:41.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164387+0000", 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164389+0000", 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164394+0000", 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7200000000000003e-06 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164406+0000", 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1426999999999999e-05 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164412+0000", 2024-06-27T21:32:41.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.438e-06 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164476+0000", 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3745999999999996e-05 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164637+0000", 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016085200000000001 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.164795+0000", 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015783299999999999 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.172613+0000", 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0078187680000000002 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.172636+0000", 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2904e-05 2024-06-27T21:32:41.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.648055+0000", 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47541857399999998 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.648129+0000", 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3852999999999999e-05 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.651606+0000", 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034773510000000001 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:152934 6.9 e532/526 ondisk, result = 0)", 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:40.168815+0000", 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.0441899809999999, 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48279854900000002, 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:32:41.325 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.168815+0000", 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.168815+0000", 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.168816+0000", 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5099999999999998e-07 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.168826+0000", 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9359999999999994e-06 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.647951+0000", 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47912462300000003 2024-06-27T21:32:41.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.647959+0000", 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1980000000000001e-06 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.648032+0000", 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3355999999999994e-05 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.648045+0000", 2024-06-27T21:32:41.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2802e-05 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:40.651614+0000", 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035687829999999999 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:32:41.329 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:32:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:41.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:41.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:40 smithi192 ceph-mon[32956]: pgmap v1389: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 9 op/s; 25220/187392 objects degraded (13.458%); 17813/187392 objects misplaced (9.506%); 28 MiB/s, 454 objects/s recovering 2024-06-27T21:32:41.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:40 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 25220/187392 objects degraded (13.458%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:41.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:41.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:41.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:41.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:41.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:41.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:41.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:41.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:41.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:41.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:41.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:42.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:42.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:42.169 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:42.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:42.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:42.218 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:42.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:42.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:42.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:42.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:42.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:42.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:42.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:42.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:42.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:42.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:42.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:42.955 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:42.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:43.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:43.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:43.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:43.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:43.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:43.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:43.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:43.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:43 smithi151 ceph-mon[28965]: pgmap v1390: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 7328/186870 objects degraded (3.921%); 28770/186870 objects misplaced (15.396%); 35 MiB/s, 552 objects/s recovering 2024-06-27T21:32:43.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:43.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:43 smithi190 ceph-mon[32247]: pgmap v1390: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 7328/186870 objects degraded (3.921%); 28770/186870 objects misplaced (15.396%); 35 MiB/s, 552 objects/s recovering 2024-06-27T21:32:43.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:43 smithi192 ceph-mon[32956]: pgmap v1390: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 7328/186870 objects degraded (3.921%); 28770/186870 objects misplaced (15.396%); 35 MiB/s, 552 objects/s recovering 2024-06-27T21:32:43.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:43.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:43.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:43.704 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:43.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:43.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:43.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:43.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:43.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:44.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:44.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:44.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:32:44.173 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:32:44.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:44.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:44.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:44.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:44.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2545882258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:32:44.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:44.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2545882258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:32:44.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2545882258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:32:44.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:44.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:44.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:44.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:44.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:44.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:44.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:44.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:45.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:45.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:45.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:45.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:45.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:45.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:45.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:45 smithi151 ceph-mon[28965]: pgmap v1391: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 44 op/s; 7328/186381 objects degraded (3.932%); 28467/186381 objects misplaced (15.274%); 32 MiB/s, 511 objects/s recovering 2024-06-27T21:32:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:45 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:32:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2545882258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:32:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:45 smithi151 ceph-mon[28965]: osdmap e533: 12 total, 12 up, 11 in 2024-06-27T21:32:45.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:45 smithi190 ceph-mon[32247]: pgmap v1391: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 44 op/s; 7328/186381 objects degraded (3.932%); 28467/186381 objects misplaced (15.274%); 32 MiB/s, 511 objects/s recovering 2024-06-27T21:32:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:45 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:32:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2545882258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:32:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:45 smithi190 ceph-mon[32247]: osdmap e533: 12 total, 12 up, 11 in 2024-06-27T21:32:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:45 smithi192 ceph-mon[32956]: pgmap v1391: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 44 op/s; 7328/186381 objects degraded (3.932%); 28467/186381 objects misplaced (15.274%); 32 MiB/s, 511 objects/s recovering 2024-06-27T21:32:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:45 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:32:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2545882258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:32:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:45 smithi192 ceph-mon[32956]: osdmap e533: 12 total, 12 up, 11 in 2024-06-27T21:32:45.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:45.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:45.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:45.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:45.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:45.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:45.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:45.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:46.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:46.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:46.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:46.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:46 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:32:46.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:46.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:46.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:46.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:46.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:46 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:32:46.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:46 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:32:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:46 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:32:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:46 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:32:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:46 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:32:46.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:46.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:46.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:46.713 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:46.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:46.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:46.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:46.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:46.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:46.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:47.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:47.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:47.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:47.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:47.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:47.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:47.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:47.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:47 smithi151 ceph-mon[28965]: pgmap v1393: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 7328/186060 objects degraded (3.939%); 27872/186060 objects misplaced (14.980%); 20 MiB/s, 324 objects/s recovering 2024-06-27T21:32:47.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:47 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/186060 objects degraded (3.939%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:47.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:47.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:47 smithi190 ceph-mon[32247]: pgmap v1393: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 7328/186060 objects degraded (3.939%); 27872/186060 objects misplaced (14.980%); 20 MiB/s, 324 objects/s recovering 2024-06-27T21:32:47.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:47 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/186060 objects degraded (3.939%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:47 smithi192 ceph-mon[32956]: pgmap v1393: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 64 op/s; 7328/186060 objects degraded (3.939%); 27872/186060 objects misplaced (14.980%); 20 MiB/s, 324 objects/s recovering 2024-06-27T21:32:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:47 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/186060 objects degraded (3.939%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:47.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:47.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:47.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:47.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:47.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:47.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:47.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:47.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:47.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:48.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:48.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:48.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:48.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:48.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:48.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:48.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:48.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:48.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:48.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:48.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:48.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:48.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:48.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:48.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:48.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:48.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:49.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:49.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:49.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:49.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:49.266 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:49 smithi151 ceph-mon[28965]: pgmap v1394: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 7328/185364 objects degraded (3.953%); 27162/185364 objects misplaced (14.653%); 20 MiB/s, 320 objects/s recovering 2024-06-27T21:32:49.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:49.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:49.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:49.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:49 smithi190 ceph-mon[32247]: pgmap v1394: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 7328/185364 objects degraded (3.953%); 27162/185364 objects misplaced (14.653%); 20 MiB/s, 320 objects/s recovering 2024-06-27T21:32:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:49 smithi192 ceph-mon[32956]: pgmap v1394: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 7328/185364 objects degraded (3.953%); 27162/185364 objects misplaced (14.653%); 20 MiB/s, 320 objects/s recovering 2024-06-27T21:32:49.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:49.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:49.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:49.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:49.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:49.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:49.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:49.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:49.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:50.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:50.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:50.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:50.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:50.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:50.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:50.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:50.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:50.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:50.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:50.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:50.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:50.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:50.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:50.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:50.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:50.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:51.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:51.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:51.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:51.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:51.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:32:51.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:51.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:51.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:51 smithi151 ceph-mon[28965]: pgmap v1395: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 7328/184974 objects degraded (3.962%); 26511/184974 objects misplaced (14.332%); 20 MiB/s, 326 objects/s recovering 2024-06-27T21:32:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:51.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:51 smithi190 ceph-mon[32247]: pgmap v1395: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 7328/184974 objects degraded (3.962%); 26511/184974 objects misplaced (14.332%); 20 MiB/s, 326 objects/s recovering 2024-06-27T21:32:51.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:51 smithi192 ceph-mon[32956]: pgmap v1395: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 80 op/s; 7328/184974 objects degraded (3.962%); 26511/184974 objects misplaced (14.332%); 20 MiB/s, 326 objects/s recovering 2024-06-27T21:32:51.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:51.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:51.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:51.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:51.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:51.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:51.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:51.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:51.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:51.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:52.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:52.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:52.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:52.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:52.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:52.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:52.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:52.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:52.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:52 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/184344 objects degraded (3.975%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:52 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/184344 objects degraded (3.975%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:52.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:52.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:52.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:52.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:52 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/184344 objects degraded (3.975%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:52.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:52.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:52.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:52.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:52.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:52.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:53.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:53.085 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:53.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:53.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:53.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:53.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:53.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:53.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:53.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:53 smithi190 ceph-mon[32247]: pgmap v1396: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 84 op/s; 7328/184344 objects degraded (3.975%); 25366/184344 objects misplaced (13.760%); 21 MiB/s, 328 objects/s recovering 2024-06-27T21:32:53.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:53 smithi192 ceph-mon[32956]: pgmap v1396: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 84 op/s; 7328/184344 objects degraded (3.975%); 25366/184344 objects misplaced (13.760%); 21 MiB/s, 328 objects/s recovering 2024-06-27T21:32:53.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:53.562 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:53 smithi151 ceph-mon[28965]: pgmap v1396: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 84 op/s; 7328/184344 objects degraded (3.975%); 25366/184344 objects misplaced (13.760%); 21 MiB/s, 328 objects/s recovering 2024-06-27T21:32:53.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:53.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:53.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:53.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:53.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:53.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:53.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:53.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:54.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:54.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:54.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:54.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:54.253 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:32:54.253 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:32:54.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:32:54.279 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:32:54.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:32:54.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:32:54.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:54.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:32:54.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:32:54.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:54.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:54.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:32:54.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:32:54.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:32:54.378 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:32:54.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:32:54.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:32:54.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:32:54.428 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:32:54.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:54.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:32:54.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:32:54.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:54.470 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:54.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:32:54.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:32:54.493 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2758795685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:54.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2758795685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:54.519 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:32:54.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:32:54.527 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:32:54.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2758795685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:32:54.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:32:54.557 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:32:54.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:54.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:54.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:54.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:54.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:54.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:54.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:54.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:54.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:55.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:55.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:55.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:55.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:55.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:55.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:55.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:55.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:55.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:55.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:55.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:32:55.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:32:55.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:55.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:55.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:55 smithi151 ceph-mon[28965]: pgmap v1397: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 7328/183876 objects degraded (3.985%); 25006/183876 objects misplaced (13.599%); 21 MiB/s, 333 objects/s recovering 2024-06-27T21:32:55.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:55.973 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:55.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:56.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:55 smithi190 ceph-mon[32247]: pgmap v1397: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 7328/183876 objects degraded (3.985%); 25006/183876 objects misplaced (13.599%); 21 MiB/s, 333 objects/s recovering 2024-06-27T21:32:56.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:55 smithi192 ceph-mon[32956]: pgmap v1397: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 7328/183876 objects degraded (3.985%); 25006/183876 objects misplaced (13.599%); 21 MiB/s, 333 objects/s recovering 2024-06-27T21:32:56.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:56.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:56.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:56.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:56.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:56.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:56.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:56.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:56.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:56.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:56.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:56.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:56.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:56.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/781844780' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-27T21:32:56.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:56.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:57.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/781844780' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-27T21:32:57.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/781844780' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-06-27T21:32:57.048 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 31 2024-06-27T21:32:57.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:57.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:57.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:57.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:57.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:57.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:57.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:32:57.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:32:57.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:32:57.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:32:57.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:32:57.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:32:57.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:57.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:57.946 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:57 smithi192 ceph-mon[32956]: pgmap v1398: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 7328/183651 objects degraded (3.990%); 24302/183651 objects misplaced (13.233%); 21 MiB/s, 338 objects/s recovering 2024-06-27T21:32:57.946 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/781844780' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-27T21:32:57.946 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:57 smithi192 ceph-mon[32956]: osdmap e534: 12 total, 12 up, 11 in 2024-06-27T21:32:57.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:57 smithi151 ceph-mon[28965]: pgmap v1398: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 7328/183651 objects degraded (3.990%); 24302/183651 objects misplaced (13.233%); 21 MiB/s, 338 objects/s recovering 2024-06-27T21:32:57.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/781844780' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-27T21:32:57.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:57 smithi151 ceph-mon[28965]: osdmap e534: 12 total, 12 up, 11 in 2024-06-27T21:32:57.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:32:57.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:57.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:32:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:57 smithi190 ceph-mon[32247]: pgmap v1398: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 7328/183651 objects degraded (3.990%); 24302/183651 objects misplaced (13.233%); 21 MiB/s, 338 objects/s recovering 2024-06-27T21:32:58.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/781844780' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-06-27T21:32:58.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:57 smithi190 ceph-mon[32247]: osdmap e534: 12 total, 12 up, 11 in 2024-06-27T21:32:58.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:58.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:58.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:58.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:58.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:58.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:58.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:58.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:58.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:58.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:58.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:58.726 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:58.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:32:58.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:32:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:58 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-27T21:32:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:58 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-27T21:32:58.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:58.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:58.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:58 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-27T21:32:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:58 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-27T21:32:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:58 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-27T21:32:59.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:58 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-06-27T21:32:59.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:59.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:59.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:59.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:32:59.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:32:59.353 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:32:59.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:32:59.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:32:59.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:32:59.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:32:59.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:32:59.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:32:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:32:59.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:32:59.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:32:59.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:59 smithi151 ceph-mon[28965]: pgmap v1400: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 7328/182877 objects degraded (4.007%); 23412/182877 objects misplaced (12.802%); 23 MiB/s, 362 objects/s recovering 2024-06-27T21:32:59.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/182877 objects degraded (4.007%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:32:59.842 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:59 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-27T21:32:59.842 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:32:59 smithi151 ceph-mon[28965]: osdmap e535: 12 total, 12 up, 11 in 2024-06-27T21:32:59.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:32:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:32:59.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:32:59.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:32:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:32:59.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:59 smithi190 ceph-mon[32247]: pgmap v1400: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 7328/182877 objects degraded (4.007%); 23412/182877 objects misplaced (12.802%); 23 MiB/s, 362 objects/s recovering 2024-06-27T21:33:00.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/182877 objects degraded (4.007%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:59 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-27T21:33:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:32:59 smithi190 ceph-mon[32247]: osdmap e535: 12 total, 12 up, 11 in 2024-06-27T21:33:00.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:59 smithi192 ceph-mon[32956]: pgmap v1400: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 28 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 82 op/s; 7328/182877 objects degraded (4.007%); 23412/182877 objects misplaced (12.802%); 23 MiB/s, 362 objects/s recovering 2024-06-27T21:33:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/182877 objects degraded (4.007%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:59 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-06-27T21:33:00.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:32:59 smithi192 ceph-mon[32956]: osdmap e535: 12 total, 12 up, 11 in 2024-06-27T21:33:00.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:00.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:00.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:00.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:00.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:00.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:00.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:00.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:00.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:00.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:00.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:00.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:00.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:00.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:00.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:00.992 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:00.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:01.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: osdmap e536: 12 total, 12 up, 11 in 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: 1.b deep-scrub starts 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: 1.b deep-scrub ok 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:33:01.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:00 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:33:01.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: osdmap e536: 12 total, 12 up, 11 in 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: 1.b deep-scrub starts 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: 1.b deep-scrub ok 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:33:01.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:00 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:33:01.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:01.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:01.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: osdmap e536: 12 total, 12 up, 11 in 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: 1.b deep-scrub starts 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: 1.b deep-scrub ok 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:33:01.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:00 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:33:01.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:01.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:01.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:01.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:01.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:01.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:01.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:01.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:01.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:01.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:01.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:01.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:01.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:01.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: pgmap v1403: 47 pgs: 6 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/182292 objects degraded (4.020%); 22640/182292 objects misplaced (12.420%); 24 MiB/s, 384 objects/s recovering 2024-06-27T21:33:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.5 deep-scrub starts 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.5 deep-scrub ok 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: osdmap e537: 12 total, 12 up, 11 in 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:33:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:01 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:33:02.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: pgmap v1403: 47 pgs: 6 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/182292 objects degraded (4.020%); 22640/182292 objects misplaced (12.420%); 24 MiB/s, 384 objects/s recovering 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.5 deep-scrub starts 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.5 deep-scrub ok 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: osdmap e537: 12 total, 12 up, 11 in 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:33:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:01 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:33:02.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:02.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: pgmap v1403: 47 pgs: 6 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/182292 objects degraded (4.020%); 22640/182292 objects misplaced (12.420%); 24 MiB/s, 384 objects/s recovering 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.5 deep-scrub starts 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.5 deep-scrub ok 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: osdmap e537: 12 total, 12 up, 11 in 2024-06-27T21:33:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:33:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:01 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:33:02.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:02.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:02.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:02.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:02.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:02.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:02.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:02.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:02.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:02.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:02.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:02.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:02.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:02.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:02.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:03.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:33:03.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: osdmap e538: 12 total, 12 up, 11 in 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:33:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:02 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:33:03.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:33:03.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: osdmap e538: 12 total, 12 up, 11 in 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:33:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:02 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:33:03.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:03.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: osdmap e538: 12 total, 12 up, 11 in 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:33:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:02 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:33:03.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:03.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:03.243 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:33:03.243 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:33:03.244 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-06-27T21:33:03.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:33:03.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:03.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:03.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:03.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:03.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:03.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:03.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:03.725 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:33:03.746 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:03.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:03.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:03.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:03.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:03.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: pgmap v1406: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 6 unknown, 8 active+recovering+undersized+degraded+remapped, 31 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 112 op/s; 7328/181521 objects degraded (4.037%); 21402/181521 objects misplaced (11.790%); 30 MiB/s, 487 objects/s recovering 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2473903766' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:33:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:03 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:33:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: pgmap v1406: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 6 unknown, 8 active+recovering+undersized+degraded+remapped, 31 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 112 op/s; 7328/181521 objects degraded (4.037%); 21402/181521 objects misplaced (11.790%); 30 MiB/s, 487 objects/s recovering 2024-06-27T21:33:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2473903766' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:33:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:03 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:33:04.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:04.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: pgmap v1406: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 6 unknown, 8 active+recovering+undersized+degraded+remapped, 31 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 112 op/s; 7328/181521 objects degraded (4.037%); 21402/181521 objects misplaced (11.790%); 30 MiB/s, 487 objects/s recovering 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:33:04.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:33:04.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2473903766' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:33:04.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:33:04.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:03 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:33:04.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:04.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:04.333 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:33:04.334 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:33:04.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:04.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:04.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:04.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:04.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:04.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:04.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:04.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:04.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:04.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:04.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:04.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:05.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: 1.17 deep-scrub starts 2024-06-27T21:33:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: 1.17 deep-scrub ok 2024-06-27T21:33:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2473903766' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:33:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: osdmap e539: 12 total, 12 up, 11 in 2024-06-27T21:33:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/181521 objects degraded (4.037%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: 1.17 deep-scrub starts 2024-06-27T21:33:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: 1.17 deep-scrub ok 2024-06-27T21:33:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2473903766' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:33:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: osdmap e539: 12 total, 12 up, 11 in 2024-06-27T21:33:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/181521 objects degraded (4.037%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:05.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:05.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:05.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: 1.17 deep-scrub starts 2024-06-27T21:33:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: 1.17 deep-scrub ok 2024-06-27T21:33:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2473903766' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:33:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: osdmap e539: 12 total, 12 up, 11 in 2024-06-27T21:33:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/181521 objects degraded (4.037%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:05.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:05.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:05.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:05.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:05.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:05.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:05.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:05.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:05.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:05.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:05.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:05.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:05.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:05.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:06.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:05 smithi190 ceph-mon[32247]: pgmap v1408: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 unknown, 8 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 7328/180993 objects degraded (4.049%); 20689/180993 objects misplaced (11.431%); 32 MiB/s, 506 objects/s recovering 2024-06-27T21:33:06.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:05 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:05 smithi192 ceph-mon[32956]: pgmap v1408: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 unknown, 8 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 7328/180993 objects degraded (4.049%); 20689/180993 objects misplaced (11.431%); 32 MiB/s, 506 objects/s recovering 2024-06-27T21:33:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:05 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:06.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:06.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:06.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:05 smithi151 ceph-mon[28965]: pgmap v1408: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 4 unknown, 8 active+recovering+undersized+degraded+remapped, 33 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 7328/180993 objects degraded (4.049%); 20689/180993 objects misplaced (11.431%); 32 MiB/s, 506 objects/s recovering 2024-06-27T21:33:06.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:05 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:06.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:06.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:06.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:06.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:06.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:06.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:06.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:06.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:06.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:06.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:06.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:06.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:06.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:06.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:07.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:07.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:07.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:07.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:07.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:07.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:07.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:07.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:07.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:07.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:33:07.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:07.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:07.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:07.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:07.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:07.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:07.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:07 smithi151 ceph-mon[28965]: pgmap v1409: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 8 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/180723 objects degraded (4.055%); 19939/180723 objects misplaced (11.033%); 27 MiB/s, 439 objects/s recovering 2024-06-27T21:33:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:07 smithi151 ceph-mon[28965]: osdmap e540: 12 total, 12 up, 11 in 2024-06-27T21:33:07.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:08.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:08.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:08.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:07 smithi190 ceph-mon[32247]: pgmap v1409: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 8 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/180723 objects degraded (4.055%); 19939/180723 objects misplaced (11.033%); 27 MiB/s, 439 objects/s recovering 2024-06-27T21:33:08.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:07 smithi190 ceph-mon[32247]: osdmap e540: 12 total, 12 up, 11 in 2024-06-27T21:33:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:07 smithi192 ceph-mon[32956]: pgmap v1409: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 8 active+recovering+undersized+degraded+remapped, 37 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/180723 objects degraded (4.055%); 19939/180723 objects misplaced (11.033%); 27 MiB/s, 439 objects/s recovering 2024-06-27T21:33:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:07 smithi192 ceph-mon[32956]: osdmap e540: 12 total, 12 up, 11 in 2024-06-27T21:33:08.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:08.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:08.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:08.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:08.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:08.365 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:08.374 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:08.380 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:08.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:08.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:08.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:08.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:08.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:08.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:08.845 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:08 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:33:08.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:08.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:08.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:08.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:09.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:09.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:08 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:33:09.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:09.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:09.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:09.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:08 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:33:09.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:09.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:09.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:09.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:09.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:09.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:09.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:09.754 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:09.780 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:09.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:09.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:09.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:09.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:09 smithi151 ceph-mon[28965]: pgmap v1411: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 7328/179898 objects degraded (4.073%); 19048/179898 objects misplaced (10.588%); 23 MiB/s, 362 objects/s recovering 2024-06-27T21:33:09.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/179898 objects degraded (4.073%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:09.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:10.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:10.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:09 smithi192 ceph-mon[32956]: pgmap v1411: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 7328/179898 objects degraded (4.073%); 19048/179898 objects misplaced (10.588%); 23 MiB/s, 362 objects/s recovering 2024-06-27T21:33:10.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/179898 objects degraded (4.073%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:10.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:10.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:10.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:10.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:10.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:10.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:09 smithi190 ceph-mon[32247]: pgmap v1411: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 7328/179898 objects degraded (4.073%); 19048/179898 objects misplaced (10.588%); 23 MiB/s, 362 objects/s recovering 2024-06-27T21:33:10.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/179898 objects degraded (4.073%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:10.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:33:10.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:10.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:10.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:10.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:10.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:10.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:10.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:10.771 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:10.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:10.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:10.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:10.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:10 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:10.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:10 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:10.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:11.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:11.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:10 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:11.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:10 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:11.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:11.145 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:11.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:11.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:10 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:11.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:10 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:11.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:11.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:11.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:11.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:11.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:11.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:11.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:11.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:11.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:11.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:11.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:11.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:11.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:12.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:11 smithi192 ceph-mon[32956]: pgmap v1412: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/179430 objects degraded (4.084%); 18243/179430 objects misplaced (10.167%); 24 MiB/s, 425 objects/s recovering 2024-06-27T21:33:12.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:11 smithi151 ceph-mon[28965]: pgmap v1412: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/179430 objects degraded (4.084%); 18243/179430 objects misplaced (10.167%); 24 MiB/s, 425 objects/s recovering 2024-06-27T21:33:12.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:12.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:12.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:11 smithi190 ceph-mon[32247]: pgmap v1412: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 87 op/s; 7328/179430 objects degraded (4.084%); 18243/179430 objects misplaced (10.167%); 24 MiB/s, 425 objects/s recovering 2024-06-27T21:33:12.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:12.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:12.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:12.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:12.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:12.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:12.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:12.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:12.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:12.873 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:33:12.874 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:33:12.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:12.898 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:33:12.898 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:33:12.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:12.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:12.924 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:33:12.924 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:33:12.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:33:12.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:33:12.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:33:12.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:33:12.982 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2486518616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:33:13.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:13.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:33:13.010 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:33:13.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:13.039 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:33:13.039 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:33:13.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:33:13.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:33:13.092 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:33:13.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:33:13.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:33:13.120 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:33:13.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:13.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:33:13.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:33:13.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:13.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:33:13.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:33:13.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:13.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2486518616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:33:13.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2486518616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:33:13.325 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:33:13.326 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:33:13.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:13.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:13.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:13.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:13.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:13.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:13.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:13.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:13.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:13.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:13.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:13.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:14.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:14.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:14.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:14.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:14.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:13 smithi151 ceph-mon[28965]: pgmap v1413: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 7328/178587 objects degraded (4.103%); 17226/178587 objects misplaced (9.646%); 25 MiB/s, 468 objects/s recovering 2024-06-27T21:33:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:13 smithi151 ceph-mon[28965]: from='client.16626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:33:14.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:13 smithi190 ceph-mon[32247]: pgmap v1413: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 7328/178587 objects degraded (4.103%); 17226/178587 objects misplaced (9.646%); 25 MiB/s, 468 objects/s recovering 2024-06-27T21:33:14.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:13 smithi190 ceph-mon[32247]: from='client.16626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:33:14.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:13 smithi192 ceph-mon[32956]: pgmap v1413: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 95 op/s; 7328/178587 objects degraded (4.103%); 17226/178587 objects misplaced (9.646%); 25 MiB/s, 468 objects/s recovering 2024-06-27T21:33:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:13 smithi192 ceph-mon[32956]: from='client.16626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:33:14.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:14.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:14.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:14.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:14.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:14.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:14.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:14.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:14.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:14.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:14.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:14.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:15.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:15.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:15.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:15.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/178587 objects degraded (4.103%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:15.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:15.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/178587 objects degraded (4.103%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:15.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:15.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/178587 objects degraded (4.103%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:15.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:15.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:15.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:15.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:15.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:15.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:15.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:15.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:15.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:15.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:15.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:15.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:16.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:16.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:16.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:15 smithi151 ceph-mon[28965]: pgmap v1414: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 7328/177960 objects degraded (4.118%); 17226/177960 objects misplaced (9.680%); 21 MiB/s, 454 objects/s recovering 2024-06-27T21:33:16.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:15 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:16.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:16.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:16.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:15 smithi190 ceph-mon[32247]: pgmap v1414: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 7328/177960 objects degraded (4.118%); 17226/177960 objects misplaced (9.680%); 21 MiB/s, 454 objects/s recovering 2024-06-27T21:33:16.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:15 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:16.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:15 smithi192 ceph-mon[32956]: pgmap v1414: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 101 op/s; 7328/177960 objects degraded (4.118%); 17226/177960 objects misplaced (9.680%); 21 MiB/s, 454 objects/s recovering 2024-06-27T21:33:16.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:15 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:16.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:16.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:16.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:16.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:16.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:16.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:16.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:16.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:16.817 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1413,"stamp":"2024-06-27T21:33:12.117969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3901784632,"num_objects":59529,"num_object_clones":0,"num_object_copies":178587,"num_objects_missing_on_primary":5183,"num_objects_missing":5183,"num_objects_degraded":7328,"num_objects_misplaced":17226,"num_objects_unfound":0,"num_objects_dirty":59529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155658,"num_write_kb":6885825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31835,"num_bytes_recovered":2046296064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158510,"ondisk_log_size":158510,"up":141,"acting":132,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17473496,"kb_used_data":12855280,"kb_used_omap":108,"kb_used_meta":4618003,"kb_avail":1013723176,"statfs":{"total":1055945392128,"available":1038052532224,"internally_reserved":0,"allocated":13163806720,"data_stored":13160505959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111086,"internal_metadata":4728835602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-52461568,"num_objects":-802,"num_object_clones":0,"num_object_copies":-2406,"num_objects_missing_on_primary":-65,"num_objects_missing":-65,"num_objects_degraded":0,"num_objects_misplaced":-3463,"num_objects_unfound":0,"num_objects_dirty":-802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":803,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3927,"num_bytes_recovered":218693632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.389737"},"pg_stats":[{"pgid":"1.1e","version":"21'92","reported_seq":23,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.744111+0000","last_change":"2024-06-27T21:33:02.987696+0000","last_active":"2024-06-27T21:33:06.744111+0000","last_peered":"2024-06-27T21:33:06.744111+0000","last_clean":"2024-06-27T21:33:06.744111+0000","last_became_active":"2024-06-27T21:33:01.707872+0000","last_became_peered":"2024-06-27T21:33:01.707872+0000","last_unstale":"2024-06-27T21:33:06.744111+0000","last_undegraded":"2024-06-27T21:33:06.744111+0000","last_fullsized":"2024-06-27T21:33:06.744111+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:23.304568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,10],"acting":[11,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":23,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.743907+0000","last_change":"2024-06-27T21:33:01.956324+0000","last_active":"2024-06-27T21:33:06.743907+0000","last_peered":"2024-06-27T21:33:06.743907+0000","last_clean":"2024-06-27T21:33:06.743907+0000","last_became_active":"2024-06-27T21:33:01.707250+0000","last_became_peered":"2024-06-27T21:33:01.707250+0000","last_unstale":"2024-06-27T21:33:06.743907+0000","last_undegraded":"2024-06-27T21:33:06.743907+0000","last_fullsized":"2024-06-27T21:33:06.743907+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:03.600384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":18,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.743735+0000","last_change":"2024-06-27T21:33:04.792903+0000","last_active":"2024-06-27T21:33:06.743735+0000","last_peered":"2024-06-27T21:33:06.743735+0000","last_clean":"2024-06-27T21:33:06.743735+0000","last_became_active":"2024-06-27T21:33:01.707734+0000","last_became_peered":"2024-06-27T21:33:01.707734+0000","last_unstale":"2024-06-27T21:33:06.743735+0000","last_undegraded":"2024-06-27T21:33:06.743735+0000","last_fullsized":"2024-06-27T21:33:06.743735+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.211775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.211775+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:08.957959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":25,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761379+0000","last_change":"2024-06-27T21:33:02.568824+0000","last_active":"2024-06-27T21:33:06.761379+0000","last_peered":"2024-06-27T21:33:06.761379+0000","last_clean":"2024-06-27T21:33:06.761379+0000","last_became_active":"2024-06-27T21:33:01.773541+0000","last_became_peered":"2024-06-27T21:33:01.773541+0000","last_unstale":"2024-06-27T21:33:06.761379+0000","last_undegraded":"2024-06-27T21:33:06.761379+0000","last_fullsized":"2024-06-27T21:33:06.761379+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:20.906343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":91,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.738670+0000","last_change":"2024-06-27T21:33:02.266680+0000","last_active":"2024-06-27T21:33:06.738670+0000","last_peered":"2024-06-27T21:33:06.738670+0000","last_clean":"2024-06-27T21:33:06.738670+0000","last_became_active":"2024-06-27T21:33:01.710059+0000","last_became_peered":"2024-06-27T21:33:01.710059+0000","last_unstale":"2024-06-27T21:33:06.738670+0000","last_undegraded":"2024-06-27T21:33:06.738670+0000","last_fullsized":"2024-06-27T21:33:06.738670+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:14.470683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":18,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761953+0000","last_change":"2024-06-27T21:33:04.900718+0000","last_active":"2024-06-27T21:33:06.761953+0000","last_peered":"2024-06-27T21:33:06.761953+0000","last_clean":"2024-06-27T21:33:06.761953+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:33:06.761953+0000","last_undegraded":"2024-06-27T21:33:06.761953+0000","last_fullsized":"2024-06-27T21:33:06.761953+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:46.150318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:30:46.150318+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:09.273261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":467,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761690+0000","last_change":"2024-06-27T21:33:02.184848+0000","last_active":"2024-06-27T21:33:06.761690+0000","last_peered":"2024-06-27T21:33:06.761690+0000","last_clean":"2024-06-27T21:33:06.761690+0000","last_became_active":"2024-06-27T21:33:01.774416+0000","last_became_peered":"2024-06-27T21:33:01.774416+0000","last_unstale":"2024-06-27T21:33:06.761690+0000","last_undegraded":"2024-06-27T21:33:06.761690+0000","last_fullsized":"2024-06-27T21:33:06.761690+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.184716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.184716+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:13.944531+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":503,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.762049+0000","last_change":"2024-06-27T21:33:03.234800+0000","last_active":"2024-06-27T21:33:06.762049+0000","last_peered":"2024-06-27T21:33:06.762049+0000","last_clean":"2024-06-27T21:33:06.762049+0000","last_became_active":"2024-06-27T21:33:01.779398+0000","last_became_peered":"2024-06-27T21:33:01.779398+0000","last_unstale":"2024-06-27T21:33:06.762049+0000","last_undegraded":"2024-06-27T21:33:06.762049+0000","last_fullsized":"2024-06-27T21:33:06.762049+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:33:03.234726+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:25.309823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":23,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761913+0000","last_change":"2024-06-27T21:33:02.510081+0000","last_active":"2024-06-27T21:33:06.761913+0000","last_peered":"2024-06-27T21:33:06.761913+0000","last_clean":"2024-06-27T21:33:06.761913+0000","last_became_active":"2024-06-27T21:33:01.775355+0000","last_became_peered":"2024-06-27T21:33:01.775355+0000","last_unstale":"2024-06-27T21:33:06.761913+0000","last_undegraded":"2024-06-27T21:33:06.761913+0000","last_fullsized":"2024-06-27T21:33:06.761913+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.509969+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.509969+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:03.505318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":23,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.748584+0000","last_change":"2024-06-27T21:33:02.531114+0000","last_active":"2024-06-27T21:33:06.748584+0000","last_peered":"2024-06-27T21:33:06.748584+0000","last_clean":"2024-06-27T21:33:06.748584+0000","last_became_active":"2024-06-27T21:33:01.810521+0000","last_became_peered":"2024-06-27T21:33:01.810521+0000","last_unstale":"2024-06-27T21:33:06.748584+0000","last_undegraded":"2024-06-27T21:33:06.748584+0000","last_fullsized":"2024-06-27T21:33:06.748584+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:17.850811+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":154,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761771+0000","last_change":"2024-06-27T21:31:19.010738+0000","last_active":"2024-06-27T21:33:06.761771+0000","last_peered":"2024-06-27T21:33:06.761771+0000","last_clean":"2024-06-27T21:33:06.761771+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:33:06.761771+0000","last_undegraded":"2024-06-27T21:33:06.761771+0000","last_fullsized":"2024-06-27T21:33:06.761771+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.010614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.010614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":154,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.743748+0000","last_change":"2024-06-27T21:31:19.025719+0000","last_active":"2024-06-27T21:33:06.743748+0000","last_peered":"2024-06-27T21:33:06.743748+0000","last_clean":"2024-06-27T21:33:06.743748+0000","last_became_active":"2024-06-27T21:29:51.673661+0000","last_became_peered":"2024-06-27T21:29:51.673661+0000","last_unstale":"2024-06-27T21:33:06.743748+0000","last_undegraded":"2024-06-27T21:33:06.743748+0000","last_fullsized":"2024-06-27T21:33:06.743748+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":162,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.810345+0000","last_change":"2024-06-27T21:32:45.802297+0000","last_active":"2024-06-27T21:33:06.810345+0000","last_peered":"2024-06-27T21:33:06.810345+0000","last_clean":"2024-06-27T21:33:06.810345+0000","last_became_active":"2024-06-27T21:29:51.676689+0000","last_became_peered":"2024-06-27T21:29:51.676689+0000","last_unstale":"2024-06-27T21:33:06.810345+0000","last_undegraded":"2024-06-27T21:33:06.810345+0000","last_fullsized":"2024-06-27T21:33:06.810345+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:45.802189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:32:45.802189+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:00.659317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":162,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761304+0000","last_change":"2024-06-27T21:32:13.808607+0000","last_active":"2024-06-27T21:33:06.761304+0000","last_peered":"2024-06-27T21:33:06.761304+0000","last_clean":"2024-06-27T21:33:06.761304+0000","last_became_active":"2024-06-27T21:32:13.808038+0000","last_became_peered":"2024-06-27T21:32:13.808038+0000","last_unstale":"2024-06-27T21:33:06.761304+0000","last_undegraded":"2024-06-27T21:33:06.761304+0000","last_fullsized":"2024-06-27T21:33:06.761304+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1],"acting":[0,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":183,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.748414+0000","last_change":"2024-06-27T21:32:13.808094+0000","last_active":"2024-06-27T21:33:06.748414+0000","last_peered":"2024-06-27T21:33:06.748414+0000","last_clean":"2024-06-27T21:33:06.748414+0000","last_became_active":"2024-06-27T21:32:13.807596+0000","last_became_peered":"2024-06-27T21:32:13.807596+0000","last_unstale":"2024-06-27T21:33:06.748414+0000","last_undegraded":"2024-06-27T21:33:06.748414+0000","last_fullsized":"2024-06-27T21:33:06.748414+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":456,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761912+0000","last_change":"2024-06-27T21:33:00.167722+0000","last_active":"2024-06-27T21:33:06.761912+0000","last_peered":"2024-06-27T21:33:06.761912+0000","last_clean":"2024-06-27T21:33:06.761912+0000","last_became_active":"2024-06-27T21:32:59.709162+0000","last_became_peered":"2024-06-27T21:32:59.709162+0000","last_unstale":"2024-06-27T21:33:06.761912+0000","last_undegraded":"2024-06-27T21:33:06.761912+0000","last_fullsized":"2024-06-27T21:33:06.761912+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.167617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.167617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:28.293116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"540'9760","reported_seq":9990,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:07.749288+0000","last_change":"2024-06-27T21:30:52.123816+0000","last_active":"2024-06-27T21:33:07.749288+0000","last_peered":"2024-06-27T21:33:07.749288+0000","last_clean":"2024-06-27T21:33:07.749288+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:33:07.749288+0000","last_undegraded":"2024-06-27T21:33:07.749288+0000","last_fullsized":"2024-06-27T21:33:07.749288+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6360","last_scrub_stamp":"2024-06-27T21:30:52.123515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.123515+0000","objects_scrubbed":6329,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249823232,"num_objects":3812,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9760,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":244,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.748716+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:33:06.748716+0000","last_peered":"2024-06-27T21:33:06.748716+0000","last_clean":"2024-06-27T21:33:06.748716+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:33:06.748716+0000","last_undegraded":"2024-06-27T21:33:06.748716+0000","last_fullsized":"2024-06-27T21:33:06.748716+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:27.450417+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"540'9819","reported_seq":10046,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:08.070879+0000","last_change":"2024-06-27T21:30:48.343185+0000","last_active":"2024-06-27T21:33:08.070879+0000","last_peered":"2024-06-27T21:33:08.070879+0000","last_clean":"2024-06-27T21:33:08.070879+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:33:08.070879+0000","last_undegraded":"2024-06-27T21:33:08.070879+0000","last_fullsized":"2024-06-27T21:33:08.070879+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6100","last_scrub_stamp":"2024-06-27T21:30:48.342993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.342993+0000","objects_scrubbed":6058,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244908032,"num_objects":3737,"num_object_clones":0,"num_object_copies":11211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9819,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":86,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.738907+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:33:06.738907+0000","last_peered":"2024-06-27T21:33:06.738907+0000","last_clean":"2024-06-27T21:33:06.738907+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:33:06.738907+0000","last_undegraded":"2024-06-27T21:33:06.738907+0000","last_fullsized":"2024-06-27T21:33:06.738907+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:18.634183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"540'9699","reported_seq":19908,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:09.888490+0000","last_change":"2024-06-27T21:31:33.570353+0000","last_active":"2024-06-27T21:33:09.888490+0000","last_peered":"2024-06-27T21:33:09.888490+0000","last_clean":"2024-06-27T21:31:31.204488+0000","last_became_active":"2024-06-27T21:31:33.525500+0000","last_became_peered":"2024-06-27T21:31:33.525500+0000","last_unstale":"2024-06-27T21:33:09.888490+0000","last_undegraded":"2024-06-27T21:31:33.354229+0000","last_fullsized":"2024-06-27T21:31:33.314507+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":9699,"log_dups_size":0,"ondisk_log_size":9699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249298968,"num_objects":3805,"num_object_clones":0,"num_object_copies":11415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":210,"num_objects_misplaced":104,"num_objects_unfound":0,"num_objects_dirty":3805,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9699,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4987,"num_bytes_recovered":326565888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1],"avail_no_missing":["10","1","5"],"object_location_counts":[{"shards":"1,5,10","objects":104},{"shards":"1,10","objects":3491}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1004,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.743786+0000","last_change":"2024-06-27T21:32:13.809592+0000","last_active":"2024-06-27T21:33:06.743786+0000","last_peered":"2024-06-27T21:33:06.743786+0000","last_clean":"2024-06-27T21:33:06.743786+0000","last_became_active":"2024-06-27T21:32:13.809197+0000","last_became_peered":"2024-06-27T21:32:13.809197+0000","last_unstale":"2024-06-27T21:33:06.743786+0000","last_undegraded":"2024-06-27T21:33:06.743786+0000","last_fullsized":"2024-06-27T21:33:06.743786+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"540'9905","reported_seq":14965,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:11.527929+0000","last_change":"2024-06-27T21:32:14.121194+0000","last_active":"2024-06-27T21:33:11.527929+0000","last_peered":"2024-06-27T21:33:11.527929+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:32:14.047570+0000","last_became_peered":"2024-06-27T21:32:14.047570+0000","last_unstale":"2024-06-27T21:33:11.527929+0000","last_undegraded":"2024-06-27T21:32:13.843384+0000","last_fullsized":"2024-06-27T21:32:13.802097+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":9905,"log_dups_size":0,"ondisk_log_size":9905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244121600,"num_objects":3725,"num_object_clones":0,"num_object_copies":11175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":2843,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9905,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2410,"num_bytes_recovered":157876224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"6,7,11","objects":2843},{"shards":"7,11","objects":876}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":581,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.764673+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:33:06.764673+0000","last_peered":"2024-06-27T21:33:06.764673+0000","last_clean":"2024-06-27T21:33:06.764673+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:33:06.764673+0000","last_undegraded":"2024-06-27T21:33:06.764673+0000","last_fullsized":"2024-06-27T21:33:06.764673+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:03.727167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"540'9542","reported_seq":9774,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:07.983064+0000","last_change":"2024-06-27T21:30:53.801878+0000","last_active":"2024-06-27T21:33:07.983064+0000","last_peered":"2024-06-27T21:33:07.983064+0000","last_clean":"2024-06-27T21:33:07.983064+0000","last_became_active":"2024-06-27T21:29:19.921089+0000","last_became_peered":"2024-06-27T21:29:19.921089+0000","last_unstale":"2024-06-27T21:33:07.983064+0000","last_undegraded":"2024-06-27T21:33:07.983064+0000","last_fullsized":"2024-06-27T21:33:07.983064+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6306","last_scrub_stamp":"2024-06-27T21:30:53.801648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:53.801648+0000","objects_scrubbed":6238,"log_size":9542,"log_dups_size":0,"ondisk_log_size":9542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":423936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":577,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.762130+0000","last_change":"2024-06-27T21:33:01.182699+0000","last_active":"2024-06-27T21:33:06.762130+0000","last_peered":"2024-06-27T21:33:06.762130+0000","last_clean":"2024-06-27T21:33:06.762130+0000","last_became_active":"2024-06-27T21:32:59.711777+0000","last_became_peered":"2024-06-27T21:32:59.711777+0000","last_unstale":"2024-06-27T21:33:06.762130+0000","last_undegraded":"2024-06-27T21:33:06.762130+0000","last_fullsized":"2024-06-27T21:33:06.762130+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:12.867934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"540'9428","reported_seq":15904,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:10.403903+0000","last_change":"2024-06-27T21:32:15.169057+0000","last_active":"2024-06-27T21:33:10.403903+0000","last_peered":"2024-06-27T21:33:10.403903+0000","last_clean":"2024-06-27T21:32:12.748576+0000","last_became_active":"2024-06-27T21:32:15.075574+0000","last_became_peered":"2024-06-27T21:32:15.075574+0000","last_unstale":"2024-06-27T21:33:10.403903+0000","last_undegraded":"2024-06-27T21:32:14.833604+0000","last_fullsized":"2024-06-27T21:32:14.802569+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"500'5961","last_scrub_stamp":"2024-06-27T21:30:49.972634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:49.972634+0000","objects_scrubbed":5918,"log_size":9428,"log_dups_size":0,"ondisk_log_size":9428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238419968,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":733,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":418112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3123,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,7],"acting":[9,10],"avail_no_missing":["9","8","10"],"object_location_counts":[{"shards":"8,9,10","objects":733},{"shards":"9,10","objects":2899}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":498,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.762222+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:33:06.762222+0000","last_peered":"2024-06-27T21:33:06.762222+0000","last_clean":"2024-06-27T21:33:06.762222+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:33:06.762222+0000","last_undegraded":"2024-06-27T21:33:06.762222+0000","last_fullsized":"2024-06-27T21:33:06.762222+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:17.860808+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"540'9794","reported_seq":20641,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:11.547168+0000","last_change":"2024-06-27T21:31:24.150930+0000","last_active":"2024-06-27T21:33:11.547168+0000","last_peered":"2024-06-27T21:33:11.547168+0000","last_clean":"2024-06-27T21:31:21.922862+0000","last_became_active":"2024-06-27T21:31:24.117632+0000","last_became_peered":"2024-06-27T21:31:24.117632+0000","last_unstale":"2024-06-27T21:33:11.547168+0000","last_undegraded":"2024-06-27T21:31:23.977903+0000","last_fullsized":"2024-06-27T21:31:23.937741+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'5888","last_scrub_stamp":"2024-06-27T21:30:45.481539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.481539+0000","objects_scrubbed":5828,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1590,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238288896,"num_objects":3636,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9794,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5309,"num_bytes_recovered":347144192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2","objects":3445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":575,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.762131+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:33:06.762131+0000","last_peered":"2024-06-27T21:33:06.762131+0000","last_clean":"2024-06-27T21:33:06.762131+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:33:06.762131+0000","last_undegraded":"2024-06-27T21:33:06.762131+0000","last_fullsized":"2024-06-27T21:33:06.762131+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"540'9729","reported_seq":21266,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:08.246187+0000","last_change":"2024-06-27T21:31:32.527777+0000","last_active":"2024-06-27T21:33:08.246187+0000","last_peered":"2024-06-27T21:33:08.246187+0000","last_clean":"2024-06-27T21:31:21.899317+0000","last_became_active":"2024-06-27T21:31:32.491754+0000","last_became_peered":"2024-06-27T21:31:32.491754+0000","last_unstale":"2024-06-27T21:33:08.246187+0000","last_undegraded":"2024-06-27T21:31:32.338579+0000","last_fullsized":"2024-06-27T21:31:32.307033+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'5932","last_scrub_stamp":"2024-06-27T21:30:46.556700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:46.556700+0000","objects_scrubbed":5888,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243597312,"num_objects":3717,"num_object_clones":0,"num_object_copies":11151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":430272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5652,"num_bytes_recovered":349634560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,7","objects":3715}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":250,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.748617+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:33:06.748617+0000","last_peered":"2024-06-27T21:33:06.748617+0000","last_clean":"2024-06-27T21:33:06.748617+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:33:06.748617+0000","last_undegraded":"2024-06-27T21:33:06.748617+0000","last_fullsized":"2024-06-27T21:33:06.748617+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"540'9659","reported_seq":16247,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:11.628433+0000","last_change":"2024-06-27T21:31:24.180605+0000","last_active":"2024-06-27T21:33:11.628433+0000","last_peered":"2024-06-27T21:33:11.628433+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:31:24.125120+0000","last_became_peered":"2024-06-27T21:31:24.125120+0000","last_unstale":"2024-06-27T21:33:11.628433+0000","last_undegraded":"2024-06-27T21:31:23.981588+0000","last_fullsized":"2024-06-27T21:31:23.941280+0000","mapping_epoch":513,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":9659,"log_dups_size":0,"ondisk_log_size":9659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234422272,"num_objects":3577,"num_object_clones":0,"num_object_copies":10731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":100,"num_objects_misplaced":2098,"num_objects_unfound":0,"num_objects_dirty":3577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":423552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3179,"num_bytes_recovered":208207872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"6,7,9","objects":2098},{"shards":"7,9","objects":1379}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":135,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.738677+0000","last_change":"2024-06-27T21:32:13.812610+0000","last_active":"2024-06-27T21:33:06.738677+0000","last_peered":"2024-06-27T21:33:06.738677+0000","last_clean":"2024-06-27T21:33:06.738677+0000","last_became_active":"2024-06-27T21:32:13.812217+0000","last_became_peered":"2024-06-27T21:32:13.812217+0000","last_unstale":"2024-06-27T21:33:06.738677+0000","last_undegraded":"2024-06-27T21:33:06.738677+0000","last_fullsized":"2024-06-27T21:33:06.738677+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,10],"acting":[4,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"540'10008","reported_seq":10236,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:08.868519+0000","last_change":"2024-06-27T21:30:48.475928+0000","last_active":"2024-06-27T21:33:08.868519+0000","last_peered":"2024-06-27T21:33:08.868519+0000","last_clean":"2024-06-27T21:33:08.868519+0000","last_became_active":"2024-06-27T21:29:19.889439+0000","last_became_peered":"2024-06-27T21:29:19.889439+0000","last_unstale":"2024-06-27T21:33:08.868519+0000","last_undegraded":"2024-06-27T21:33:08.868519+0000","last_fullsized":"2024-06-27T21:33:08.868519+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248119296,"num_objects":3786,"num_object_clones":0,"num_object_copies":11358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":441408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":247,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.748538+0000","last_change":"2024-06-27T21:29:56.956056+0000","last_active":"2024-06-27T21:33:06.748538+0000","last_peered":"2024-06-27T21:33:06.748538+0000","last_clean":"2024-06-27T21:33:06.748538+0000","last_became_active":"2024-06-27T21:29:49.671863+0000","last_became_peered":"2024-06-27T21:29:49.671863+0000","last_unstale":"2024-06-27T21:33:06.748538+0000","last_undegraded":"2024-06-27T21:33:06.748538+0000","last_fullsized":"2024-06-27T21:33:06.748538+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.955983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.955983+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"540'9593","reported_seq":9866,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:07.799773+0000","last_change":"2024-06-27T21:32:25.997427+0000","last_active":"2024-06-27T21:33:07.799773+0000","last_peered":"2024-06-27T21:33:07.799773+0000","last_clean":"2024-06-27T21:33:07.799773+0000","last_became_active":"2024-06-27T21:29:19.822526+0000","last_became_peered":"2024-06-27T21:29:19.822526+0000","last_unstale":"2024-06-27T21:33:07.799773+0000","last_undegraded":"2024-06-27T21:33:07.799773+0000","last_fullsized":"2024-06-27T21:33:07.799773+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'9415","last_scrub_stamp":"2024-06-27T21:32:25.997194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:25.997194+0000","objects_scrubbed":3847,"log_size":9593,"log_dups_size":0,"ondisk_log_size":9593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:52.701904+0000","scrub_duration":1334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451584,"num_objects":3669,"num_object_clones":0,"num_object_copies":11007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9593,"num_write_kb":424384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":763,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.810519+0000","last_change":"2024-06-27T21:32:26.712793+0000","last_active":"2024-06-27T21:33:06.810519+0000","last_peered":"2024-06-27T21:33:06.810519+0000","last_clean":"2024-06-27T21:33:06.810519+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:33:06.810519+0000","last_undegraded":"2024-06-27T21:33:06.810519+0000","last_fullsized":"2024-06-27T21:33:06.810519+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:26.712657+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:32:26.712657+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:31.938171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"540'9822","reported_seq":10050,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:11.264713+0000","last_change":"2024-06-27T21:30:45.505788+0000","last_active":"2024-06-27T21:33:11.264713+0000","last_peered":"2024-06-27T21:33:11.264713+0000","last_clean":"2024-06-27T21:33:11.264713+0000","last_became_active":"2024-06-27T21:29:19.921198+0000","last_became_peered":"2024-06-27T21:29:19.921198+0000","last_unstale":"2024-06-27T21:33:11.264713+0000","last_undegraded":"2024-06-27T21:33:11.264713+0000","last_fullsized":"2024-06-27T21:33:11.264713+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'5955","last_scrub_stamp":"2024-06-27T21:30:45.505543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.505543+0000","objects_scrubbed":5927,"log_size":9822,"log_dups_size":0,"ondisk_log_size":9822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243662848,"num_objects":3718,"num_object_clones":0,"num_object_copies":11154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9822,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":167,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.738732+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:33:06.738732+0000","last_peered":"2024-06-27T21:33:06.738732+0000","last_clean":"2024-06-27T21:33:06.738732+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:33:06.738732+0000","last_undegraded":"2024-06-27T21:33:06.738732+0000","last_fullsized":"2024-06-27T21:33:06.738732+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:19.543339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"540'9655","reported_seq":9891,"reported_epoch":540,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:10.955705+0000","last_change":"2024-06-27T21:32:15.062048+0000","last_active":"2024-06-27T21:33:10.955705+0000","last_peered":"2024-06-27T21:33:10.955705+0000","last_clean":"2024-06-27T21:32:12.787863+0000","last_became_active":"2024-06-27T21:32:15.039338+0000","last_became_peered":"2024-06-27T21:32:15.039338+0000","last_unstale":"2024-06-27T21:33:10.955705+0000","last_undegraded":"2024-06-27T21:32:14.836078+0000","last_fullsized":"2024-06-27T21:32:14.801629+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6309","last_scrub_stamp":"2024-06-27T21:30:52.261532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.261532+0000","objects_scrubbed":6289,"log_size":9655,"log_dups_size":0,"ondisk_log_size":9655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251592704,"num_objects":3839,"num_object_clones":0,"num_object_copies":11517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9655,"num_write_kb":431808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9],"avail_no_missing":["0","8","9"],"object_location_counts":[{"shards":"0,8,9","objects":3839},{"shards":"0,9","objects":-195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":159,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.748729+0000","last_change":"2024-06-27T21:33:01.399273+0000","last_active":"2024-06-27T21:33:06.748729+0000","last_peered":"2024-06-27T21:33:06.748729+0000","last_clean":"2024-06-27T21:33:06.748729+0000","last_became_active":"2024-06-27T21:32:59.709705+0000","last_became_peered":"2024-06-27T21:32:59.709705+0000","last_unstale":"2024-06-27T21:33:06.748729+0000","last_undegraded":"2024-06-27T21:33:06.748729+0000","last_fullsized":"2024-06-27T21:33:06.748729+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:23.108874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"540'9685","reported_seq":20836,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:11.549583+0000","last_change":"2024-06-27T21:31:33.655323+0000","last_active":"2024-06-27T21:33:11.549583+0000","last_peered":"2024-06-27T21:33:11.549583+0000","last_clean":"2024-06-27T21:31:31.288940+0000","last_became_active":"2024-06-27T21:31:33.577869+0000","last_became_peered":"2024-06-27T21:31:33.577869+0000","last_unstale":"2024-06-27T21:33:11.549583+0000","last_undegraded":"2024-06-27T21:31:33.336543+0000","last_fullsized":"2024-06-27T21:31:33.308088+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243335168,"num_objects":3713,"num_object_clones":0,"num_object_copies":11139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":66,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9685,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5458,"num_bytes_recovered":340197376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,6","objects":3647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"540'9557","reported_seq":9826,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.926005+0000","last_change":"2024-06-27T21:32:26.025196+0000","last_active":"2024-06-27T21:33:06.926005+0000","last_peered":"2024-06-27T21:33:06.926005+0000","last_clean":"2024-06-27T21:33:06.926005+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:33:06.926005+0000","last_undegraded":"2024-06-27T21:33:06.926005+0000","last_fullsized":"2024-06-27T21:33:06.926005+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'9389","last_scrub_stamp":"2024-06-27T21:32:26.024974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:26.024974+0000","objects_scrubbed":3795,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:33:31.641075+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237699072,"num_objects":3627,"num_object_clones":0,"num_object_copies":10881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":175,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.761688+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:33:06.761688+0000","last_peered":"2024-06-27T21:33:06.761688+0000","last_clean":"2024-06-27T21:33:06.761688+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:33:06.761688+0000","last_undegraded":"2024-06-27T21:33:06.761688+0000","last_fullsized":"2024-06-27T21:33:06.761688+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:11.332879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"540'10003","reported_seq":15435,"reported_epoch":540,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:12.004909+0000","last_change":"2024-06-27T21:32:36.645550+0000","last_active":"2024-06-27T21:33:12.004909+0000","last_peered":"2024-06-27T21:33:12.004909+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:32:34.904596+0000","last_became_peered":"2024-06-27T21:32:34.904596+0000","last_unstale":"2024-06-27T21:33:12.004909+0000","last_undegraded":"2024-06-27T21:32:22.691354+0000","last_fullsized":"2024-06-27T21:32:22.682941+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250413056,"num_objects":3821,"num_object_clones":0,"num_object_copies":11463,"num_objects_missing_on_primary":5183,"num_objects_missing":5183,"num_objects_degraded":3,"num_objects_misplaced":11448,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10003,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1714,"num_bytes_recovered":112001024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10],"avail_no_missing":["5","6","8"],"object_location_counts":[{"shards":"5,6,7,8","objects":15},{"shards":"5,6,8","objects":3806},{"shards":"7,10","objects":-3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":173,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-06-27T21:33:06.743643+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:33:06.743643+0000","last_peered":"2024-06-27T21:33:06.743643+0000","last_clean":"2024-06-27T21:33:06.743643+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:33:06.743643+0000","last_undegraded":"2024-06-27T21:33:06.743643+0000","last_fullsized":"2024-06-27T21:33:06.743643+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:22.883827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3901030424,"num_objects":59526,"num_object_clones":0,"num_object_copies":178578,"num_objects_missing_on_primary":5183,"num_objects_missing":5183,"num_objects_degraded":7328,"num_objects_misplaced":17226,"num_objects_unfound":0,"num_objects_dirty":59526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155658,"num_write_kb":6885825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31835,"num_bytes_recovered":2046296064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13938212864,"data_stored":13938196576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155658,"ondisk_log_size":155658,"up":48,"acting":39,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":754208,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253151,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509588,"kb_used_data":955020,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92235564,"statfs":{"total":95995035648,"available":94449217536,"internally_reserved":0,"allocated":977940480,"data_stored":977642813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514787,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1353944,"kb_used_data":939600,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92391208,"statfs":{"total":95995035648,"available":94608596992,"internally_reserved":0,"allocated":962150400,"data_stored":961848661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743720,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1628692,"kb_used_data":1196300,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92116460,"statfs":{"total":95995035648,"available":94327255040,"internally_reserved":0,"allocated":1225011200,"data_stored":1224713533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":266287972651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266990,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1845468,"kb_used_data":1363092,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91899684,"statfs":{"total":95995035648,"available":94105276416,"internally_reserved":0,"allocated":1395806208,"data_stored":1395504989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528630,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2130068,"kb_used_data":1633804,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91615084,"statfs":{"total":95995035648,"available":93813846016,"internally_reserved":0,"allocated":1673015296,"data_stored":1672717629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757563,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199964,"kb_used_data":1848080,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91545188,"statfs":{"total":95995035648,"available":93742272512,"internally_reserved":0,"allocated":1892433920,"data_stored":1892132181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161267,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167892,"kb_used_data":950028,"kb_used_omap":92,"kb_used_meta":217763,"kb_avail":92577260,"statfs":{"total":95995035648,"available":94799114240,"internally_reserved":0,"allocated":972828672,"data_stored":972531005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":222989410},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182723,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063832,"kb_used_data":707788,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92681320,"statfs":{"total":95995035648,"available":94905671680,"internally_reserved":0,"allocated":724774912,"data_stored":724477245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1641952,"kb_used_data":1285908,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92103200,"statfs":{"total":95995035648,"available":94313676800,"internally_reserved":0,"allocated":1316769792,"data_stored":1316468573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575181,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454244,"kb_used_data":957976,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92290908,"statfs":{"total":95995035648,"available":94505889792,"internally_reserved":0,"allocated":980967424,"data_stored":980662133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738706,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1477852,"kb_used_data":1017684,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92267300,"statfs":{"total":95995035648,"available":94481715200,"internally_reserved":0,"allocated":1042108416,"data_stored":1041807197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1039273984,"data_stored":1039269912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":979439616,"data_stored":979435544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1315241984,"data_stored":1315241984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":723779584,"data_stored":723779584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":972095488,"data_stored":972095488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1891569664,"data_stored":1891565592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1671168000,"data_stored":1671168000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1398013952,"data_stored":1398013952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":784138240,"data_stored":784138240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1221918720,"data_stored":1221918720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":965611520,"data_stored":965607448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":975962112,"data_stored":975962112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:33:16.820 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-27T21:33:16.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-06-27T21:33:16.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:16.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:16.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:16.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:17.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:17.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:17.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:17.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:16 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:17.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:17.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:17.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:16 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:17.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:16 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:17.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:17.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:17.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:17.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:17.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:17.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:17.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:17.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:17.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:17.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:17.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:17.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:17.989 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:33:17.989 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:33:17.989 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:33:17.989 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:33:17.989 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:33:18.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:18.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:18.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:18.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:17 smithi151 ceph-mon[28965]: pgmap v1415: 47 pgs: 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 36 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 7328/177633 objects degraded (4.125%); 16680/177633 objects misplaced (9.390%); 20 MiB/s, 475 objects/s recovering 2024-06-27T21:33:18.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:18.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:17 smithi190 ceph-mon[32247]: pgmap v1415: 47 pgs: 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 36 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 7328/177633 objects degraded (4.125%); 16680/177633 objects misplaced (9.390%); 20 MiB/s, 475 objects/s recovering 2024-06-27T21:33:18.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:17 smithi192 ceph-mon[32956]: pgmap v1415: 47 pgs: 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 36 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 102 op/s; 7328/177633 objects degraded (4.125%); 16680/177633 objects misplaced (9.390%); 20 MiB/s, 475 objects/s recovering 2024-06-27T21:33:18.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:18.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:18.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:18.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:18.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:18.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:18.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:18.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:18.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:18.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:18.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:18.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:33:18.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:19.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:19.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:19.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:18 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:18 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:19.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:18 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:18 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:19.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:18 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:19.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:18 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:19.320 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:19.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:19.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:19.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:19.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:19.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:19.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:19.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:19.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:19.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:19.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:19.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:19.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:19.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:19 smithi151 ceph-mon[28965]: pgmap v1416: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 8 active+recovering+undersized+degraded+remapped, 36 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 104 op/s; 7328/176868 objects degraded (4.143%); 16581/176868 objects misplaced (9.375%); 13 MiB/s, 436 objects/s recovering 2024-06-27T21:33:19.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:19.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:19.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:19 smithi151 ceph-mon[28965]: osdmap e541: 12 total, 12 up, 11 in 2024-06-27T21:33:19.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/176868 objects degraded (4.143%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:20.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:20.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:20.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:20.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:20.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:20.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:19 smithi190 ceph-mon[32247]: pgmap v1416: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 8 active+recovering+undersized+degraded+remapped, 36 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 104 op/s; 7328/176868 objects degraded (4.143%); 16581/176868 objects misplaced (9.375%); 13 MiB/s, 436 objects/s recovering 2024-06-27T21:33:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:19 smithi190 ceph-mon[32247]: osdmap e541: 12 total, 12 up, 11 in 2024-06-27T21:33:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/176868 objects degraded (4.143%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:20.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:19 smithi192 ceph-mon[32956]: pgmap v1416: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 8 active+recovering+undersized+degraded+remapped, 36 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 104 op/s; 7328/176868 objects degraded (4.143%); 16581/176868 objects misplaced (9.375%); 13 MiB/s, 436 objects/s recovering 2024-06-27T21:33:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:20.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:19 smithi192 ceph-mon[32956]: osdmap e541: 12 total, 12 up, 11 in 2024-06-27T21:33:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/176868 objects degraded (4.143%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:20.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:20.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:20.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:20.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:20.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:20.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:20.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:20.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:20.838 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:20.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:20.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:20.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:20.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:21.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:21.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:20 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:21.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:21.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:20 smithi151 ceph-mon[28965]: osdmap e542: 12 total, 12 up, 11 in 2024-06-27T21:33:21.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:21.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:21.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:20 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:21.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:20 smithi190 ceph-mon[32247]: osdmap e542: 12 total, 12 up, 11 in 2024-06-27T21:33:21.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:21.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:20 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:21.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:20 smithi192 ceph-mon[32956]: osdmap e542: 12 total, 12 up, 11 in 2024-06-27T21:33:21.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:33:21.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:21.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:21.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:21.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:21.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:21.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:21.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:21.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:21.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:21.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:21.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:21.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:21 smithi151 ceph-mon[28965]: pgmap v1419: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 7328/176451 objects degraded (4.153%); 16581/176451 objects misplaced (9.397%); 4.5 MiB/s, 418 objects/s recovering 2024-06-27T21:33:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:21 smithi151 ceph-mon[28965]: osdmap e543: 12 total, 12 up, 11 in 2024-06-27T21:33:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:21 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:21 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:21.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:22.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:22.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:22.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:22.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:21 smithi190 ceph-mon[32247]: pgmap v1419: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 7328/176451 objects degraded (4.153%); 16581/176451 objects misplaced (9.397%); 4.5 MiB/s, 418 objects/s recovering 2024-06-27T21:33:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:21 smithi190 ceph-mon[32247]: osdmap e543: 12 total, 12 up, 11 in 2024-06-27T21:33:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:21 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:21 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:22.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:21 smithi192 ceph-mon[32956]: pgmap v1419: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 88 op/s; 7328/176451 objects degraded (4.153%); 16581/176451 objects misplaced (9.397%); 4.5 MiB/s, 418 objects/s recovering 2024-06-27T21:33:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:21 smithi192 ceph-mon[32956]: osdmap e543: 12 total, 12 up, 11 in 2024-06-27T21:33:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:21 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:21 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:22.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:22.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:22.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:22.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:22.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:22.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:22.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:22.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:22.839 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 6 pg_num to 26 2024-06-27T21:33:22.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:22.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:22.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:22.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:23.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:23.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:23.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:23.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:22 smithi151 ceph-mon[28965]: osdmap e544: 12 total, 12 up, 11 in 2024-06-27T21:33:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1406726791' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:33:23.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:23.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:22 smithi190 ceph-mon[32247]: osdmap e544: 12 total, 12 up, 11 in 2024-06-27T21:33:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1406726791' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:33:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:22 smithi192 ceph-mon[32956]: osdmap e544: 12 total, 12 up, 11 in 2024-06-27T21:33:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1406726791' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:33:23.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:23.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:23.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:23.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:23.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:23.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:23.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:23.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:23.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:23.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:23.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:23.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:24.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:24.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:24.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:24.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:23 smithi151 ceph-mon[28965]: pgmap v1422: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 52 op/s; 7328/176238 objects degraded (4.158%); 16290/176238 objects misplaced (9.243%); 4.5 MiB/s, 598 objects/s recovering 2024-06-27T21:33:24.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1406726791' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-06-27T21:33:24.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:23 smithi151 ceph-mon[28965]: osdmap e545: 12 total, 12 up, 11 in 2024-06-27T21:33:24.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:23 smithi190 ceph-mon[32247]: pgmap v1422: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 52 op/s; 7328/176238 objects degraded (4.158%); 16290/176238 objects misplaced (9.243%); 4.5 MiB/s, 598 objects/s recovering 2024-06-27T21:33:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1406726791' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-06-27T21:33:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:23 smithi190 ceph-mon[32247]: osdmap e545: 12 total, 12 up, 11 in 2024-06-27T21:33:24.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:23 smithi192 ceph-mon[32956]: pgmap v1422: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 38 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 52 op/s; 7328/176238 objects degraded (4.158%); 16290/176238 objects misplaced (9.243%); 4.5 MiB/s, 598 objects/s recovering 2024-06-27T21:33:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1406726791' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-06-27T21:33:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:23 smithi192 ceph-mon[32956]: osdmap e545: 12 total, 12 up, 11 in 2024-06-27T21:33:24.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:24.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:24.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:24.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:24.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:24.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:24.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:24.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:24.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:24.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:24.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:25.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:25.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:25.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:24 smithi151 ceph-mon[28965]: osdmap e546: 12 total, 12 up, 11 in 2024-06-27T21:33:25.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:24 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:25.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:24 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:25.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7328/176238 objects degraded (4.158%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:25.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:24 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-27T21:33:25.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:24 smithi151 ceph-mon[28965]: osdmap e547: 12 total, 12 up, 11 in 2024-06-27T21:33:25.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:25.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:24 smithi190 ceph-mon[32247]: osdmap e546: 12 total, 12 up, 11 in 2024-06-27T21:33:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:24 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:24 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7328/176238 objects degraded (4.158%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:24 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-27T21:33:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:24 smithi190 ceph-mon[32247]: osdmap e547: 12 total, 12 up, 11 in 2024-06-27T21:33:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:24 smithi192 ceph-mon[32956]: osdmap e546: 12 total, 12 up, 11 in 2024-06-27T21:33:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:24 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:24 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7328/176238 objects degraded (4.158%), 9 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:24 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-27T21:33:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:24 smithi192 ceph-mon[32956]: osdmap e547: 12 total, 12 up, 11 in 2024-06-27T21:33:25.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:25.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:25.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:25.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:25.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:25.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:25.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:25.690 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:33:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:25.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:25.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:25.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:25.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:25.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:25 smithi151 ceph-mon[28965]: pgmap v1425: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 39 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 7326/175500 objects degraded (4.174%); 16194/175500 objects misplaced (9.227%); 5.1 MiB/s, 514 objects/s recovering 2024-06-27T21:33:25.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1504507369' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:33:25.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:25 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:33:25.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:25 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:25.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:25 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:33:25.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:25 smithi151 ceph-mon[28965]: osdmap e548: 12 total, 12 up, 11 in 2024-06-27T21:33:25.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:25.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:26.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:26.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:26.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:26.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:25 smithi190 ceph-mon[32247]: pgmap v1425: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 39 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 7326/175500 objects degraded (4.174%); 16194/175500 objects misplaced (9.227%); 5.1 MiB/s, 514 objects/s recovering 2024-06-27T21:33:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1504507369' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:33:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:25 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:33:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:25 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:25 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:33:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:25 smithi190 ceph-mon[32247]: osdmap e548: 12 total, 12 up, 11 in 2024-06-27T21:33:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:25 smithi192 ceph-mon[32956]: pgmap v1425: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 39 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 79 op/s; 7326/175500 objects degraded (4.174%); 16194/175500 objects misplaced (9.227%); 5.1 MiB/s, 514 objects/s recovering 2024-06-27T21:33:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1504507369' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:33:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:25 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:33:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:25 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:25 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:33:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:25 smithi192 ceph-mon[32956]: osdmap e548: 12 total, 12 up, 11 in 2024-06-27T21:33:26.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:26.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:26.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:26.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:26.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:26.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:26.646 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:26.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:26.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:26.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:26.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:26.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:26.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:26.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-27T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:26 smithi151 ceph-mon[28965]: osdmap e549: 12 total, 12 up, 11 in 2024-06-27T21:33:27.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:27.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:27.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:33:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:27.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:33:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:33:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-27T21:33:27.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:26 smithi192 ceph-mon[32956]: osdmap e549: 12 total, 12 up, 11 in 2024-06-27T21:33:27.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-27T21:33:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:26 smithi190 ceph-mon[32247]: osdmap e549: 12 total, 12 up, 11 in 2024-06-27T21:33:27.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:27.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:27.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:27.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:27.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:27.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:27.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:27.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:27.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:27.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:27.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:27.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:27.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:28.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:28.088 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:28.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: pgmap v1428: 57 pgs: 1 remapped+peering, 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 7235/175296 objects degraded (4.127%); 16194/175296 objects misplaced (9.238%); 528 KiB/s, 454 objects/s recovering 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:33:28.146 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:27 smithi151 ceph-mon[28965]: osdmap e550: 12 total, 12 up, 11 in 2024-06-27T21:33:28.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:28.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: pgmap v1428: 57 pgs: 1 remapped+peering, 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 7235/175296 objects degraded (4.127%); 16194/175296 objects misplaced (9.238%); 528 KiB/s, 454 objects/s recovering 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:33:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:27 smithi190 ceph-mon[32247]: osdmap e550: 12 total, 12 up, 11 in 2024-06-27T21:33:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: pgmap v1428: 57 pgs: 1 remapped+peering, 10 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 39 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 78 op/s; 7235/175296 objects degraded (4.127%); 16194/175296 objects misplaced (9.238%); 528 KiB/s, 454 objects/s recovering 2024-06-27T21:33:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:33:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:33:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:33:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:33:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:27 smithi192 ceph-mon[32956]: osdmap e550: 12 total, 12 up, 11 in 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25312.0:163707 4.f 4:fa8ede5c:::benchmark_data_smithi192_88961_object72968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.945784+0000", 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 584.34845770499999, 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72990378199999995, 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.297 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25312", 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/4222178936", 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 163707 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945784+0000", 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945784+0000", 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945785+0000", 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7699999999999992e-07 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945788+0000", 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4809999999999998e-06 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945799+0000", 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0295000000000001e-05 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945804+0000", 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945854+0000", 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0015999999999997e-05 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.945956+0000", 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101816 2024-06-27T21:33:28.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,11", 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.946034+0000", 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8271999999999999e-05 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950082+0000", 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040472379999999999 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.950119+0000", 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7231999999999999e-05 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675577+0000", 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72545799 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675639+0000", 2024-06-27T21:33:28.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2314e-05 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675688+0000", 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8671000000000002e-05 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25312.0:163707 4.f e362/282 ondisk, result = 0)", 2024-06-27T21:33:28.304 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.949869+0000", 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 584.344372249, 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72582532300000002, 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949869+0000", 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949869+0000", 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949870+0000", 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8100000000000002e-07 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.949874+0000", 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.551e-06 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675482+0000", 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72560794399999995 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675505+0000", 2024-06-27T21:33:28.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3059000000000001e-05 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675558+0000", 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3541000000000001e-05 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675569+0000", 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1051e-05 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.675695+0000", 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012549600000000001 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25312.0:163716 4.3 e362/282 4:ce21cfad:::benchmark_data_smithi192_88961_object72977:head v 362'10240, mlcod=362'10238)", 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:23:43.970556+0000", 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 584.32368580699995, 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70433514200000003, 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.309 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970556+0000", 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970556+0000", 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970557+0000", 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4169999999999999e-06 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:43.970568+0000", 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0637e-05 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669850+0000", 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.699282395 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669855+0000", 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1599999999999997e-06 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669941+0000", 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5082999999999998e-05 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.669964+0000", 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3575999999999999e-05 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.674844+0000", 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048802710000000003 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:23:44.674891+0000", 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6603000000000001e-05 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e376,346 4:29d172ee:::benchmark_data_smithi192_88961_object19814:head,337'6947)", 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:26.382105+0000", 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.91213626000001, 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51754427000000003, 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.314 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382105+0000", 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382105+0000", 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382106+0000", 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3300000000000002e-07 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.382109+0000", 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0089999999999999e-06 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899025+0000", 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51691604099999999 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899044+0000", 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9266000000000002e-05 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899208+0000", 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016374799999999999 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:26.899650+0000", 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044167300000000001 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2c9cc615:::benchmark_data_smithi192_88961_object31150:head,343'7618)", 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:33.676427+0000", 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 534.61781455000005, 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36238720299999999, 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676427+0000", 2024-06-27T21:33:28.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676427+0000", 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676428+0000", 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5700000000000002e-07 2024-06-27T21:33:28.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:33.676439+0000", 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1600000000000001e-05 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038278+0000", 2024-06-27T21:33:28.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36183875399999998 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038295+0000", 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6702999999999999e-05 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038413+0000", 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118225 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.038814+0000", 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040116400000000002 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:24b2f406:::benchmark_data_smithi192_88961_object31387:head,343'7636)", 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:34.126182+0000", 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 534.16805932900002, 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85182796800000005, 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126182+0000", 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126182+0000", 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126183+0000", 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3700000000000001e-07 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.126186+0000", 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.676e-06 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.977601+0000", 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.85141412299999997 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.977612+0000", 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1447000000000001e-05 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.977730+0000", 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011793 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:34.978010+0000", 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028035499999999998 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2bef463e:::benchmark_data_smithi192_88961_object31885:head,343'7665)", 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:35.105338+0000", 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 533.18890393200002, 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48611961300000001, 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105338+0000", 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105338+0000", 2024-06-27T21:33:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105345+0000", 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1339999999999998e-06 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.105355+0000", 2024-06-27T21:33:28.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:28.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0266e-05 2024-06-27T21:33:28.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.590991+0000", 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48563563900000001 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.590999+0000", 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0450000000000001e-06 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.591087+0000", 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7956999999999998e-05 2024-06-27T21:33:28.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:35.591457+0000", 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000370572 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2c917cec:::benchmark_data_smithi192_88961_object33714:head,343'7766)", 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:37.211636+0000", 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 531.08260575899999, 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.76569309100000005, 2024-06-27T21:33:28.335 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211636+0000", 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211636+0000", 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211637+0000", 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.457e-06 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.211648+0000", 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0370000000000001e-05 2024-06-27T21:33:28.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977014+0000", 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.76536581000000004 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977027+0000", 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2965e-05 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977093+0000", 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6544000000000005e-05 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:37.977329+0000", 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023594499999999999 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:24984d7a:::benchmark_data_smithi192_88961_object34428:head,343'7813)", 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:38.197021+0000", 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 530.09722025999997, 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41096396699999999, 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197021+0000", 2024-06-27T21:33:28.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197021+0000", 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197023+0000", 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.46e-06 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.197032+0000", 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5000000000000005e-06 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607487+0000", 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41045458499999998 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607505+0000", 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7867e-05 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607701+0000", 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000195818 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.607985+0000", 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028473700000000002 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(4.4 e377,346 4:2e7e20f1:::benchmark_data_smithi192_88961_object34450:head,343'7814)", 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:24:38.611847+0000", 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 529.68239508399995, 2024-06-27T21:33:28.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42814653200000002, 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611847+0000", 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611847+0000", 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611855+0000", 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8689999999999993e-06 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:38.611867+0000", 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.112e-05 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039635+0000", 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42776847200000001 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039665+0000", 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0187000000000001e-05 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039768+0000", 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102796 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:24:39.039993+0000", 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000225088 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:131672 6.4 6:2c24218f:::benchmark_data_smithi192_117994_object24078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e515)", 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:31:31.348332+0000", 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 116.945910024, 2024-06-27T21:33:28.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57927920200000005, 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 131672 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348332+0000", 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348332+0000", 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348333+0000", 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.72e-06 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348341+0000", 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2010000000000002e-06 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348355+0000", 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4592e-05 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348366+0000", 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0740000000000001e-05 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348440+0000", 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3638000000000002e-05 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348580+0000", 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014018799999999999 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.348701+0000", 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121391 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.352988+0000", 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042868840000000004 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.599975+0000", 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.246986973 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.927497+0000", 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32752223699999999 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.927561+0000", 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4136999999999999e-05 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:31.927611+0000", 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9500999999999999e-05 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.357 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:139487 6.d e518/516 6:b4f24ea2:::benchmark_data_smithi192_117994_object31893:head v 518'8717, mlcod=480'1965)", 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:31:48.418546+0000", 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 99.875695716999999, 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28863029499999998, 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.418546+0000", 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.418546+0000", 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.418547+0000", 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.418552+0000", 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9620000000000001e-06 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.702910+0000", 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.284358 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.702930+0000", 2024-06-27T21:33:28.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0177999999999999e-05 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.702992+0000", 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2459999999999995e-05 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.703017+0000", 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5000000000000001e-05 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.707123+0000", 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041055010000000001 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:31:48.707176+0000", 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3335999999999999e-05 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:152686 6.4 6:2910d715:::benchmark_data_smithi192_117994_object45092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:36.940278+0000", 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 51.353963989999997, 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.892650622, 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152686 2024-06-27T21:33:28.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940278+0000", 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940278+0000", 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940280+0000", 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.956e-06 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940286+0000", 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4010000000000002e-06 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940303+0000", 2024-06-27T21:33:28.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7294000000000001e-05 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940314+0000", 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0998e-05 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940379+0000", 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4171000000000006e-05 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940475+0000", 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6858e-05 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.940550+0000", 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5020999999999999e-05 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.944516+0000", 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039659329999999996 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.944842+0000", 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032594500000000001 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832856+0000", 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.88801343700000002 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832892+0000", 2024-06-27T21:33:28.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6134999999999999e-05 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832928+0000", 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6473000000000001e-05 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:152686 6.4 e531/467 ondisk, result = 0)", 2024-06-27T21:33:28.369 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:36.944706+0000", 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 51.349536163000003, 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.888226615, 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.944706+0000", 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.944706+0000", 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.944706+0000", 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9100000000000002e-07 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:36.944712+0000", 2024-06-27T21:33:28.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4709999999999998e-06 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832816+0000", 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.88810414299999996 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832819+0000", 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.303e-06 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832847+0000", 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7946999999999999e-05 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832852+0000", 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9439999999999996e-06 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:37.832932+0000", 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9915999999999999e-05 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:153162 6.4 6:2decceea:::benchmark_data_smithi192_117994_object45568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:42.325944+0000", 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.968297945000003, 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47418280400000001, 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.374 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 153162 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.325944+0000", 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.325944+0000", 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.325946+0000", 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5409999999999999e-06 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.325956+0000", 2024-06-27T21:33:28.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2699999999999993e-06 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.325976+0000", 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.088e-05 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.325987+0000", 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.043e-05 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.326058+0000", 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1342999999999998e-05 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.326197+0000", 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000139097 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-06-27T21:33:28.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.326320+0000", 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012239099999999999 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.330248+0000", 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039284070000000001 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.331335+0000", 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001086931 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.800017+0000", 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46868203899999999 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.800073+0000", 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5862000000000001e-05 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.800127+0000", 2024-06-27T21:33:28.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3613000000000003e-05 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:153162 6.4 e532/467 ondisk, result = 0)", 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:42.330793+0000", 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.963448407999998, 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46933793200000001, 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.381 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.330793+0000", 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.330793+0000", 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.330795+0000", 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.449e-06 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.330802+0000", 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000003e-06 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.799884+0000", 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46908136 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.799891+0000", 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1559999999999998e-06 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.799999+0000", 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108367 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.800012+0000", 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3105000000000001e-05 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:42.800131+0000", 2024-06-27T21:33:28.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119005 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:153990 6.e e533/467 6:76763d4e:::benchmark_data_smithi192_117994_object46396:head v 533'9704, mlcod=533'9701)", 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:32:52.638144+0000", 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 35.656097551999999, 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.339264697, 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.638144+0000", 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.638144+0000", 2024-06-27T21:33:28.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.638145+0000", 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2279999999999999e-06 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.638156+0000", 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0767999999999999e-05 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.972701+0000", 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33454467300000001 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.972710+0000", 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0310000000000004e-06 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.972775+0000", 2024-06-27T21:33:28.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5222000000000007e-05 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.972799+0000", 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4102000000000001e-05 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.977353+0000", 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045539079999999997 2024-06-27T21:33:28.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:32:52.977409+0000", 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5764999999999999e-05 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:156266 6.d e540/516 6:bd78bb73:::benchmark_data_smithi192_117994_object48672:head v 540'9742, mlcod=508'7204)", 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:15.929327+0000", 2024-06-27T21:33:28.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 12.364914717, 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.165410522, 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.929327+0000", 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.929327+0000", 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.929328+0000", 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.083e-06 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.929335+0000", 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3010000000000001e-06 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:16.090843+0000", 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.161507503 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:16.090857+0000", 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3720000000000001e-05 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:16.091004+0000", 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014702899999999999 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:16.091053+0000", 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9206000000000003e-05 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:16.094703+0000", 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036499459999999998 2024-06-27T21:33:28.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:16.094737+0000", 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4734000000000003e-05 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157108 6.19 6.5fbcb8f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:26.762363+0000", 2024-06-27T21:33:28.396 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.5318788240000001, 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93288493800000005, 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157108 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762363+0000", 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762363+0000", 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762365+0000", 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8649999999999999e-06 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762373+0000", 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8150000000000006e-06 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762389+0000", 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6154000000000002e-05 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762397+0000", 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9759999999999995e-06 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762451+0000", 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4725000000000002e-05 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762458+0000", 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.404e-06 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695237+0000", 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93277944899999998 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695248+0000", 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0550000000000001e-05 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157114 6.19 6.78756779 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:26.886811+0000", 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.407430312, 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80846879699999996, 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157114 2024-06-27T21:33:28.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886811+0000", 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886811+0000", 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886813+0000", 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886820+0000", 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.973e-06 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886836+0000", 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6116999999999998e-05 2024-06-27T21:33:28.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886844+0000", 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9689999999999994e-06 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886896+0000", 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1560000000000001e-05 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:33:28.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886901+0000", 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5979999999999996e-06 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695272+0000", 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80837113599999999 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695280+0000", 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8380000000000002e-06 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.406 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:28.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:28.407 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:33:28.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:28.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:28.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:28.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:28.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:28.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:28.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:28.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:28.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:28.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:29.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:29.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:29.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:33:29.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:28 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:33:29.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:28 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:29.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:28 smithi151 ceph-mon[28965]: osdmap e551: 12 total, 12 up, 11 in 2024-06-27T21:33:29.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:29.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:28 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:33:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:28 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:28 smithi190 ceph-mon[32247]: osdmap e551: 12 total, 12 up, 11 in 2024-06-27T21:33:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:28 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:33:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:28 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:28 smithi192 ceph-mon[32956]: osdmap e551: 12 total, 12 up, 11 in 2024-06-27T21:33:29.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:29.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:29.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:29.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:29.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:29.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:29.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:29.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:29.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:29.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:29.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:29.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:29.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:29.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:30.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:30.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:30.132 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:33:30.132 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:33:30.132 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-27T21:33:30.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:33:30.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:30.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:30.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: pgmap v1431: 57 pgs: 1 peering, 1 remapped+peering, 8 unknown, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering, 42 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 128/158340 objects degraded (0.081%); 15336/158340 objects misplaced (9.685%); 0 B/s, 238 objects/s recovering 2024-06-27T21:33:30.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: 6.12 scrub starts 2024-06-27T21:33:30.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: 6.14 starting backfill to osd.7 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 128/158340 objects degraded (0.081%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:30.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:30.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: pgmap v1431: 57 pgs: 1 peering, 1 remapped+peering, 8 unknown, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering, 42 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 128/158340 objects degraded (0.081%); 15336/158340 objects misplaced (9.685%); 0 B/s, 238 objects/s recovering 2024-06-27T21:33:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: 6.12 scrub starts 2024-06-27T21:33:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: 6.14 starting backfill to osd.7 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 128/158340 objects degraded (0.081%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:30.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:30.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:30.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: pgmap v1431: 57 pgs: 1 peering, 1 remapped+peering, 8 unknown, 2 active+recovering+undersized+degraded+remapped, 3 active+recovering, 42 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 128/158340 objects degraded (0.081%); 15336/158340 objects misplaced (9.685%); 0 B/s, 238 objects/s recovering 2024-06-27T21:33:30.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: 6.12 scrub starts 2024-06-27T21:33:30.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: 6.14 starting backfill to osd.2 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: 6.14 starting backfill to osd.4 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: 6.14 starting backfill to osd.7 from (0'0,0'0] MAX to 546'10105 2024-06-27T21:33:30.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 128/158340 objects degraded (0.081%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:33:30.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:30.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:30.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:30.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:30.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:30.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:30.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:30.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:30.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:30.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:30.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:30.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:30.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:31.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:31.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:30 smithi151 ceph-mon[28965]: pgmap v1433: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 4 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3/163299 objects degraded (0.002%); 24504/163299 objects misplaced (15.006%); 4.8 MiB/s, 213 objects/s recovering 2024-06-27T21:33:31.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:30 smithi151 ceph-mon[28965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:33:31.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:31.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:31.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:30 smithi190 ceph-mon[32247]: pgmap v1433: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 4 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3/163299 objects degraded (0.002%); 24504/163299 objects misplaced (15.006%); 4.8 MiB/s, 213 objects/s recovering 2024-06-27T21:33:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:30 smithi190 ceph-mon[32247]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:33:31.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:30 smithi192 ceph-mon[32956]: pgmap v1433: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 remapped+peering, 4 unknown, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3/163299 objects degraded (0.002%); 24504/163299 objects misplaced (15.006%); 4.8 MiB/s, 213 objects/s recovering 2024-06-27T21:33:31.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:30 smithi192 ceph-mon[32956]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:33:31.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:31.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.361 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:31.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:31.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:31.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:31.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:31.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:31.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:31.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:31.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:31.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:31.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:31.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:32.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:32.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:32.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:31 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:32.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:32.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:32.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:31 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:31 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:32.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:32.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:32.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:32.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:32.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:32.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:32.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:32.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:32.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:32.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:32.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:32.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:32.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:33.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:33.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:33.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:32 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:33.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:32 smithi151 ceph-mon[28965]: pgmap v1434: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 3 unknown, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6529/162864 objects degraded (4.009%); 34144/162864 objects misplaced (20.965%); 10 MiB/s, 334 objects/s recovering 2024-06-27T21:33:33.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:32 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:33.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:33.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:33.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:32 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:33.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:32 smithi190 ceph-mon[32247]: pgmap v1434: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 3 unknown, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6529/162864 objects degraded (4.009%); 34144/162864 objects misplaced (20.965%); 10 MiB/s, 334 objects/s recovering 2024-06-27T21:33:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:32 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:33.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:32 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:32 smithi192 ceph-mon[32956]: pgmap v1434: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 3 unknown, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 6529/162864 objects degraded (4.009%); 34144/162864 objects misplaced (20.965%); 10 MiB/s, 334 objects/s recovering 2024-06-27T21:33:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:32 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:33.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:33.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:33.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:33.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:33.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:33.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:33.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:33.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:33.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:33.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:33.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:33.982 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:33.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:34.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:34.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:33 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:33:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:33 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:33 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:33:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:33 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:33:34.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:34.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:33 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:33:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:33 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:33 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:33:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:33 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:33:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:33 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:33 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:33 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:33 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:33:34.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:34.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:34.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:34.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:34.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:34.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:34.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:34.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:34.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:34.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:34.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:34.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:35.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:35.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:35.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: 6.12 scrub starts 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: pgmap v1435: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 42 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 23034/172764 objects degraded (13.333%); 41794/172764 objects misplaced (24.191%); 14 MiB/s, 294 objects/s recovering 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6529/162864 objects degraded (4.009%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:34 smithi151 ceph-mon[28965]: osdmap e552: 12 total, 12 up, 11 in 2024-06-27T21:33:35.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:35.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:35.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: 6.12 scrub starts 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: pgmap v1435: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 42 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 23034/172764 objects degraded (13.333%); 41794/172764 objects misplaced (24.191%); 14 MiB/s, 294 objects/s recovering 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6529/162864 objects degraded (4.009%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:34 smithi190 ceph-mon[32247]: osdmap e552: 12 total, 12 up, 11 in 2024-06-27T21:33:35.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:33:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:33:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: 6.12 scrub starts 2024-06-27T21:33:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: pgmap v1435: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 42 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 23034/172764 objects degraded (13.333%); 41794/172764 objects misplaced (24.191%); 14 MiB/s, 294 objects/s recovering 2024-06-27T21:33:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:33:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6529/162864 objects degraded (4.009%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:34 smithi192 ceph-mon[32956]: osdmap e552: 12 total, 12 up, 11 in 2024-06-27T21:33:35.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:35.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:35.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:35.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:35.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:35.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:35.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:35.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:35.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:35.856 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:35.860 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:35.879 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:35.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:35.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:35.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:36.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:36.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:36.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:36.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:36.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:36.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:36.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:36.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:36.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:36.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:36.634 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:33:36.634 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:33:36.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:36.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:33:36.662 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:33:36.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:33:36.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:33:36.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:33:36.713 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:33:36.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:33:36.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:33:36.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:36.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:36.769 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:33:36.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:33:36.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:33:36.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:33:36.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:33:36.825 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:33:36.851 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:33:36.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:33:36.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:36.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:33:36.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:33:36.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:36.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:36.908 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:33:36.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:33:36.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:36 smithi151 ceph-mon[28965]: osdmap e553: 12 total, 12 up, 11 in 2024-06-27T21:33:36.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1403097300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:33:36.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:36 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:33:36.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:33:36.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:33:36.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:37.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:37.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:36 smithi190 ceph-mon[32247]: osdmap e553: 12 total, 12 up, 11 in 2024-06-27T21:33:37.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1403097300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:33:37.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:36 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:33:37.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:36 smithi192 ceph-mon[32956]: osdmap e553: 12 total, 12 up, 11 in 2024-06-27T21:33:37.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1403097300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:33:37.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:36 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:33:37.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:37.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:37.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:37.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:37.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:37.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:37.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:37.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:37.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:37.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:37.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:37.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:37.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:37.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:37.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:37.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:37 smithi151 ceph-mon[28965]: pgmap v1438: 57 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 42 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 113 MiB/s wr, 2.73k op/s; 23034/172332 objects degraded (13.366%); 41117/172332 objects misplaced (23.859%); 26 MiB/s, 534 objects/s recovering 2024-06-27T21:33:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:37 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:37 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:37.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:38.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:38.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:37 smithi190 ceph-mon[32247]: pgmap v1438: 57 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 42 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 113 MiB/s wr, 2.73k op/s; 23034/172332 objects degraded (13.366%); 41117/172332 objects misplaced (23.859%); 26 MiB/s, 534 objects/s recovering 2024-06-27T21:33:38.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:37 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:38.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:37 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:37 smithi192 ceph-mon[32956]: pgmap v1438: 57 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 42 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 113 MiB/s wr, 2.73k op/s; 23034/172332 objects degraded (13.366%); 41117/172332 objects misplaced (23.859%); 26 MiB/s, 534 objects/s recovering 2024-06-27T21:33:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:37 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:38.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:37 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:38.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:38.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:38.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:38.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:38.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:38.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:38.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:38.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:38.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:38.636 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:38.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:38.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:38.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:38.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:38.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:38.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:38 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:33:38.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:38 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:38.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:38 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:33:38.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:39.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:39.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:38 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:33:39.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:38 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:39.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:38 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:33:39.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:38 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:33:39.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:38 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:39.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:38 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:33:39.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:39.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:39.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:39.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:39.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:39.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:39.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:39.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:39.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:39.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:39.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:39.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:39.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:39.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:39.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:39.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:39.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:40.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:39 smithi190 ceph-mon[32247]: pgmap v1439: 57 pgs: 1 peering, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 41 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 52 MiB/s wr, 1.33k op/s; 22969/171441 objects degraded (13.398%); 40267/171441 objects misplaced (23.487%); 19 MiB/s, 405 objects/s recovering 2024-06-27T21:33:40.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:39 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:33:40.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:39 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:33:40.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 22969/171441 objects degraded (13.398%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:40.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:40.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:39 smithi192 ceph-mon[32956]: pgmap v1439: 57 pgs: 1 peering, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 41 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 52 MiB/s wr, 1.33k op/s; 22969/171441 objects degraded (13.398%); 40267/171441 objects misplaced (23.487%); 19 MiB/s, 405 objects/s recovering 2024-06-27T21:33:40.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:39 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:33:40.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:39 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:33:40.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 22969/171441 objects degraded (13.398%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:40.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:40.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:40.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:39 smithi151 ceph-mon[28965]: pgmap v1439: 57 pgs: 1 peering, 2 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 41 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 52 MiB/s wr, 1.33k op/s; 22969/171441 objects degraded (13.398%); 40267/171441 objects misplaced (23.487%); 19 MiB/s, 405 objects/s recovering 2024-06-27T21:33:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:39 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:33:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:39 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:33:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 22969/171441 objects degraded (13.398%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:40.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:40.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:40.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:40.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:40.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:40.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:40.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:40.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:40.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:40.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:40.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:40.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:40.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:33:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: 6.12 scrub starts 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:40.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:33:40.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:41.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: 6.12 scrub starts 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:33:41.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:41.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:41.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:33:41.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:33:41.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:33:41.083 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: 6.12 scrub starts 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:33:41.084 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:33:41.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:41.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:41.251 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:41.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:41.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:41.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:41.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:41.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:41.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:41.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:41.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:41.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:41.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:41.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:41.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:41.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:42.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:42.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:41 smithi190 ceph-mon[32247]: pgmap v1440: 57 pgs: 1 peering, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 39 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 52 MiB/s wr, 1.35k op/s; 22969/170736 objects degraded (13.453%); 39294/170736 objects misplaced (23.014%); 19 MiB/s, 347 objects/s recovering 2024-06-27T21:33:42.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:42.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:41 smithi192 ceph-mon[32956]: pgmap v1440: 57 pgs: 1 peering, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 39 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 52 MiB/s wr, 1.35k op/s; 22969/170736 objects degraded (13.453%); 39294/170736 objects misplaced (23.014%); 19 MiB/s, 347 objects/s recovering 2024-06-27T21:33:42.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:42.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:42.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:41 smithi151 ceph-mon[28965]: pgmap v1440: 57 pgs: 1 peering, 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 39 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 52 MiB/s wr, 1.35k op/s; 22969/170736 objects degraded (13.453%); 39294/170736 objects misplaced (23.014%); 19 MiB/s, 347 objects/s recovering 2024-06-27T21:33:42.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:42.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:42.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:42.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:42.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:42.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:42.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:42.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:42.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:42.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:42.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:42.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:42.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:43.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:43.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:43.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:43.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:43.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: osdmap e554: 12 total, 12 up, 11 in 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: pgmap v1442: 57 pgs: 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 40 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 22969/170319 objects degraded (13.486%); 38028/170319 objects misplaced (22.328%); 20 MiB/s, 377 objects/s recovering 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:42 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:43.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:43.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:43.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: osdmap e554: 12 total, 12 up, 11 in 2024-06-27T21:33:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: pgmap v1442: 57 pgs: 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 40 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 22969/170319 objects degraded (13.486%); 38028/170319 objects misplaced (22.328%); 20 MiB/s, 377 objects/s recovering 2024-06-27T21:33:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:33:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:42 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:43.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:43.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:43.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:43.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: osdmap e554: 12 total, 12 up, 11 in 2024-06-27T21:33:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: pgmap v1442: 57 pgs: 4 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 40 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 103 op/s; 22969/170319 objects degraded (13.486%); 38028/170319 objects misplaced (22.328%); 20 MiB/s, 377 objects/s recovering 2024-06-27T21:33:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:33:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:42 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:43.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:43.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:43.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:43.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:43.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:43.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:43.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:43.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:43.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:43.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:43.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:43.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:44.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:44.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:44.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:44.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:44.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:43 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:33:44.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:43 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:33:44.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:44.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:44.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:44.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:43 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:33:44.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:44.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:44.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:44.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:44.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:44.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:44.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:44.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:44.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:44.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:44.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:44.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:44.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:45.018 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:45.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:45.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:44 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:33:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:44 smithi151 ceph-mon[28965]: osdmap e555: 12 total, 12 up, 11 in 2024-06-27T21:33:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:44 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:44 smithi151 ceph-mon[28965]: pgmap v1444: 57 pgs: 3 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 41 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 22879/169314 objects degraded (13.513%); 37420/169314 objects misplaced (22.101%); 21 MiB/s, 333 objects/s recovering 2024-06-27T21:33:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 22969/170319 objects degraded (13.486%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:45.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:44 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:33:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:44 smithi190 ceph-mon[32247]: osdmap e555: 12 total, 12 up, 11 in 2024-06-27T21:33:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:44 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:44 smithi190 ceph-mon[32247]: pgmap v1444: 57 pgs: 3 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 41 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 22879/169314 objects degraded (13.513%); 37420/169314 objects misplaced (22.101%); 21 MiB/s, 333 objects/s recovering 2024-06-27T21:33:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 22969/170319 objects degraded (13.486%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:45.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:44 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:33:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:44 smithi192 ceph-mon[32956]: osdmap e555: 12 total, 12 up, 11 in 2024-06-27T21:33:45.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:44 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:44 smithi192 ceph-mon[32956]: pgmap v1444: 57 pgs: 3 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 41 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 22879/169314 objects degraded (13.513%); 37420/169314 objects misplaced (22.101%); 21 MiB/s, 333 objects/s recovering 2024-06-27T21:33:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 22969/170319 objects degraded (13.486%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:45.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:45.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:45.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:45.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:45.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:45.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:45.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:45.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:45.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:45.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:45.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:45.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:46.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:46.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:46.133 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:46.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:45 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:33:46.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:45 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:33:46.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:45 smithi151 ceph-mon[28965]: 6.12 scrub starts 2024-06-27T21:33:46.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:46.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:46.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:45 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:33:46.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:45 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:33:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:45 smithi190 ceph-mon[32247]: 6.12 scrub starts 2024-06-27T21:33:46.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:45 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:33:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:45 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:33:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:45 smithi192 ceph-mon[32956]: 6.12 scrub starts 2024-06-27T21:33:46.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:46.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:46.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:46.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:46.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:46.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:46.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:46.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:46.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:46.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:46.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:46.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:47.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:47.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:47.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:46 smithi151 ceph-mon[28965]: pgmap v1445: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 104 op/s; 22879/168939 objects degraded (13.543%); 36324/168939 objects misplaced (21.501%); 20 MiB/s, 316 objects/s recovering 2024-06-27T21:33:47.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:47.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:46 smithi190 ceph-mon[32247]: pgmap v1445: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 104 op/s; 22879/168939 objects degraded (13.543%); 36324/168939 objects misplaced (21.501%); 20 MiB/s, 316 objects/s recovering 2024-06-27T21:33:47.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:47.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:46 smithi192 ceph-mon[32956]: pgmap v1445: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 104 op/s; 22879/168939 objects degraded (13.543%); 36324/168939 objects misplaced (21.501%); 20 MiB/s, 316 objects/s recovering 2024-06-27T21:33:47.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:47.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:47.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:47.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:47.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:47.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:47.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:47.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:47.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:47.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:47.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:47.946 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:33:48.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:48.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:48.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:48.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:48.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:47 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:48.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3705412205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:33:48.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:47 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3705412205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 556, 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:33:47.939912+0000", 2024-06-27T21:33:48.265 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:23:43.806967+0000", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:32:12.792267+0000", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:33:48.266 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 27, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 6, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:33:48.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:33:48.268 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:33:48.269 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "537", 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "535", 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:33:48.270 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:33:48.271 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:33:48.272 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:33:48.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.3199999332427979, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.3199999332427979, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.130000114440918, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.130000114440918, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.80101770162582397, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.80000001192092896, 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 6, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_4", 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:29:17.908064+0000", 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:33:48.275 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 26, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 26, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 26, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 26, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 26, 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:33:48.276 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "549", 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "547", 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:33:48.277 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:33:48.278 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:33:48.279 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:33:48.280 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.2999999523162842, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.2200000286102295, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.119999885559082, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.9600000381469727, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.80101770162582397, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.80000001192092896, 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.80000001192092896 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.282 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.283 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.284 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:33:48.287 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.1994171142578125, 2024-06-27T21:33:48.288 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:33:48.290 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.291 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.292 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:33:48.293 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.294 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 549, 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:33:48.296 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:33:48.297 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.298 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:33:48.299 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.3845977783203125, 2024-06-27T21:33:48.300 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.305 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:33:48.307 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.3238067626953125, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 31, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 347, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 362, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 348, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.308 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.309 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.310 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.311 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:33:48.312 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/3587059633", 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/3587059633", 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/3587059633", 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/3587059633", 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.5222625732421875, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 547, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.315 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:33:48.316 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.317 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.319 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6059112548828125, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.0889892578125, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.320 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 479, 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.321 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:33:48.323 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.324 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:33:48.325 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.326 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:33:48.328 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.329 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.330 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:33:48.331 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.332 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 525, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.333 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.334 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.336 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.337 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.338 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6501922607421875, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 552, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.339 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.340 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.341 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:33:48.344 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.346 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.348 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:33:48.349 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 550, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:33:48.351 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.352 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.353 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.354 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:33:48.355 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.356 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:27.988549+0000", 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.357 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:32:25.628008+0000", 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.358 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:47.714312+0000", 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.359 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:18.157624+0000", 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:06.051545+0000", 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 347 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:20.900647+0000", 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.361 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:19.451176+0000", 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.362 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:32:38.389622+0000", 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:18.196397+0000", 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:48.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:48.364 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:47 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:48.364 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3705412205' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.365 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:41.427919+0000", 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:33:43.055657+0000", 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:32:33.968031+0000", 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-06-27T21:33:48.367 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.c", 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: 10 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.10", 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.368 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: 1 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.11", 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: 10 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.13", 2024-06-27T21:33:48.369 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: 1 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.14", 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: 5, 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.15", 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: 4 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.16", 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-06-27T21:33:48.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.17", 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: 3, 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.18", 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: 3, 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-06-27T21:33:48.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.19", 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:33:48.373 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:33:48.374 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:33:48.375 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:33:48.376 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:33:48.376 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:33:48.376 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:33:48.376 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:33:48.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:33:48.376 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:33:48.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:48.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:48.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:33:48.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:48.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:48.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:48.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:48.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:48.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:48.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:48.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:49.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:49.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:49.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3705412205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: osdmap e556: 12 total, 12 up, 11 in 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: pgmap v1447: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 106 op/s; 22843/168336 objects degraded (13.570%); 35307/168336 objects misplaced (20.974%); 18 MiB/s, 293 objects/s recovering 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3060531654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:33:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:33:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:49.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:49.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:49.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3705412205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: osdmap e556: 12 total, 12 up, 11 in 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: pgmap v1447: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 106 op/s; 22843/168336 objects degraded (13.570%); 35307/168336 objects misplaced (20.974%); 18 MiB/s, 293 objects/s recovering 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3060531654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3705412205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: osdmap e556: 12 total, 12 up, 11 in 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: pgmap v1447: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 106 op/s; 22843/168336 objects degraded (13.570%); 35307/168336 objects misplaced (20.974%); 18 MiB/s, 293 objects/s recovering 2024-06-27T21:33:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3060531654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:33:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:33:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:49.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:49.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:49.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:49.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:49.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:49.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:49.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:49.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:49.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:49.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:49.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:49.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:50.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:50.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:50.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: 6.1 scrub ok 2024-06-27T21:33:50.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: osdmap e557: 12 total, 12 up, 11 in 2024-06-27T21:33:50.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:50.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:50.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:33:50.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 22843/168336 objects degraded (13.570%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:50.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:33:50.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:49 smithi151 ceph-mon[28965]: 6.7 scrub ok 2024-06-27T21:33:50.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:50.184 INFO:tasks.thrashosds.thrasher:j is {'epoch': 556, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:33:47.939912+0000', 'last_up_change': '2024-06-27T21:23:43.806967+0000', 'last_in_change': '2024-06-27T21:32:12.792267+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '537', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '535', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.801017701625824, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.800000011920929}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-06-27T21:29:17.908064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '549', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '547', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.299999952316284, 'score_stable': 3.2200000286102295, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.119999885559082, 'raw_score_stable': 2.9600000381469727, 'primary_affinity_weighted': 0.801017701625824, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.800000011920929}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 550, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1994171142578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 550, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 0.3845977783203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 550, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3238067626953125, 'last_clean_begin': 31, 'last_clean_end': 347, 'up_from': 362, 'up_thru': 550, 'down_at': 348, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 3587059633}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 3587059633}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 3587059633}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 3587059633}]}, 'public_addr': '172.21.15.190:6801/3587059633', 'cluster_addr': '172.21.15.190:6803/3587059633', 'heartbeat_back_addr': '172.21.15.190:6807/3587059633', 'heartbeat_front_addr': '172.21.15.190:6805/3587059633', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 0.5222625732421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 547, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 0.6059112548828125, 'primary_affinity': 0.0889892578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 479, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 550, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 0.6501922607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 552, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 550, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 550, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:27.988549+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:32:25.628008+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:47.714312+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:18.157624+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:06.051545+0000', 'dead_epoch': 347}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:20.900647+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:19.451176+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:32:38.389622+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-27T21:33:18.196397+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:41.427919+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:33:43.055657+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:32:33.968031+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.c', 'osds': [7, 10]}, {'pgid': '6.10', 'osds': [0, 1]}, {'pgid': '6.11', 'osds': [9, 10]}, {'pgid': '6.13', 'osds': [11, 1]}, {'pgid': '6.14', 'osds': [1, 5, 9]}, {'pgid': '6.15', 'osds': [7, 4]}, {'pgid': '6.16', 'osds': [4, 7]}, {'pgid': '6.17', 'osds': [3, 2]}, {'pgid': '6.18', 'osds': [3, 7]}, {'pgid': '6.19', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:33:50.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:33:50.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:50.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: 6.1 scrub ok 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: osdmap e557: 12 total, 12 up, 11 in 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 22843/168336 objects degraded (13.570%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:33:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:49 smithi190 ceph-mon[32247]: 6.7 scrub ok 2024-06-27T21:33:50.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: 6.1 scrub ok 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: osdmap e557: 12 total, 12 up, 11 in 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 22843/168336 objects degraded (13.570%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:33:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:49 smithi192 ceph-mon[32956]: 6.7 scrub ok 2024-06-27T21:33:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:50.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:50.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:50.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:50.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:50.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:50.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:50.704 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:33:50.704 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:33:50.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:50.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:50.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:50.918 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:50.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:50.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:51.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:51.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:51.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:51.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: 6.8 scrub ok 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: osdmap e558: 12 total, 12 up, 11 in 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: pgmap v1450: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 22843/167844 objects degraded (13.610%); 34063/167844 objects misplaced (20.294%); 22 MiB/s, 347 objects/s recovering 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: 6.4 scrub ok 2024-06-27T21:33:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:50 smithi190 ceph-mon[32247]: 6.6 scrub starts 2024-06-27T21:33:51.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:33:51.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:33:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: 6.8 scrub ok 2024-06-27T21:33:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: osdmap e558: 12 total, 12 up, 11 in 2024-06-27T21:33:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:33:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:33:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:33:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:33:51.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: pgmap v1450: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 22843/167844 objects degraded (13.610%); 34063/167844 objects misplaced (20.294%); 22 MiB/s, 347 objects/s recovering 2024-06-27T21:33:51.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: 6.4 scrub ok 2024-06-27T21:33:51.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:50 smithi192 ceph-mon[32956]: 6.6 scrub starts 2024-06-27T21:33:51.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: 6.8 scrub ok 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: osdmap e558: 12 total, 12 up, 11 in 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: pgmap v1450: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 81 op/s; 22843/167844 objects degraded (13.610%); 34063/167844 objects misplaced (20.294%); 22 MiB/s, 347 objects/s recovering 2024-06-27T21:33:51.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: 6.4 scrub ok 2024-06-27T21:33:51.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:50 smithi151 ceph-mon[28965]: 6.6 scrub starts 2024-06-27T21:33:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:51.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:51.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:51.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:51.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:51.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:51.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:51.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:51.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:51.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:51.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:51.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:51.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:51.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:52.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:52.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:52.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:52.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:51 smithi151 ceph-mon[28965]: 6.6 scrub ok 2024-06-27T21:33:52.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:51 smithi151 ceph-mon[28965]: osdmap e559: 12 total, 12 up, 11 in 2024-06-27T21:33:52.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:51 smithi151 ceph-mon[28965]: 6.12 scrub starts 2024-06-27T21:33:52.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:51 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:33:52.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:51 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:33:52.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:51 smithi190 ceph-mon[32247]: 6.6 scrub ok 2024-06-27T21:33:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:51 smithi190 ceph-mon[32247]: osdmap e559: 12 total, 12 up, 11 in 2024-06-27T21:33:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:51 smithi190 ceph-mon[32247]: 6.12 scrub starts 2024-06-27T21:33:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:51 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:33:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:51 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:33:52.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:52.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:51 smithi192 ceph-mon[32956]: 6.6 scrub ok 2024-06-27T21:33:52.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:51 smithi192 ceph-mon[32956]: osdmap e559: 12 total, 12 up, 11 in 2024-06-27T21:33:52.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:51 smithi192 ceph-mon[32956]: 6.12 scrub starts 2024-06-27T21:33:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:51 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:33:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:51 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:33:52.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:52.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:52.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:52.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:52.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:52.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:52.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:33:52.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:52.635 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:33:52.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:52.680 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:52.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:52.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:52.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:52.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:52.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:53.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:53.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:53.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:53.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:53.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:52 smithi151 ceph-mon[28965]: 6.12 scrub ok 2024-06-27T21:33:53.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:52 smithi151 ceph-mon[28965]: pgmap v1452: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22843/167604 objects degraded (13.629%); 32818/167604 objects misplaced (19.581%) 2024-06-27T21:33:53.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:52 smithi190 ceph-mon[32247]: 6.12 scrub ok 2024-06-27T21:33:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:52 smithi190 ceph-mon[32247]: pgmap v1452: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22843/167604 objects degraded (13.629%); 32818/167604 objects misplaced (19.581%) 2024-06-27T21:33:53.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:53.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:52 smithi192 ceph-mon[32956]: 6.12 scrub ok 2024-06-27T21:33:53.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:52 smithi192 ceph-mon[32956]: pgmap v1452: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 44 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22843/167604 objects degraded (13.629%); 32818/167604 objects misplaced (19.581%) 2024-06-27T21:33:53.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:53.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:53.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:53.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:53.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:53.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:53.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:53.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:53.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:53.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:53.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:53.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:54.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.057 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:54.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:54.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:54.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:54.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:53 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:33:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:53 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:33:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:53 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:33:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:53 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:33:54.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:53 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:33:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:53 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:33:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:53 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:33:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:53 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:33:54.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:54.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:54.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:54.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:53 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:33:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:33:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:33:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:33:54.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:54.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:54.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:54.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:54.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:33:54.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:33:54.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:54.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:54.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:55.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:55.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:55.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:55.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:55.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:54 smithi190 ceph-mon[32247]: pgmap v1453: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22779/166704 objects degraded (13.664%); 32369/166704 objects misplaced (19.417%) 2024-06-27T21:33:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:54 smithi190 ceph-mon[32247]: 6.2 scrub ok 2024-06-27T21:33:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 22843/167604 objects degraded (13.629%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:54 smithi190 ceph-mon[32247]: 6.0 scrub ok 2024-06-27T21:33:55.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:55.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:54 smithi192 ceph-mon[32956]: pgmap v1453: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22779/166704 objects degraded (13.664%); 32369/166704 objects misplaced (19.417%) 2024-06-27T21:33:55.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:54 smithi192 ceph-mon[32956]: 6.2 scrub ok 2024-06-27T21:33:55.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 22843/167604 objects degraded (13.629%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:54 smithi192 ceph-mon[32956]: 6.0 scrub ok 2024-06-27T21:33:55.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:55.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:55.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:55.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: pgmap v1453: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22779/166704 objects degraded (13.664%); 32369/166704 objects misplaced (19.417%) 2024-06-27T21:33:55.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: 6.2 scrub ok 2024-06-27T21:33:55.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 22843/167604 objects degraded (13.629%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:33:55.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:54 smithi151 ceph-mon[28965]: 6.0 scrub ok 2024-06-27T21:33:55.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:55.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:55.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:33:55.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:33:55.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:55.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:55.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:55.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:55.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:56.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:56.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:56.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:56.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:56.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:33:56.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:33:56.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:56.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:56.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:56.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:56.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:56.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:56.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:56.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:56.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:56.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:56.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:57.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:57.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:57.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:57.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:57.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:57.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:57.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:57.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:57 smithi151 ceph-mon[28965]: pgmap v1454: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22779/166005 objects degraded (13.722%); 31519/166005 objects misplaced (18.987%) 2024-06-27T21:33:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:57 smithi190 ceph-mon[32247]: pgmap v1454: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22779/166005 objects degraded (13.722%); 31519/166005 objects misplaced (18.987%) 2024-06-27T21:33:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:57 smithi192 ceph-mon[32956]: pgmap v1454: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22779/166005 objects degraded (13.722%); 31519/166005 objects misplaced (18.987%) 2024-06-27T21:33:57.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:57.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:57.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:57.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:57.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:33:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:57.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:33:57.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:57.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:57.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:58.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:58.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:58.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:58.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:58.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:33:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:58.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:33:58.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:58.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:58.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:58.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:58.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:58.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:58.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:33:58.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:33:58.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:58.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:58.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:59.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:59.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:59.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:33:59.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:33:59.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:33:59.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:33:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:33:59.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:33:59 smithi151 ceph-mon[28965]: pgmap v1455: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22707/164979 objects degraded (13.764%); 31055/164979 objects misplaced (18.824%) 2024-06-27T21:33:59.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:33:59.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:33:59.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:33:59 smithi190 ceph-mon[32247]: pgmap v1455: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22707/164979 objects degraded (13.764%); 31055/164979 objects misplaced (18.824%) 2024-06-27T21:33:59.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:33:59 smithi192 ceph-mon[32956]: pgmap v1455: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering, 43 active+clean; 3.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22707/164979 objects degraded (13.764%); 31055/164979 objects misplaced (18.824%) 2024-06-27T21:33:59.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:59.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:33:59.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:33:59.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:33:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:33:59.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:33:59.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:33:59.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:33:59.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:33:59.918 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:33:59.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:33:59.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:33:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:33:59.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:33:59.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:33:59.945 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:33:59.969 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:33:59.969 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:33:59.998 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:33:59.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:34:00.031 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:34:00.032 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:34:00.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:00.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:34:00.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:34:00.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:00.103 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:34:00.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:34:00.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:34:00.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:34:00.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:34:00.170 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:34:00.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:00.196 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:00.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:34:00.212 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:34:00.248 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:34:00.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:34:00.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:00.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:00.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:00 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 22707/164979 objects degraded (13.764%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2910934910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:34:00.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:00.496 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:34:00.497 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:34:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:00 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 22707/164979 objects degraded (13.764%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:00.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2910934910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:34:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:00 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 22707/164979 objects degraded (13.764%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:00.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2910934910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:34:00.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:00.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:00.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:00.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:00.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:00.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:00.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:00.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:00.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:01.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:01.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:01.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:01.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:01.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:01.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:01.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:01.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:01.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:01.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:01.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:01.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:01.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:01.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:01.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:01.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:01 smithi151 ceph-mon[28965]: 6.9 scrub starts 2024-06-27T21:34:01.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:01 smithi151 ceph-mon[28965]: pgmap v1456: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering, 44 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22707/164295 objects degraded (13.821%); 30372/164295 objects misplaced (18.486%) 2024-06-27T21:34:01.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:01 smithi151 ceph-mon[28965]: 6.9 scrub ok 2024-06-27T21:34:01.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:01 smithi151 ceph-mon[28965]: from='client.26329 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:34:01.971 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:01.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:01 smithi190 ceph-mon[32247]: 6.9 scrub starts 2024-06-27T21:34:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:01 smithi190 ceph-mon[32247]: pgmap v1456: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering, 44 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22707/164295 objects degraded (13.821%); 30372/164295 objects misplaced (18.486%) 2024-06-27T21:34:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:01 smithi190 ceph-mon[32247]: 6.9 scrub ok 2024-06-27T21:34:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:01 smithi190 ceph-mon[32247]: from='client.26329 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:34:02.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:01 smithi192 ceph-mon[32956]: 6.9 scrub starts 2024-06-27T21:34:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:01 smithi192 ceph-mon[32956]: pgmap v1456: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering, 44 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22707/164295 objects degraded (13.821%); 30372/164295 objects misplaced (18.486%) 2024-06-27T21:34:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:01 smithi192 ceph-mon[32956]: 6.9 scrub ok 2024-06-27T21:34:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:01 smithi192 ceph-mon[32956]: from='client.26329 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:34:02.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:02.096 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:02.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:02.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:02.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:02.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:02.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:02.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:02.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:02.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:02.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:02.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:02.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:02.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:02.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:02.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:02.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:03.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:03.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:03.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:03.217 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:03.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:03.262 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1456,"stamp":"2024-06-27T21:34:00.129856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3589407288,"num_objects":54765,"num_object_clones":0,"num_object_copies":164295,"num_objects_missing_on_primary":4895,"num_objects_missing":4895,"num_objects_degraded":22707,"num_objects_misplaced":30372,"num_objects_unfound":0,"num_objects_dirty":54765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120788,"num_write_kb":5173281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40940,"num_bytes_recovered":2213937176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260633,"ondisk_log_size":260633,"up":171,"acting":162,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":198,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":16792972,"kb_used_data":11804148,"kb_used_omap":108,"kb_used_meta":4988627,"kb_avail":1014403700,"statfs":{"total":1055945392128,"available":1038749388800,"internally_reserved":0,"allocated":12087447552,"data_stored":12083277097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111086,"internal_metadata":5108354578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-77692928,"num_objects":-1183,"num_object_clones":0,"num_object_copies":-3549,"num_objects_missing_on_primary":-69,"num_objects_missing":-69,"num_objects_degraded":-136,"num_objects_misplaced":-3691,"num_objects_unfound":0,"num_objects_dirty":-1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-38451,"num_write_kb":-1712544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5982,"num_bytes_recovered":-341901312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002688"},"pg_stats":[{"pgid":"6.18","version":"559'9996","reported_seq":10272,"reported_epoch":559,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:57.299962+0000","last_change":"2024-06-27T21:33:29.189177+0000","last_active":"2024-06-27T21:33:57.299962+0000","last_peered":"2024-06-27T21:33:57.299962+0000","last_clean":"2024-06-27T21:33:26.530184+0000","last_became_active":"2024-06-27T21:33:29.166421+0000","last_became_peered":"2024-06-27T21:33:29.166421+0000","last_unstale":"2024-06-27T21:33:57.299962+0000","last_undegraded":"2024-06-27T21:33:28.891033+0000","last_fullsized":"2024-06-27T21:33:28.693201+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'5955","last_scrub_stamp":"2024-06-27T21:30:45.505543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.505543+0000","objects_scrubbed":5927,"log_size":9996,"log_dups_size":0,"ondisk_log_size":9996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112328704,"num_objects":1714,"num_object_clones":0,"num_object_copies":5142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3341,"num_objects_misplaced":3341,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":216640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[3,7],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"3,7","objects":-82},{"shards":"3,7,11","objects":1684}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"559'9830","reported_seq":10467,"reported_epoch":559,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:56.309265+0000","last_change":"2024-06-27T21:33:29.154889+0000","last_active":"2024-06-27T21:33:56.309265+0000","last_peered":"2024-06-27T21:33:56.309265+0000","last_clean":"2024-06-27T21:32:12.787863+0000","last_became_active":"2024-06-27T21:33:29.125343+0000","last_became_peered":"2024-06-27T21:33:29.125343+0000","last_unstale":"2024-06-27T21:33:56.309265+0000","last_undegraded":"2024-06-27T21:33:28.949030+0000","last_fullsized":"2024-06-27T21:33:28.693282+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"502'6309","last_scrub_stamp":"2024-06-27T21:30:52.261532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.261532+0000","objects_scrubbed":6289,"log_size":9830,"log_dups_size":0,"ondisk_log_size":9830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116752384,"num_objects":1782,"num_object_clones":0,"num_object_copies":5346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6521,"num_objects_misplaced":3318,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":215904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":7667712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[0,2],"avail_no_missing":["0","8","9"],"object_location_counts":[{"shards":"0,2","objects":-66},{"shards":"0,2,8,9","objects":4},{"shards":"0,8,9","objects":1745}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":61,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.975244+0000","last_change":"2024-06-27T21:33:02.987696+0000","last_active":"2024-06-27T21:33:50.975244+0000","last_peered":"2024-06-27T21:33:50.975244+0000","last_clean":"2024-06-27T21:33:50.975244+0000","last_became_active":"2024-06-27T21:33:01.707872+0000","last_became_peered":"2024-06-27T21:33:01.707872+0000","last_unstale":"2024-06-27T21:33:50.975244+0000","last_undegraded":"2024-06-27T21:33:50.975244+0000","last_fullsized":"2024-06-27T21:33:50.975244+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:23.304568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,10],"acting":[11,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":61,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.975217+0000","last_change":"2024-06-27T21:33:01.956324+0000","last_active":"2024-06-27T21:33:50.975217+0000","last_peered":"2024-06-27T21:33:50.975217+0000","last_clean":"2024-06-27T21:33:50.975217+0000","last_became_active":"2024-06-27T21:33:01.707250+0000","last_became_peered":"2024-06-27T21:33:01.707250+0000","last_unstale":"2024-06-27T21:33:50.975217+0000","last_undegraded":"2024-06-27T21:33:50.975217+0000","last_fullsized":"2024-06-27T21:33:50.975217+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:03.600384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":88,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:52.993204+0000","last_change":"2024-06-27T21:33:52.993226+0000","last_active":"2024-06-27T21:33:52.993204+0000","last_peered":"2024-06-27T21:33:52.993204+0000","last_clean":"2024-06-27T21:33:52.993204+0000","last_became_active":"2024-06-27T21:33:01.707734+0000","last_became_peered":"2024-06-27T21:33:01.707734+0000","last_unstale":"2024-06-27T21:33:52.993204+0000","last_undegraded":"2024-06-27T21:33:52.993204+0000","last_fullsized":"2024-06-27T21:33:52.993204+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:52.993157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:52.993157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:03.807236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":63,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.976224+0000","last_change":"2024-06-27T21:33:02.568824+0000","last_active":"2024-06-27T21:33:50.976224+0000","last_peered":"2024-06-27T21:33:50.976224+0000","last_clean":"2024-06-27T21:33:50.976224+0000","last_became_active":"2024-06-27T21:33:01.773541+0000","last_became_peered":"2024-06-27T21:33:01.773541+0000","last_unstale":"2024-06-27T21:33:50.976224+0000","last_undegraded":"2024-06-27T21:33:50.976224+0000","last_fullsized":"2024-06-27T21:33:50.976224+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:20.906343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":129,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.970616+0000","last_change":"2024-06-27T21:33:02.266680+0000","last_active":"2024-06-27T21:33:50.970616+0000","last_peered":"2024-06-27T21:33:50.970616+0000","last_clean":"2024-06-27T21:33:50.970616+0000","last_became_active":"2024-06-27T21:33:01.710059+0000","last_became_peered":"2024-06-27T21:33:01.710059+0000","last_unstale":"2024-06-27T21:33:50.970616+0000","last_undegraded":"2024-06-27T21:33:50.970616+0000","last_fullsized":"2024-06-27T21:33:50.970616+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:14.470683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":85,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.983135+0000","last_change":"2024-06-27T21:33:50.003594+0000","last_active":"2024-06-27T21:33:50.983135+0000","last_peered":"2024-06-27T21:33:50.983135+0000","last_clean":"2024-06-27T21:33:50.983135+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:33:50.983135+0000","last_undegraded":"2024-06-27T21:33:50.983135+0000","last_fullsized":"2024-06-27T21:33:50.983135+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:50.003522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.003522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:05.414708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":505,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.980864+0000","last_change":"2024-06-27T21:33:02.184848+0000","last_active":"2024-06-27T21:33:50.980864+0000","last_peered":"2024-06-27T21:33:50.980864+0000","last_clean":"2024-06-27T21:33:50.980864+0000","last_became_active":"2024-06-27T21:33:01.774416+0000","last_became_peered":"2024-06-27T21:33:01.774416+0000","last_unstale":"2024-06-27T21:33:50.980864+0000","last_undegraded":"2024-06-27T21:33:50.980864+0000","last_fullsized":"2024-06-27T21:33:50.980864+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.184716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.184716+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:13.944531+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"559'9935","reported_seq":26962,"reported_epoch":559,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:33:56.557402+0000","last_change":"2024-06-27T21:33:29.146762+0000","last_active":"2024-06-27T21:33:56.557402+0000","last_peered":"2024-06-27T21:33:56.557402+0000","last_clean":"2024-06-27T21:33:26.559592+0000","last_became_active":"2024-06-27T21:33:29.111994+0000","last_became_peered":"2024-06-27T21:33:29.111994+0000","last_unstale":"2024-06-27T21:33:56.557402+0000","last_undegraded":"2024-06-27T21:33:56.557402+0000","last_fullsized":"2024-06-27T21:33:28.693330+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'5888","last_scrub_stamp":"2024-06-27T21:30:45.481539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.481539+0000","objects_scrubbed":5828,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1590,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111804416,"num_objects":1706,"num_object_clones":0,"num_object_copies":5118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1366,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5009,"num_write_kb":214880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4463,"num_bytes_recovered":245989376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1],"avail_no_missing":["0","2","10"],"object_location_counts":[{"shards":"0,1","objects":1023},{"shards":"0,2,10","objects":683}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":541,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.983350+0000","last_change":"2024-06-27T21:33:03.234800+0000","last_active":"2024-06-27T21:33:50.983350+0000","last_peered":"2024-06-27T21:33:50.983350+0000","last_clean":"2024-06-27T21:33:50.983350+0000","last_became_active":"2024-06-27T21:33:01.779398+0000","last_became_peered":"2024-06-27T21:33:01.779398+0000","last_unstale":"2024-06-27T21:33:50.983350+0000","last_undegraded":"2024-06-27T21:33:50.983350+0000","last_fullsized":"2024-06-27T21:33:50.983350+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:33:03.234726+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:25.309823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.11","version":"559'9616","reported_seq":23093,"reported_epoch":559,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:55.414400+0000","last_change":"2024-06-27T21:33:27.873461+0000","last_active":"2024-06-27T21:33:55.414400+0000","last_peered":"2024-06-27T21:33:55.414400+0000","last_clean":"2024-06-27T21:32:12.748576+0000","last_became_active":"2024-06-27T21:33:27.840847+0000","last_became_peered":"2024-06-27T21:33:27.840847+0000","last_unstale":"2024-06-27T21:33:55.414400+0000","last_undegraded":"2024-06-27T21:33:27.783158+0000","last_fullsized":"2024-06-27T21:33:27.691166+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"500'5961","last_scrub_stamp":"2024-06-27T21:30:49.972634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:49.972634+0000","objects_scrubbed":5918,"log_size":9616,"log_dups_size":0,"ondisk_log_size":9616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1550,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":1122,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4861,"num_write_kb":209056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3600,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[9,10],"avail_no_missing":["9","8","10"],"object_location_counts":[{"shards":"7,8,9,10","objects":559},{"shards":"9,10","objects":1109}],"blocked_by":[],"up_primary":0,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":61,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.970355+0000","last_change":"2024-06-27T21:33:02.510081+0000","last_active":"2024-06-27T21:33:50.970355+0000","last_peered":"2024-06-27T21:33:50.970355+0000","last_clean":"2024-06-27T21:33:50.970355+0000","last_became_active":"2024-06-27T21:33:01.775355+0000","last_became_peered":"2024-06-27T21:33:01.775355+0000","last_unstale":"2024-06-27T21:33:50.970355+0000","last_undegraded":"2024-06-27T21:33:50.970355+0000","last_fullsized":"2024-06-27T21:33:50.970355+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.509969+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.509969+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:03.505318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"559'9748","reported_seq":10067,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:59.435430+0000","last_change":"2024-06-27T21:33:51.180577+0000","last_active":"2024-06-27T21:33:59.435430+0000","last_peered":"2024-06-27T21:33:59.435430+0000","last_clean":"2024-06-27T21:33:59.435430+0000","last_became_active":"2024-06-27T21:33:27.698780+0000","last_became_peered":"2024-06-27T21:33:27.698780+0000","last_unstale":"2024-06-27T21:33:59.435430+0000","last_undegraded":"2024-06-27T21:33:59.435430+0000","last_fullsized":"2024-06-27T21:33:59.435430+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'9712","last_scrub_stamp":"2024-06-27T21:33:51.180247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.180247+0000","objects_scrubbed":1680,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:57.699175+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107741184,"num_objects":1644,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2],"acting":[11,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":61,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.973989+0000","last_change":"2024-06-27T21:33:02.531114+0000","last_active":"2024-06-27T21:33:50.973989+0000","last_peered":"2024-06-27T21:33:50.973989+0000","last_clean":"2024-06-27T21:33:50.973989+0000","last_became_active":"2024-06-27T21:33:01.810521+0000","last_became_peered":"2024-06-27T21:33:01.810521+0000","last_unstale":"2024-06-27T21:33:50.973989+0000","last_undegraded":"2024-06-27T21:33:50.973989+0000","last_fullsized":"2024-06-27T21:33:50.973989+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:17.850811+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"559'10098","reported_seq":20345,"reported_epoch":559,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:33:59.797675+0000","last_change":"2024-06-27T21:33:29.113997+0000","last_active":"2024-06-27T21:33:59.797675+0000","last_peered":"2024-06-27T21:33:59.797675+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:33:29.069929+0000","last_became_peered":"2024-06-27T21:33:29.069929+0000","last_unstale":"2024-06-27T21:33:59.797675+0000","last_undegraded":"2024-06-27T21:33:59.797675+0000","last_fullsized":"2024-06-27T21:33:28.692763+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112001024,"num_objects":1709,"num_object_clones":0,"num_object_copies":5127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":322,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":218080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2923,"num_bytes_recovered":191496192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,1],"avail_no_missing":["11","6","7"],"object_location_counts":[{"shards":"1,11","objects":1548},{"shards":"6,7,11","objects":161}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":192,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.981226+0000","last_change":"2024-06-27T21:31:19.010738+0000","last_active":"2024-06-27T21:33:50.981226+0000","last_peered":"2024-06-27T21:33:50.981226+0000","last_clean":"2024-06-27T21:33:50.981226+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:33:50.981226+0000","last_undegraded":"2024-06-27T21:33:50.981226+0000","last_fullsized":"2024-06-27T21:33:50.981226+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.010614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.010614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"559'10227","reported_seq":10511,"reported_epoch":559,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-06-27T21:33:58.995450+0000","last_change":"2024-06-27T21:33:58.995450+0000","last_active":"2024-06-27T21:33:58.995450+0000","last_peered":"2024-06-27T21:33:58.995450+0000","last_clean":"2024-06-27T21:33:25.735988+0000","last_became_active":"2024-06-27T21:33:28.958742+0000","last_became_peered":"2024-06-27T21:33:28.958742+0000","last_unstale":"2024-06-27T21:33:58.995450+0000","last_undegraded":"2024-06-27T21:33:58.995450+0000","last_fullsized":"2024-06-27T21:33:58.995450+0000","mapping_epoch":550,"log_start":"470'100","ondisk_log_start":"470'100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":10127,"log_dups_size":0,"ondisk_log_size":10127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112885760,"num_objects":1723,"num_object_clones":0,"num_object_copies":5169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5169,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":220704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[1,5,9],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5,9","objects":1723}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":192,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.976151+0000","last_change":"2024-06-27T21:31:19.025719+0000","last_active":"2024-06-27T21:33:50.976151+0000","last_peered":"2024-06-27T21:33:50.976151+0000","last_clean":"2024-06-27T21:33:50.976151+0000","last_became_active":"2024-06-27T21:29:51.673661+0000","last_became_peered":"2024-06-27T21:29:51.673661+0000","last_unstale":"2024-06-27T21:33:50.976151+0000","last_undegraded":"2024-06-27T21:33:50.976151+0000","last_fullsized":"2024-06-27T21:33:50.976151+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.15","version":"559'9831","reported_seq":16831,"reported_epoch":559,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:57.389797+0000","last_change":"2024-06-27T21:33:28.935649+0000","last_active":"2024-06-27T21:33:57.389797+0000","last_peered":"2024-06-27T21:33:57.389797+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:28.917741+0000","last_became_peered":"2024-06-27T21:33:28.917741+0000","last_unstale":"2024-06-27T21:33:57.389797+0000","last_undegraded":"2024-06-27T21:33:28.805628+0000","last_fullsized":"2024-06-27T21:33:28.693311+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":9831,"log_dups_size":0,"ondisk_log_size":9831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108429312,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1721,"num_bytes_recovered":112721920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,10],"acting":[7,4],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,6,7,9","objects":607},{"shards":"4,7","objects":48},{"shards":"6,7,9","objects":1000}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":200,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:51.049087+0000","last_change":"2024-06-27T21:32:45.802297+0000","last_active":"2024-06-27T21:33:51.049087+0000","last_peered":"2024-06-27T21:33:51.049087+0000","last_clean":"2024-06-27T21:33:51.049087+0000","last_became_active":"2024-06-27T21:29:51.676689+0000","last_became_peered":"2024-06-27T21:29:51.676689+0000","last_unstale":"2024-06-27T21:33:51.049087+0000","last_undegraded":"2024-06-27T21:33:51.049087+0000","last_fullsized":"2024-06-27T21:33:51.049087+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:45.802189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:32:45.802189+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:00.659317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"559'9907","reported_seq":29676,"reported_epoch":559,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:58.207793+0000","last_change":"2024-06-27T21:33:29.009115+0000","last_active":"2024-06-27T21:33:58.207793+0000","last_peered":"2024-06-27T21:33:58.207793+0000","last_clean":"2024-06-27T21:33:25.724063+0000","last_became_active":"2024-06-27T21:33:28.945463+0000","last_became_peered":"2024-06-27T21:33:28.945463+0000","last_unstale":"2024-06-27T21:33:58.207793+0000","last_undegraded":"2024-06-27T21:33:28.715920+0000","last_fullsized":"2024-06-27T21:33:28.692016+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'5932","last_scrub_stamp":"2024-06-27T21:30:46.556700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:46.556700+0000","objects_scrubbed":5888,"log_size":9907,"log_dups_size":0,"ondisk_log_size":9907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112754688,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5003,"num_write_kb":215136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6292,"num_bytes_recovered":295796736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[4,7],"avail_no_missing":["4","6","7","11"],"object_location_counts":[{"shards":"4,7","objects":1719}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":200,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.977523+0000","last_change":"2024-06-27T21:32:13.808607+0000","last_active":"2024-06-27T21:33:50.977523+0000","last_peered":"2024-06-27T21:33:50.977523+0000","last_clean":"2024-06-27T21:33:50.977523+0000","last_became_active":"2024-06-27T21:32:13.808038+0000","last_became_peered":"2024-06-27T21:32:13.808038+0000","last_unstale":"2024-06-27T21:33:50.977523+0000","last_undegraded":"2024-06-27T21:33:50.977523+0000","last_fullsized":"2024-06-27T21:33:50.977523+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1],"acting":[0,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"559'9793","reported_seq":10521,"reported_epoch":559,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:57.388443+0000","last_change":"2024-06-27T21:33:29.113557+0000","last_active":"2024-06-27T21:33:57.388443+0000","last_peered":"2024-06-27T21:33:57.388443+0000","last_clean":"2024-06-27T21:33:26.047542+0000","last_became_active":"2024-06-27T21:33:29.082411+0000","last_became_peered":"2024-06-27T21:33:29.082411+0000","last_unstale":"2024-06-27T21:33:57.388443+0000","last_undegraded":"2024-06-27T21:33:28.958717+0000","last_fullsized":"2024-06-27T21:33:28.692997+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"527'9415","last_scrub_stamp":"2024-06-27T21:32:25.997194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:25.997194+0000","objects_scrubbed":3847,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110166016,"num_objects":1681,"num_object_clones":0,"num_object_copies":5043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6472,"num_objects_misplaced":3236,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4950,"num_write_kb":212192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":6684672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[3,2],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"2,3","objects":17},{"shards":"3,4,5","objects":1664}],"blocked_by":[],"up_primary":10,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":221,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.973733+0000","last_change":"2024-06-27T21:32:13.808094+0000","last_active":"2024-06-27T21:33:50.973733+0000","last_peered":"2024-06-27T21:33:50.973733+0000","last_clean":"2024-06-27T21:33:50.973733+0000","last_became_active":"2024-06-27T21:32:13.807596+0000","last_became_peered":"2024-06-27T21:32:13.807596+0000","last_unstale":"2024-06-27T21:33:50.973733+0000","last_undegraded":"2024-06-27T21:33:50.973733+0000","last_fullsized":"2024-06-27T21:33:50.973733+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":495,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.981010+0000","last_change":"2024-06-27T21:33:00.167722+0000","last_active":"2024-06-27T21:33:50.981010+0000","last_peered":"2024-06-27T21:33:50.981010+0000","last_clean":"2024-06-27T21:33:50.981010+0000","last_became_active":"2024-06-27T21:32:59.709162+0000","last_became_peered":"2024-06-27T21:32:59.709162+0000","last_unstale":"2024-06-27T21:33:50.981010+0000","last_undegraded":"2024-06-27T21:33:50.981010+0000","last_fullsized":"2024-06-27T21:33:50.981010+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.167617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.167617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:28.293116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"559'10105","reported_seq":10377,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:58.341825+0000","last_change":"2024-06-27T21:30:52.123816+0000","last_active":"2024-06-27T21:33:58.341825+0000","last_peered":"2024-06-27T21:33:58.341825+0000","last_clean":"2024-06-27T21:33:58.341825+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:33:58.341825+0000","last_undegraded":"2024-06-27T21:33:58.341825+0000","last_fullsized":"2024-06-27T21:33:58.341825+0000","mapping_epoch":467,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6360","last_scrub_stamp":"2024-06-27T21:30:52.123515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.123515+0000","objects_scrubbed":6329,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227213312,"num_objects":3467,"num_object_clones":0,"num_object_copies":10401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10105,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":282,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.973799+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:33:50.973799+0000","last_peered":"2024-06-27T21:33:50.973799+0000","last_clean":"2024-06-27T21:33:50.973799+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:33:50.973799+0000","last_undegraded":"2024-06-27T21:33:50.973799+0000","last_fullsized":"2024-06-27T21:33:50.973799+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:27.450417+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"559'10134","reported_seq":10403,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:57.529983+0000","last_change":"2024-06-27T21:30:48.343185+0000","last_active":"2024-06-27T21:33:57.529983+0000","last_peered":"2024-06-27T21:33:57.529983+0000","last_clean":"2024-06-27T21:33:57.529983+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:33:57.529983+0000","last_undegraded":"2024-06-27T21:33:57.529983+0000","last_fullsized":"2024-06-27T21:33:57.529983+0000","mapping_epoch":467,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6100","last_scrub_stamp":"2024-06-27T21:30:48.342993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.342993+0000","objects_scrubbed":6058,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224264192,"num_objects":3422,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":125,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.970807+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:33:50.970807+0000","last_peered":"2024-06-27T21:33:50.970807+0000","last_clean":"2024-06-27T21:33:50.970807+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:33:50.970807+0000","last_undegraded":"2024-06-27T21:33:50.970807+0000","last_fullsized":"2024-06-27T21:33:50.970807+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:18.634183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"559'10050","reported_seq":23838,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:59.444733+0000","last_change":"2024-06-27T21:33:28.733557+0000","last_active":"2024-06-27T21:33:59.444733+0000","last_peered":"2024-06-27T21:33:59.444733+0000","last_clean":"2024-06-27T21:33:59.444733+0000","last_became_active":"2024-06-27T21:33:28.698375+0000","last_became_peered":"2024-06-27T21:33:28.698375+0000","last_unstale":"2024-06-27T21:33:59.444733+0000","last_undegraded":"2024-06-27T21:33:59.444733+0000","last_fullsized":"2024-06-27T21:33:59.444733+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226295832,"num_objects":3454,"num_object_clones":0,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10050,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6752,"num_bytes_recovered":333316120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1043,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.975210+0000","last_change":"2024-06-27T21:32:13.809592+0000","last_active":"2024-06-27T21:33:50.975210+0000","last_peered":"2024-06-27T21:33:50.975210+0000","last_clean":"2024-06-27T21:33:50.975210+0000","last_became_active":"2024-06-27T21:32:13.809197+0000","last_became_peered":"2024-06-27T21:32:13.809197+0000","last_unstale":"2024-06-27T21:33:50.975210+0000","last_undegraded":"2024-06-27T21:33:50.975210+0000","last_fullsized":"2024-06-27T21:33:50.975210+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"559'10083","reported_seq":15684,"reported_epoch":559,"state":"active+recovering","last_fresh":"2024-06-27T21:33:57.794661+0000","last_change":"2024-06-27T21:33:26.890778+0000","last_active":"2024-06-27T21:33:57.794661+0000","last_peered":"2024-06-27T21:33:57.794661+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:33:26.864503+0000","last_became_peered":"2024-06-27T21:33:26.864503+0000","last_unstale":"2024-06-27T21:33:57.794661+0000","last_undegraded":"2024-06-27T21:33:57.794661+0000","last_fullsized":"2024-06-27T21:33:57.794661+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1152,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":218080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1449,"num_bytes_recovered":94961664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"0,7,11","objects":572},{"shards":"6,7,11","objects":1152}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":620,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.983271+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:33:50.983271+0000","last_peered":"2024-06-27T21:33:50.983271+0000","last_clean":"2024-06-27T21:33:50.983271+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:33:50.983271+0000","last_undegraded":"2024-06-27T21:33:50.983271+0000","last_fullsized":"2024-06-27T21:33:50.983271+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:03.727167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"559'9760","reported_seq":10083,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:57.785464+0000","last_change":"2024-06-27T21:33:54.377162+0000","last_active":"2024-06-27T21:33:57.785464+0000","last_peered":"2024-06-27T21:33:57.785464+0000","last_clean":"2024-06-27T21:33:57.785464+0000","last_became_active":"2024-06-27T21:33:25.695260+0000","last_became_peered":"2024-06-27T21:33:25.695260+0000","last_unstale":"2024-06-27T21:33:57.785464+0000","last_undegraded":"2024-06-27T21:33:57.785464+0000","last_fullsized":"2024-06-27T21:33:57.785464+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9736","last_scrub_stamp":"2024-06-27T21:33:54.376905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.376905+0000","objects_scrubbed":1765,"log_size":9760,"log_dups_size":0,"ondisk_log_size":9760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:55.143355+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113901568,"num_objects":1738,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":615,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.983262+0000","last_change":"2024-06-27T21:33:01.182699+0000","last_active":"2024-06-27T21:33:50.983262+0000","last_peered":"2024-06-27T21:33:50.983262+0000","last_clean":"2024-06-27T21:33:50.983262+0000","last_became_active":"2024-06-27T21:32:59.711777+0000","last_became_peered":"2024-06-27T21:32:59.711777+0000","last_unstale":"2024-06-27T21:33:50.983262+0000","last_undegraded":"2024-06-27T21:33:50.983262+0000","last_fullsized":"2024-06-27T21:33:50.983262+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:12.867934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"559'9610","reported_seq":21150,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:55.044006+0000","last_change":"2024-06-27T21:33:48.555029+0000","last_active":"2024-06-27T21:33:55.044006+0000","last_peered":"2024-06-27T21:33:55.044006+0000","last_clean":"2024-06-27T21:33:55.044006+0000","last_became_active":"2024-06-27T21:33:35.694586+0000","last_became_peered":"2024-06-27T21:33:35.694586+0000","last_unstale":"2024-06-27T21:33:55.044006+0000","last_undegraded":"2024-06-27T21:33:55.044006+0000","last_fullsized":"2024-06-27T21:33:55.044006+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'9582","last_scrub_stamp":"2024-06-27T21:33:48.554929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:48.554929+0000","objects_scrubbed":1728,"log_size":9610,"log_dups_size":0,"ondisk_log_size":9610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:03.341335+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,7],"acting":[9,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":536,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.983416+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:33:50.983416+0000","last_peered":"2024-06-27T21:33:50.983416+0000","last_clean":"2024-06-27T21:33:50.983416+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:33:50.983416+0000","last_undegraded":"2024-06-27T21:33:50.983416+0000","last_fullsized":"2024-06-27T21:33:50.983416+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:17.860808+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"559'9937","reported_seq":23688,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:55.706397+0000","last_change":"2024-06-27T21:33:54.883116+0000","last_active":"2024-06-27T21:33:55.706397+0000","last_peered":"2024-06-27T21:33:55.706397+0000","last_clean":"2024-06-27T21:33:55.706397+0000","last_became_active":"2024-06-27T21:33:25.693582+0000","last_became_peered":"2024-06-27T21:33:25.693582+0000","last_unstale":"2024-06-27T21:33:55.706397+0000","last_undegraded":"2024-06-27T21:33:55.706397+0000","last_fullsized":"2024-06-27T21:33:55.706397+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":1712,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:18.128885+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":614,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.983260+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:33:50.983260+0000","last_peered":"2024-06-27T21:33:50.983260+0000","last_clean":"2024-06-27T21:33:50.983260+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:33:50.983260+0000","last_undegraded":"2024-06-27T21:33:50.983260+0000","last_fullsized":"2024-06-27T21:33:50.983260+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"559'9915","reported_seq":24037,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:57.457068+0000","last_change":"2024-06-27T21:33:50.968993+0000","last_active":"2024-06-27T21:33:57.457068+0000","last_peered":"2024-06-27T21:33:57.457068+0000","last_clean":"2024-06-27T21:33:57.457068+0000","last_became_active":"2024-06-27T21:33:25.692464+0000","last_became_peered":"2024-06-27T21:33:25.692464+0000","last_unstale":"2024-06-27T21:33:57.457068+0000","last_undegraded":"2024-06-27T21:33:57.457068+0000","last_fullsized":"2024-06-27T21:33:57.457068+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'9882","last_scrub_stamp":"2024-06-27T21:33:50.968852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.968852+0000","objects_scrubbed":1716,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:17.284729+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110297088,"num_objects":1683,"num_object_clones":0,"num_object_copies":5049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":289,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.973857+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:33:50.973857+0000","last_peered":"2024-06-27T21:33:50.973857+0000","last_clean":"2024-06-27T21:33:50.973857+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:33:50.973857+0000","last_undegraded":"2024-06-27T21:33:50.973857+0000","last_fullsized":"2024-06-27T21:33:50.973857+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"559'9845","reported_seq":17078,"reported_epoch":559,"state":"active+recovering","last_fresh":"2024-06-27T21:33:57.273793+0000","last_change":"2024-06-27T21:33:26.778860+0000","last_active":"2024-06-27T21:33:57.273793+0000","last_peered":"2024-06-27T21:33:57.273793+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:26.702107+0000","last_became_peered":"2024-06-27T21:33:26.702107+0000","last_unstale":"2024-06-27T21:33:57.273793+0000","last_undegraded":"2024-06-27T21:33:57.273793+0000","last_fullsized":"2024-06-27T21:33:57.273793+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":9845,"log_dups_size":0,"ondisk_log_size":9845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107511808,"num_objects":1640,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":764,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4977,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1886,"num_bytes_recovered":123535360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,7,9","objects":876},{"shards":"6,7,9","objects":764}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":173,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.970707+0000","last_change":"2024-06-27T21:32:13.812610+0000","last_active":"2024-06-27T21:33:50.970707+0000","last_peered":"2024-06-27T21:33:50.970707+0000","last_clean":"2024-06-27T21:33:50.970707+0000","last_became_active":"2024-06-27T21:32:13.812217+0000","last_became_peered":"2024-06-27T21:32:13.812217+0000","last_unstale":"2024-06-27T21:33:50.970707+0000","last_undegraded":"2024-06-27T21:33:50.970707+0000","last_fullsized":"2024-06-27T21:33:50.970707+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,10],"acting":[4,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"559'10212","reported_seq":10529,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:58.962826+0000","last_change":"2024-06-27T21:33:50.128755+0000","last_active":"2024-06-27T21:33:58.962826+0000","last_peered":"2024-06-27T21:33:58.962826+0000","last_clean":"2024-06-27T21:33:58.962826+0000","last_became_active":"2024-06-27T21:33:25.695186+0000","last_became_peered":"2024-06-27T21:33:25.695186+0000","last_unstale":"2024-06-27T21:33:58.962826+0000","last_undegraded":"2024-06-27T21:33:58.962826+0000","last_fullsized":"2024-06-27T21:33:58.962826+0000","mapping_epoch":467,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'10169","last_scrub_stamp":"2024-06-27T21:33:50.128579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.128579+0000","objects_scrubbed":1771,"log_size":10112,"log_dups_size":0,"ondisk_log_size":10112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:56.697028+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":286,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.973866+0000","last_change":"2024-06-27T21:29:56.956056+0000","last_active":"2024-06-27T21:33:50.973866+0000","last_peered":"2024-06-27T21:33:50.973866+0000","last_clean":"2024-06-27T21:33:50.973866+0000","last_became_active":"2024-06-27T21:29:49.671863+0000","last_became_peered":"2024-06-27T21:29:49.671863+0000","last_unstale":"2024-06-27T21:33:50.973866+0000","last_undegraded":"2024-06-27T21:33:50.973866+0000","last_fullsized":"2024-06-27T21:33:50.973866+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:29:56.955983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:29:56.955983+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"559'9789","reported_seq":10149,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:57.523803+0000","last_change":"2024-06-27T21:33:49.922777+0000","last_active":"2024-06-27T21:33:57.523803+0000","last_peered":"2024-06-27T21:33:57.523803+0000","last_clean":"2024-06-27T21:33:57.523803+0000","last_became_active":"2024-06-27T21:33:25.697304+0000","last_became_peered":"2024-06-27T21:33:25.697304+0000","last_unstale":"2024-06-27T21:33:57.523803+0000","last_undegraded":"2024-06-27T21:33:57.523803+0000","last_fullsized":"2024-06-27T21:33:57.523803+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9759","last_scrub_stamp":"2024-06-27T21:33:49.922636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.922636+0000","objects_scrubbed":1732,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:10.096662+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542272,"num_objects":1702,"num_object_clones":0,"num_object_copies":5106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":809,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:51.930912+0000","last_change":"2024-06-27T21:33:51.930934+0000","last_active":"2024-06-27T21:33:51.930912+0000","last_peered":"2024-06-27T21:33:51.930912+0000","last_clean":"2024-06-27T21:33:51.930912+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:33:51.930912+0000","last_undegraded":"2024-06-27T21:33:51.930912+0000","last_fullsized":"2024-06-27T21:33:51.930912+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:51.930851+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.930851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:00.975914+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"559'9988","reported_seq":10303,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:57.442138+0000","last_change":"2024-06-27T21:33:49.851182+0000","last_active":"2024-06-27T21:33:57.442138+0000","last_peered":"2024-06-27T21:33:57.442138+0000","last_clean":"2024-06-27T21:33:57.442138+0000","last_became_active":"2024-06-27T21:33:25.736643+0000","last_became_peered":"2024-06-27T21:33:25.736643+0000","last_unstale":"2024-06-27T21:33:57.442138+0000","last_undegraded":"2024-06-27T21:33:57.442138+0000","last_fullsized":"2024-06-27T21:33:57.442138+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9962","last_scrub_stamp":"2024-06-27T21:33:49.850981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.850981+0000","objects_scrubbed":1778,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:08.929859+0000","scrub_duration":516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819072,"num_objects":1752,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":205,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.970976+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:33:50.970976+0000","last_peered":"2024-06-27T21:33:50.970976+0000","last_clean":"2024-06-27T21:33:50.970976+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:33:50.970976+0000","last_undegraded":"2024-06-27T21:33:50.970976+0000","last_fullsized":"2024-06-27T21:33:50.970976+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:19.543339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"559'9838","reported_seq":16091,"reported_epoch":559,"state":"active+recovering","last_fresh":"2024-06-27T21:33:56.584080+0000","last_change":"2024-06-27T21:33:26.772423+0000","last_active":"2024-06-27T21:33:56.584080+0000","last_peered":"2024-06-27T21:33:56.584080+0000","last_clean":"2024-06-27T21:32:12.787863+0000","last_became_active":"2024-06-27T21:33:26.705196+0000","last_became_peered":"2024-06-27T21:33:26.705196+0000","last_unstale":"2024-06-27T21:33:56.584080+0000","last_undegraded":"2024-06-27T21:33:56.584080+0000","last_fullsized":"2024-06-27T21:33:56.584080+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6309","last_scrub_stamp":"2024-06-27T21:30:52.261532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.261532+0000","objects_scrubbed":6289,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116228096,"num_objects":1773,"num_object_clones":0,"num_object_copies":5319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4973,"num_write_kb":215904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2930,"num_bytes_recovered":125960192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":["0","8","9"],"object_location_counts":[{"shards":"0,2,9","objects":1773}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":197,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.974083+0000","last_change":"2024-06-27T21:33:01.399273+0000","last_active":"2024-06-27T21:33:50.974083+0000","last_peered":"2024-06-27T21:33:50.974083+0000","last_clean":"2024-06-27T21:33:50.974083+0000","last_became_active":"2024-06-27T21:32:59.709705+0000","last_became_peered":"2024-06-27T21:32:59.709705+0000","last_unstale":"2024-06-27T21:33:50.974083+0000","last_undegraded":"2024-06-27T21:33:50.974083+0000","last_fullsized":"2024-06-27T21:33:50.974083+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:23.108874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"559'9976","reported_seq":23657,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:56.538174+0000","last_change":"2024-06-27T21:33:23.873709+0000","last_active":"2024-06-27T21:33:56.538174+0000","last_peered":"2024-06-27T21:33:56.538174+0000","last_clean":"2024-06-27T21:33:56.538174+0000","last_became_active":"2024-06-27T21:33:23.851489+0000","last_became_peered":"2024-06-27T21:33:23.851489+0000","last_unstale":"2024-06-27T21:33:56.538174+0000","last_undegraded":"2024-06-27T21:33:56.538174+0000","last_fullsized":"2024-06-27T21:33:56.538174+0000","mapping_epoch":545,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":9976,"log_dups_size":0,"ondisk_log_size":9976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224264192,"num_objects":3422,"num_object_clones":0,"num_object_copies":10266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6699,"num_bytes_recovered":340197376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"559'9867","reported_seq":10178,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:57.416458+0000","last_change":"2024-06-27T21:32:26.025196+0000","last_active":"2024-06-27T21:33:57.416458+0000","last_peered":"2024-06-27T21:33:57.416458+0000","last_clean":"2024-06-27T21:33:57.416458+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:33:57.416458+0000","last_undegraded":"2024-06-27T21:33:57.416458+0000","last_fullsized":"2024-06-27T21:33:57.416458+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'9389","last_scrub_stamp":"2024-06-27T21:32:26.024974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:26.024974+0000","objects_scrubbed":3795,"log_size":9867,"log_dups_size":0,"ondisk_log_size":9867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9867,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":213,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.976205+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:33:50.976205+0000","last_peered":"2024-06-27T21:33:50.976205+0000","last_clean":"2024-06-27T21:33:50.976205+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:33:50.976205+0000","last_undegraded":"2024-06-27T21:33:50.976205+0000","last_fullsized":"2024-06-27T21:33:50.976205+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:11.332879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"559'10291","reported_seq":16919,"reported_epoch":559,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:33:57.852179+0000","last_change":"2024-06-27T21:32:36.645550+0000","last_active":"2024-06-27T21:33:57.852179+0000","last_peered":"2024-06-27T21:33:57.852179+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:32:34.904596+0000","last_became_peered":"2024-06-27T21:32:34.904596+0000","last_unstale":"2024-06-27T21:33:57.852179+0000","last_undegraded":"2024-06-27T21:32:22.691354+0000","last_fullsized":"2024-06-27T21:32:22.682941+0000","mapping_epoch":526,"log_start":"471'200","ondisk_log_start":"471'200","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231538688,"num_objects":3533,"num_object_clones":0,"num_object_copies":10599,"num_objects_missing_on_primary":4895,"num_objects_missing":4895,"num_objects_degraded":3,"num_objects_misplaced":10582,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2003,"num_bytes_recovered":130940928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10],"avail_no_missing":["5","6","8"],"object_location_counts":[{"shards":"5,6,7,8","objects":14},{"shards":"5,6,8","objects":3518},{"shards":"7,10","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":211,"reported_epoch":559,"state":"active+clean","last_fresh":"2024-06-27T21:33:50.976122+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:33:50.976122+0000","last_peered":"2024-06-27T21:33:50.976122+0000","last_clean":"2024-06-27T21:33:50.976122+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:33:50.976122+0000","last_undegraded":"2024-06-27T21:33:50.976122+0000","last_fullsized":"2024-06-27T21:33:50.976122+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:22.883827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":3588816920,"num_objects":54763,"num_object_clones":0,"num_object_copies":164289,"num_objects_missing_on_primary":4895,"num_objects_missing":4895,"num_objects_degraded":22707,"num_objects_misplaced":30372,"num_objects_unfound":0,"num_objects_dirty":54763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120788,"num_write_kb":5173281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40940,"num_bytes_recovered":2213937176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12815577088,"data_stored":12815564872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":257781,"ondisk_log_size":257781,"up":78,"acting":69,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253161,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549304,"kb_used_data":972528,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92195848,"statfs":{"total":95995035648,"available":94408548352,"internally_reserved":0,"allocated":995868672,"data_stored":995491619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514797,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1334140,"kb_used_data":879540,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92411012,"statfs":{"total":95995035648,"available":94628876288,"internally_reserved":0,"allocated":900648960,"data_stored":900267835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":465500608},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743729,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573880,"kb_used_data":1115120,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":92171272,"statfs":{"total":95995035648,"available":94383382528,"internally_reserved":0,"allocated":1141882880,"data_stored":1141505827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":469760448},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":266287972661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266999,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1826112,"kb_used_data":1313208,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":91919040,"statfs":{"total":95995035648,"available":94125096960,"internally_reserved":0,"allocated":1344724992,"data_stored":1344344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":525203914},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2046456,"kb_used_data":1550192,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91698696,"statfs":{"total":95995035648,"available":93899464704,"internally_reserved":0,"allocated":1587396608,"data_stored":1587019555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757573,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180280,"kb_used_data":788144,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92564872,"statfs":{"total":95995035648,"available":94786428928,"internally_reserved":0,"allocated":807059456,"data_stored":806682403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161278,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106552,"kb_used_data":878960,"kb_used_omap":92,"kb_used_meta":227491,"kb_avail":92638600,"statfs":{"total":95995035648,"available":94861926400,"internally_reserved":0,"allocated":900055040,"data_stored":899677987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":232950882},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182733,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1097208,"kb_used_data":714800,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92647944,"statfs":{"total":95995035648,"available":94871494656,"internally_reserved":0,"allocated":731955200,"data_stored":731578147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":391576007},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411666,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1807296,"kb_used_data":1415160,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91937856,"statfs":{"total":95995035648,"available":94144364544,"internally_reserved":0,"allocated":1449123840,"data_stored":1448743235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575191,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576964,"kb_used_data":1040444,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92168188,"statfs":{"total":95995035648,"available":94380224512,"internally_reserved":0,"allocated":1065414656,"data_stored":1065029979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738715,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1694780,"kb_used_data":1136052,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92050372,"statfs":{"total":95995035648,"available":94259580928,"internally_reserved":0,"allocated":1163317248,"data_stored":1162936123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1159794688,"data_stored":1159790616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1065488384,"data_stored":1065484312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1448280064,"data_stored":1448280064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":730988544,"data_stored":730988544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":898695168,"data_stored":898695168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":805961728,"data_stored":805961728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1585643520,"data_stored":1585643520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1342177280,"data_stored":1342177280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":740950016,"data_stored":740950016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1140195328,"data_stored":1140195328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":899289088,"data_stored":899285016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":998113280,"data_stored":998113280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:34:03.266 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [1, 9, 10] 2024-06-27T21:34:03.266 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '1', '9', '10'] 2024-06-27T21:34:03.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap 1.0 1 9 10 2024-06-27T21:34:03.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:03.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:03.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:34:03.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:03.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:03.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:03.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:03.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:03.715 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:34:03.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:03.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:03 smithi151 ceph-mon[28965]: pgmap v1457: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22679/163941 objects degraded (13.834%); 29362/163941 objects misplaced (17.910%) 2024-06-27T21:34:03.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:03 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:34:03.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:03 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:34:03.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3560836288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:34:03.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:34:03.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:03.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:03.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:03.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:03.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:04.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:03 smithi190 ceph-mon[32247]: pgmap v1457: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22679/163941 objects degraded (13.834%); 29362/163941 objects misplaced (17.910%) 2024-06-27T21:34:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:03 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:34:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:03 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:34:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3560836288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:34:04.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:34:04.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:03 smithi192 ceph-mon[32956]: pgmap v1457: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 22679/163941 objects degraded (13.834%); 29362/163941 objects misplaced (17.910%) 2024-06-27T21:34:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:03 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:34:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:03 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:34:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3560836288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:34:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:34:04.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:04.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:04.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:04.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:04.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:04.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:04.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:04.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:04.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:04.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:04.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:04.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:04.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:04.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:04.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:04.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:04 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:34:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:34:05.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:04 smithi190 ceph-mon[32247]: osdmap e560: 12 total, 12 up, 11 in 2024-06-27T21:34:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 22679/163941 objects degraded (13.834%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:05.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:04 smithi190 ceph-mon[32247]: osdmap e561: 12 total, 12 up, 11 in 2024-06-27T21:34:05.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:04 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:34:05.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:34:05.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:04 smithi151 ceph-mon[28965]: osdmap e560: 12 total, 12 up, 11 in 2024-06-27T21:34:05.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 22679/163941 objects degraded (13.834%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:05.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:04 smithi151 ceph-mon[28965]: osdmap e561: 12 total, 12 up, 11 in 2024-06-27T21:34:05.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:04 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:34:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:34:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:04 smithi192 ceph-mon[32956]: osdmap e560: 12 total, 12 up, 11 in 2024-06-27T21:34:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 22679/163941 objects degraded (13.834%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:05.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:04 smithi192 ceph-mon[32956]: osdmap e561: 12 total, 12 up, 11 in 2024-06-27T21:34:05.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:05.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:05.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:05.233 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:05.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:05.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:05.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:05.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:05.548 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:05.560 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:05.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:05.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:05.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:05.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:05.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:05.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:05.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:05.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:05.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:05.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:05 smithi151 ceph-mon[28965]: pgmap v1459: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16173/163005 objects degraded (9.922%); 30670/163005 objects misplaced (18.815%) 2024-06-27T21:34:05.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:05 smithi151 ceph-mon[28965]: osdmap e562: 12 total, 12 up, 11 in 2024-06-27T21:34:06.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:05 smithi190 ceph-mon[32247]: pgmap v1459: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16173/163005 objects degraded (9.922%); 30670/163005 objects misplaced (18.815%) 2024-06-27T21:34:06.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:05 smithi190 ceph-mon[32247]: osdmap e562: 12 total, 12 up, 11 in 2024-06-27T21:34:06.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:05 smithi192 ceph-mon[32956]: pgmap v1459: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16173/163005 objects degraded (9.922%); 30670/163005 objects misplaced (18.815%) 2024-06-27T21:34:06.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:05 smithi192 ceph-mon[32956]: osdmap e562: 12 total, 12 up, 11 in 2024-06-27T21:34:06.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:06.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:06.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:06.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:06.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:06.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:06.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:06.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:06.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:06.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:06.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:06.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:06.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:06.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:06.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:06.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:06 smithi151 ceph-mon[28965]: 6.16 scrub starts 2024-06-27T21:34:06.953 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:34:06.954 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:34:06.954 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:34:06.954 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:34:06.954 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:34:06.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:06.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:06.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:07.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:06 smithi190 ceph-mon[32247]: 6.16 scrub starts 2024-06-27T21:34:07.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:06 smithi192 ceph-mon[32956]: 6.16 scrub starts 2024-06-27T21:34:07.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:07.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:07.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:07.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:07.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:07.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:07.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:07.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:07.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:07.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:07.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:34:07.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:07.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:07.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:07.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:07.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:07.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:07 smithi151 ceph-mon[28965]: pgmap v1462: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16173/162543 objects degraded (9.950%); 29792/162543 objects misplaced (18.329%) 2024-06-27T21:34:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:07 smithi151 ceph-mon[28965]: 6.16 scrub ok 2024-06-27T21:34:07.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:07.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:07.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:08.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:07 smithi190 ceph-mon[32247]: pgmap v1462: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16173/162543 objects degraded (9.950%); 29792/162543 objects misplaced (18.329%) 2024-06-27T21:34:08.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:07 smithi190 ceph-mon[32247]: 6.16 scrub ok 2024-06-27T21:34:08.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:07 smithi192 ceph-mon[32956]: pgmap v1462: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 16173/162543 objects degraded (9.950%); 29792/162543 objects misplaced (18.329%) 2024-06-27T21:34:08.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:07 smithi192 ceph-mon[32956]: 6.16 scrub ok 2024-06-27T21:34:08.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:08.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:08.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:08.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:08.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:08.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:08.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:08.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:08.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:08.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:08.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:08.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:08.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:08.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:08.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:08.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:08.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:09.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:09.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:09.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:09.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:09.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:09.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:09.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:09.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:09.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:09.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:09.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:09.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:09.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:09.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:09.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:09.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:09.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:10.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:09 smithi190 ceph-mon[32247]: pgmap v1463: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 136 op/s; 16101/161481 objects degraded (9.971%); 28192/161481 objects misplaced (17.458%); 15 MiB/s, 401 objects/s recovering 2024-06-27T21:34:10.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 16101/161481 objects degraded (9.971%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:10.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:09 smithi190 ceph-mon[32247]: osdmap e563: 12 total, 12 up, 11 in 2024-06-27T21:34:10.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:09 smithi192 ceph-mon[32956]: pgmap v1463: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 136 op/s; 16101/161481 objects degraded (9.971%); 28192/161481 objects misplaced (17.458%); 15 MiB/s, 401 objects/s recovering 2024-06-27T21:34:10.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 16101/161481 objects degraded (9.971%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:10.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:09 smithi192 ceph-mon[32956]: osdmap e563: 12 total, 12 up, 11 in 2024-06-27T21:34:10.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:10.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:09 smithi151 ceph-mon[28965]: pgmap v1463: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 136 op/s; 16101/161481 objects degraded (9.971%); 28192/161481 objects misplaced (17.458%); 15 MiB/s, 401 objects/s recovering 2024-06-27T21:34:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 16101/161481 objects degraded (9.971%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:09 smithi151 ceph-mon[28965]: osdmap e563: 12 total, 12 up, 11 in 2024-06-27T21:34:10.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:10.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:10.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:10.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:10.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:10.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:10.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:10.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:10.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:10.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:10.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:10.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:10.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:10.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:10.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:11.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:11.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:11.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:11.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:11.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:11.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:11.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:11.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:11.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:11.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:11.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:11.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:11.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:11 smithi151 ceph-mon[28965]: pgmap v1465: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 111 op/s; 16101/160998 objects degraded (10.001%); 28141/160998 objects misplaced (17.479%); 13 MiB/s, 296 objects/s recovering 2024-06-27T21:34:11.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:11.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:11.969 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:11.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:11.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:12.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:11 smithi190 ceph-mon[32247]: pgmap v1465: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 111 op/s; 16101/160998 objects degraded (10.001%); 28141/160998 objects misplaced (17.479%); 13 MiB/s, 296 objects/s recovering 2024-06-27T21:34:12.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:11 smithi192 ceph-mon[32956]: pgmap v1465: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 45 active+clean; 3.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 111 op/s; 16101/160998 objects degraded (10.001%); 28141/160998 objects misplaced (17.479%); 13 MiB/s, 296 objects/s recovering 2024-06-27T21:34:12.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:12.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:12.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:12.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:12.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:12.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:12.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:12.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:12.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:12.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:12.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:12.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:12.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:12.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:12.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:12.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:12.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:13.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:13.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:13.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:13.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:13.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:13.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:13.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:13.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:13.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:13.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:13.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:13.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:13.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:13.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:13.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:13.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:13.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:14.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:13 smithi190 ceph-mon[32247]: pgmap v1466: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16059/160443 objects degraded (10.009%); 27847/160443 objects misplaced (17.356%) 2024-06-27T21:34:14.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:13 smithi192 ceph-mon[32956]: pgmap v1466: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16059/160443 objects degraded (10.009%); 27847/160443 objects misplaced (17.356%) 2024-06-27T21:34:14.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:14.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:13 smithi151 ceph-mon[28965]: pgmap v1466: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16059/160443 objects degraded (10.009%); 27847/160443 objects misplaced (17.356%) 2024-06-27T21:34:14.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:14.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:14.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:14.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:14.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:14.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:14.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:14.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:14.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:14.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:14.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:14.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:14.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:14.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:14.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:15.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 16059/160443 objects degraded (10.009%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:15.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 16059/160443 objects degraded (10.009%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:15.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:15.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:15.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 16059/160443 objects degraded (10.009%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:15.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:15.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:15.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:15.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:15.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:15.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:15.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:15.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:15.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:15.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:15.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:15.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:15.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:15.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:15.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:15 smithi190 ceph-mon[32247]: pgmap v1467: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16021/159477 objects degraded (10.046%); 27121/159477 objects misplaced (17.006%) 2024-06-27T21:34:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:15 smithi190 ceph-mon[32247]: osdmap e564: 12 total, 12 up, 11 in 2024-06-27T21:34:16.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:15 smithi192 ceph-mon[32956]: pgmap v1467: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16021/159477 objects degraded (10.046%); 27121/159477 objects misplaced (17.006%) 2024-06-27T21:34:16.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:16.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:16.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:15 smithi192 ceph-mon[32956]: osdmap e564: 12 total, 12 up, 11 in 2024-06-27T21:34:16.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:16.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:16.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:15 smithi151 ceph-mon[28965]: pgmap v1467: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16021/159477 objects degraded (10.046%); 27121/159477 objects misplaced (17.006%) 2024-06-27T21:34:16.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:16.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:16.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:15 smithi151 ceph-mon[28965]: osdmap e564: 12 total, 12 up, 11 in 2024-06-27T21:34:16.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:16.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:16.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:16.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:16.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:16.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:16.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:16.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:16.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:16.724 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:16.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:16.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:16.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:16.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:16.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:17.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:17.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:17.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:17.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:17.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:17.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:17.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:17.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:17.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:34:17.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:17.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:17.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:17.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:17.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:17 smithi151 ceph-mon[28965]: pgmap v1469: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16021/159093 objects degraded (10.070%); 27070/159093 objects misplaced (17.015%) 2024-06-27T21:34:17.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:17.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:17.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:17.901 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:34:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:34:17.931 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:34:17.931 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:34:17.959 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:34:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:34:17.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:34:17.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:34:17.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:18.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:17 smithi190 ceph-mon[32247]: pgmap v1469: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16021/159093 objects degraded (10.070%); 27070/159093 objects misplaced (17.015%) 2024-06-27T21:34:18.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:18.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:34:18.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:34:18.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:17 smithi192 ceph-mon[32956]: pgmap v1469: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16021/159093 objects degraded (10.070%); 27070/159093 objects misplaced (17.015%) 2024-06-27T21:34:18.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:34:18.057 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:34:18.087 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:34:18.087 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:34:18.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:18.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:18.192 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:34:18.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:34:18.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:34:18.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:34:18.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:18.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:34:18.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:34:18.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:18.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:34:18.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:34:18.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:34:18.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:34:18.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:18.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:18.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:18.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:18.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:18.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:18.690 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:18.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:18.763 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.0 pg_upmap mapping to [1,9,10] 2024-06-27T21:34:18.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:18.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:18.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:18.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:19.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:18 smithi190 ceph-mon[32247]: osdmap e565: 12 total, 12 up, 11 in 2024-06-27T21:34:19.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/623951938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:34:19.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/361353814' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 9, 10]}]: dispatch 2024-06-27T21:34:19.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:19.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:19.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:18 smithi192 ceph-mon[32956]: osdmap e565: 12 total, 12 up, 11 in 2024-06-27T21:34:19.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/623951938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:34:19.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/361353814' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 9, 10]}]: dispatch 2024-06-27T21:34:19.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:19.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:18 smithi151 ceph-mon[28965]: osdmap e565: 12 total, 12 up, 11 in 2024-06-27T21:34:19.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/623951938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:34:19.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/361353814' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 9, 10]}]: dispatch 2024-06-27T21:34:19.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:19.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:19.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:19.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:19.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:19.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:19.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:19.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:19.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:19.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:19.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:19.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:19.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:19.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:20.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:19 smithi192 ceph-mon[32956]: pgmap v1471: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16001/158610 objects degraded (10.088%); 26413/158610 objects misplaced (16.653%) 2024-06-27T21:34:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/361353814' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 9, 10]}]': finished 2024-06-27T21:34:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:19 smithi192 ceph-mon[32956]: osdmap e566: 12 total, 12 up, 11 in 2024-06-27T21:34:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:19 smithi192 ceph-mon[32956]: 6.13 scrub starts 2024-06-27T21:34:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 16001/158610 objects degraded (10.088%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:19 smithi192 ceph-mon[32956]: osdmap e567: 12 total, 12 up, 11 in 2024-06-27T21:34:20.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:20.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:20.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:20.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:19 smithi151 ceph-mon[28965]: pgmap v1471: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16001/158610 objects degraded (10.088%); 26413/158610 objects misplaced (16.653%) 2024-06-27T21:34:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/361353814' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 9, 10]}]': finished 2024-06-27T21:34:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:19 smithi151 ceph-mon[28965]: osdmap e566: 12 total, 12 up, 11 in 2024-06-27T21:34:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:19 smithi151 ceph-mon[28965]: 6.13 scrub starts 2024-06-27T21:34:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 16001/158610 objects degraded (10.088%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:19 smithi151 ceph-mon[28965]: osdmap e567: 12 total, 12 up, 11 in 2024-06-27T21:34:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:19 smithi190 ceph-mon[32247]: pgmap v1471: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 46 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 16001/158610 objects degraded (10.088%); 26413/158610 objects misplaced (16.653%) 2024-06-27T21:34:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/361353814' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [1, 9, 10]}]': finished 2024-06-27T21:34:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:19 smithi190 ceph-mon[32247]: osdmap e566: 12 total, 12 up, 11 in 2024-06-27T21:34:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:19 smithi190 ceph-mon[32247]: 6.13 scrub starts 2024-06-27T21:34:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 16001/158610 objects degraded (10.088%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:19 smithi190 ceph-mon[32247]: osdmap e567: 12 total, 12 up, 11 in 2024-06-27T21:34:20.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:20.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:20.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:20.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:20.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:20.553 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:20.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:20.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:20.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:20.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:20.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:20.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:20.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:21.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:21.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:20 smithi192 ceph-mon[32956]: 6.13 scrub ok 2024-06-27T21:34:21.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:20 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:34:21.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:20 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:34:21.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:21.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:21.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:20 smithi151 ceph-mon[28965]: 6.13 scrub ok 2024-06-27T21:34:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:20 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:34:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:20 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:34:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:20 smithi190 ceph-mon[32247]: 6.13 scrub ok 2024-06-27T21:34:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:20 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:34:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:20 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:34:21.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:21.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:21.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:21.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:21.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:21.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:21.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:21.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:21.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:21.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:21.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:21.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:21.953 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:21.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:21 smithi192 ceph-mon[32956]: pgmap v1474: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15961/157758 objects degraded (10.117%); 26238/157758 objects misplaced (16.632%) 2024-06-27T21:34:22.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:21 smithi192 ceph-mon[32956]: osdmap e568: 12 total, 12 up, 11 in 2024-06-27T21:34:22.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:22.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:22.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:22.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:21 smithi151 ceph-mon[28965]: pgmap v1474: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15961/157758 objects degraded (10.117%); 26238/157758 objects misplaced (16.632%) 2024-06-27T21:34:22.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:21 smithi151 ceph-mon[28965]: osdmap e568: 12 total, 12 up, 11 in 2024-06-27T21:34:22.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:21 smithi190 ceph-mon[32247]: pgmap v1474: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 15961/157758 objects degraded (10.117%); 26238/157758 objects misplaced (16.632%) 2024-06-27T21:34:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:21 smithi190 ceph-mon[32247]: osdmap e568: 12 total, 12 up, 11 in 2024-06-27T21:34:22.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:22.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:22.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:22.450 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:22.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:22.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:22.573 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:22.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:22.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:22.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:22.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:22.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:22.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:22.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:23.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:23.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:22 smithi192 ceph-mon[32956]: osdmap e569: 12 total, 12 up, 11 in 2024-06-27T21:34:23.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:23.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:23.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:23.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:22 smithi151 ceph-mon[28965]: osdmap e569: 12 total, 12 up, 11 in 2024-06-27T21:34:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:22 smithi190 ceph-mon[32247]: osdmap e569: 12 total, 12 up, 11 in 2024-06-27T21:34:23.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:23.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:23.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:23.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:23.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:23.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:23.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:23.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:23.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:23.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:23.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:23.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:23.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:24.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:24.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:23 smithi192 ceph-mon[32956]: 6.11 scrub starts 2024-06-27T21:34:24.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:23 smithi192 ceph-mon[32956]: pgmap v1477: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 9663/157311 objects degraded (6.143%); 27947/157311 objects misplaced (17.765%) 2024-06-27T21:34:24.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:23 smithi192 ceph-mon[32956]: 6.11 scrub ok 2024-06-27T21:34:24.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:24.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:24.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:24.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:23 smithi151 ceph-mon[28965]: 6.11 scrub starts 2024-06-27T21:34:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:23 smithi151 ceph-mon[28965]: pgmap v1477: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 9663/157311 objects degraded (6.143%); 27947/157311 objects misplaced (17.765%) 2024-06-27T21:34:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:23 smithi151 ceph-mon[28965]: 6.11 scrub ok 2024-06-27T21:34:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:23 smithi190 ceph-mon[32247]: 6.11 scrub starts 2024-06-27T21:34:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:23 smithi190 ceph-mon[32247]: pgmap v1477: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 9663/157311 objects degraded (6.143%); 27947/157311 objects misplaced (17.765%) 2024-06-27T21:34:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:23 smithi190 ceph-mon[32247]: 6.11 scrub ok 2024-06-27T21:34:24.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:24.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:24.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:24.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:24.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:24.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:24.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:24.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:24.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:24.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:24.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:24.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:24.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:24.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:25.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 9663/157311 objects degraded (6.143%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:25.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:25.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:25.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:25.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 9663/157311 objects degraded (6.143%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 9663/157311 objects degraded (6.143%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:25.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:25.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:25.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:25.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:25.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:25.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:25.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:25.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:25.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:25.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:25.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:25.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:26.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:26.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:25 smithi192 ceph-mon[32956]: pgmap v1478: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/156357 objects degraded (4.137%); 26764/156357 objects misplaced (17.117%) 2024-06-27T21:34:26.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:26.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:26.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:26.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:25 smithi151 ceph-mon[28965]: pgmap v1478: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/156357 objects degraded (4.137%); 26764/156357 objects misplaced (17.117%) 2024-06-27T21:34:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:25 smithi190 ceph-mon[32247]: pgmap v1478: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/156357 objects degraded (4.137%); 26764/156357 objects misplaced (17.117%) 2024-06-27T21:34:26.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:26.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:26.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:26.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:26.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:26.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:26.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:26.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:26.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:26.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:26.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:26.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:26.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:27.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:27.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:27.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:27.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:27.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:27.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:27.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:27.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:27.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:27.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:27.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:27.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:27.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:27.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:27.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:27.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:27.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:28.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:28.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:27 smithi192 ceph-mon[32956]: pgmap v1479: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/156012 objects degraded (4.146%); 26725/156012 objects misplaced (17.130%) 2024-06-27T21:34:28.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:28.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:28.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:28.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:27 smithi151 ceph-mon[28965]: pgmap v1479: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/156012 objects degraded (4.146%); 26725/156012 objects misplaced (17.130%) 2024-06-27T21:34:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:27 smithi190 ceph-mon[32247]: pgmap v1479: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 47 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/156012 objects degraded (4.146%); 26725/156012 objects misplaced (17.130%) 2024-06-27T21:34:28.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:28.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:28.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:28.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:28.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:28.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:28.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:28.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:28.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:28.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:28.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:29.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:29.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:29.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:29.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:29.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:29.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:29.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:29.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:29.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:29.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:29.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:29.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:29.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:29.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:29.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:29.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:29.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:30.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:30.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:30.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:30.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:29 smithi151 ceph-mon[28965]: pgmap v1480: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/155385 objects degraded (4.163%); 25384/155385 objects misplaced (16.336%) 2024-06-27T21:34:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6469/155385 objects degraded (4.163%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:29 smithi190 ceph-mon[32247]: pgmap v1480: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/155385 objects degraded (4.163%); 25384/155385 objects misplaced (16.336%) 2024-06-27T21:34:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6469/155385 objects degraded (4.163%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:30.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:29 smithi192 ceph-mon[32956]: pgmap v1480: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6469/155385 objects degraded (4.163%); 25384/155385 objects misplaced (16.336%) 2024-06-27T21:34:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6469/155385 objects degraded (4.163%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:34:30.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:30.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:30.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:30.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:30.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:30.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:30.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:30.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:30.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:30.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:30.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:30.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:30.982 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:34:30.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:34:30.983 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-27T21:34:30.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:34:31.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:31.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:31.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:31.081 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:31.130 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:31.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:31.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:31.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:31.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:31.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:31.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:31.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:31.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:31.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:31.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:31.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:31.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:31.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:31.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:31.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:32.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:32.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:32.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:32.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:31 smithi151 ceph-mon[28965]: pgmap v1481: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6449/154701 objects degraded (4.169%); 25241/154701 objects misplaced (16.316%) 2024-06-27T21:34:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:31 smithi151 ceph-mon[28965]: osdmap e570: 12 total, 12 up, 11 in 2024-06-27T21:34:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:31 smithi190 ceph-mon[32247]: pgmap v1481: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6449/154701 objects degraded (4.169%); 25241/154701 objects misplaced (16.316%) 2024-06-27T21:34:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:31 smithi190 ceph-mon[32247]: osdmap e570: 12 total, 12 up, 11 in 2024-06-27T21:34:32.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:31 smithi192 ceph-mon[32956]: pgmap v1481: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6449/154701 objects degraded (4.169%); 25241/154701 objects misplaced (16.316%) 2024-06-27T21:34:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:31 smithi192 ceph-mon[32956]: osdmap e570: 12 total, 12 up, 11 in 2024-06-27T21:34:32.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:32.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:32.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:32.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:32.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:32.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:32.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:32.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:32.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:32.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:32.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:32.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:33.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:33.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:33.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:34:33.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:33.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:33.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:33.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:33.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:33.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:33.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:33.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:33.640 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:33.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:33.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:33.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:33.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:33.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:33 smithi151 ceph-mon[28965]: pgmap v1483: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6449/154377 objects degraded (4.177%); 24668/154377 objects misplaced (15.979%) 2024-06-27T21:34:33.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:33.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:33.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:34.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:34.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:34.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:33 smithi190 ceph-mon[32247]: pgmap v1483: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6449/154377 objects degraded (4.177%); 24668/154377 objects misplaced (15.979%) 2024-06-27T21:34:34.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:34.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:34.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:33 smithi192 ceph-mon[32956]: pgmap v1483: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6449/154377 objects degraded (4.177%); 24668/154377 objects misplaced (15.979%) 2024-06-27T21:34:34.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:34.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:34.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:34.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:34.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:34.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:34.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:34.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:34.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:34.825 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:34:34.825 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:34:34.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:34.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:34.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:35.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:35.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:35.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6449/154377 objects degraded (4.177%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:35.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:34 smithi151 ceph-mon[28965]: osdmap e571: 12 total, 12 up, 11 in 2024-06-27T21:34:35.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:35.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6449/154377 objects degraded (4.177%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:34 smithi190 ceph-mon[32247]: osdmap e571: 12 total, 12 up, 11 in 2024-06-27T21:34:35.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6449/154377 objects degraded (4.177%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:34 smithi192 ceph-mon[32956]: osdmap e571: 12 total, 12 up, 11 in 2024-06-27T21:34:35.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:35.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:35.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:35.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:35.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:35.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:35.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:35.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:35.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:35.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:35.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:35.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:36.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:36.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:36.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:35 smithi151 ceph-mon[28965]: pgmap v1484: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6429/153798 objects degraded (4.180%); 23543/153798 objects misplaced (15.308%) 2024-06-27T21:34:36.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:35 smithi151 ceph-mon[28965]: osdmap e572: 12 total, 12 up, 11 in 2024-06-27T21:34:36.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:36.208 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:35 smithi190 ceph-mon[32247]: pgmap v1484: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6429/153798 objects degraded (4.180%); 23543/153798 objects misplaced (15.308%) 2024-06-27T21:34:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:35 smithi190 ceph-mon[32247]: osdmap e572: 12 total, 12 up, 11 in 2024-06-27T21:34:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:35 smithi192 ceph-mon[32956]: pgmap v1484: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6429/153798 objects degraded (4.180%); 23543/153798 objects misplaced (15.308%) 2024-06-27T21:34:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:35 smithi192 ceph-mon[32956]: osdmap e572: 12 total, 12 up, 11 in 2024-06-27T21:34:36.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:36.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:36.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:36.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:36.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:36.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:36.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:36.724 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:36.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:36.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:36.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:36.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:36.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:37.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:34:37.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:37.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:37.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:37.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:37.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:37.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:37.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:37.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:37.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:37.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:37.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:37.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:37.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:37.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:37.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:37 smithi151 ceph-mon[28965]: 6.10 deep-scrub starts 2024-06-27T21:34:37.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:37 smithi151 ceph-mon[28965]: pgmap v1487: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 55 op/s; 6429/153696 objects degraded (4.183%); 23522/153696 objects misplaced (15.304%); 6.8 MiB/s, 158 objects/s recovering 2024-06-27T21:34:37.934 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:34:37.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:37.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:37.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:38.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:38.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:38.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:38.241 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:37 smithi190 ceph-mon[32247]: 6.10 deep-scrub starts 2024-06-27T21:34:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:37 smithi190 ceph-mon[32247]: pgmap v1487: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 55 op/s; 6429/153696 objects degraded (4.183%); 23522/153696 objects misplaced (15.304%); 6.8 MiB/s, 158 objects/s recovering 2024-06-27T21:34:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:37 smithi192 ceph-mon[32956]: 6.10 deep-scrub starts 2024-06-27T21:34:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:37 smithi192 ceph-mon[32956]: pgmap v1487: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 55 op/s; 6429/153696 objects degraded (4.183%); 23522/153696 objects misplaced (15.304%); 6.8 MiB/s, 158 objects/s recovering 2024-06-27T21:34:38.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:38.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:38.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:38.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:38.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:38.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:38.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:38.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:38.831 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:34:38.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:38.864 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:38.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:38.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:38.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:39.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:39.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:39.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/981246461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:34:39.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:38 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:34:39.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:39.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/981246461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:34:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:38 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:34:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/981246461' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:34:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:38 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:34:39.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:39.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:39.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:39.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:39.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:39.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:39.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:39.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:39.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:39.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:39.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:39.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:39 smithi151 ceph-mon[28965]: pgmap v1488: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 6429/153666 objects degraded (4.184%); 21787/153666 objects misplaced (14.178%); 11 MiB/s, 246 objects/s recovering 2024-06-27T21:34:39.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:39 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:34:39.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:39 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:34:39.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:39 smithi151 ceph-mon[28965]: osdmap e573: 12 total, 12 up, 11 in 2024-06-27T21:34:39.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6429/153666 objects degraded (4.184%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:39.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:40.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:40.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:40.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:40.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:40.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:39 smithi190 ceph-mon[32247]: pgmap v1488: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 6429/153666 objects degraded (4.184%); 21787/153666 objects misplaced (14.178%); 11 MiB/s, 246 objects/s recovering 2024-06-27T21:34:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:39 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:34:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:39 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:34:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:39 smithi190 ceph-mon[32247]: osdmap e573: 12 total, 12 up, 11 in 2024-06-27T21:34:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6429/153666 objects degraded (4.184%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:39 smithi192 ceph-mon[32956]: pgmap v1488: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 45 op/s; 6429/153666 objects degraded (4.184%); 21787/153666 objects misplaced (14.178%); 11 MiB/s, 246 objects/s recovering 2024-06-27T21:34:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:39 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:34:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:39 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:34:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:39 smithi192 ceph-mon[32956]: osdmap e573: 12 total, 12 up, 11 in 2024-06-27T21:34:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6429/153666 objects degraded (4.184%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:40.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:40.387 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:40.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:40.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:40.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:40.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:40.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:40.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:40.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:40.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:40.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:40.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:41.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:41.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:41.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:34:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:40 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:41.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:41.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:34:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:40 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:41.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:41.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:34:41.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:40 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:41.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:41.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:41.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:41.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:41.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:41.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:41.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:41.767 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:41.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:41.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:41.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:41.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:42.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:42.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:42.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:41 smithi151 ceph-mon[28965]: pgmap v1490: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 6429/153666 objects degraded (4.184%); 21787/153666 objects misplaced (14.178%); 6.8 MiB/s, 143 objects/s recovering 2024-06-27T21:34:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:41 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:41 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:42.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:42.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:41 smithi190 ceph-mon[32247]: pgmap v1490: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 6429/153666 objects degraded (4.184%); 21787/153666 objects misplaced (14.178%); 6.8 MiB/s, 143 objects/s recovering 2024-06-27T21:34:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:41 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:41 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:42.270 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:42.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:41 smithi192 ceph-mon[32956]: pgmap v1490: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 6429/153666 objects degraded (4.184%); 21787/153666 objects misplaced (14.178%); 6.8 MiB/s, 143 objects/s recovering 2024-06-27T21:34:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:41 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:41 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:34:42.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:42.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:42.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:42.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:42.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:42.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:42.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:42.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:42.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:42.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:42.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:43.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:43.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:43.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:43.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:43.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:43.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:43.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:43.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:43.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:43.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:43.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:43.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:43.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:43.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:43.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:43.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:43.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:44.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:44.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:43 smithi151 ceph-mon[28965]: pgmap v1491: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 10 op/s; 6429/153525 objects degraded (4.188%); 21376/153525 objects misplaced (13.923%); 6.7 MiB/s, 133 objects/s recovering 2024-06-27T21:34:44.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:44.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:43 smithi190 ceph-mon[32247]: pgmap v1491: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 10 op/s; 6429/153525 objects degraded (4.188%); 21376/153525 objects misplaced (13.923%); 6.7 MiB/s, 133 objects/s recovering 2024-06-27T21:34:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:43 smithi192 ceph-mon[32956]: pgmap v1491: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 10 op/s; 6429/153525 objects degraded (4.188%); 21376/153525 objects misplaced (13.923%); 6.7 MiB/s, 133 objects/s recovering 2024-06-27T21:34:44.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:44.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:44.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:44.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:44.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:44.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:44.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:44.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:44.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:44.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:44.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:44.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:44.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:45.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:45.113 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:45.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2048369188,config=2353167246,health=4027701229,kv=30032894}) 2024-06-27T21:34:45.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {kv=12,logm=88} crc {kv=2917721341,logm=4086529796}) 2024-06-27T21:34:45.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=315843321}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963456066}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827970617}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351872174}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2081773633}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407569803}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1891360028}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103918422}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362441234}) 2024-06-27T21:34:45.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837764817}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2341034527}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792474168}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=641908584}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=314746746}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795358174}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2068418312}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781133834}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2005628724}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1009315012}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469233921}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2332801385}) 2024-06-27T21:34:45.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984824782}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1438703003}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2236903905}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2441392569}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606756661}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3898654898}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2866409017}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138077735}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303223994}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4029622981}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204423188}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2592638477}) 2024-06-27T21:34:45.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373094346}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3853145367}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595110208}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1201384389}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3776066671}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1042293855}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714999820}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170375216}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756160331}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1579112627}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=693366999}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041693784}) 2024-06-27T21:34:45.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383433430}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060209472}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537134581}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538535469}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658048882}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685707574}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594715312}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899296613}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375639617}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=62,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=4} crc {logm=3661658851,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=4057847186,mon_config_key=400410321}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1163677583}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3878262644}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=2719639702,monmap=3596043196,osd_metadata=2082607930}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2856723407,osd_pg_creating=689344543,osdmap=640473547}) 2024-06-27T21:34:45.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=1743660220}) 2024-06-27T21:34:45.140 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6429/153525 objects degraded (4.188%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:45.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:45.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2048369188,config=2353167246,health=4027701229,kv=30032894}) 2024-06-27T21:34:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {kv=12,logm=88} crc {kv=2917721341,logm=4086529796}) 2024-06-27T21:34:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=315843321}) 2024-06-27T21:34:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963456066}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827970617}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351872174}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2081773633}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407569803}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1891360028}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103918422}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362441234}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837764817}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2341034527}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792474168}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=641908584}) 2024-06-27T21:34:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=314746746}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795358174}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2068418312}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781133834}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2005628724}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1009315012}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469233921}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2332801385}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984824782}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1438703003}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2236903905}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2441392569}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606756661}) 2024-06-27T21:34:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3898654898}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2866409017}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138077735}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303223994}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4029622981}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204423188}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2592638477}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373094346}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3853145367}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595110208}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1201384389}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3776066671}) 2024-06-27T21:34:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1042293855}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714999820}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170375216}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756160331}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1579112627}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=693366999}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041693784}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383433430}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060209472}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537134581}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538535469}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658048882}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685707574}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594715312}) 2024-06-27T21:34:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899296613}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375639617}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=62,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=4} crc {logm=3661658851,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=4057847186,mon_config_key=400410321}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1163677583}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3878262644}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=2719639702,monmap=3596043196,osd_metadata=2082607930}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2856723407,osd_pg_creating=689344543,osdmap=640473547}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=1743660220}) 2024-06-27T21:34:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6429/153525 objects degraded (4.188%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=2048369188,config=2353167246,health=4027701229,kv=30032894}) 2024-06-27T21:34:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {kv=12,logm=88} crc {kv=2917721341,logm=4086529796}) 2024-06-27T21:34:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=315843321}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3963456066}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827970617}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=351872174}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2081773633}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407569803}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1891360028}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=103918422}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362441234}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837764817}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2341034527}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792474168}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=641908584}) 2024-06-27T21:34:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=314746746}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795358174}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2068418312}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781133834}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2005628724}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1009315012}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469233921}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2332801385}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1984824782}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1438703003}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2236903905}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2441392569}) 2024-06-27T21:34:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606756661}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3898654898}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2866409017}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138077735}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303223994}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4029622981}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4204423188}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2592638477}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=373094346}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3853145367}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3595110208}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1201384389}) 2024-06-27T21:34:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3776066671}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1042293855}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714999820}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170375216}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756160331}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1579112627}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=693366999}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041693784}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383433430}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060209472}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=537134581}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3538535469}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2658048882}) 2024-06-27T21:34:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=685707574}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594715312}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899296613}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=375639617}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=62,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=4} crc {logm=3661658851,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=4057847186,mon_config_key=400410321}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1163677583}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3878262644}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=2719639702,monmap=3596043196,osd_metadata=2082607930}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2856723407,osd_pg_creating=689344543,osdmap=640473547}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=79} crc {osdmap=1743660220}) 2024-06-27T21:34:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6429/153525 objects degraded (4.188%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:45.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:45.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:45.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:45.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:45.607 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:45.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:45.730 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:45.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:45.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:45.903 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:34:45.904 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:34:45.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:45.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:45.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:46.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:46.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:46.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:45 smithi151 ceph-mon[28965]: pgmap v1492: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 6396/152904 objects degraded (4.183%); 20520/152904 objects misplaced (13.420%); 8.5 MiB/s, 169 objects/s recovering 2024-06-27T21:34:46.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:46.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:46.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:45 smithi190 ceph-mon[32247]: pgmap v1492: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 6396/152904 objects degraded (4.183%); 20520/152904 objects misplaced (13.420%); 8.5 MiB/s, 169 objects/s recovering 2024-06-27T21:34:46.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:45 smithi192 ceph-mon[32956]: pgmap v1492: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 31 op/s; 6396/152904 objects degraded (4.183%); 20520/152904 objects misplaced (13.420%); 8.5 MiB/s, 169 objects/s recovering 2024-06-27T21:34:46.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:46.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:46.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:46.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:46.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:46.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:46.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:46.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:46.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:46.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:46.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:46.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:46.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:47.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:47.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:46 smithi151 ceph-mon[28965]: from='client.26291 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:34:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:46 smithi151 ceph-mon[28965]: osdmap e574: 12 total, 12 up, 11 in 2024-06-27T21:34:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:47.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:47.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:46 smithi190 ceph-mon[32247]: from='client.26291 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:34:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:46 smithi190 ceph-mon[32247]: osdmap e574: 12 total, 12 up, 11 in 2024-06-27T21:34:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:46 smithi192 ceph-mon[32956]: from='client.26291 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:34:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:46 smithi192 ceph-mon[32956]: osdmap e574: 12 total, 12 up, 11 in 2024-06-27T21:34:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:47.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:47.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:47.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:47.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:47.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:47.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:47.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:47.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:47.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:47.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:47.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:47.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:47.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:48.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.122 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:48.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:48.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:47 smithi151 ceph-mon[28965]: 6.10 deep-scrub ok 2024-06-27T21:34:48.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:47 smithi151 ceph-mon[28965]: pgmap v1494: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 47 op/s; 6396/152532 objects degraded (4.193%); 20463/152532 objects misplaced (13.416%); 3.9 MiB/s, 71 objects/s recovering 2024-06-27T21:34:48.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:47 smithi151 ceph-mon[28965]: osdmap e575: 12 total, 12 up, 11 in 2024-06-27T21:34:48.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:48.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:47 smithi190 ceph-mon[32247]: 6.10 deep-scrub ok 2024-06-27T21:34:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:47 smithi190 ceph-mon[32247]: pgmap v1494: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 47 op/s; 6396/152532 objects degraded (4.193%); 20463/152532 objects misplaced (13.416%); 3.9 MiB/s, 71 objects/s recovering 2024-06-27T21:34:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:47 smithi190 ceph-mon[32247]: osdmap e575: 12 total, 12 up, 11 in 2024-06-27T21:34:48.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:47 smithi192 ceph-mon[32956]: 6.10 deep-scrub ok 2024-06-27T21:34:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:47 smithi192 ceph-mon[32956]: pgmap v1494: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 48 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 47 op/s; 6396/152532 objects degraded (4.193%); 20463/152532 objects misplaced (13.416%); 3.9 MiB/s, 71 objects/s recovering 2024-06-27T21:34:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:47 smithi192 ceph-mon[32956]: osdmap e575: 12 total, 12 up, 11 in 2024-06-27T21:34:48.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:48.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:48.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:48.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:48.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:48.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:48.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:48.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:48.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:48.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:48.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:48.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:48.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:49.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:49.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:49.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:49.232 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:49.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:49.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:49.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:49.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:49.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:49.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:49.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:49.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:49.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:49.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:49.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:49.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:49.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:50.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:50.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:49 smithi151 ceph-mon[28965]: pgmap v1496: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6395/151824 objects degraded (4.212%); 19568/151824 objects misplaced (12.889%) 2024-06-27T21:34:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:49 smithi151 ceph-mon[28965]: osdmap e576: 12 total, 12 up, 11 in 2024-06-27T21:34:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6395/151824 objects degraded (4.212%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:49 smithi151 ceph-mon[28965]: osdmap e577: 12 total, 12 up, 11 in 2024-06-27T21:34:50.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:50.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:50.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:50.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:49 smithi190 ceph-mon[32247]: pgmap v1496: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6395/151824 objects degraded (4.212%); 19568/151824 objects misplaced (12.889%) 2024-06-27T21:34:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:49 smithi190 ceph-mon[32247]: osdmap e576: 12 total, 12 up, 11 in 2024-06-27T21:34:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6395/151824 objects degraded (4.212%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:49 smithi190 ceph-mon[32247]: osdmap e577: 12 total, 12 up, 11 in 2024-06-27T21:34:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:49 smithi192 ceph-mon[32956]: pgmap v1496: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6395/151824 objects degraded (4.212%); 19568/151824 objects misplaced (12.889%) 2024-06-27T21:34:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:49 smithi192 ceph-mon[32956]: osdmap e576: 12 total, 12 up, 11 in 2024-06-27T21:34:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6395/151824 objects degraded (4.212%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:49 smithi192 ceph-mon[32956]: osdmap e577: 12 total, 12 up, 11 in 2024-06-27T21:34:50.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:50.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:50.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:50.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:50.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:50.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:50.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:50.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:50.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:50.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:50.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:50.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:50.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:50.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:51.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:51.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:51.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:51.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:51.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:51.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:51.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:51.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:51.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:51.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:51.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:51.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:51.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:51.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:51.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:51.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:52.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:52.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:52.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:52.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:51 smithi151 ceph-mon[28965]: pgmap v1499: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6355/150990 objects degraded (4.209%); 19403/150990 objects misplaced (12.851%) 2024-06-27T21:34:52.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:52.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:52.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:51 smithi190 ceph-mon[32247]: pgmap v1499: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6355/150990 objects degraded (4.209%); 19403/150990 objects misplaced (12.851%) 2024-06-27T21:34:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:51 smithi192 ceph-mon[32956]: pgmap v1499: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6355/150990 objects degraded (4.209%); 19403/150990 objects misplaced (12.851%) 2024-06-27T21:34:52.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:52.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:52.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:52.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:52.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:52.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:52.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:52.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:52.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:52.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:52.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:52.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:53.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:53.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:53.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:53.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:53.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:53.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:53.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:53.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:53.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:53.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:53.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:53.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:53.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:53.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:53.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:53.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:53.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:54.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:54.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:53 smithi151 ceph-mon[28965]: pgmap v1500: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6355/150627 objects degraded (4.219%); 18921/150627 objects misplaced (12.561%) 2024-06-27T21:34:54.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:54.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:53 smithi190 ceph-mon[32247]: pgmap v1500: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6355/150627 objects degraded (4.219%); 18921/150627 objects misplaced (12.561%) 2024-06-27T21:34:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:53 smithi192 ceph-mon[32956]: pgmap v1500: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 6355/150627 objects degraded (4.219%); 18921/150627 objects misplaced (12.561%) 2024-06-27T21:34:54.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:54.372 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:54.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:54.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:54.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:54.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:54.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:54.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:54.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:54.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:54.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:55.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:55.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:55.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6355/150627 objects degraded (4.219%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:55.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:55.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:55.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6355/150627 objects degraded (4.219%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:55.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6355/150627 objects degraded (4.219%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:34:55.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:55.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:55.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:55.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:55.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:55.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:55.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:55.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:55.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:55.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:55.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:55.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:55.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:56.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:56.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:55 smithi151 ceph-mon[28965]: pgmap v1501: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 6317/149934 objects degraded (4.213%); 18319/149934 objects misplaced (12.218%); 6.3 MiB/s, 194 objects/s recovering 2024-06-27T21:34:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:56.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:56.240 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:55 smithi190 ceph-mon[32247]: pgmap v1501: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 6317/149934 objects degraded (4.213%); 18319/149934 objects misplaced (12.218%); 6.3 MiB/s, 194 objects/s recovering 2024-06-27T21:34:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:56.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:55 smithi192 ceph-mon[32956]: pgmap v1501: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 98 op/s; 6317/149934 objects degraded (4.213%); 18319/149934 objects misplaced (12.218%); 6.3 MiB/s, 194 objects/s recovering 2024-06-27T21:34:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:56.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:56.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:56.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:56.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:56.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:56.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:56.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:56.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:56.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:56.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:56.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:56.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:56.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:57.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:57.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:57.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:56 smithi192 ceph-mon[32956]: osdmap e578: 12 total, 12 up, 11 in 2024-06-27T21:34:57.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:56 smithi192 ceph-mon[32956]: pgmap v1503: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 6317/149598 objects degraded (4.223%); 18280/149598 objects misplaced (12.219%); 4.0 MiB/s, 139 objects/s recovering 2024-06-27T21:34:57.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:56 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:57.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:56 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:56 smithi151 ceph-mon[28965]: osdmap e578: 12 total, 12 up, 11 in 2024-06-27T21:34:57.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:56 smithi151 ceph-mon[28965]: pgmap v1503: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 6317/149598 objects degraded (4.223%); 18280/149598 objects misplaced (12.219%); 4.0 MiB/s, 139 objects/s recovering 2024-06-27T21:34:57.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:56 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:57.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:56 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:57.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:34:57.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:34:57.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:56 smithi190 ceph-mon[32247]: osdmap e578: 12 total, 12 up, 11 in 2024-06-27T21:34:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:56 smithi190 ceph-mon[32247]: pgmap v1503: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 77 op/s; 6317/149598 objects degraded (4.223%); 18280/149598 objects misplaced (12.219%); 4.0 MiB/s, 139 objects/s recovering 2024-06-27T21:34:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:56 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:56 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:57.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:34:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:57.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:34:57.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:57.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:57.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:57.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:57.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:57.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:57.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:34:57.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:34:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:57.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:34:57.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:34:57.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:57.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:34:58.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:58.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:57 smithi151 ceph-mon[28965]: osdmap e579: 12 total, 12 up, 11 in 2024-06-27T21:34:58.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:34:58.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:34:58.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:57 smithi190 ceph-mon[32247]: osdmap e579: 12 total, 12 up, 11 in 2024-06-27T21:34:58.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:57 smithi192 ceph-mon[32956]: osdmap e579: 12 total, 12 up, 11 in 2024-06-27T21:34:58.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:34:58.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:34:58.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:58.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:58.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:58.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:58.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:58.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:34:58.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:58.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:58.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:58.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:58.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:59.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:34:59.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:34:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:34:59.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:59.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:58 smithi190 ceph-mon[32247]: pgmap v1505: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 6319/148929 objects degraded (4.243%); 17612/148929 objects misplaced (11.826%); 5.8 MiB/s, 245 objects/s recovering 2024-06-27T21:34:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:34:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:34:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:58 smithi192 ceph-mon[32956]: pgmap v1505: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 6319/148929 objects degraded (4.243%); 17612/148929 objects misplaced (11.826%); 5.8 MiB/s, 245 objects/s recovering 2024-06-27T21:34:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:34:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:34:59.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:59.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:59.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:58 smithi151 ceph-mon[28965]: pgmap v1505: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 6319/148929 objects degraded (4.243%); 17612/148929 objects misplaced (11.826%); 5.8 MiB/s, 245 objects/s recovering 2024-06-27T21:34:59.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:34:59.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:34:59.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:34:59.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:34:59.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:34:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:34:59.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:34:59.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:34:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:34:59.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:34:59.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:34:59.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:34:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:34:59.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:34:59.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:00.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:00.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:59 smithi151 ceph-mon[28965]: osdmap e580: 12 total, 12 up, 11 in 2024-06-27T21:35:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6319/148929 objects degraded (4.243%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:34:59 smithi151 ceph-mon[28965]: osdmap e581: 12 total, 12 up, 11 in 2024-06-27T21:35:00.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:00.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:59 smithi190 ceph-mon[32247]: osdmap e580: 12 total, 12 up, 11 in 2024-06-27T21:35:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6319/148929 objects degraded (4.243%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:34:59 smithi190 ceph-mon[32247]: osdmap e581: 12 total, 12 up, 11 in 2024-06-27T21:35:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:59 smithi192 ceph-mon[32956]: osdmap e580: 12 total, 12 up, 11 in 2024-06-27T21:35:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6319/148929 objects degraded (4.243%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:34:59 smithi192 ceph-mon[32956]: osdmap e581: 12 total, 12 up, 11 in 2024-06-27T21:35:00.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:00.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:00.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:00.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:00.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:00.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:00.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:00.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:00.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:00.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:00.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:00.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:01.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:01.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:01.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:01.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:01.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:01.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:01.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:00 smithi190 ceph-mon[32247]: pgmap v1508: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 132 op/s; 6289/147921 objects degraded (4.252%); 17375/147921 objects misplaced (11.746%); 5.2 MiB/s, 244 objects/s recovering 2024-06-27T21:35:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:00 smithi192 ceph-mon[32956]: pgmap v1508: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 132 op/s; 6289/147921 objects degraded (4.252%); 17375/147921 objects misplaced (11.746%); 5.2 MiB/s, 244 objects/s recovering 2024-06-27T21:35:01.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:01.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:01.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:00 smithi151 ceph-mon[28965]: pgmap v1508: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 132 op/s; 6289/147921 objects degraded (4.252%); 17375/147921 objects misplaced (11.746%); 5.2 MiB/s, 244 objects/s recovering 2024-06-27T21:35:01.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:01.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:01.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:01.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:01.736 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:01.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:01.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:01.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:01.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:01.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:02.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:02.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:02.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:02.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:02.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:02.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:01 smithi190 ceph-mon[32247]: osdmap e582: 12 total, 12 up, 11 in 2024-06-27T21:35:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:35:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:35:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:01 smithi192 ceph-mon[32956]: osdmap e582: 12 total, 12 up, 11 in 2024-06-27T21:35:02.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:35:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:35:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:02.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:01 smithi151 ceph-mon[28965]: osdmap e582: 12 total, 12 up, 11 in 2024-06-27T21:35:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:35:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:35:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:35:02.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:02.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:02.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:02.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:02.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:02.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:02.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:02.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:02.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:02.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:03.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:03.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:03.145 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:03.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:02 smithi190 ceph-mon[32247]: osdmap e583: 12 total, 12 up, 11 in 2024-06-27T21:35:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:02 smithi190 ceph-mon[32247]: pgmap v1511: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 6289/147438 objects degraded (4.266%); 16766/147438 objects misplaced (11.372%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:35:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:02 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:03.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:02 smithi192 ceph-mon[32956]: osdmap e583: 12 total, 12 up, 11 in 2024-06-27T21:35:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:02 smithi192 ceph-mon[32956]: pgmap v1511: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 6289/147438 objects degraded (4.266%); 16766/147438 objects misplaced (11.372%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:35:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:02 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:03.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:03.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:02 smithi151 ceph-mon[28965]: osdmap e583: 12 total, 12 up, 11 in 2024-06-27T21:35:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:02 smithi151 ceph-mon[28965]: pgmap v1511: 57 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 6289/147438 objects degraded (4.266%); 16766/147438 objects misplaced (11.372%); 4.4 MiB/s, 70 objects/s recovering 2024-06-27T21:35:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:02 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:03.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:03.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:03.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:03.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:03.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:03.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:03.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:03.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:03.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:03.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:04.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:04.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:04.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:04.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:04.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:04.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:35:04.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:35:04.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:03 smithi190 ceph-mon[32247]: osdmap e584: 12 total, 12 up, 11 in 2024-06-27T21:35:04.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:35:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:35:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:03 smithi192 ceph-mon[32956]: osdmap e584: 12 total, 12 up, 11 in 2024-06-27T21:35:04.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:04.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:04.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:35:04.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:35:04.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:03 smithi151 ceph-mon[28965]: osdmap e584: 12 total, 12 up, 11 in 2024-06-27T21:35:04.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:04.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:04.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:04.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:04.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:04.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:04.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:04.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:04.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:04.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:05.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:05.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:05.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:05.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:05.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:04 smithi190 ceph-mon[32247]: pgmap v1513: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 6269/146478 objects degraded (4.280%); 16512/146478 objects misplaced (11.273%); 5.6 MiB/s, 181 objects/s recovering 2024-06-27T21:35:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6289/147438 objects degraded (4.266%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:04 smithi190 ceph-mon[32247]: osdmap e585: 12 total, 12 up, 11 in 2024-06-27T21:35:05.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:05.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:04 smithi192 ceph-mon[32956]: pgmap v1513: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 6269/146478 objects degraded (4.280%); 16512/146478 objects misplaced (11.273%); 5.6 MiB/s, 181 objects/s recovering 2024-06-27T21:35:05.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6289/147438 objects degraded (4.266%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:04 smithi192 ceph-mon[32956]: osdmap e585: 12 total, 12 up, 11 in 2024-06-27T21:35:05.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:05.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:05.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:04 smithi151 ceph-mon[28965]: pgmap v1513: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 49 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 6269/146478 objects degraded (4.280%); 16512/146478 objects misplaced (11.273%); 5.6 MiB/s, 181 objects/s recovering 2024-06-27T21:35:05.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6289/147438 objects degraded (4.266%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:05.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:04 smithi151 ceph-mon[28965]: osdmap e585: 12 total, 12 up, 11 in 2024-06-27T21:35:05.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:05.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:05.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:05.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:05.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:05.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:05.812 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1492,"stamp":"2024-06-27T21:34:44.140810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3340567096,"num_objects":50968,"num_object_clones":0,"num_object_copies":152904,"num_objects_missing_on_primary":4629,"num_objects_missing":4629,"num_objects_degraded":6396,"num_objects_misplaced":20520,"num_objects_unfound":0,"num_objects_dirty":50968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104498,"num_write_kb":4315105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31865,"num_bytes_recovered":1724940312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":264088,"ondisk_log_size":264088,"up":171,"acting":166,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":196,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":16416840,"kb_used_data":11382176,"kb_used_omap":108,"kb_used_meta":5034451,"kb_avail":1014779832,"statfs":{"total":1055945392128,"available":1039134547968,"internally_reserved":0,"allocated":11655348224,"data_stored":11650523184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111086,"internal_metadata":5155278354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17301504,"num_objects":-264,"num_object_clones":0,"num_object_copies":-792,"num_objects_missing_on_primary":-38,"num_objects_missing":-38,"num_objects_degraded":-33,"num_objects_misplaced":-3002,"num_objects_unfound":0,"num_objects_dirty":-264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1429,"num_bytes_recovered":75563008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.436271"},"pg_stats":[{"pgid":"6.18","version":"573'10128","reported_seq":13440,"reported_epoch":573,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:34:42.487748+0000","last_change":"2024-06-27T21:34:17.536692+0000","last_active":"2024-06-27T21:34:42.487748+0000","last_peered":"2024-06-27T21:34:42.487748+0000","last_clean":"2024-06-27T21:33:26.530184+0000","last_became_active":"2024-06-27T21:33:29.166421+0000","last_became_peered":"2024-06-27T21:33:29.166421+0000","last_unstale":"2024-06-27T21:34:42.487748+0000","last_undegraded":"2024-06-27T21:33:28.891033+0000","last_fullsized":"2024-06-27T21:33:28.693201+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'5955","last_scrub_stamp":"2024-06-27T21:30:45.505543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.505543+0000","objects_scrubbed":5927,"log_size":10128,"log_dups_size":0,"ondisk_log_size":10128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":189,"num_objects_misplaced":1399,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":216640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1002,"num_bytes_recovered":65667072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[3,7],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"3,7","objects":788},{"shards":"3,7,11","objects":605}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"573'9940","reported_seq":13608,"reported_epoch":573,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:34:41.368453+0000","last_change":"2024-06-27T21:34:20.281801+0000","last_active":"2024-06-27T21:34:41.368453+0000","last_peered":"2024-06-27T21:34:41.368453+0000","last_clean":"2024-06-27T21:32:12.787863+0000","last_became_active":"2024-06-27T21:33:29.125343+0000","last_became_peered":"2024-06-27T21:33:29.125343+0000","last_unstale":"2024-06-27T21:34:41.368453+0000","last_undegraded":"2024-06-27T21:33:28.949030+0000","last_fullsized":"2024-06-27T21:33:28.693282+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"502'6309","last_scrub_stamp":"2024-06-27T21:30:52.261532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.261532+0000","objects_scrubbed":6289,"log_size":9940,"log_dups_size":0,"ondisk_log_size":9940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109543424,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":3087,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":215904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":870,"num_bytes_recovered":56819712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[0,2],"avail_no_missing":["0","8","9"],"object_location_counts":[{"shards":"0,2","objects":574},{"shards":"0,8,9","objects":994},{"shards":"0,8,9,10","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":89,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875951+0000","last_change":"2024-06-27T21:33:02.987696+0000","last_active":"2024-06-27T21:34:38.875951+0000","last_peered":"2024-06-27T21:34:38.875951+0000","last_clean":"2024-06-27T21:34:38.875951+0000","last_became_active":"2024-06-27T21:33:01.707872+0000","last_became_peered":"2024-06-27T21:33:01.707872+0000","last_unstale":"2024-06-27T21:34:38.875951+0000","last_undegraded":"2024-06-27T21:34:38.875951+0000","last_fullsized":"2024-06-27T21:34:38.875951+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,10],"acting":[11,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":89,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875898+0000","last_change":"2024-06-27T21:33:01.956324+0000","last_active":"2024-06-27T21:34:38.875898+0000","last_peered":"2024-06-27T21:34:38.875898+0000","last_clean":"2024-06-27T21:34:38.875898+0000","last_became_active":"2024-06-27T21:33:01.707250+0000","last_became_peered":"2024-06-27T21:33:01.707250+0000","last_unstale":"2024-06-27T21:34:38.875898+0000","last_undegraded":"2024-06-27T21:34:38.875898+0000","last_fullsized":"2024-06-27T21:34:38.875898+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":116,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875947+0000","last_change":"2024-06-27T21:33:52.993226+0000","last_active":"2024-06-27T21:34:38.875947+0000","last_peered":"2024-06-27T21:34:38.875947+0000","last_clean":"2024-06-27T21:34:38.875947+0000","last_became_active":"2024-06-27T21:33:01.707734+0000","last_became_peered":"2024-06-27T21:33:01.707734+0000","last_unstale":"2024-06-27T21:34:38.875947+0000","last_undegraded":"2024-06-27T21:34:38.875947+0000","last_fullsized":"2024-06-27T21:34:38.875947+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:52.993157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:52.993157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:03.807236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":91,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.851018+0000","last_change":"2024-06-27T21:33:02.568824+0000","last_active":"2024-06-27T21:34:38.851018+0000","last_peered":"2024-06-27T21:34:38.851018+0000","last_clean":"2024-06-27T21:34:38.851018+0000","last_became_active":"2024-06-27T21:33:01.773541+0000","last_became_peered":"2024-06-27T21:33:01.773541+0000","last_unstale":"2024-06-27T21:34:38.851018+0000","last_undegraded":"2024-06-27T21:34:38.851018+0000","last_fullsized":"2024-06-27T21:34:38.851018+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":157,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.829463+0000","last_change":"2024-06-27T21:33:02.266680+0000","last_active":"2024-06-27T21:34:38.829463+0000","last_peered":"2024-06-27T21:34:38.829463+0000","last_clean":"2024-06-27T21:34:38.829463+0000","last_became_active":"2024-06-27T21:33:01.710059+0000","last_became_peered":"2024-06-27T21:33:01.710059+0000","last_unstale":"2024-06-27T21:34:38.829463+0000","last_undegraded":"2024-06-27T21:34:38.829463+0000","last_fullsized":"2024-06-27T21:34:38.829463+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":113,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875600+0000","last_change":"2024-06-27T21:33:50.003594+0000","last_active":"2024-06-27T21:34:38.875600+0000","last_peered":"2024-06-27T21:34:38.875600+0000","last_clean":"2024-06-27T21:34:38.875600+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:34:38.875600+0000","last_undegraded":"2024-06-27T21:34:38.875600+0000","last_fullsized":"2024-06-27T21:34:38.875600+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:50.003522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.003522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:05.414708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":533,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876227+0000","last_change":"2024-06-27T21:33:02.184848+0000","last_active":"2024-06-27T21:34:38.876227+0000","last_peered":"2024-06-27T21:34:38.876227+0000","last_clean":"2024-06-27T21:34:38.876227+0000","last_became_active":"2024-06-27T21:33:01.774416+0000","last_became_peered":"2024-06-27T21:33:01.774416+0000","last_unstale":"2024-06-27T21:34:38.876227+0000","last_undegraded":"2024-06-27T21:34:38.876227+0000","last_fullsized":"2024-06-27T21:34:38.876227+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.184716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.184716+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"573'10831","reported_seq":34704,"reported_epoch":573,"state":"active+clean+scrubbing+deep","last_fresh":"2024-06-27T21:34:41.368913+0000","last_change":"2024-06-27T21:34:35.920228+0000","last_active":"2024-06-27T21:34:41.368913+0000","last_peered":"2024-06-27T21:34:41.368913+0000","last_clean":"2024-06-27T21:34:41.368913+0000","last_became_active":"2024-06-27T21:34:35.709291+0000","last_became_peered":"2024-06-27T21:34:35.709291+0000","last_unstale":"2024-06-27T21:34:41.368913+0000","last_undegraded":"2024-06-27T21:34:41.368913+0000","last_fullsized":"2024-06-27T21:34:41.368913+0000","mapping_epoch":571,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'5888","last_scrub_stamp":"2024-06-27T21:30:45.481539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:45.481539+0000","objects_scrubbed":774,"log_size":10831,"log_dups_size":0,"ondisk_log_size":10831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":1590,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5131,"num_write_kb":214880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6723,"num_bytes_recovered":290750464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":569,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876554+0000","last_change":"2024-06-27T21:33:03.234800+0000","last_active":"2024-06-27T21:34:38.876554+0000","last_peered":"2024-06-27T21:34:38.876554+0000","last_clean":"2024-06-27T21:34:38.876554+0000","last_became_active":"2024-06-27T21:33:01.779398+0000","last_became_peered":"2024-06-27T21:33:01.779398+0000","last_unstale":"2024-06-27T21:34:38.876554+0000","last_undegraded":"2024-06-27T21:34:38.876554+0000","last_fullsized":"2024-06-27T21:34:38.876554+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:33:03.234726+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.11","version":"573'9741","reported_seq":29717,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:41.293556+0000","last_change":"2024-06-27T21:34:22.233121+0000","last_active":"2024-06-27T21:34:41.293556+0000","last_peered":"2024-06-27T21:34:41.293556+0000","last_clean":"2024-06-27T21:34:41.293556+0000","last_became_active":"2024-06-27T21:34:21.788051+0000","last_became_peered":"2024-06-27T21:34:21.788051+0000","last_unstale":"2024-06-27T21:34:41.293556+0000","last_undegraded":"2024-06-27T21:34:41.293556+0000","last_fullsized":"2024-06-27T21:34:41.293556+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"569'9717","last_scrub_stamp":"2024-06-27T21:34:22.232987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:22.232987+0000","objects_scrubbed":1552,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:43.717699+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100073472,"num_objects":1527,"num_object_clones":0,"num_object_copies":4581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":89,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.828857+0000","last_change":"2024-06-27T21:33:02.510081+0000","last_active":"2024-06-27T21:34:38.828857+0000","last_peered":"2024-06-27T21:34:38.828857+0000","last_clean":"2024-06-27T21:34:38.828857+0000","last_became_active":"2024-06-27T21:33:01.775355+0000","last_became_peered":"2024-06-27T21:33:01.775355+0000","last_unstale":"2024-06-27T21:34:38.828857+0000","last_undegraded":"2024-06-27T21:34:38.828857+0000","last_fullsized":"2024-06-27T21:34:38.828857+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.509969+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.509969+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"570'9840","reported_seq":10187,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875900+0000","last_change":"2024-06-27T21:33:51.180577+0000","last_active":"2024-06-27T21:34:38.875900+0000","last_peered":"2024-06-27T21:34:38.875900+0000","last_clean":"2024-06-27T21:34:38.875900+0000","last_became_active":"2024-06-27T21:33:27.698780+0000","last_became_peered":"2024-06-27T21:33:27.698780+0000","last_unstale":"2024-06-27T21:34:38.875900+0000","last_undegraded":"2024-06-27T21:34:38.875900+0000","last_fullsized":"2024-06-27T21:34:38.875900+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'9712","last_scrub_stamp":"2024-06-27T21:33:51.180247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.180247+0000","objects_scrubbed":1680,"log_size":9840,"log_dups_size":0,"ondisk_log_size":9840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:57.699175+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101711872,"num_objects":1552,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2],"acting":[11,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":89,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.831348+0000","last_change":"2024-06-27T21:33:02.531114+0000","last_active":"2024-06-27T21:34:38.831348+0000","last_peered":"2024-06-27T21:34:38.831348+0000","last_clean":"2024-06-27T21:34:38.831348+0000","last_became_active":"2024-06-27T21:33:01.810521+0000","last_became_peered":"2024-06-27T21:33:01.810521+0000","last_unstale":"2024-06-27T21:34:38.831348+0000","last_undegraded":"2024-06-27T21:34:38.831348+0000","last_fullsized":"2024-06-27T21:34:38.831348+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"570'10207","reported_seq":25835,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875926+0000","last_change":"2024-06-27T21:34:19.641926+0000","last_active":"2024-06-27T21:34:38.875926+0000","last_peered":"2024-06-27T21:34:38.875926+0000","last_clean":"2024-06-27T21:34:38.875926+0000","last_became_active":"2024-06-27T21:34:18.803849+0000","last_became_peered":"2024-06-27T21:34:18.803849+0000","last_unstale":"2024-06-27T21:34:38.875926+0000","last_undegraded":"2024-06-27T21:34:38.875926+0000","last_fullsized":"2024-06-27T21:34:38.875926+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":5,"last_scrub":"566'10180","last_scrub_stamp":"2024-06-27T21:34:19.641768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:19.641768+0000","objects_scrubbed":1673,"log_size":10207,"log_dups_size":0,"ondisk_log_size":10207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:29.432904+0000","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":1642,"num_object_clones":0,"num_object_copies":4926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":221,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876971+0000","last_change":"2024-06-27T21:31:19.010738+0000","last_active":"2024-06-27T21:34:38.876971+0000","last_peered":"2024-06-27T21:34:38.876971+0000","last_clean":"2024-06-27T21:34:38.876971+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:34:38.876971+0000","last_undegraded":"2024-06-27T21:34:38.876971+0000","last_fullsized":"2024-06-27T21:34:38.876971+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.010614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.010614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"570'10330","reported_seq":10642,"reported_epoch":573,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-06-27T21:34:38.828785+0000","last_change":"2024-06-27T21:33:58.995450+0000","last_active":"2024-06-27T21:34:38.828785+0000","last_peered":"2024-06-27T21:34:38.828785+0000","last_clean":"2024-06-27T21:33:25.735988+0000","last_became_active":"2024-06-27T21:33:28.958742+0000","last_became_peered":"2024-06-27T21:33:28.958742+0000","last_unstale":"2024-06-27T21:34:38.828785+0000","last_undegraded":"2024-06-27T21:34:38.828785+0000","last_fullsized":"2024-06-27T21:34:38.828785+0000","mapping_epoch":550,"log_start":"470'100","ondisk_log_start":"470'100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":10230,"log_dups_size":0,"ondisk_log_size":10230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106135552,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4860,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5278,"num_write_kb":220704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[1,5,9],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5,9","objects":1620}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":221,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876143+0000","last_change":"2024-06-27T21:31:19.025719+0000","last_active":"2024-06-27T21:34:38.876143+0000","last_peered":"2024-06-27T21:34:38.876143+0000","last_clean":"2024-06-27T21:34:38.876143+0000","last_became_active":"2024-06-27T21:29:51.673661+0000","last_became_peered":"2024-06-27T21:29:51.673661+0000","last_unstale":"2024-06-27T21:34:38.876143+0000","last_undegraded":"2024-06-27T21:34:38.876143+0000","last_fullsized":"2024-06-27T21:34:38.876143+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.15","version":"573'9963","reported_seq":17387,"reported_epoch":573,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:34:43.408513+0000","last_change":"2024-06-27T21:33:28.935649+0000","last_active":"2024-06-27T21:34:43.408513+0000","last_peered":"2024-06-27T21:34:43.408513+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:28.917741+0000","last_became_peered":"2024-06-27T21:33:28.917741+0000","last_unstale":"2024-06-27T21:34:43.408513+0000","last_undegraded":"2024-06-27T21:33:28.805628+0000","last_fullsized":"2024-06-27T21:33:28.693311+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":9963,"log_dups_size":0,"ondisk_log_size":9963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99778560,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":121372672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,10],"acting":[7,4],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,6,7,9","objects":475},{"shards":"4,7","objects":48},{"shards":"6,7,9","objects":1000}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":237,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:39.501548+0000","last_change":"2024-06-27T21:34:02.673681+0000","last_active":"2024-06-27T21:34:39.501548+0000","last_peered":"2024-06-27T21:34:39.501548+0000","last_clean":"2024-06-27T21:34:39.501548+0000","last_became_active":"2024-06-27T21:29:51.676689+0000","last_became_peered":"2024-06-27T21:29:51.676689+0000","last_unstale":"2024-06-27T21:34:39.501548+0000","last_undegraded":"2024-06-27T21:34:39.501548+0000","last_fullsized":"2024-06-27T21:34:39.501548+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:34:02.673610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:34:02.673610+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:07.427971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"573'10023","reported_seq":30928,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:41.058025+0000","last_change":"2024-06-27T21:34:06.592362+0000","last_active":"2024-06-27T21:34:41.058025+0000","last_peered":"2024-06-27T21:34:41.058025+0000","last_clean":"2024-06-27T21:34:41.058025+0000","last_became_active":"2024-06-27T21:34:05.705454+0000","last_became_peered":"2024-06-27T21:34:05.705454+0000","last_unstale":"2024-06-27T21:34:41.058025+0000","last_undegraded":"2024-06-27T21:34:41.058025+0000","last_fullsized":"2024-06-27T21:34:41.058025+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":5,"last_scrub":"562'9943","last_scrub_stamp":"2024-06-27T21:34:06.592264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:06.592264+0000","objects_scrubbed":1716,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:08.150584+0000","scrub_duration":830,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107085824,"num_objects":1634,"num_object_clones":0,"num_object_copies":4902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":229,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.849136+0000","last_change":"2024-06-27T21:32:13.808607+0000","last_active":"2024-06-27T21:34:38.849136+0000","last_peered":"2024-06-27T21:34:38.849136+0000","last_clean":"2024-06-27T21:34:38.849136+0000","last_became_active":"2024-06-27T21:32:13.808038+0000","last_became_peered":"2024-06-27T21:32:13.808038+0000","last_unstale":"2024-06-27T21:34:38.849136+0000","last_undegraded":"2024-06-27T21:34:38.849136+0000","last_fullsized":"2024-06-27T21:34:38.849136+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1],"acting":[0,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"573'9923","reported_seq":17635,"reported_epoch":573,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:34:42.421515+0000","last_change":"2024-06-27T21:34:42.422497+0000","last_active":"2024-06-27T21:34:42.421515+0000","last_peered":"2024-06-27T21:34:42.421515+0000","last_clean":"2024-06-27T21:33:26.047542+0000","last_became_active":"2024-06-27T21:33:29.082411+0000","last_became_peered":"2024-06-27T21:33:29.082411+0000","last_unstale":"2024-06-27T21:34:42.421515+0000","last_undegraded":"2024-06-27T21:34:42.421515+0000","last_fullsized":"2024-06-27T21:33:28.692997+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"527'9415","last_scrub_stamp":"2024-06-27T21:32:25.997194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:25.997194+0000","objects_scrubbed":3847,"log_size":9923,"log_dups_size":0,"ondisk_log_size":9923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1334,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101646336,"num_objects":1551,"num_object_clones":0,"num_object_copies":4653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5080,"num_write_kb":212192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":115736576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[3,2],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"2,3","objects":1550},{"shards":"10,11","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":250,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.830640+0000","last_change":"2024-06-27T21:32:13.808094+0000","last_active":"2024-06-27T21:34:38.830640+0000","last_peered":"2024-06-27T21:34:38.830640+0000","last_clean":"2024-06-27T21:34:38.830640+0000","last_became_active":"2024-06-27T21:32:13.807596+0000","last_became_peered":"2024-06-27T21:32:13.807596+0000","last_unstale":"2024-06-27T21:34:38.830640+0000","last_undegraded":"2024-06-27T21:34:38.830640+0000","last_fullsized":"2024-06-27T21:34:38.830640+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":523,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876375+0000","last_change":"2024-06-27T21:33:00.167722+0000","last_active":"2024-06-27T21:34:38.876375+0000","last_peered":"2024-06-27T21:34:38.876375+0000","last_clean":"2024-06-27T21:34:38.876375+0000","last_became_active":"2024-06-27T21:32:59.709162+0000","last_became_peered":"2024-06-27T21:32:59.709162+0000","last_unstale":"2024-06-27T21:34:38.876375+0000","last_undegraded":"2024-06-27T21:34:38.876375+0000","last_fullsized":"2024-06-27T21:34:38.876375+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.167617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.167617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"573'10366","reported_seq":10666,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:43.416711+0000","last_change":"2024-06-27T21:30:52.123816+0000","last_active":"2024-06-27T21:34:43.416711+0000","last_peered":"2024-06-27T21:34:43.416711+0000","last_clean":"2024-06-27T21:34:43.416711+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:34:43.416711+0000","last_undegraded":"2024-06-27T21:34:43.416711+0000","last_fullsized":"2024-06-27T21:34:43.416711+0000","mapping_epoch":467,"log_start":"471'300","ondisk_log_start":"471'300","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6360","last_scrub_stamp":"2024-06-27T21:30:52.123515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.123515+0000","objects_scrubbed":6329,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210108416,"num_objects":3206,"num_object_clones":0,"num_object_copies":9618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10366,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":311,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.830879+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:34:38.830879+0000","last_peered":"2024-06-27T21:34:38.830879+0000","last_clean":"2024-06-27T21:34:38.830879+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:34:38.830879+0000","last_undegraded":"2024-06-27T21:34:38.830879+0000","last_fullsized":"2024-06-27T21:34:38.830879+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"573'10381","reported_seq":10678,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:43.511959+0000","last_change":"2024-06-27T21:30:48.343185+0000","last_active":"2024-06-27T21:34:43.511959+0000","last_peered":"2024-06-27T21:34:43.511959+0000","last_clean":"2024-06-27T21:34:43.511959+0000","last_became_active":"2024-06-27T21:29:19.821666+0000","last_became_peered":"2024-06-27T21:29:19.821666+0000","last_unstale":"2024-06-27T21:34:43.511959+0000","last_undegraded":"2024-06-27T21:34:43.511959+0000","last_fullsized":"2024-06-27T21:34:43.511959+0000","mapping_epoch":467,"log_start":"471'300","ondisk_log_start":"471'300","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6100","last_scrub_stamp":"2024-06-27T21:30:48.342993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.342993+0000","objects_scrubbed":6058,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208076800,"num_objects":3175,"num_object_clones":0,"num_object_copies":9525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10381,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":153,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.829705+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:34:38.829705+0000","last_peered":"2024-06-27T21:34:38.829705+0000","last_clean":"2024-06-27T21:34:38.829705+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:34:38.829705+0000","last_undegraded":"2024-06-27T21:34:38.829705+0000","last_fullsized":"2024-06-27T21:34:38.829705+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"570'10233","reported_seq":24049,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875794+0000","last_change":"2024-06-27T21:33:28.733557+0000","last_active":"2024-06-27T21:34:38.875794+0000","last_peered":"2024-06-27T21:34:38.875794+0000","last_clean":"2024-06-27T21:34:38.875794+0000","last_became_active":"2024-06-27T21:33:28.698375+0000","last_became_peered":"2024-06-27T21:33:28.698375+0000","last_unstale":"2024-06-27T21:34:38.875794+0000","last_undegraded":"2024-06-27T21:34:38.875794+0000","last_fullsized":"2024-06-27T21:34:38.875794+0000","mapping_epoch":550,"log_start":"471'200","ondisk_log_start":"471'200","created":467,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214302744,"num_objects":3271,"num_object_clones":0,"num_object_copies":9813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10233,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6752,"num_bytes_recovered":333316120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1071,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876020+0000","last_change":"2024-06-27T21:32:13.809592+0000","last_active":"2024-06-27T21:34:38.876020+0000","last_peered":"2024-06-27T21:34:38.876020+0000","last_clean":"2024-06-27T21:34:38.876020+0000","last_became_active":"2024-06-27T21:32:13.809197+0000","last_became_peered":"2024-06-27T21:32:13.809197+0000","last_unstale":"2024-06-27T21:34:38.876020+0000","last_undegraded":"2024-06-27T21:34:38.876020+0000","last_fullsized":"2024-06-27T21:34:38.876020+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"573'10212","reported_seq":16309,"reported_epoch":573,"state":"active+recovering","last_fresh":"2024-06-27T21:34:43.250686+0000","last_change":"2024-06-27T21:33:26.890778+0000","last_active":"2024-06-27T21:34:43.250686+0000","last_peered":"2024-06-27T21:34:43.250686+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:33:26.864503+0000","last_became_peered":"2024-06-27T21:33:26.864503+0000","last_unstale":"2024-06-27T21:34:43.250686+0000","last_undegraded":"2024-06-27T21:34:43.250686+0000","last_fullsized":"2024-06-27T21:34:43.250686+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":10212,"log_dups_size":0,"ondisk_log_size":10212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104529920,"num_objects":1595,"num_object_clones":0,"num_object_copies":4785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":918,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":218080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":110297088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"0,7,11","objects":677},{"shards":"6,7,11","objects":918}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":648,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876235+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:34:38.876235+0000","last_peered":"2024-06-27T21:34:38.876235+0000","last_clean":"2024-06-27T21:34:38.876235+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:34:38.876235+0000","last_undegraded":"2024-06-27T21:34:38.876235+0000","last_fullsized":"2024-06-27T21:34:38.876235+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"573'9885","reported_seq":10236,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:43.461697+0000","last_change":"2024-06-27T21:33:54.377162+0000","last_active":"2024-06-27T21:34:43.461697+0000","last_peered":"2024-06-27T21:34:43.461697+0000","last_clean":"2024-06-27T21:34:43.461697+0000","last_became_active":"2024-06-27T21:33:25.695260+0000","last_became_peered":"2024-06-27T21:33:25.695260+0000","last_unstale":"2024-06-27T21:34:43.461697+0000","last_undegraded":"2024-06-27T21:34:43.461697+0000","last_fullsized":"2024-06-27T21:34:43.461697+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9736","last_scrub_stamp":"2024-06-27T21:33:54.376905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.376905+0000","objects_scrubbed":1765,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:55.143355+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105709568,"num_objects":1613,"num_object_clones":0,"num_object_copies":4839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":644,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876226+0000","last_change":"2024-06-27T21:33:01.182699+0000","last_active":"2024-06-27T21:34:38.876226+0000","last_peered":"2024-06-27T21:34:38.876226+0000","last_clean":"2024-06-27T21:34:38.876226+0000","last_became_active":"2024-06-27T21:32:59.711777+0000","last_became_peered":"2024-06-27T21:32:59.711777+0000","last_unstale":"2024-06-27T21:34:38.876226+0000","last_undegraded":"2024-06-27T21:34:38.876226+0000","last_fullsized":"2024-06-27T21:34:38.876226+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"570'9723","reported_seq":21292,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:39.572821+0000","last_change":"2024-06-27T21:33:48.555029+0000","last_active":"2024-06-27T21:34:39.572821+0000","last_peered":"2024-06-27T21:34:39.572821+0000","last_clean":"2024-06-27T21:34:39.572821+0000","last_became_active":"2024-06-27T21:33:35.694586+0000","last_became_peered":"2024-06-27T21:33:35.694586+0000","last_unstale":"2024-06-27T21:34:39.572821+0000","last_undegraded":"2024-06-27T21:34:39.572821+0000","last_fullsized":"2024-06-27T21:34:39.572821+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'9582","last_scrub_stamp":"2024-06-27T21:33:48.554929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:48.554929+0000","objects_scrubbed":1728,"log_size":9723,"log_dups_size":0,"ondisk_log_size":9723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:03.341335+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874560,"num_objects":1585,"num_object_clones":0,"num_object_copies":4755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,7],"acting":[9,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":564,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876621+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:34:38.876621+0000","last_peered":"2024-06-27T21:34:38.876621+0000","last_clean":"2024-06-27T21:34:38.876621+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:34:38.876621+0000","last_undegraded":"2024-06-27T21:34:38.876621+0000","last_fullsized":"2024-06-27T21:34:38.876621+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"573'10069","reported_seq":23848,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:41.124439+0000","last_change":"2024-06-27T21:33:54.883116+0000","last_active":"2024-06-27T21:34:41.124439+0000","last_peered":"2024-06-27T21:34:41.124439+0000","last_clean":"2024-06-27T21:34:41.124439+0000","last_became_active":"2024-06-27T21:33:25.693582+0000","last_became_peered":"2024-06-27T21:33:25.693582+0000","last_unstale":"2024-06-27T21:34:41.124439+0000","last_undegraded":"2024-06-27T21:34:41.124439+0000","last_fullsized":"2024-06-27T21:34:41.124439+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":1712,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:18.128885+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":642,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.875773+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:34:38.875773+0000","last_peered":"2024-06-27T21:34:38.875773+0000","last_clean":"2024-06-27T21:34:38.875773+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:34:38.875773+0000","last_undegraded":"2024-06-27T21:34:38.875773+0000","last_fullsized":"2024-06-27T21:34:38.875773+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"573'10030","reported_seq":24180,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:42.431953+0000","last_change":"2024-06-27T21:33:50.968993+0000","last_active":"2024-06-27T21:34:42.431953+0000","last_peered":"2024-06-27T21:34:42.431953+0000","last_clean":"2024-06-27T21:34:42.431953+0000","last_became_active":"2024-06-27T21:33:25.692464+0000","last_became_peered":"2024-06-27T21:33:25.692464+0000","last_unstale":"2024-06-27T21:34:42.431953+0000","last_undegraded":"2024-06-27T21:34:42.431953+0000","last_fullsized":"2024-06-27T21:34:42.431953+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'9882","last_scrub_stamp":"2024-06-27T21:33:50.968852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.968852+0000","objects_scrubbed":1716,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:17.284729+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102760448,"num_objects":1568,"num_object_clones":0,"num_object_copies":4704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":148,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":317,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.830823+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:34:38.830823+0000","last_peered":"2024-06-27T21:34:38.830823+0000","last_clean":"2024-06-27T21:34:38.830823+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:34:38.830823+0000","last_undegraded":"2024-06-27T21:34:38.830823+0000","last_fullsized":"2024-06-27T21:34:38.830823+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"573'9987","reported_seq":17818,"reported_epoch":573,"state":"active+recovering","last_fresh":"2024-06-27T21:34:43.306515+0000","last_change":"2024-06-27T21:33:26.778860+0000","last_active":"2024-06-27T21:34:43.306515+0000","last_peered":"2024-06-27T21:34:43.306515+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:26.702107+0000","last_became_peered":"2024-06-27T21:33:26.702107+0000","last_unstale":"2024-06-27T21:34:43.306515+0000","last_undegraded":"2024-06-27T21:34:43.306515+0000","last_fullsized":"2024-06-27T21:34:43.306515+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":9987,"log_dups_size":0,"ondisk_log_size":9987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98205696,"num_objects":1498,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":479,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":142213120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,7,9","objects":1019},{"shards":"6,7,9","objects":479}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":202,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.829624+0000","last_change":"2024-06-27T21:32:13.812610+0000","last_active":"2024-06-27T21:34:38.829624+0000","last_peered":"2024-06-27T21:34:38.829624+0000","last_clean":"2024-06-27T21:34:38.829624+0000","last_became_active":"2024-06-27T21:32:13.812217+0000","last_became_peered":"2024-06-27T21:32:13.812217+0000","last_unstale":"2024-06-27T21:34:38.829624+0000","last_undegraded":"2024-06-27T21:34:38.829624+0000","last_fullsized":"2024-06-27T21:34:38.829624+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,10],"acting":[4,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"570'10328","reported_seq":10673,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.828858+0000","last_change":"2024-06-27T21:33:50.128755+0000","last_active":"2024-06-27T21:34:38.828858+0000","last_peered":"2024-06-27T21:34:38.828858+0000","last_clean":"2024-06-27T21:34:38.828858+0000","last_became_active":"2024-06-27T21:33:25.695186+0000","last_became_peered":"2024-06-27T21:33:25.695186+0000","last_unstale":"2024-06-27T21:34:38.828858+0000","last_undegraded":"2024-06-27T21:34:38.828858+0000","last_fullsized":"2024-06-27T21:34:38.828858+0000","mapping_epoch":467,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'10169","last_scrub_stamp":"2024-06-27T21:33:50.128579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.128579+0000","objects_scrubbed":1771,"log_size":10228,"log_dups_size":0,"ondisk_log_size":10228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:34:56.697028+0000","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,9],"acting":[1,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":31,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:39.572712+0000","last_change":"2024-06-27T21:34:19.947020+0000","last_active":"2024-06-27T21:34:39.572712+0000","last_peered":"2024-06-27T21:34:39.572712+0000","last_clean":"2024-06-27T21:34:39.572712+0000","last_became_active":"2024-06-27T21:34:19.712448+0000","last_became_peered":"2024-06-27T21:34:19.712448+0000","last_unstale":"2024-06-27T21:34:39.572712+0000","last_undegraded":"2024-06-27T21:34:39.572712+0000","last_fullsized":"2024-06-27T21:34:39.572712+0000","mapping_epoch":566,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":567,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:34:19.946945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:34:19.946945+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:49.544208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.7","version":"573'9904","reported_seq":10292,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:42.662228+0000","last_change":"2024-06-27T21:33:49.922777+0000","last_active":"2024-06-27T21:34:42.662228+0000","last_peered":"2024-06-27T21:34:42.662228+0000","last_clean":"2024-06-27T21:34:42.662228+0000","last_became_active":"2024-06-27T21:33:25.697304+0000","last_became_peered":"2024-06-27T21:33:25.697304+0000","last_unstale":"2024-06-27T21:34:42.662228+0000","last_undegraded":"2024-06-27T21:34:42.662228+0000","last_fullsized":"2024-06-27T21:34:42.662228+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9759","last_scrub_stamp":"2024-06-27T21:33:49.922636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.922636+0000","objects_scrubbed":1732,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:10.096662+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005632,"num_objects":1587,"num_object_clones":0,"num_object_copies":4761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":837,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:39.501585+0000","last_change":"2024-06-27T21:33:51.930934+0000","last_active":"2024-06-27T21:34:39.501585+0000","last_peered":"2024-06-27T21:34:39.501585+0000","last_clean":"2024-06-27T21:34:39.501585+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:34:39.501585+0000","last_undegraded":"2024-06-27T21:34:39.501585+0000","last_fullsized":"2024-06-27T21:34:39.501585+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:51.930851+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.930851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:00.975914+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"573'10107","reported_seq":10450,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:43.304679+0000","last_change":"2024-06-27T21:33:49.851182+0000","last_active":"2024-06-27T21:34:43.304679+0000","last_peered":"2024-06-27T21:34:43.304679+0000","last_clean":"2024-06-27T21:34:43.304679+0000","last_became_active":"2024-06-27T21:33:25.736643+0000","last_became_peered":"2024-06-27T21:33:25.736643+0000","last_unstale":"2024-06-27T21:34:43.304679+0000","last_undegraded":"2024-06-27T21:34:43.304679+0000","last_fullsized":"2024-06-27T21:34:43.304679+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9962","last_scrub_stamp":"2024-06-27T21:33:49.850981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.850981+0000","objects_scrubbed":1778,"log_size":10107,"log_dups_size":0,"ondisk_log_size":10107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:08.929859+0000","scrub_duration":516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":4899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":234,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.829868+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:34:38.829868+0000","last_peered":"2024-06-27T21:34:38.829868+0000","last_clean":"2024-06-27T21:34:38.829868+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:34:38.829868+0000","last_undegraded":"2024-06-27T21:34:38.829868+0000","last_fullsized":"2024-06-27T21:34:38.829868+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"573'9984","reported_seq":17279,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:41.065052+0000","last_change":"2024-06-27T21:34:00.348871+0000","last_active":"2024-06-27T21:34:41.065052+0000","last_peered":"2024-06-27T21:34:41.065052+0000","last_clean":"2024-06-27T21:34:41.065052+0000","last_became_active":"2024-06-27T21:33:26.705196+0000","last_became_peered":"2024-06-27T21:33:26.705196+0000","last_unstale":"2024-06-27T21:34:41.065052+0000","last_undegraded":"2024-06-27T21:34:41.065052+0000","last_fullsized":"2024-06-27T21:34:41.065052+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9854","last_scrub_stamp":"2024-06-27T21:34:00.348580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:00.348580+0000","objects_scrubbed":1790,"log_size":9984,"log_dups_size":0,"ondisk_log_size":9984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:35:25.852625+0000","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":226,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.831455+0000","last_change":"2024-06-27T21:33:01.399273+0000","last_active":"2024-06-27T21:34:38.831455+0000","last_peered":"2024-06-27T21:34:38.831455+0000","last_clean":"2024-06-27T21:34:38.831455+0000","last_became_active":"2024-06-27T21:32:59.709705+0000","last_became_peered":"2024-06-27T21:32:59.709705+0000","last_unstale":"2024-06-27T21:34:38.831455+0000","last_undegraded":"2024-06-27T21:34:38.831455+0000","last_fullsized":"2024-06-27T21:34:38.831455+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"573'10234","reported_seq":23943,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:41.330480+0000","last_change":"2024-06-27T21:33:23.873709+0000","last_active":"2024-06-27T21:34:41.330480+0000","last_peered":"2024-06-27T21:34:41.330480+0000","last_clean":"2024-06-27T21:34:41.330480+0000","last_became_active":"2024-06-27T21:33:23.851489+0000","last_became_peered":"2024-06-27T21:33:23.851489+0000","last_unstale":"2024-06-27T21:34:41.330480+0000","last_undegraded":"2024-06-27T21:34:41.330480+0000","last_fullsized":"2024-06-27T21:34:41.330480+0000","mapping_epoch":545,"log_start":"471'200","ondisk_log_start":"471'200","created":467,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207355904,"num_objects":3164,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6699,"num_bytes_recovered":340197376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"573'10107","reported_seq":10446,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:42.399199+0000","last_change":"2024-06-27T21:32:26.025196+0000","last_active":"2024-06-27T21:34:42.399199+0000","last_peered":"2024-06-27T21:34:42.399199+0000","last_clean":"2024-06-27T21:34:42.399199+0000","last_became_active":"2024-06-27T21:29:19.819725+0000","last_became_peered":"2024-06-27T21:29:19.819725+0000","last_unstale":"2024-06-27T21:34:42.399199+0000","last_undegraded":"2024-06-27T21:34:42.399199+0000","last_fullsized":"2024-06-27T21:34:42.399199+0000","mapping_epoch":467,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'9389","last_scrub_stamp":"2024-06-27T21:32:26.024974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:32:26.024974+0000","objects_scrubbed":3795,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201654272,"num_objects":3077,"num_object_clones":0,"num_object_copies":9231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10107,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":242,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.850445+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:34:38.850445+0000","last_peered":"2024-06-27T21:34:38.850445+0000","last_clean":"2024-06-27T21:34:38.850445+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:34:38.850445+0000","last_undegraded":"2024-06-27T21:34:38.850445+0000","last_fullsized":"2024-06-27T21:34:38.850445+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"573'10560","reported_seq":18291,"reported_epoch":573,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:34:43.475157+0000","last_change":"2024-06-27T21:32:36.645550+0000","last_active":"2024-06-27T21:34:43.475157+0000","last_peered":"2024-06-27T21:34:43.475157+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:32:34.904596+0000","last_became_peered":"2024-06-27T21:32:34.904596+0000","last_unstale":"2024-06-27T21:34:43.475157+0000","last_undegraded":"2024-06-27T21:32:22.691354+0000","last_fullsized":"2024-06-27T21:32:22.682941+0000","mapping_epoch":526,"log_start":"471'500","ondisk_log_start":"471'500","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":4629,"num_objects_missing":4629,"num_objects_degraded":3,"num_objects_misplaced":9777,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10560,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2272,"num_bytes_recovered":148570112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10],"avail_no_missing":["5","6","8"],"object_location_counts":[{"shards":"4,5,6,7,8","objects":1},{"shards":"5,6,7,8","objects":10},{"shards":"5,6,8","objects":3252},{"shards":"7,10","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":240,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-06-27T21:34:38.876042+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:34:38.876042+0000","last_peered":"2024-06-27T21:34:38.876042+0000","last_clean":"2024-06-27T21:34:38.876042+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:34:38.876042+0000","last_undegraded":"2024-06-27T21:34:38.876042+0000","last_fullsized":"2024-06-27T21:34:38.876042+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":3339976728,"num_objects":50966,"num_object_clones":0,"num_object_copies":152898,"num_objects_missing_on_primary":4629,"num_objects_missing":4629,"num_objects_degraded":6396,"num_objects_misplaced":20520,"num_objects_unfound":0,"num_objects_dirty":50966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104498,"num_write_kb":4315105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31865,"num_bytes_recovered":1724940312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12171948032,"data_stored":12171935816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261236,"ondisk_log_size":261236,"up":78,"acting":73,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253169,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509732,"kb_used_data":923224,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92235420,"statfs":{"total":95995035648,"available":94449070080,"internally_reserved":0,"allocated":945381376,"data_stored":944944421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514805,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1306536,"kb_used_data":851932,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92438616,"statfs":{"total":95995035648,"available":94657142784,"internally_reserved":0,"allocated":872378368,"data_stored":871937341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":465500608},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743738,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460836,"kb_used_data":1014552,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92284316,"statfs":{"total":95995035648,"available":94499139584,"internally_reserved":0,"allocated":1038901248,"data_stored":1038464293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":266287972670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753192,"kb_used_data":1230560,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91991960,"statfs":{"total":95995035648,"available":94199767040,"internally_reserved":0,"allocated":1260093440,"data_stored":1259652933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":535165386},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528648,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1838244,"kb_used_data":1337816,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91906908,"statfs":{"total":95995035648,"available":94112673792,"internally_reserved":0,"allocated":1369923584,"data_stored":1369486629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757582,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161696,"kb_used_data":765400,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92583456,"statfs":{"total":95995035648,"available":94805458944,"internally_reserved":0,"allocated":783769600,"data_stored":783332645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161287,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1062176,"kb_used_data":826264,"kb_used_omap":92,"kb_used_meta":235811,"kb_avail":92682976,"statfs":{"total":95995035648,"available":94907367424,"internally_reserved":0,"allocated":846094336,"data_stored":845657381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":241470562},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182742,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1079712,"kb_used_data":705624,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92665440,"statfs":{"total":95995035648,"available":94889410560,"internally_reserved":0,"allocated":722558976,"data_stored":722122021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":383056327},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411675,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1873576,"kb_used_data":1477280,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91871576,"statfs":{"total":95995035648,"available":94076493824,"internally_reserved":0,"allocated":1512734720,"data_stored":1512294213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575199,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1682848,"kb_used_data":1119960,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92062304,"statfs":{"total":95995035648,"available":94271799296,"internally_reserved":0,"allocated":1146839040,"data_stored":1146398798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738724,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1688292,"kb_used_data":1129564,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92056860,"statfs":{"total":95995035648,"available":94266224640,"internally_reserved":0,"allocated":1156673536,"data_stored":1156232509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1153241088,"data_stored":1153237016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1147932672,"data_stored":1147928600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1511063552,"data_stored":1511063552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":719126528,"data_stored":719126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":844038144,"data_stored":844038144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":782368768,"data_stored":782368768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1368391680,"data_stored":1368391680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1258487808,"data_stored":1258487808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":527106048,"data_stored":527106048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1036910592,"data_stored":1036910592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":875630592,"data_stored":875626520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":947650560,"data_stored":947650560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:35:05.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-06-27T21:35:05.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:05.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:05.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:05.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:05.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:06.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:06.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:06.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:06.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:06.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:06.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:06.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:06.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:06.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:06.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:06 smithi190 ceph-mon[32247]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:35:06.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:06 smithi192 ceph-mon[32956]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:35:06.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:06.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:06.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:06 smithi151 ceph-mon[28965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:35:06.711 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:06.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:06.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:06.793 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:06.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:06.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:06.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:06.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:07.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:07.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:07.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:07.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:07.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:07.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:07.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:07.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:07.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:07.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:07 smithi190 ceph-mon[32247]: pgmap v1515: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 122 op/s; 6267/145947 objects degraded (4.294%); 16440/145947 objects misplaced (11.264%); 4.8 MiB/s, 220 objects/s recovering 2024-06-27T21:35:07.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:07 smithi192 ceph-mon[32956]: pgmap v1515: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 122 op/s; 6267/145947 objects degraded (4.294%); 16440/145947 objects misplaced (11.264%); 4.8 MiB/s, 220 objects/s recovering 2024-06-27T21:35:07.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:07.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:07.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:07 smithi151 ceph-mon[28965]: pgmap v1515: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 122 op/s; 6267/145947 objects degraded (4.294%); 16440/145947 objects misplaced (11.264%); 4.8 MiB/s, 220 objects/s recovering 2024-06-27T21:35:07.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:07.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:07.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:07.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:07.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:07.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:07.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:08.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:08.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:08.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:08.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:08.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:08.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:08.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:08.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:08 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:08.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:35:08.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:35:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:08 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:08.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:35:08.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:35:08.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:08.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:08.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:08 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:08.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:35:08.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:35:08.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:08.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:08.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:08.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:08.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:08.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:08.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:09.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:09.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:09.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:09.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:09.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:09.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:09.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:09.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:09 smithi190 ceph-mon[32247]: pgmap v1516: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 123 op/s; 6267/145143 objects degraded (4.318%); 15768/145143 objects misplaced (10.864%); 4.2 MiB/s, 250 objects/s recovering 2024-06-27T21:35:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:09 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:35:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:09 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:35:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:09 smithi190 ceph-mon[32247]: osdmap e586: 12 total, 12 up, 11 in 2024-06-27T21:35:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:09 smithi192 ceph-mon[32956]: pgmap v1516: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 123 op/s; 6267/145143 objects degraded (4.318%); 15768/145143 objects misplaced (10.864%); 4.2 MiB/s, 250 objects/s recovering 2024-06-27T21:35:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:09 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:35:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:09 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:35:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:09 smithi192 ceph-mon[32956]: osdmap e586: 12 total, 12 up, 11 in 2024-06-27T21:35:09.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:09.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:09.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:09 smithi151 ceph-mon[28965]: pgmap v1516: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 3.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 123 op/s; 6267/145143 objects degraded (4.318%); 15768/145143 objects misplaced (10.864%); 4.2 MiB/s, 250 objects/s recovering 2024-06-27T21:35:09.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:09 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:35:09.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:09 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:35:09.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:09 smithi151 ceph-mon[28965]: osdmap e586: 12 total, 12 up, 11 in 2024-06-27T21:35:09.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:09.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:09.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:09.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:09.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:09.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:09.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:10.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:10.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:10.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:10.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:10.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:10.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:10.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:10.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:10 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6267/145143 objects degraded (4.318%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:10 smithi190 ceph-mon[32247]: osdmap e587: 12 total, 12 up, 11 in 2024-06-27T21:35:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:10 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6267/145143 objects degraded (4.318%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:10 smithi192 ceph-mon[32956]: osdmap e587: 12 total, 12 up, 11 in 2024-06-27T21:35:10.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:10.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:10.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:10 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6267/145143 objects degraded (4.318%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:10.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:10 smithi151 ceph-mon[28965]: osdmap e587: 12 total, 12 up, 11 in 2024-06-27T21:35:10.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:10.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:10.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:10.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:10.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:10.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:10.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:11.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:11.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:11.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:11.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:11.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:11.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:11.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:11.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:11.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:11.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:11.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:11.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:11.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:11.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:11.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:11.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:12.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:11 smithi190 ceph-mon[32247]: pgmap v1519: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 142 op/s; 6231/143913 objects degraded (4.330%); 15581/143913 objects misplaced (10.827%); 3.0 MiB/s, 168 objects/s recovering 2024-06-27T21:35:12.037 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:11 smithi151 ceph-mon[28965]: pgmap v1519: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 142 op/s; 6231/143913 objects degraded (4.330%); 15581/143913 objects misplaced (10.827%); 3.0 MiB/s, 168 objects/s recovering 2024-06-27T21:35:12.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:11 smithi192 ceph-mon[32956]: pgmap v1519: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 142 op/s; 6231/143913 objects degraded (4.330%); 15581/143913 objects misplaced (10.827%); 3.0 MiB/s, 168 objects/s recovering 2024-06-27T21:35:12.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:12.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:12.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:12.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:12.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:12.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:12.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:12.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:12.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:12.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:12.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:12.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:12.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:12.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:12.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:12.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:12.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:13.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:13.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:13.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:13.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:13.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:13.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:13.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:13.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:13.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:35:13.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:35:13.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:35:13.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:35:13.639 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:35:13.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:35:13.660 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:35:13.660 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:35:13.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:13.689 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:35:13.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:35:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:13.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:13.723 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:35:13.723 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:35:13.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:13 smithi151 ceph-mon[28965]: pgmap v1520: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 134 op/s; 6231/143355 objects degraded (4.347%); 15201/143355 objects misplaced (10.604%); 3.5 MiB/s, 181 objects/s recovering 2024-06-27T21:35:13.747 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/65357124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:13.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:35:13.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:35:13.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:13.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:13.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:13.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:13.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:14.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:13 smithi190 ceph-mon[32247]: pgmap v1520: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 134 op/s; 6231/143355 objects degraded (4.347%); 15201/143355 objects misplaced (10.604%); 3.5 MiB/s, 181 objects/s recovering 2024-06-27T21:35:14.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/65357124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:14.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:13 smithi192 ceph-mon[32956]: pgmap v1520: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 134 op/s; 6231/143355 objects degraded (4.347%); 15201/143355 objects misplaced (10.604%); 3.5 MiB/s, 181 objects/s recovering 2024-06-27T21:35:14.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/65357124' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:14.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:14.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:14.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:14.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:14.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:14.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:14.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:14.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:14.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:14.572 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:35:14.572 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:35:14.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:14.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:35:14.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:35:14.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:35:14.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:35:14.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:35:14.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:35:14.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:35:14.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:35:14.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:14.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:14.781 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:35:14.781 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:35:14.781 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:35:14.781 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:35:14.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:14.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:14.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:14.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:14 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:14.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6231/143355 objects degraded (4.347%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:14.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:14.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:15.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:14 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:15.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6231/143355 objects degraded (4.347%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:15.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:14 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:15.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6231/143355 objects degraded (4.347%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:15.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:15.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:15.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:15.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:15.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:15.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:15.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:15.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:15.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:15.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:15.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:15.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:15.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:15.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:15.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:15.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:15 smithi151 ceph-mon[28965]: pgmap v1521: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 146 op/s; 6199/142437 objects degraded (4.352%); 15082/142437 objects misplaced (10.589%); 3.7 MiB/s, 196 objects/s recovering 2024-06-27T21:35:15.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:15.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:15 smithi190 ceph-mon[32247]: pgmap v1521: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 146 op/s; 6199/142437 objects degraded (4.352%); 15082/142437 objects misplaced (10.589%); 3.7 MiB/s, 196 objects/s recovering 2024-06-27T21:35:16.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:15 smithi192 ceph-mon[32956]: pgmap v1521: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 146 op/s; 6199/142437 objects degraded (4.352%); 15082/142437 objects misplaced (10.589%); 3.7 MiB/s, 196 objects/s recovering 2024-06-27T21:35:16.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:16.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:16.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:16.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:16.237 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:16.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:16.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:16.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:16.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:16.549 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:16.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:16.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:16.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:16.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:35:16.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:16.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:16.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:16.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:16.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:17.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:17.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:17.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:17.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:17.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:17.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:17.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:17.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:17.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:17.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:17.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:17.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:17.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:17.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:17.916 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:17.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:17.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:17.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:17 smithi151 ceph-mon[28965]: pgmap v1522: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 6199/142047 objects degraded (4.364%); 15034/142047 objects misplaced (10.584%); 2.0 MiB/s, 120 objects/s recovering 2024-06-27T21:35:18.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:17 smithi190 ceph-mon[32247]: pgmap v1522: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 6199/142047 objects degraded (4.364%); 15034/142047 objects misplaced (10.584%); 2.0 MiB/s, 120 objects/s recovering 2024-06-27T21:35:18.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:18.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:17 smithi192 ceph-mon[32956]: pgmap v1522: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 6199/142047 objects degraded (4.364%); 15034/142047 objects misplaced (10.584%); 2.0 MiB/s, 120 objects/s recovering 2024-06-27T21:35:18.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:35:18.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:18.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:18.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:18.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:18.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:18.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:18.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:18.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:18.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:18.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:18.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:18.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:18.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:18.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:18.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:18.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:19.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:18 smithi190 ceph-mon[32247]: osdmap e588: 12 total, 12 up, 11 in 2024-06-27T21:35:19.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:18 smithi192 ceph-mon[32956]: osdmap e588: 12 total, 12 up, 11 in 2024-06-27T21:35:19.049 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:19.070 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:19.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:19.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:19.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:18 smithi151 ceph-mon[28965]: osdmap e588: 12 total, 12 up, 11 in 2024-06-27T21:35:19.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:19.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:19.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:19.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:19.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:19.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:19.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:19.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:19.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:19.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:19.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:19.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:20.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:19 smithi190 ceph-mon[32247]: pgmap v1524: 57 pgs: 1 active+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 112 op/s; 6012/141339 objects degraded (4.254%); 14845/141339 objects misplaced (10.503%); 1.4 MiB/s, 160 objects/s recovering 2024-06-27T21:35:20.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:19 smithi190 ceph-mon[32247]: osdmap e589: 12 total, 12 up, 11 in 2024-06-27T21:35:20.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:35:20.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:35:20.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6012/141339 objects degraded (4.254%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:20.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:19 smithi192 ceph-mon[32956]: pgmap v1524: 57 pgs: 1 active+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 112 op/s; 6012/141339 objects degraded (4.254%); 14845/141339 objects misplaced (10.503%); 1.4 MiB/s, 160 objects/s recovering 2024-06-27T21:35:20.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:19 smithi192 ceph-mon[32956]: osdmap e589: 12 total, 12 up, 11 in 2024-06-27T21:35:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:35:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:35:20.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6012/141339 objects degraded (4.254%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:20.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:20.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:20.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:19 smithi151 ceph-mon[28965]: pgmap v1524: 57 pgs: 1 active+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 112 op/s; 6012/141339 objects degraded (4.254%); 14845/141339 objects misplaced (10.503%); 1.4 MiB/s, 160 objects/s recovering 2024-06-27T21:35:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:19 smithi151 ceph-mon[28965]: osdmap e589: 12 total, 12 up, 11 in 2024-06-27T21:35:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:35:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:35:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6012/141339 objects degraded (4.254%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:20.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:20.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:20.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:20.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:20.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:20.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:20.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:20.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:20.757 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:35:20.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:20.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:20.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:20.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:20.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:21.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:20 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:21.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:20 smithi190 ceph-mon[32247]: 6.18 scrub starts 2024-06-27T21:35:21.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:20 smithi190 ceph-mon[32247]: osdmap e590: 12 total, 12 up, 11 in 2024-06-27T21:35:21.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3746789330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:21.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:21.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:20 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:21.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:20 smithi192 ceph-mon[32956]: 6.18 scrub starts 2024-06-27T21:35:21.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:20 smithi192 ceph-mon[32956]: osdmap e590: 12 total, 12 up, 11 in 2024-06-27T21:35:21.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3746789330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:21.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:21.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:21.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:20 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:20 smithi151 ceph-mon[28965]: 6.18 scrub starts 2024-06-27T21:35:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:20 smithi151 ceph-mon[28965]: osdmap e590: 12 total, 12 up, 11 in 2024-06-27T21:35:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3746789330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:21.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:21.302 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:21.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:21.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:21.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:21.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:21.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:21.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:21.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:21.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:21.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:21.923 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:21.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:21 smithi151 ceph-mon[28965]: pgmap v1527: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 131 op/s; 5982/140076 objects degraded (4.271%); 14644/140076 objects misplaced (10.454%); 896 KiB/s, 89 objects/s recovering 2024-06-27T21:35:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:21 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:35:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3746789330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:35:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:21 smithi151 ceph-mon[28965]: osdmap e591: 12 total, 12 up, 11 in 2024-06-27T21:35:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:21 smithi190 ceph-mon[32247]: pgmap v1527: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 131 op/s; 5982/140076 objects degraded (4.271%); 14644/140076 objects misplaced (10.454%); 896 KiB/s, 89 objects/s recovering 2024-06-27T21:35:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:21 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:35:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3746789330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:35:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:21 smithi190 ceph-mon[32247]: osdmap e591: 12 total, 12 up, 11 in 2024-06-27T21:35:22.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:21 smithi192 ceph-mon[32956]: pgmap v1527: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 131 op/s; 5982/140076 objects degraded (4.271%); 14644/140076 objects misplaced (10.454%); 896 KiB/s, 89 objects/s recovering 2024-06-27T21:35:22.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:21 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:35:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3746789330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:35:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:21 smithi192 ceph-mon[32956]: osdmap e591: 12 total, 12 up, 11 in 2024-06-27T21:35:22.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:22.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:22.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:22.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:22.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:22.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:22.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:22.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:22.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:22.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:22.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:22.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:22.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:22.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:22.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:22.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:22.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:23.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:23.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:23.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:23.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:23.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:23.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:23.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:23.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:23.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:23.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:23.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:23.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:23.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:23.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:23.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:23.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:23.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:23.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:24.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:23 smithi190 ceph-mon[32247]: pgmap v1529: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 134 op/s; 5982/139629 objects degraded (4.284%); 14644/139629 objects misplaced (10.488%); 1.1 MiB/s, 102 objects/s recovering 2024-06-27T21:35:24.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:24.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:24.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:23 smithi192 ceph-mon[32956]: pgmap v1529: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 134 op/s; 5982/139629 objects degraded (4.284%); 14644/139629 objects misplaced (10.488%); 1.1 MiB/s, 102 objects/s recovering 2024-06-27T21:35:24.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:24.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:23 smithi151 ceph-mon[28965]: pgmap v1529: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 134 op/s; 5982/139629 objects degraded (4.284%); 14644/139629 objects misplaced (10.488%); 1.1 MiB/s, 102 objects/s recovering 2024-06-27T21:35:24.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:24.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:24.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:24.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:24.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:24.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:24.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:24.664 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:24.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:24.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:24.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:24.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:24.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:25.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:25.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:25.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5982/139629 objects degraded (4.284%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:25.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:25.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:25.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5982/139629 objects degraded (4.284%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5982/139629 objects degraded (4.284%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:25.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:25.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:25.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:25.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:25.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:25.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:25.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:25.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:25.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:25.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:25.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:25.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:25.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:26.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:26.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:26.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:25 smithi192 ceph-mon[32956]: pgmap v1530: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 145 op/s; 5928/138726 objects degraded (4.273%); 14518/138726 objects misplaced (10.465%); 1.6 MiB/s, 87 objects/s recovering 2024-06-27T21:35:26.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:25 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:26.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:26.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:26.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:25 smithi151 ceph-mon[28965]: pgmap v1530: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 145 op/s; 5928/138726 objects degraded (4.273%); 14518/138726 objects misplaced (10.465%); 1.6 MiB/s, 87 objects/s recovering 2024-06-27T21:35:26.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:25 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:26.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:25 smithi190 ceph-mon[32247]: pgmap v1530: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 145 op/s; 5928/138726 objects degraded (4.273%); 14518/138726 objects misplaced (10.465%); 1.6 MiB/s, 87 objects/s recovering 2024-06-27T21:35:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:25 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:26.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:26.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:26.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:26.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:26.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:26.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:26.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:26.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:26.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:26.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:26.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:26.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:27.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:27.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:27.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:35:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:35:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:26 smithi192 ceph-mon[32956]: 6.18 scrub starts 2024-06-27T21:35:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:26 smithi192 ceph-mon[32956]: osdmap e592: 12 total, 12 up, 11 in 2024-06-27T21:35:27.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:27.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:35:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:35:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:26 smithi151 ceph-mon[28965]: 6.18 scrub starts 2024-06-27T21:35:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:26 smithi151 ceph-mon[28965]: osdmap e592: 12 total, 12 up, 11 in 2024-06-27T21:35:27.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:35:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:35:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:26 smithi190 ceph-mon[32247]: 6.18 scrub starts 2024-06-27T21:35:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:26 smithi190 ceph-mon[32247]: osdmap e592: 12 total, 12 up, 11 in 2024-06-27T21:35:27.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:27.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:27.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:27.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:27.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:27.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:27.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:27.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:27.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:27.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:27.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:27.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:27.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:28.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:27 smithi192 ceph-mon[32956]: pgmap v1532: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 148 op/s; 5928/137583 objects degraded (4.309%); 14452/137583 objects misplaced (10.504%); 740 KiB/s, 158 objects/s recovering 2024-06-27T21:35:28.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:28.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:27 smithi151 ceph-mon[28965]: pgmap v1532: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 148 op/s; 5928/137583 objects degraded (4.309%); 14452/137583 objects misplaced (10.504%); 740 KiB/s, 158 objects/s recovering 2024-06-27T21:35:28.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:27 smithi190 ceph-mon[32247]: pgmap v1532: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 148 op/s; 5928/137583 objects degraded (4.309%); 14452/137583 objects misplaced (10.504%); 740 KiB/s, 158 objects/s recovering 2024-06-27T21:35:28.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:28.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:28.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:28.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:28.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:28.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:28.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:28.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:28.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:28.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:28.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:28.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:28.996 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:29.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:29.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:29.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:29.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:29.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:29.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:29.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:29.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:29.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:29.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:29.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:29.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:29.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:29.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:29.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:29.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:30.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:30.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:29 smithi192 ceph-mon[32956]: pgmap v1533: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 110 op/s; 5928/137415 objects degraded (4.314%); 14452/137415 objects misplaced (10.517%); 592 KiB/s, 126 objects/s recovering 2024-06-27T21:35:30.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:29 smithi192 ceph-mon[32956]: osdmap e593: 12 total, 12 up, 11 in 2024-06-27T21:35:30.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5928/137415 objects degraded (4.314%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:30.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:30.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:29 smithi151 ceph-mon[28965]: pgmap v1533: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 110 op/s; 5928/137415 objects degraded (4.314%); 14452/137415 objects misplaced (10.517%); 592 KiB/s, 126 objects/s recovering 2024-06-27T21:35:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:29 smithi151 ceph-mon[28965]: osdmap e593: 12 total, 12 up, 11 in 2024-06-27T21:35:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5928/137415 objects degraded (4.314%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:30.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:30.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:29 smithi190 ceph-mon[32247]: pgmap v1533: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 110 op/s; 5928/137415 objects degraded (4.314%); 14452/137415 objects misplaced (10.517%); 592 KiB/s, 126 objects/s recovering 2024-06-27T21:35:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:29 smithi190 ceph-mon[32247]: osdmap e593: 12 total, 12 up, 11 in 2024-06-27T21:35:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5928/137415 objects degraded (4.314%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:30.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:30.377 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:30.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:30.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:30.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:30.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:30.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:30.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:30.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:30.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:30.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:30.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:30.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:31.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:30 smithi192 ceph-mon[32956]: osdmap e594: 12 total, 12 up, 11 in 2024-06-27T21:35:31.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:30 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:31.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:31.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:31.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:30 smithi151 ceph-mon[28965]: osdmap e594: 12 total, 12 up, 11 in 2024-06-27T21:35:31.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:30 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:31.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:31.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:31.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:30 smithi190 ceph-mon[32247]: osdmap e594: 12 total, 12 up, 11 in 2024-06-27T21:35:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:30 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:31.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:31.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:31.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:31.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:31.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:31.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:31.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:31.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:31.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:31.912 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:31 smithi151 ceph-mon[28965]: pgmap v1536: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 143 op/s; 5779/136143 objects degraded (4.245%); 14060/136143 objects misplaced (10.327%); 1.6 MiB/s, 153 objects/s recovering 2024-06-27T21:35:31.912 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:31 smithi151 ceph-mon[28965]: 6.19 scrub starts 2024-06-27T21:35:31.918 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:31.958 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:31.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:31.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:31.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:31 smithi192 ceph-mon[32956]: pgmap v1536: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 143 op/s; 5779/136143 objects degraded (4.245%); 14060/136143 objects misplaced (10.327%); 1.6 MiB/s, 153 objects/s recovering 2024-06-27T21:35:32.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:31 smithi192 ceph-mon[32956]: 6.19 scrub starts 2024-06-27T21:35:32.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:32.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:32.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:32.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:31 smithi190 ceph-mon[32247]: pgmap v1536: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 50 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 143 op/s; 5779/136143 objects degraded (4.245%); 14060/136143 objects misplaced (10.327%); 1.6 MiB/s, 153 objects/s recovering 2024-06-27T21:35:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:31 smithi190 ceph-mon[32247]: 6.19 scrub starts 2024-06-27T21:35:32.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:32.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:32.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:32.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:32.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:32.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:32.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:32.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:32.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:32.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:32.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:32.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:33.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:32 smithi151 ceph-mon[28965]: 6.18 scrub starts 2024-06-27T21:35:33.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:33.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:33.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:33.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:32 smithi190 ceph-mon[32247]: 6.18 scrub starts 2024-06-27T21:35:33.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:32 smithi192 ceph-mon[32956]: 6.18 scrub starts 2024-06-27T21:35:33.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:33.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:33.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:33.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:33.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:33.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:33.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:35:33.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:33.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:33.803 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pgp_num to 31 2024-06-27T21:35:33.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:33.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:33.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:33.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:33 smithi151 ceph-mon[28965]: pgmap v1537: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 96 op/s; 5779/135735 objects degraded (4.258%); 14060/135735 objects misplaced (10.358%); 1.4 MiB/s, 55 objects/s recovering 2024-06-27T21:35:33.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2159669711' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:35:33.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:33.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:34.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:34.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:34.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:33 smithi190 ceph-mon[32247]: pgmap v1537: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 96 op/s; 5779/135735 objects degraded (4.258%); 14060/135735 objects misplaced (10.358%); 1.4 MiB/s, 55 objects/s recovering 2024-06-27T21:35:34.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2159669711' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:35:34.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:34.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:33 smithi192 ceph-mon[32956]: pgmap v1537: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 96 op/s; 5779/135735 objects degraded (4.258%); 14060/135735 objects misplaced (10.358%); 1.4 MiB/s, 55 objects/s recovering 2024-06-27T21:35:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2159669711' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:35:34.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:34.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:34.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.461 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:34.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:34.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:34.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:34.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:34.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:34.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:34.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:34.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:34.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2159669711' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:35:34.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:34 smithi151 ceph-mon[28965]: osdmap e595: 12 total, 12 up, 11 in 2024-06-27T21:35:34.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5779/135735 objects degraded (4.258%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:34.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:34.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:35.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:35.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:35.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:35.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:35.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2159669711' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:35:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:34 smithi190 ceph-mon[32247]: osdmap e595: 12 total, 12 up, 11 in 2024-06-27T21:35:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5779/135735 objects degraded (4.258%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2159669711' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:35:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:34 smithi192 ceph-mon[32956]: osdmap e595: 12 total, 12 up, 11 in 2024-06-27T21:35:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5779/135735 objects degraded (4.258%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:35.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:35.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:35.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:35.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:35.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:35.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:35.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:35.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:35.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:35.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:35.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:35.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:35 smithi151 ceph-mon[28965]: pgmap v1539: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 5779/135345 objects degraded (4.270%); 14060/135345 objects misplaced (10.388%); 1.5 MiB/s, 58 objects/s recovering 2024-06-27T21:35:35.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:35.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:36.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:36.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:36.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:36.172 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:36.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:36.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:35 smithi190 ceph-mon[32247]: pgmap v1539: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 5779/135345 objects degraded (4.270%); 14060/135345 objects misplaced (10.388%); 1.5 MiB/s, 58 objects/s recovering 2024-06-27T21:35:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:35 smithi192 ceph-mon[32956]: pgmap v1539: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 51 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 5779/135345 objects degraded (4.270%); 14060/135345 objects misplaced (10.388%); 1.5 MiB/s, 58 objects/s recovering 2024-06-27T21:35:36.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:36.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:36.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:36.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:36.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.569 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:36.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:36.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:36.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:36.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:36.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:36.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:36.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:36.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:36.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:36 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:36.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3784841034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:37.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:35:37.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:35:37.037 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:35:37.037 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:35:37.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:37.060 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:35:37.060 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:35:37.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:37.082 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:35:37.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:35:37.105 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:35:37.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:35:37.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:35:37.126 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:35:37.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:35:37.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:35:37.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:37.188 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:37.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:35:37.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:35:37.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:35:37.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:35:37.261 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:35:37.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:35:37.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:36 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3784841034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:37.284 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:35:37.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:35:37.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:36 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3784841034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:37.307 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:35:37.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:35:37.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:37.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:37.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:37.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:37.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:37.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:37.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:37.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:37.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:37.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:37.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:37.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:37 smithi151 ceph-mon[28965]: pgmap v1540: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 130 op/s; 5747/134118 objects degraded (4.285%); 13745/134118 objects misplaced (10.248%); 2.8 MiB/s, 73 objects/s recovering 2024-06-27T21:35:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:37 smithi151 ceph-mon[28965]: 6.18 scrub starts 2024-06-27T21:35:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:37 smithi151 ceph-mon[28965]: 6.19 scrub starts 2024-06-27T21:35:38.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:38.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:38.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:38.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:37 smithi190 ceph-mon[32247]: pgmap v1540: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 130 op/s; 5747/134118 objects degraded (4.285%); 13745/134118 objects misplaced (10.248%); 2.8 MiB/s, 73 objects/s recovering 2024-06-27T21:35:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:37 smithi190 ceph-mon[32247]: 6.18 scrub starts 2024-06-27T21:35:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:37 smithi190 ceph-mon[32247]: 6.19 scrub starts 2024-06-27T21:35:38.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:37 smithi192 ceph-mon[32956]: pgmap v1540: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 130 op/s; 5747/134118 objects degraded (4.285%); 13745/134118 objects misplaced (10.248%); 2.8 MiB/s, 73 objects/s recovering 2024-06-27T21:35:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:37 smithi192 ceph-mon[32956]: 6.18 scrub starts 2024-06-27T21:35:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:37 smithi192 ceph-mon[32956]: 6.19 scrub starts 2024-06-27T21:35:38.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:38.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:38.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:38.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:38.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:38.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:38.661 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:35:38.661 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:35:38.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:38.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:38.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:38.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:38.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:38.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:39.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:39.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:39.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:39.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:39.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:39.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:39.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:39.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:39.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:39.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:39.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:39.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:35:39.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:35:39.671 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 3, 4, 5, 6, 9, 11, 10, 2, 7] 2024-06-27T21:35:39.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 1 2024-06-27T21:35:39.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:39.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:39.828 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:35:39.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:39.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:39.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:40.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:40.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:39 smithi151 ceph-mon[28965]: pgmap v1541: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 5747/133920 objects degraded (4.291%); 13745/133920 objects misplaced (10.264%); 1.4 MiB/s, 22 objects/s recovering 2024-06-27T21:35:40.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1946306465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:35:40.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5747/133920 objects degraded (4.291%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:40.034 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:40.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:35:40.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:40.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:40.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:40.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:39 smithi190 ceph-mon[32247]: pgmap v1541: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 5747/133920 objects degraded (4.291%); 13745/133920 objects misplaced (10.264%); 1.4 MiB/s, 22 objects/s recovering 2024-06-27T21:35:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1946306465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:35:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5747/133920 objects degraded (4.291%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:40.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:39 smithi192 ceph-mon[32956]: pgmap v1541: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 5747/133920 objects degraded (4.291%); 13745/133920 objects misplaced (10.264%); 1.4 MiB/s, 22 objects/s recovering 2024-06-27T21:35:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1946306465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:35:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5747/133920 objects degraded (4.291%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:40.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:35:40.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:40.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:40.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:40.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:40.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:40.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:40.749 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:40.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:40.776 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:40.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:40.781 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:40.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:40.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:40.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:40.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:40 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:35:40.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1946306465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:35:40.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:40 smithi151 ceph-mon[28965]: osdmap e596: 12 total, 12 up, 11 in 2024-06-27T21:35:41.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:41.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:41.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:41.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:41.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:40 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:35:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1946306465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:35:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:40 smithi190 ceph-mon[32247]: osdmap e596: 12 total, 12 up, 11 in 2024-06-27T21:35:41.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:40 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:35:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1946306465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:35:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:40 smithi192 ceph-mon[32956]: osdmap e596: 12 total, 12 up, 11 in 2024-06-27T21:35:41.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:41.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:41.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:41.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:41.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:41.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:41.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:41.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:41.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:41.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:41.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:41.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:42.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:42.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:42.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:41 smithi151 ceph-mon[28965]: pgmap v1543: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 5711/132750 objects degraded (4.302%); 13530/132750 objects misplaced (10.192%); 2.3 MiB/s, 36 objects/s recovering 2024-06-27T21:35:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:41 smithi151 ceph-mon[28965]: 6.17 scrub starts 2024-06-27T21:35:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:41 smithi151 ceph-mon[28965]: 6.17 scrub ok 2024-06-27T21:35:42.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:42.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:42.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:41 smithi190 ceph-mon[32247]: pgmap v1543: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 5711/132750 objects degraded (4.302%); 13530/132750 objects misplaced (10.192%); 2.3 MiB/s, 36 objects/s recovering 2024-06-27T21:35:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:41 smithi190 ceph-mon[32247]: 6.17 scrub starts 2024-06-27T21:35:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:41 smithi190 ceph-mon[32247]: 6.17 scrub ok 2024-06-27T21:35:42.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:41 smithi192 ceph-mon[32956]: pgmap v1543: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 124 op/s; 5711/132750 objects degraded (4.302%); 13530/132750 objects misplaced (10.192%); 2.3 MiB/s, 36 objects/s recovering 2024-06-27T21:35:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:41 smithi192 ceph-mon[32956]: 6.17 scrub starts 2024-06-27T21:35:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:41 smithi192 ceph-mon[32956]: 6.17 scrub ok 2024-06-27T21:35:42.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:42.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:42.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:42.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:42.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:42.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:42.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:35:42.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:42.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:42.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:42.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:42.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:42.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:43.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:42 smithi151 ceph-mon[28965]: 6.19 scrub starts 2024-06-27T21:35:43.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:43.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:43.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:43.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:43.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:42 smithi190 ceph-mon[32247]: 6.19 scrub starts 2024-06-27T21:35:43.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:42 smithi192 ceph-mon[32956]: 6.19 scrub starts 2024-06-27T21:35:43.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:43.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:43.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:43.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:43.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:43.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:43.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:43.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:43.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:43.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:43.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:43.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:44.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:44.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:43 smithi151 ceph-mon[28965]: pgmap v1544: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 5711/132189 objects degraded (4.320%); 13530/132189 objects misplaced (10.235%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:35:44.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:43 smithi151 ceph-mon[28965]: 6.18 scrub starts 2024-06-27T21:35:44.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:43 smithi151 ceph-mon[28965]: 6.18 scrub ok 2024-06-27T21:35:44.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:43 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:35:44.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:43 smithi151 ceph-mon[28965]: 6.19 scrub ok 2024-06-27T21:35:44.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:44.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:44.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:44.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:43 smithi190 ceph-mon[32247]: pgmap v1544: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 5711/132189 objects degraded (4.320%); 13530/132189 objects misplaced (10.235%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:35:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:43 smithi190 ceph-mon[32247]: 6.18 scrub starts 2024-06-27T21:35:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:43 smithi190 ceph-mon[32247]: 6.18 scrub ok 2024-06-27T21:35:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:43 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:35:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:43 smithi190 ceph-mon[32247]: 6.19 scrub ok 2024-06-27T21:35:44.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:43 smithi192 ceph-mon[32956]: pgmap v1544: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 125 op/s; 5711/132189 objects degraded (4.320%); 13530/132189 objects misplaced (10.235%); 2.2 MiB/s, 34 objects/s recovering 2024-06-27T21:35:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:43 smithi192 ceph-mon[32956]: 6.18 scrub starts 2024-06-27T21:35:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:43 smithi192 ceph-mon[32956]: 6.18 scrub ok 2024-06-27T21:35:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:43 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:35:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:43 smithi192 ceph-mon[32956]: 6.19 scrub ok 2024-06-27T21:35:44.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:44.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:44.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:44.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:44.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:44.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:44.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:44.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:44.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:44.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:44 smithi151 ceph-mon[28965]: 6.b scrub ok 2024-06-27T21:35:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5711/132189 objects degraded (4.320%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:44 smithi151 ceph-mon[28965]: osdmap e597: 12 total, 12 up, 11 in 2024-06-27T21:35:44.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:44.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:44.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:45.045 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:35:45.045 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:35:45.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:35:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:35:45.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:45.095 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:35:45.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:35:45.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:45.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:35:45.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:35:45.142 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:35:45.142 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:35:45.167 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:35:45.167 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:35:45.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:35:45.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:35:45.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:45.219 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:35:45.219 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:35:45.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:45.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:35:45.242 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:35:45.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:44 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:35:44.982+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e597 definitely_dead 0 2024-06-27T21:35:45.264 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:35:45.264 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:35:45.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:44 smithi190 ceph-mon[32247]: 6.b scrub ok 2024-06-27T21:35:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5711/132189 objects degraded (4.320%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:44 smithi190 ceph-mon[32247]: osdmap e597: 12 total, 12 up, 11 in 2024-06-27T21:35:45.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:35:45.289 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:35:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:44 smithi192 ceph-mon[32956]: 6.b scrub ok 2024-06-27T21:35:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5711/132189 objects degraded (4.320%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:44 smithi192 ceph-mon[32956]: osdmap e597: 12 total, 12 up, 11 in 2024-06-27T21:35:45.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:35:45.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:35:45.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:45.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:45.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:45.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:45.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:45.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:45.701 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:45.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:45.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:45.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:45.843 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.1. 2024-06-27T21:35:45.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:45.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:45.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:45.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:46.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:45 smithi151 ceph-mon[28965]: pgmap v1545: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 5711/131400 objects degraded (4.346%); 13464/131400 objects misplaced (10.247%) 2024-06-27T21:35:46.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/609865234' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-06-27T21:35:46.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:45 smithi151 ceph-mon[28965]: Client client.admin marked osd.1 out, while it was still marked up 2024-06-27T21:35:46.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3841353189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:46.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:46.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:46.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:46.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:45 smithi190 ceph-mon[32247]: pgmap v1545: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 5711/131400 objects degraded (4.346%); 13464/131400 objects misplaced (10.247%) 2024-06-27T21:35:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/609865234' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-06-27T21:35:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:45 smithi190 ceph-mon[32247]: Client client.admin marked osd.1 out, while it was still marked up 2024-06-27T21:35:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3841353189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:45 smithi192 ceph-mon[32956]: pgmap v1545: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering, 52 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 5711/131400 objects degraded (4.346%); 13464/131400 objects misplaced (10.247%) 2024-06-27T21:35:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/609865234' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-06-27T21:35:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:45 smithi192 ceph-mon[32956]: Client client.admin marked osd.1 out, while it was still marked up 2024-06-27T21:35:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3841353189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:46.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:46.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:46.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:46.477 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:46.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:46.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:46.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:46.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:46.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:46.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:46.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:46.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:46.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:47.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:47.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:47.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/609865234' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-06-27T21:35:47.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:46 smithi151 ceph-mon[28965]: osdmap e598: 12 total, 12 up, 10 in 2024-06-27T21:35:47.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:46 smithi151 ceph-mon[28965]: osdmap e599: 12 total, 12 up, 10 in 2024-06-27T21:35:47.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/609865234' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-06-27T21:35:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:46 smithi190 ceph-mon[32247]: osdmap e598: 12 total, 12 up, 10 in 2024-06-27T21:35:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:46 smithi190 ceph-mon[32247]: osdmap e599: 12 total, 12 up, 10 in 2024-06-27T21:35:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/609865234' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-06-27T21:35:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:46 smithi192 ceph-mon[32956]: osdmap e598: 12 total, 12 up, 10 in 2024-06-27T21:35:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:46 smithi192 ceph-mon[32956]: osdmap e599: 12 total, 12 up, 10 in 2024-06-27T21:35:47.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:47.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:47.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:47.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:47.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:47.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:47.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:47.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:47.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:47.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:47.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:47.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:47.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:48.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:48.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:47 smithi151 ceph-mon[28965]: pgmap v1548: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering, 52 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 5689/130014 objects degraded (4.376%); 13339/130014 objects misplaced (10.260%) 2024-06-27T21:35:48.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:47 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:35:48.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:47 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:35:48.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:47 smithi151 ceph-mon[28965]: osdmap e600: 12 total, 12 up, 10 in 2024-06-27T21:35:48.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:48.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:48.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:47 smithi190 ceph-mon[32247]: pgmap v1548: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering, 52 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 5689/130014 objects degraded (4.376%); 13339/130014 objects misplaced (10.260%) 2024-06-27T21:35:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:47 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:35:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:47 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:35:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:47 smithi190 ceph-mon[32247]: osdmap e600: 12 total, 12 up, 10 in 2024-06-27T21:35:48.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:47 smithi192 ceph-mon[32956]: pgmap v1548: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering, 52 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 5689/130014 objects degraded (4.376%); 13339/130014 objects misplaced (10.260%) 2024-06-27T21:35:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:47 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:35:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:47 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:35:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:47 smithi192 ceph-mon[32956]: osdmap e600: 12 total, 12 up, 10 in 2024-06-27T21:35:48.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:48.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:48.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:48.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:48.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:48.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:48.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:48.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:48.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:48.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:35:48.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:48.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:48.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:48.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:49.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:49.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 6.19 starting backfill to osd.0 from (0'0,592'10135] MIN to 597'10207 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 6.d starting backfill to osd.11 from (0'0,0'0] MAX to 597'10754 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 6.b starting backfill to osd.6 from (0'0,0'0] MAX to 597'10553 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 6.4 starting backfill to osd.11 from (0'0,0'0] MAX to 597'10561 2024-06-27T21:35:49.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:48 smithi151 ceph-mon[28965]: 6.e starting backfill to osd.7 from (0'0,0'0] MAX to 597'10840 2024-06-27T21:35:49.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:49.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:49.263 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 6.19 starting backfill to osd.0 from (0'0,592'10135] MIN to 597'10207 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 6.d starting backfill to osd.11 from (0'0,0'0] MAX to 597'10754 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 6.b starting backfill to osd.6 from (0'0,0'0] MAX to 597'10553 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 6.4 starting backfill to osd.11 from (0'0,0'0] MAX to 597'10561 2024-06-27T21:35:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:48 smithi190 ceph-mon[32247]: 6.e starting backfill to osd.7 from (0'0,0'0] MAX to 597'10840 2024-06-27T21:35:49.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:35:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:35:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:35:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:35:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 6.19 starting backfill to osd.0 from (0'0,592'10135] MIN to 597'10207 2024-06-27T21:35:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 6.d starting backfill to osd.11 from (0'0,0'0] MAX to 597'10754 2024-06-27T21:35:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 6.b starting backfill to osd.6 from (0'0,0'0] MAX to 597'10553 2024-06-27T21:35:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 6.4 starting backfill to osd.11 from (0'0,0'0] MAX to 597'10561 2024-06-27T21:35:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:48 smithi192 ceph-mon[32956]: 6.e starting backfill to osd.7 from (0'0,0'0] MAX to 597'10840 2024-06-27T21:35:49.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:49.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:49.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:49.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:49.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:49.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:49.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:49.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:49.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:49.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:49.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:49.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:49.961 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:49 smithi151 ceph-mon[28965]: pgmap v1551: 57 pgs: 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering, 50 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 5689/129903 objects degraded (4.379%); 14713/129903 objects misplaced (11.326%) 2024-06-27T21:35:49.961 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5689/129903 objects degraded (4.379%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:49.961 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:49 smithi151 ceph-mon[28965]: osdmap e601: 12 total, 12 up, 10 in 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:51265 6.b 6:df23569f:::benchmark_data_smithi192_117994_object51264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:30:07.361725+0000", 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.59649051600002, 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45991362600000002, 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:49.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51265 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.361725+0000", 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.361725+0000", 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.361726+0000", 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.92e-07 2024-06-27T21:35:49.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.361783+0000", 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6716999999999999e-05 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.361794+0000", 2024-06-27T21:35:49.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1657999999999999e-05 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.361802+0000", 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7840000000000004e-06 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.362260+0000", 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045766399999999999 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.362333+0000", 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2948999999999996e-05 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.362420+0000", 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7892999999999993e-05 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.363342+0000", 2024-06-27T21:35:49.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092132900000000003 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.366393+0000", 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030509589999999998 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821571+0000", 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.455178052 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821609+0000", 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7774000000000001e-05 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821639+0000", 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0054999999999999e-05 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:51265 6.b e486/467 ondisk, result = 0)", 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:30:07.363963+0000", 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.594252541, 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45768162600000001, 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:49.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.363963+0000", 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.363963+0000", 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.363964+0000", 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8100000000000003e-07 2024-06-27T21:35:49.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.363967+0000", 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3620000000000002e-06 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821525+0000", 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45755806399999999 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821530+0000", 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.468e-06 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821560+0000", 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9569000000000001e-05 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821566+0000", 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3969999999999999e-06 2024-06-27T21:35:49.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.821645+0000", 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8184999999999998e-05 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:51270 6.8 e486/467 6:136c287d:::benchmark_data_smithi192_117994_object51269:head v 486'3238, mlcod=486'3236)", 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:30:07.366158+0000", 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 342.59205800900003, 2024-06-27T21:35:49.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37208370600000001, 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.366158+0000", 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.366158+0000", 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.366158+0000", 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8299999999999996e-07 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.366309+0000", 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015028799999999999 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:49.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.736463+0000", 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37015440700000002 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.736481+0000", 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7980000000000001e-05 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.736595+0000", 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114455 2024-06-27T21:35:49.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.736637+0000", 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2126999999999998e-05 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.738201+0000", 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015631760000000001 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:49.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.738241+0000", 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0590000000000003e-05 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:156425 6.b 6:d1a93836:::benchmark_data_smithi192_117994_object48831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:18.073205+0000", 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 151.88501041399999, 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.442289233, 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:49.980 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156425 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073205+0000", 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073205+0000", 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073206+0000", 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8100000000000001e-07 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073211+0000", 2024-06-27T21:35:49.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6670000000000004e-06 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073221+0000", 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0669999999999999e-05 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073227+0000", 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4920000000000001e-06 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073279+0000", 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1612e-05 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073399+0000", 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120425 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:35:49.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.073513+0000", 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011379000000000001 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.077172+0000", 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036590469999999999 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.078434+0000", 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012620870000000001 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515326+0000", 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43689227000000003 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515423+0000", 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7069000000000002e-05 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:49.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515494+0000", 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1123000000000003e-05 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:156425 6.b e540/467 ondisk, result = 0)", 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:18.078248+0000", 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 151.87996739100001, 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43725826299999998, 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:35:49.987 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.078248+0000", 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.078248+0000", 2024-06-27T21:35:49.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.078249+0000", 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8000000000000004e-07 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.078254+0000", 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6879999999999998e-06 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515113+0000", 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43685896200000002 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515134+0000", 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1166999999999999e-05 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515299+0000", 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016511100000000001 2024-06-27T21:35:49.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515316+0000", 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7244e-05 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:18.515506+0000", 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000190211 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:49.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157136 6.18 6:188b89eb:::benchmark_data_smithi192_117994_object49542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:28.891537+0000", 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 141.066678852, 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33585609799999999, 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157136 2024-06-27T21:35:49.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891537+0000", 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891537+0000", 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:49.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891537+0000", 2024-06-27T21:35:49.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8699999999999995e-07 2024-06-27T21:35:49.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:49.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891540+0000", 2024-06-27T21:35:49.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4150000000000002e-06 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891546+0000", 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5810000000000002e-06 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891551+0000", 2024-06-27T21:35:49.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0300000000000001e-06 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.098157+0000", 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206605543 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.098165+0000", 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8059999999999995e-06 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.193448+0000", 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.095283628999999995 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.193594+0000", 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014556000000000001 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,1,7", 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.193735+0000", 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000141299 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.201494+0000", 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077590840000000003 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.201525+0000", 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1269000000000002e-05 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.218938+0000", 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017412548999999999 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.225716+0000", 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006777796 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:49.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.225772+0000", 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6224999999999998e-05 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.227393+0000", 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001620825 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.000 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157143 6.18 6:1c866899:::benchmark_data_smithi192_117994_object49549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:28.891561+0000", 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 141.06665497700001, 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33732965799999998, 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157143 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891561+0000", 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891561+0000", 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891561+0000", 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6699999999999999e-07 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891564+0000", 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5730000000000002e-06 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891569+0000", 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5759999999999996e-06 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891574+0000", 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.13e-06 2024-06-27T21:35:50.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.098193+0000", 2024-06-27T21:35:50.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20661865600000001 2024-06-27T21:35:50.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.098200+0000", 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.498e-06 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.195573+0000", 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.097372607 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.195727+0000", 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015400799999999999 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,1,7", 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.195873+0000", 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014567200000000001 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.201564+0000", 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0056915619999999998 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.206747+0000", 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051829880000000004 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.218971+0000", 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012223357000000001 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.227445+0000", 2024-06-27T21:35:50.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0084749409999999997 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.227498+0000", 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2410999999999997e-05 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.228890+0000", 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013923119999999999 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.009 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157149 6.18 6:1b733dbc:::benchmark_data_smithi192_117994_object49555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:28.891581+0000", 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 141.06663483700001, 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34081883400000001, 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157149 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891581+0000", 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891581+0000", 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891581+0000", 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1e-07 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891583+0000", 2024-06-27T21:35:50.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4310000000000001e-06 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891589+0000", 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8449999999999999e-06 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:28.891594+0000", 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.098962+0000", 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20736765700000001 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.098972+0000", 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9119999999999993e-06 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.201594+0000", 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.102622167 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.201749+0000", 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015513699999999999 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,1,7", 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.201903+0000", 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015363900000000001 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.206758+0000", 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048554009999999996 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.208219+0000", 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014612169999999999 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.219004+0000", 2024-06-27T21:35:50.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010785177 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.230776+0000", 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011772039 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.230833+0000", 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6784999999999999e-05 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.232400+0000", 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015663719999999999 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.10 553/550 [PushOp(6:0ccb8fbe:::benchmark_data_smithi192_117994_object58114:head, version: 487'3656, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:0ccb8fbe:::benchmark_data_smithi192_117994_object58114:head@487'3656, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:35.987293+0000", 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 133.97092254200001, 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.700879637, 2024-06-27T21:35:50.018 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987293+0000", 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987293+0000", 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987294+0000", 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000005e-07 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987341+0000", 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7268000000000002e-05 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686231+0000", 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69889012800000005 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686248+0000", 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6549999999999999e-05 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686347+0000", 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9213e-05 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686357+0000", 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8109999999999997e-06 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.688173+0000", 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018159179999999999 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:161535 6.17 6:ed224b72:::benchmark_data_smithi192_117994_object53941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:06.882690+0000", 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 103.075525588, 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32460927000000001, 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 161535 2024-06-27T21:35:50.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882690+0000", 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882690+0000", 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882692+0000", 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.979e-06 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882705+0000", 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030999999999999e-05 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882723+0000", 2024-06-27T21:35:50.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8253999999999998e-05 2024-06-27T21:35:50.026 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882732+0000", 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4060000000000005e-06 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882804+0000", 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2768000000000005e-05 2024-06-27T21:35:50.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.882983+0000", 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017827899999999999 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10,11", 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.883094+0000", 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111556 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.884880+0000", 2024-06-27T21:35:50.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001785926 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.886478+0000", 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015982349999999999 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:06.886750+0000", 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000272107 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:07.206837+0000", 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32008680499999997 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:07.206883+0000", 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5806999999999997e-05 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:07.207299+0000", 2024-06-27T21:35:50.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041611699999999997 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.10 e563,550 6:0802b08f:::benchmark_data_smithi192_117994_object3626:head,507'6948)", 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:12.425128+0000", 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 97.533087555999998, 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35620019000000003, 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:35:50.031 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.425128+0000", 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.425128+0000", 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.425129+0000", 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5600000000000005e-07 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.425133+0000", 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5549999999999996e-06 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.781183+0000", 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35604965300000002 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.781208+0000", 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5514e-05 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.781266+0000", 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7164999999999999e-05 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.781272+0000", 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6139999999999997e-06 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.781328+0000", 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6032999999999997e-05 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.035 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.17 564/550 [PushReplyOp(6:eb05b92d:::benchmark_data_smithi192_117994_object70401:head)])", 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:14.798559+0000", 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 95.159656439000003, 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34160877899999997, 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.798559+0000", 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.798559+0000", 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.798560+0000", 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.798565+0000", 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2109999999999997e-06 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:15.138824+0000", 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34025897500000002 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:15.138832+0000", 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9759999999999995e-06 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:15.138893+0000", 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1388000000000002e-05 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:15.140168+0000", 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012744220000000001 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.17 564/550 [PushReplyOp(6:ed0f1f09:::benchmark_data_smithi192_117994_object74567:head)])", 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 92.835743958999998, 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35817200500000002, 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999996e-07 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122479+0000", 2024-06-27T21:35:50.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1570000000000003e-06 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.479318+0000", 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35683908399999997 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.479326+0000", 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9289999999999998e-06 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.479393+0000", 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6268999999999999e-05 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.480644+0000", 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012509000000000001 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.043 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:168233 6.18 6:1d52b39c:::benchmark_data_smithi192_117994_object60639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:15.087596+0000", 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 34.870619388999998, 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31978306200000001, 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168233 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087596+0000", 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087596+0000", 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087598+0000", 2024-06-27T21:35:50.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7099999999999999e-06 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087605+0000", 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9360000000000002e-06 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087621+0000", 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6075000000000001e-05 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087628+0000", 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4800000000000004e-06 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087703+0000", 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4462999999999995e-05 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.087874+0000", 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000171168 2024-06-27T21:35:50.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,1,7", 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.088268+0000", 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039370799999999999 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.090195+0000", 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019268219999999999 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:50.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.090237+0000", 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2432000000000003e-05 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.100148+0000", 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0099111289999999994 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.407151+0000", 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30700248099999999 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.407196+0000", 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5932e-05 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.407379+0000", 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000182726 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:169458 6.b 6:dedf206c:::benchmark_data_smithi192_117994_object61864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e591)", 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:24.371330+0000", 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.586885987999999, 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38649861699999999, 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 169458 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.051 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371330+0000", 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371330+0000", 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371332+0000", 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2560000000000001e-06 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371342+0000", 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0227000000000001e-05 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371372+0000", 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9578999999999999e-05 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371387+0000", 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5063e-05 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371499+0000", 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011266599999999999 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371621+0000", 2024-06-27T21:35:50.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121459 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371696+0000", 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4849000000000001e-05 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.375339+0000", 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003643271 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.375961+0000", 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00062231000000000005 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.757766+0000", 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38180487400000002 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.757800+0000", 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3738999999999997e-05 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.757828+0000", 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8323999999999999e-05 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172061 6.b 6.70fd271b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:46.120027+0000", 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.8381884080000002, 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72584151200000002, 2024-06-27T21:35:50.057 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172061 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120027+0000", 2024-06-27T21:35:50.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120027+0000", 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120029+0000", 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4479999999999999e-06 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120035+0000", 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.883e-06 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120051+0000", 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5925000000000001e-05 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120059+0000", 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.272e-06 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120126+0000", 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7019000000000003e-05 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120135+0000", 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6290000000000005e-06 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845829+0000", 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72569375300000005 2024-06-27T21:35:50.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845869+0000", 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9582999999999997e-05 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172064 6.b 6.ad4cb37b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:46.318009+0000", 2024-06-27T21:35:50.062 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.6402069790000002, 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52793522800000003, 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172064 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318009+0000", 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318009+0000", 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318010+0000", 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7069999999999999e-06 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318018+0000", 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3560000000000004e-06 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318035+0000", 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7164e-05 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318043+0000", 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4270000000000008e-06 2024-06-27T21:35:50.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318118+0000", 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4550999999999998e-05 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318132+0000", 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3878e-05 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845915+0000", 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52778372299999998 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845944+0000", 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8422e-05 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172327 6.b 6:db698ede:::benchmark_data_smithi192_117994_object64733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-06-27T21:35:50.067 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.569266+0000", 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.388949338, 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.49585006199999998, 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172327 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569266+0000", 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569266+0000", 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569268+0000", 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5039999999999999e-06 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569275+0000", 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9079999999999998e-06 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569290+0000", 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.49e-05 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569297+0000", 2024-06-27T21:35:50.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8900000000000007e-06 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569369+0000", 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1151999999999999e-05 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569514+0000", 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014560699999999999 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,6,7", 2024-06-27T21:35:50.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569673+0000", 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158899 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.573853+0000", 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041802389999999997 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.574777+0000", 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092400700000000002 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.682001+0000", 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.107223204 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065010+0000", 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38300970000000001 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065050+0000", 2024-06-27T21:35:50.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9743999999999998e-05 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065116+0000", 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6308000000000003e-05 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:172327 6.b e600/599 ondisk, result = 0)", 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.575752+0000", 2024-06-27T21:35:50.074 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.3824636690000001, 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48936887200000001, 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575752+0000", 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575752+0000", 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575753+0000", 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575760+0000", 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4580000000000004e-06 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.064940+0000", 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48918015500000001 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.064947+0000", 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7320000000000001e-06 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.064997+0000", 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9684000000000001e-05 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065006+0000", 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3559999999999999e-06 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065121+0000", 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114815 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.b 600/599 [PushReplyOp(6:d020eeb0:::benchmark_data_smithi192_117994_object74198:head)])", 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.681894+0000", 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.2763213709999999, 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.322229661, 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.681894+0000", 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.681894+0000", 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.681894+0000", 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4399999999999999e-07 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.681897+0000", 2024-06-27T21:35:50.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8839999999999998e-06 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.003852+0000", 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.321955092 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.003863+0000", 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1164e-05 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:35:50.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.003944+0000", 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0595999999999999e-05 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.004124+0000", 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017978100000000001 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:35:50.082 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:35:50.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:50.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:50.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:50.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:50.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:49 smithi190 ceph-mon[32247]: pgmap v1551: 57 pgs: 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering, 50 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 5689/129903 objects degraded (4.379%); 14713/129903 objects misplaced (11.326%) 2024-06-27T21:35:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5689/129903 objects degraded (4.379%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:49 smithi190 ceph-mon[32247]: osdmap e601: 12 total, 12 up, 10 in 2024-06-27T21:35:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:49 smithi192 ceph-mon[32956]: pgmap v1551: 57 pgs: 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+recovering, 50 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 5689/129903 objects degraded (4.379%); 14713/129903 objects misplaced (11.326%) 2024-06-27T21:35:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5689/129903 objects degraded (4.379%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:49 smithi192 ceph-mon[32956]: osdmap e601: 12 total, 12 up, 10 in 2024-06-27T21:35:50.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:50.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:50.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:50.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:50.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:50.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:50.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:50.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:50.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:50.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:50.876 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:35:50.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:50.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:50.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:51.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:51.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:51.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:35:51.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:35:51.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2221485796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:51.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:51.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:51.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:51.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:35:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:35:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:35:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2221485796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:35:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:35:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2221485796' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:35:51.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:51.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:51.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:51.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:51.551 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:35:51.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:51.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:51.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:51.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:51.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:51.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:51.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:51 smithi190 ceph-mon[32247]: pgmap v1553: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 5657/128889 objects degraded (4.389%); 20552/128889 objects misplaced (15.946%); 13 MiB/s, 344 objects/s recovering 2024-06-27T21:35:51.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:51 smithi190 ceph-mon[32247]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:35:51.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:51 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:35:51.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:35:51.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:51 smithi190 ceph-mon[32247]: osdmap e602: 12 total, 12 up, 10 in 2024-06-27T21:35:51.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:51.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:51.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:52.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:52.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:51 smithi151 ceph-mon[28965]: pgmap v1553: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 5657/128889 objects degraded (4.389%); 20552/128889 objects misplaced (15.946%); 13 MiB/s, 344 objects/s recovering 2024-06-27T21:35:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:51 smithi151 ceph-mon[28965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:35:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:51 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:35:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:35:52.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:51 smithi151 ceph-mon[28965]: osdmap e602: 12 total, 12 up, 10 in 2024-06-27T21:35:52.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.229 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:52.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:51 smithi192 ceph-mon[32956]: pgmap v1553: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 5657/128889 objects degraded (4.389%); 20552/128889 objects misplaced (15.946%); 13 MiB/s, 344 objects/s recovering 2024-06-27T21:35:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:51 smithi192 ceph-mon[32956]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:35:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:51 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:35:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:35:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:51 smithi192 ceph-mon[32956]: osdmap e602: 12 total, 12 up, 10 in 2024-06-27T21:35:52.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:52.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:52.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:52.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:52.558 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:35:52.559 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:35:52.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:35:52.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:52.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:52.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:52.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:52.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:52.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:52.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:52.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:52.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:53.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:53.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:53.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:53.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:52 smithi151 ceph-mon[28965]: osdmap e603: 12 total, 12 up, 10 in 2024-06-27T21:35:53.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:53.237 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:52 smithi190 ceph-mon[32247]: osdmap e603: 12 total, 12 up, 10 in 2024-06-27T21:35:53.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:53.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:52 smithi192 ceph-mon[32956]: osdmap e603: 12 total, 12 up, 10 in 2024-06-27T21:35:53.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:53.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:53.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:53.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:53.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:53.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:53.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:53.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:53.797 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:53.800 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:35:53.800 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:35:53.800 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:35:53.801 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:35:53.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:53.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:53.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:53.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:54.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:54.082 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:53 smithi151 ceph-mon[28965]: pgmap v1556: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 44 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 5679/128397 objects degraded (4.423%); 23002/128397 objects misplaced (17.915%); 15 MiB/s, 411 objects/s recovering 2024-06-27T21:35:54.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:53 smithi151 ceph-mon[28965]: osdmap e604: 12 total, 12 up, 10 in 2024-06-27T21:35:54.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:54.125 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:53 smithi190 ceph-mon[32247]: pgmap v1556: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 44 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 5679/128397 objects degraded (4.423%); 23002/128397 objects misplaced (17.915%); 15 MiB/s, 411 objects/s recovering 2024-06-27T21:35:54.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:53 smithi190 ceph-mon[32247]: osdmap e604: 12 total, 12 up, 10 in 2024-06-27T21:35:54.145 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:54.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:54.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:54.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:53 smithi192 ceph-mon[32956]: pgmap v1556: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 44 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 5679/128397 objects degraded (4.423%); 23002/128397 objects misplaced (17.915%); 15 MiB/s, 411 objects/s recovering 2024-06-27T21:35:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:53 smithi192 ceph-mon[32956]: osdmap e604: 12 total, 12 up, 10 in 2024-06-27T21:35:54.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:54.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:54.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:54.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:54.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:54.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:35:54.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:54.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:54.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:54.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:54.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:54.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:54.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:35:54.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:35:54.938 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:35:54.944 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:35:54.971 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:35:54.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:35:54.998 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:35:54.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:35:55.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:55.022 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:55.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:35:55.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:35:55.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:35:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:35:55.079 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:35:55.079 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:35:55.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5679/128397 objects degraded (4.423%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:55.080 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:54 smithi151 ceph-mon[28965]: osdmap e605: 12 total, 12 up, 10 in 2024-06-27T21:35:55.103 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:35:55.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:35:55.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:55.125 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:35:55.125 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:35:55.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:35:55.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:35:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:55.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:35:55.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:35:55.180 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5679/128397 objects degraded (4.423%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:55.180 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:54 smithi190 ceph-mon[32247]: osdmap e605: 12 total, 12 up, 10 in 2024-06-27T21:35:55.199 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:35:55.200 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:35:55.224 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:35:55.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:35:55.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:55.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5679/128397 objects degraded (4.423%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:35:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:54 smithi192 ceph-mon[32956]: osdmap e605: 12 total, 12 up, 10 in 2024-06-27T21:35:55.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:35:55.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:35:55.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:55.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:55.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:55.646 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:55.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:55.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:55.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:55.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:56.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:56.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:56.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:56.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:56.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:55 smithi190 ceph-mon[32247]: pgmap v1558: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 44 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 5679/127800 objects degraded (4.444%); 23187/127800 objects misplaced (18.143%); 38 MiB/s, 677 objects/s recovering 2024-06-27T21:35:56.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4292807604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:55 smithi151 ceph-mon[28965]: pgmap v1558: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 44 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 5679/127800 objects degraded (4.444%); 23187/127800 objects misplaced (18.143%); 38 MiB/s, 677 objects/s recovering 2024-06-27T21:35:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4292807604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:56.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:56.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:56.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:55 smithi192 ceph-mon[32956]: pgmap v1558: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 44 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 5679/127800 objects degraded (4.444%); 23187/127800 objects misplaced (18.143%); 38 MiB/s, 677 objects/s recovering 2024-06-27T21:35:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4292807604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:35:56.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:56.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:56.456 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:35:56.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:56.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:56.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:56.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:56.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:35:56.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:56.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:35:56.871 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:35:56.871 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:35:56.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:56.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:56.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:57.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:57.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:57.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:57.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:57.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:35:57.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:35:57.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:57.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:57.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:57.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:57.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:35:57.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:35:57.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:57.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:57.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:57.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:57.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:58.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:58.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:58.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:58.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:57 smithi151 ceph-mon[28965]: pgmap v1560: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 5671/126699 objects degraded (4.476%); 21877/126699 objects misplaced (17.267%); 41 MiB/s, 801 objects/s recovering 2024-06-27T21:35:58.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:57 smithi190 ceph-mon[32247]: pgmap v1560: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 5671/126699 objects degraded (4.476%); 21877/126699 objects misplaced (17.267%); 41 MiB/s, 801 objects/s recovering 2024-06-27T21:35:58.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:35:58.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:35:58.295 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:35:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:57 smithi192 ceph-mon[32956]: pgmap v1560: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 107 op/s; 5671/126699 objects degraded (4.476%); 21877/126699 objects misplaced (17.267%); 41 MiB/s, 801 objects/s recovering 2024-06-27T21:35:58.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:58.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:35:58.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:35:58.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:35:58.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:35:58.568 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:35:58.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:58.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:58.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:58.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:58.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:35:58.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:35:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:58.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:35:58.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:58 smithi190 ceph-mon[32247]: 6.4 deep-scrub starts 2024-06-27T21:35:58.952 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:58 smithi190 ceph-mon[32247]: osdmap e606: 12 total, 12 up, 10 in 2024-06-27T21:35:59.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:35:59.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:35:59.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:59.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:59.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:58 smithi151 ceph-mon[28965]: 6.4 deep-scrub starts 2024-06-27T21:35:59.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:58 smithi151 ceph-mon[28965]: osdmap e606: 12 total, 12 up, 10 in 2024-06-27T21:35:59.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:59.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:58 smithi192 ceph-mon[32956]: 6.4 deep-scrub starts 2024-06-27T21:35:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:58 smithi192 ceph-mon[32956]: osdmap e606: 12 total, 12 up, 10 in 2024-06-27T21:35:59.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:59.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:35:59.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:35:59.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:35:59.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:59.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:35:59.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:59.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:59.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:35:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:35:59.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:35:59.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:35:59.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:35:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:35:59.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:59 smithi190 ceph-mon[32247]: pgmap v1562: 57 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 5671/126489 objects degraded (4.483%); 20559/126489 objects misplaced (16.254%); 50 MiB/s, 920 objects/s recovering 2024-06-27T21:36:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:59 smithi190 ceph-mon[32247]: osdmap e607: 12 total, 12 up, 10 in 2024-06-27T21:36:00.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:35:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5671/126489 objects degraded (4.483%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:00.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.055 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:00.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:00.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:00.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:59 smithi151 ceph-mon[28965]: pgmap v1562: 57 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 5671/126489 objects degraded (4.483%); 20559/126489 objects misplaced (16.254%); 50 MiB/s, 920 objects/s recovering 2024-06-27T21:36:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:59 smithi151 ceph-mon[28965]: osdmap e607: 12 total, 12 up, 10 in 2024-06-27T21:36:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:35:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5671/126489 objects degraded (4.483%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:00.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:59 smithi192 ceph-mon[32956]: pgmap v1562: 57 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 5671/126489 objects degraded (4.483%); 20559/126489 objects misplaced (16.254%); 50 MiB/s, 920 objects/s recovering 2024-06-27T21:36:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:59 smithi192 ceph-mon[32956]: osdmap e607: 12 total, 12 up, 10 in 2024-06-27T21:36:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:35:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5671/126489 objects degraded (4.483%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:00.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:00.310 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:00.310 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:00.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:00.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:00.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:00.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:00.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:00.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:00.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:00.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:00.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:00.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:00.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:01.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:36:01.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:01.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:01.166 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:36:01.167 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:36:01.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:01.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:01.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:01.318 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:01.331 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:01.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:01.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:01.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:01.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:01.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:01.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:01.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:01.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:01.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:01.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:01.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: pgmap v1564: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 134 op/s; 5633/125373 objects degraded (4.493%); 18828/125373 objects misplaced (15.018%); 42 MiB/s, 794 objects/s recovering 2024-06-27T21:36:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:36:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: from='client.16755 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:36:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: 6.4 scrub ok 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: osdmap e608: 12 total, 12 up, 10 in 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:01 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:02.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:02.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: pgmap v1564: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 134 op/s; 5633/125373 objects degraded (4.493%); 18828/125373 objects misplaced (15.018%); 42 MiB/s, 794 objects/s recovering 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: from='client.16755 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: 6.4 scrub ok 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: osdmap e608: 12 total, 12 up, 10 in 2024-06-27T21:36:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:36:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:01 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: pgmap v1564: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 134 op/s; 5633/125373 objects degraded (4.493%); 18828/125373 objects misplaced (15.018%); 42 MiB/s, 794 objects/s recovering 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: from='client.16755 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: 6.4 scrub ok 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: osdmap e608: 12 total, 12 up, 10 in 2024-06-27T21:36:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:36:02.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:01 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:02.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:02.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:02.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:02.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:02.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:02.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:02.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:02.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:02.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:02.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:02.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:02.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:02.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:03.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:03.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:03.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:03.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:02 smithi151 ceph-mon[28965]: osdmap e609: 12 total, 12 up, 10 in 2024-06-27T21:36:03.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:02 smithi190 ceph-mon[32247]: osdmap e609: 12 total, 12 up, 10 in 2024-06-27T21:36:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:02 smithi192 ceph-mon[32956]: osdmap e609: 12 total, 12 up, 10 in 2024-06-27T21:36:03.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:03.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:03.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:03.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:03.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:03.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:03.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:03.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:03.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:03.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:03.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:03.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:03.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:03.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:04.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:04.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:04.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:04.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:03 smithi190 ceph-mon[32247]: pgmap v1567: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 5669/124731 objects degraded (4.545%); 18246/124731 objects misplaced (14.628%); 38 MiB/s, 706 objects/s recovering 2024-06-27T21:36:04.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:04.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:03 smithi151 ceph-mon[28965]: pgmap v1567: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 5669/124731 objects degraded (4.545%); 18246/124731 objects misplaced (14.628%); 38 MiB/s, 706 objects/s recovering 2024-06-27T21:36:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:04.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:03 smithi192 ceph-mon[32956]: pgmap v1567: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 45 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 5669/124731 objects degraded (4.545%); 18246/124731 objects misplaced (14.628%); 38 MiB/s, 706 objects/s recovering 2024-06-27T21:36:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:04.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:04.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:04.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:04.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:04.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:04.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:04.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:04.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:04.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:04.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:04.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:04.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:04.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:05.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:05.062 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:05.062 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:05.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:05.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5669/124731 objects degraded (4.545%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:05.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5669/124731 objects degraded (4.545%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:05.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5669/124731 objects degraded (4.545%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:05.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:05.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:05.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:05.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:05.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:05.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:05.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:05.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:05.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:05.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:36:05.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:05.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:05.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:05.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:06.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:06.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:06.099 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:06.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:06.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:06.222 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:05 smithi190 ceph-mon[32247]: pgmap v1568: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 5669/123891 objects degraded (4.576%); 16669/123891 objects misplaced (13.455%); 37 MiB/s, 660 objects/s recovering 2024-06-27T21:36:06.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:05 smithi190 ceph-mon[32247]: osdmap e610: 12 total, 12 up, 10 in 2024-06-27T21:36:06.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:05 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:36:06.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:06.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:06.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:05 smithi190 ceph-mon[32247]: 6.7 scrub ok 2024-06-27T21:36:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:05 smithi192 ceph-mon[32956]: pgmap v1568: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 5669/123891 objects degraded (4.576%); 16669/123891 objects misplaced (13.455%); 37 MiB/s, 660 objects/s recovering 2024-06-27T21:36:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:05 smithi192 ceph-mon[32956]: osdmap e610: 12 total, 12 up, 10 in 2024-06-27T21:36:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:05 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:36:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:05 smithi192 ceph-mon[32956]: 6.7 scrub ok 2024-06-27T21:36:06.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:06.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:06.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:05 smithi151 ceph-mon[28965]: pgmap v1568: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 144 op/s; 5669/123891 objects degraded (4.576%); 16669/123891 objects misplaced (13.455%); 37 MiB/s, 660 objects/s recovering 2024-06-27T21:36:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:05 smithi151 ceph-mon[28965]: osdmap e610: 12 total, 12 up, 10 in 2024-06-27T21:36:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:05 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:36:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:05 smithi151 ceph-mon[28965]: 6.7 scrub ok 2024-06-27T21:36:06.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:06.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:06.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:06.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:06.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:06.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:06.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:06.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:06.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:06.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:07.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:07.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:07.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:07.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:06 smithi190 ceph-mon[32247]: osdmap e611: 12 total, 12 up, 10 in 2024-06-27T21:36:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:06 smithi190 ceph-mon[32247]: pgmap v1571: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 5640/122529 objects degraded (4.603%); 15651/122529 objects misplaced (12.773%); 28 MiB/s, 679 objects/s recovering 2024-06-27T21:36:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:06 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:36:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:06 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:36:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:06 smithi192 ceph-mon[32956]: osdmap e611: 12 total, 12 up, 10 in 2024-06-27T21:36:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:06 smithi192 ceph-mon[32956]: pgmap v1571: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 5640/122529 objects degraded (4.603%); 15651/122529 objects misplaced (12.773%); 28 MiB/s, 679 objects/s recovering 2024-06-27T21:36:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:06 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:36:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:06 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:36:07.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:07.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:07.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:07.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:06 smithi151 ceph-mon[28965]: osdmap e611: 12 total, 12 up, 10 in 2024-06-27T21:36:07.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:07 smithi151 ceph-mon[28965]: pgmap v1571: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 5640/122529 objects degraded (4.603%); 15651/122529 objects misplaced (12.773%); 28 MiB/s, 679 objects/s recovering 2024-06-27T21:36:07.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:07 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:36:07.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:07 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:36:07.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:07.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:07.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:07.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:07.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:07.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:07.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:07.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:07.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:07.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:08.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:08.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:08.105 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:08.105 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:08.105 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:07 smithi190 ceph-mon[32247]: osdmap e612: 12 total, 12 up, 10 in 2024-06-27T21:36:08.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:08.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:07 smithi192 ceph-mon[32956]: osdmap e612: 12 total, 12 up, 10 in 2024-06-27T21:36:08.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:08.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:08.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:08.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:07 smithi151 ceph-mon[28965]: osdmap e612: 12 total, 12 up, 10 in 2024-06-27T21:36:08.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:08.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:08.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:08.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:08.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:08.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:08.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:08.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:36:08.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:08.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:08.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:09.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:08 smithi190 ceph-mon[32247]: osdmap e613: 12 total, 12 up, 10 in 2024-06-27T21:36:09.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:08 smithi190 ceph-mon[32247]: pgmap v1574: 57 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 5640/122346 objects degraded (4.610%); 15263/122346 objects misplaced (12.475%); 20 MiB/s, 614 objects/s recovering 2024-06-27T21:36:09.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:08 smithi190 ceph-mon[32247]: 6.5 scrub starts 2024-06-27T21:36:09.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:08 smithi190 ceph-mon[32247]: 6.5 scrub ok 2024-06-27T21:36:09.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:09.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:09.142 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:09.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:09.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:08 smithi151 ceph-mon[28965]: osdmap e613: 12 total, 12 up, 10 in 2024-06-27T21:36:09.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:08 smithi151 ceph-mon[28965]: pgmap v1574: 57 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 5640/122346 objects degraded (4.610%); 15263/122346 objects misplaced (12.475%); 20 MiB/s, 614 objects/s recovering 2024-06-27T21:36:09.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:08 smithi151 ceph-mon[28965]: 6.5 scrub starts 2024-06-27T21:36:09.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:08 smithi151 ceph-mon[28965]: 6.5 scrub ok 2024-06-27T21:36:09.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:08 smithi192 ceph-mon[32956]: osdmap e613: 12 total, 12 up, 10 in 2024-06-27T21:36:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:08 smithi192 ceph-mon[32956]: pgmap v1574: 57 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+recovering, 46 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 5640/122346 objects degraded (4.610%); 15263/122346 objects misplaced (12.475%); 20 MiB/s, 614 objects/s recovering 2024-06-27T21:36:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:08 smithi192 ceph-mon[32956]: 6.5 scrub starts 2024-06-27T21:36:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:08 smithi192 ceph-mon[32956]: 6.5 scrub ok 2024-06-27T21:36:09.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:09.327 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:09.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:09.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:09.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:09.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:09.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:09.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:09.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:09.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:09.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:09.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:09.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:09.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:10.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:10.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:10.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:10.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5640/122346 objects degraded (4.610%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5640/122346 objects degraded (4.610%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:10.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:10.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:10.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:10.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:10.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5640/122346 objects degraded (4.610%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:10.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.569 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:10.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:10.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:10.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:10.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:10.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:10.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:10.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:10.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:11.023 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:11.024 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:11.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:11.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:11.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:11.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:11.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:11.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:11.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:10 smithi151 ceph-mon[28965]: osdmap e614: 12 total, 12 up, 10 in 2024-06-27T21:36:11.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:11.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:11.262 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:10 smithi151 ceph-mon[28965]: pgmap v1576: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 49 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 100/121302 objects degraded (0.082%); 16704/121302 objects misplaced (13.771%) 2024-06-27T21:36:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:10 smithi190 ceph-mon[32247]: osdmap e614: 12 total, 12 up, 10 in 2024-06-27T21:36:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:10 smithi190 ceph-mon[32247]: pgmap v1576: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 49 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 100/121302 objects degraded (0.082%); 16704/121302 objects misplaced (13.771%) 2024-06-27T21:36:11.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:10 smithi192 ceph-mon[32956]: osdmap e614: 12 total, 12 up, 10 in 2024-06-27T21:36:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:10 smithi192 ceph-mon[32956]: pgmap v1576: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 49 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 100/121302 objects degraded (0.082%); 16704/121302 objects misplaced (13.771%) 2024-06-27T21:36:11.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:11.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:11.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:11.478 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1564,"stamp":"2024-06-27T21:36:00.157950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2739175992,"num_objects":41791,"num_object_clones":0,"num_object_copies":125373,"num_objects_missing_on_primary":4036,"num_objects_missing":4036,"num_objects_degraded":5633,"num_objects_misplaced":18828,"num_objects_unfound":0,"num_objects_dirty":41791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92413,"num_write_kb":3455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31438,"num_bytes_recovered":1754468944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270597,"ondisk_log_size":270597,"up":171,"acting":167,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":13210596,"kb_used_data":8479208,"kb_used_omap":106,"kb_used_meta":4731157,"kb_avail":924240924,"statfs":{"total":959950356480,"available":946422706176,"internally_reserved":0,"allocated":8682708992,"data_stored":8676919673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109491,"internal_metadata":4844704845},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-53018624,"num_objects":-809,"num_object_clones":0,"num_object_copies":-2427,"num_objects_missing_on_primary":-80,"num_objects_missing":-80,"num_objects_degraded":-46,"num_objects_misplaced":-4359,"num_objects_unfound":0,"num_objects_dirty":-809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":809,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4770,"num_bytes_recovered":263520280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001355"},"pg_stats":[{"pgid":"6.18","version":"605'10400","reported_seq":22044,"reported_epoch":605,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:35:55.442015+0000","last_change":"2024-06-27T21:35:48.037773+0000","last_active":"2024-06-27T21:35:55.442015+0000","last_peered":"2024-06-27T21:35:55.442015+0000","last_clean":"2024-06-27T21:35:45.797225+0000","last_became_active":"2024-06-27T21:35:48.005941+0000","last_became_peered":"2024-06-27T21:35:48.005941+0000","last_unstale":"2024-06-27T21:35:55.442015+0000","last_undegraded":"2024-06-27T21:35:47.879684+0000","last_fullsized":"2024-06-27T21:35:47.851720+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":589,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10353","last_scrub_stamp":"2024-06-27T21:35:42.524197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.524197+0000","objects_scrubbed":1327,"log_size":10400,"log_dups_size":0,"ondisk_log_size":10400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":1280,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":703,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":39518208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":703},{"shards":"0,7","objects":576}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"607'10260","reported_seq":24195,"reported_epoch":607,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:35:59.711575+0000","last_change":"2024-06-27T21:35:47.968691+0000","last_active":"2024-06-27T21:35:59.711575+0000","last_peered":"2024-06-27T21:35:59.711575+0000","last_clean":"2024-06-27T21:35:45.223031+0000","last_became_active":"2024-06-27T21:35:47.950730+0000","last_became_peered":"2024-06-27T21:35:47.950730+0000","last_unstale":"2024-06-27T21:35:59.711575+0000","last_undegraded":"2024-06-27T21:35:59.711575+0000","last_fullsized":"2024-06-27T21:35:59.711575+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10195","last_scrub_stamp":"2024-06-27T21:35:42.766433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.766433+0000","objects_scrubbed":1384,"log_size":10260,"log_dups_size":0,"ondisk_log_size":10260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1319,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,1],"avail_no_missing":["10","1","2","8","9"],"object_location_counts":[{"shards":"1,2,10","objects":1319}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":159,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.987313+0000","last_change":"2024-06-27T21:33:02.987696+0000","last_active":"2024-06-27T21:35:58.987313+0000","last_peered":"2024-06-27T21:35:58.987313+0000","last_clean":"2024-06-27T21:35:58.987313+0000","last_became_active":"2024-06-27T21:33:01.707872+0000","last_became_peered":"2024-06-27T21:33:01.707872+0000","last_unstale":"2024-06-27T21:35:58.987313+0000","last_undegraded":"2024-06-27T21:35:58.987313+0000","last_fullsized":"2024-06-27T21:35:58.987313+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,10],"acting":[11,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":165,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.987327+0000","last_change":"2024-06-27T21:35:46.856033+0000","last_active":"2024-06-27T21:35:58.987327+0000","last_peered":"2024-06-27T21:35:58.987327+0000","last_clean":"2024-06-27T21:35:58.987327+0000","last_became_active":"2024-06-27T21:35:46.855054+0000","last_became_peered":"2024-06-27T21:35:46.855054+0000","last_unstale":"2024-06-27T21:35:58.987327+0000","last_undegraded":"2024-06-27T21:35:58.987327+0000","last_fullsized":"2024-06-27T21:35:58.987327+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,10],"acting":[11,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":192,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.991664+0000","last_change":"2024-06-27T21:35:46.857696+0000","last_active":"2024-06-27T21:35:58.991664+0000","last_peered":"2024-06-27T21:35:58.991664+0000","last_clean":"2024-06-27T21:35:58.991664+0000","last_became_active":"2024-06-27T21:35:46.857381+0000","last_became_peered":"2024-06-27T21:35:46.857381+0000","last_unstale":"2024-06-27T21:35:58.991664+0000","last_undegraded":"2024-06-27T21:35:58.991664+0000","last_fullsized":"2024-06-27T21:35:58.991664+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:52.993157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:52.993157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,5],"acting":[11,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":157,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.725916+0000","last_change":"2024-06-27T21:33:02.568824+0000","last_active":"2024-06-27T21:35:54.725916+0000","last_peered":"2024-06-27T21:35:54.725916+0000","last_clean":"2024-06-27T21:35:54.725916+0000","last_became_active":"2024-06-27T21:33:01.773541+0000","last_became_peered":"2024-06-27T21:33:01.773541+0000","last_unstale":"2024-06-27T21:35:54.725916+0000","last_undegraded":"2024-06-27T21:35:54.725916+0000","last_fullsized":"2024-06-27T21:35:54.725916+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":231,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.907812+0000","last_change":"2024-06-27T21:35:46.858389+0000","last_active":"2024-06-27T21:35:57.907812+0000","last_peered":"2024-06-27T21:35:57.907812+0000","last_clean":"2024-06-27T21:35:57.907812+0000","last_became_active":"2024-06-27T21:35:46.858050+0000","last_became_peered":"2024-06-27T21:35:46.858050+0000","last_unstale":"2024-06-27T21:35:57.907812+0000","last_undegraded":"2024-06-27T21:35:57.907812+0000","last_fullsized":"2024-06-27T21:35:57.907812+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":183,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.913660+0000","last_change":"2024-06-27T21:33:50.003594+0000","last_active":"2024-06-27T21:35:58.913660+0000","last_peered":"2024-06-27T21:35:58.913660+0000","last_clean":"2024-06-27T21:35:58.913660+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:35:58.913660+0000","last_undegraded":"2024-06-27T21:35:58.913660+0000","last_fullsized":"2024-06-27T21:35:58.913660+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:50.003522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.003522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":611,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.919295+0000","last_change":"2024-06-27T21:35:46.862167+0000","last_active":"2024-06-27T21:35:57.919295+0000","last_peered":"2024-06-27T21:35:57.919295+0000","last_clean":"2024-06-27T21:35:57.919295+0000","last_became_active":"2024-06-27T21:35:46.859076+0000","last_became_peered":"2024-06-27T21:35:46.859076+0000","last_unstale":"2024-06-27T21:35:57.919295+0000","last_undegraded":"2024-06-27T21:35:57.919295+0000","last_fullsized":"2024-06-27T21:35:57.919295+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.184716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.184716+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,11],"acting":[2,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"604'11930","reported_seq":36963,"reported_epoch":605,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:35:55.442485+0000","last_change":"2024-06-27T21:35:48.122275+0000","last_active":"2024-06-27T21:35:55.442485+0000","last_peered":"2024-06-27T21:35:55.442485+0000","last_clean":"2024-06-27T21:35:45.791479+0000","last_became_active":"2024-06-27T21:35:48.071184+0000","last_became_peered":"2024-06-27T21:35:48.071184+0000","last_unstale":"2024-06-27T21:35:55.442485+0000","last_undegraded":"2024-06-27T21:35:47.908221+0000","last_fullsized":"2024-06-27T21:35:47.880861+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":572,"parent":"0.0","parent_split_bits":5,"last_scrub":"574'11661","last_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_deep_scrub":"574'11661","last_deep_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_clean_scrub_stamp":"2024-06-27T21:34:46.137011+0000","objects_scrubbed":1586,"log_size":11930,"log_dups_size":0,"ondisk_log_size":11930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"no scrub is scheduled","scrub_duration":10216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85065728,"num_objects":1298,"num_object_clones":0,"num_object_copies":3894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":1298,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,9],"acting":[0,3],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1,3","objects":1298},{"shards":"0,3","objects":-37}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":639,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.913373+0000","last_change":"2024-06-27T21:33:03.234800+0000","last_active":"2024-06-27T21:35:58.913373+0000","last_peered":"2024-06-27T21:35:58.913373+0000","last_clean":"2024-06-27T21:35:58.913373+0000","last_became_active":"2024-06-27T21:33:01.779398+0000","last_became_peered":"2024-06-27T21:33:01.779398+0000","last_unstale":"2024-06-27T21:35:58.913373+0000","last_undegraded":"2024-06-27T21:35:58.913373+0000","last_fullsized":"2024-06-27T21:35:58.913373+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:33:03.234726+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.11","version":"605'10014","reported_seq":30054,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.725807+0000","last_change":"2024-06-27T21:34:22.233121+0000","last_active":"2024-06-27T21:35:54.725807+0000","last_peered":"2024-06-27T21:35:54.725807+0000","last_clean":"2024-06-27T21:35:54.725807+0000","last_became_active":"2024-06-27T21:34:21.788051+0000","last_became_peered":"2024-06-27T21:34:21.788051+0000","last_unstale":"2024-06-27T21:35:54.725807+0000","last_undegraded":"2024-06-27T21:35:54.725807+0000","last_fullsized":"2024-06-27T21:35:54.725807+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"569'9717","last_scrub_stamp":"2024-06-27T21:34:22.232987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:22.232987+0000","objects_scrubbed":1552,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":298,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":33,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.726250+0000","last_change":"2024-06-27T21:35:47.430217+0000","last_active":"2024-06-27T21:35:54.726250+0000","last_peered":"2024-06-27T21:35:54.726250+0000","last_clean":"2024-06-27T21:35:54.726250+0000","last_became_active":"2024-06-27T21:35:46.855888+0000","last_became_peered":"2024-06-27T21:35:46.855888+0000","last_unstale":"2024-06-27T21:35:54.726250+0000","last_undegraded":"2024-06-27T21:35:54.726250+0000","last_fullsized":"2024-06-27T21:35:54.726250+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.430103+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.430103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:48.754970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"607'10118","reported_seq":10534,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:59.429538+0000","last_change":"2024-06-27T21:33:51.180577+0000","last_active":"2024-06-27T21:35:59.429538+0000","last_peered":"2024-06-27T21:35:59.429538+0000","last_clean":"2024-06-27T21:35:59.429538+0000","last_became_active":"2024-06-27T21:33:27.698780+0000","last_became_peered":"2024-06-27T21:33:27.698780+0000","last_unstale":"2024-06-27T21:35:59.429538+0000","last_undegraded":"2024-06-27T21:35:59.429538+0000","last_fullsized":"2024-06-27T21:35:59.429538+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'9712","last_scrub_stamp":"2024-06-27T21:33:51.180247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.180247+0000","objects_scrubbed":1680,"log_size":10118,"log_dups_size":0,"ondisk_log_size":10118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":406,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2],"acting":[11,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":165,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.919882+0000","last_change":"2024-06-27T21:35:46.859131+0000","last_active":"2024-06-27T21:35:58.919882+0000","last_peered":"2024-06-27T21:35:58.919882+0000","last_clean":"2024-06-27T21:35:58.919882+0000","last_became_active":"2024-06-27T21:35:46.858265+0000","last_became_peered":"2024-06-27T21:35:46.858265+0000","last_unstale":"2024-06-27T21:35:58.919882+0000","last_undegraded":"2024-06-27T21:35:58.919882+0000","last_fullsized":"2024-06-27T21:35:58.919882+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"607'10509","reported_seq":27152,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:59.024610+0000","last_change":"2024-06-27T21:35:52.894201+0000","last_active":"2024-06-27T21:35:59.024610+0000","last_peered":"2024-06-27T21:35:59.024610+0000","last_clean":"2024-06-27T21:35:59.024610+0000","last_became_active":"2024-06-27T21:35:52.878734+0000","last_became_peered":"2024-06-27T21:35:52.878734+0000","last_unstale":"2024-06-27T21:35:59.024610+0000","last_undegraded":"2024-06-27T21:35:59.024610+0000","last_fullsized":"2024-06-27T21:35:59.024610+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"566'10180","last_scrub_stamp":"2024-06-27T21:34:19.641768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:19.641768+0000","objects_scrubbed":1673,"log_size":10509,"log_dups_size":0,"ondisk_log_size":10509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":333,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":288,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.917561+0000","last_change":"2024-06-27T21:31:19.010738+0000","last_active":"2024-06-27T21:35:57.917561+0000","last_peered":"2024-06-27T21:35:57.917561+0000","last_clean":"2024-06-27T21:35:57.917561+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:35:57.917561+0000","last_undegraded":"2024-06-27T21:35:57.917561+0000","last_fullsized":"2024-06-27T21:35:57.917561+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.010614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.010614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"606'10627","reported_seq":11148,"reported_epoch":606,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:35:58.466676+0000","last_change":"2024-06-27T21:35:46.879240+0000","last_active":"2024-06-27T21:35:58.466676+0000","last_peered":"2024-06-27T21:35:58.466676+0000","last_clean":"2024-06-27T21:33:25.735988+0000","last_became_active":"2024-06-27T21:33:28.958742+0000","last_became_peered":"2024-06-27T21:33:28.958742+0000","last_unstale":"2024-06-27T21:35:58.466676+0000","last_undegraded":"2024-06-27T21:35:58.466676+0000","last_fullsized":"2024-06-27T21:35:58.466676+0000","mapping_epoch":550,"log_start":"470'100","ondisk_log_start":"470'100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":10527,"log_dups_size":0,"ondisk_log_size":10527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86671360,"num_objects":1323,"num_object_clones":0,"num_object_copies":3969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3876,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5575,"num_write_kb":220704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2228224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[1,5,9],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5,9","objects":1323}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":296,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.987202+0000","last_change":"2024-06-27T21:35:46.853442+0000","last_active":"2024-06-27T21:35:58.987202+0000","last_peered":"2024-06-27T21:35:58.987202+0000","last_clean":"2024-06-27T21:35:58.987202+0000","last_became_active":"2024-06-27T21:35:46.853087+0000","last_became_peered":"2024-06-27T21:35:46.853087+0000","last_unstale":"2024-06-27T21:35:58.987202+0000","last_undegraded":"2024-06-27T21:35:58.987202+0000","last_fullsized":"2024-06-27T21:35:58.987202+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.15","version":"607'10217","reported_seq":18472,"reported_epoch":607,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:35:59.356765+0000","last_change":"2024-06-27T21:33:28.935649+0000","last_active":"2024-06-27T21:35:59.356765+0000","last_peered":"2024-06-27T21:35:59.356765+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:28.917741+0000","last_became_peered":"2024-06-27T21:33:28.917741+0000","last_unstale":"2024-06-27T21:35:59.356765+0000","last_undegraded":"2024-06-27T21:33:28.805628+0000","last_fullsized":"2024-06-27T21:33:28.693311+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":10217,"log_dups_size":0,"ondisk_log_size":10217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83132416,"num_objects":1269,"num_object_clones":0,"num_object_copies":3807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2107,"num_bytes_recovered":138018816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,10],"acting":[7,4],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,6,7,9","objects":221},{"shards":"4,7","objects":48},{"shards":"6,7,9","objects":1000}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":310,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.911235+0000","last_change":"2024-06-27T21:35:46.862654+0000","last_active":"2024-06-27T21:35:57.911235+0000","last_peered":"2024-06-27T21:35:57.911235+0000","last_clean":"2024-06-27T21:35:57.911235+0000","last_became_active":"2024-06-27T21:35:46.862310+0000","last_became_peered":"2024-06-27T21:35:46.862310+0000","last_unstale":"2024-06-27T21:35:57.911235+0000","last_undegraded":"2024-06-27T21:35:57.911235+0000","last_fullsized":"2024-06-27T21:35:57.911235+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:34:02.673610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:34:02.673610+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"605'10309","reported_seq":31278,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.977088+0000","last_change":"2024-06-27T21:34:06.592362+0000","last_active":"2024-06-27T21:35:54.977088+0000","last_peered":"2024-06-27T21:35:54.977088+0000","last_clean":"2024-06-27T21:35:54.977088+0000","last_became_active":"2024-06-27T21:34:05.705454+0000","last_became_peered":"2024-06-27T21:34:05.705454+0000","last_unstale":"2024-06-27T21:35:54.977088+0000","last_undegraded":"2024-06-27T21:35:54.977088+0000","last_fullsized":"2024-06-27T21:35:54.977088+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":5,"last_scrub":"562'9943","last_scrub_stamp":"2024-06-27T21:34:06.592264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:06.592264+0000","objects_scrubbed":1716,"log_size":10309,"log_dups_size":0,"ondisk_log_size":10309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":830,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":300,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.725844+0000","last_change":"2024-06-27T21:35:46.855364+0000","last_active":"2024-06-27T21:35:54.725844+0000","last_peered":"2024-06-27T21:35:54.725844+0000","last_clean":"2024-06-27T21:35:54.725844+0000","last_became_active":"2024-06-27T21:35:46.854914+0000","last_became_peered":"2024-06-27T21:35:46.854914+0000","last_unstale":"2024-06-27T21:35:54.725844+0000","last_undegraded":"2024-06-27T21:35:54.725844+0000","last_fullsized":"2024-06-27T21:35:54.725844+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,10],"acting":[0,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"607'10218","reported_seq":24035,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:59.173872+0000","last_change":"2024-06-27T21:35:41.151637+0000","last_active":"2024-06-27T21:35:59.173872+0000","last_peered":"2024-06-27T21:35:59.173872+0000","last_clean":"2024-06-27T21:35:59.173872+0000","last_became_active":"2024-06-27T21:35:01.964347+0000","last_became_peered":"2024-06-27T21:35:01.964347+0000","last_unstale":"2024-06-27T21:35:59.173872+0000","last_undegraded":"2024-06-27T21:35:59.173872+0000","last_fullsized":"2024-06-27T21:35:59.173872+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10152","last_scrub_stamp":"2024-06-27T21:35:41.151550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:41.151550+0000","objects_scrubbed":1305,"log_size":10218,"log_dups_size":0,"ondisk_log_size":10218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:56.383804+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":319,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.919560+0000","last_change":"2024-06-27T21:32:13.808094+0000","last_active":"2024-06-27T21:35:58.919560+0000","last_peered":"2024-06-27T21:35:58.919560+0000","last_clean":"2024-06-27T21:35:58.919560+0000","last_became_active":"2024-06-27T21:32:13.807596+0000","last_became_peered":"2024-06-27T21:32:13.807596+0000","last_unstale":"2024-06-27T21:35:58.919560+0000","last_undegraded":"2024-06-27T21:35:58.919560+0000","last_fullsized":"2024-06-27T21:35:58.919560+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":591,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.919481+0000","last_change":"2024-06-27T21:33:00.167722+0000","last_active":"2024-06-27T21:35:57.919481+0000","last_peered":"2024-06-27T21:35:57.919481+0000","last_clean":"2024-06-27T21:35:57.919481+0000","last_became_active":"2024-06-27T21:32:59.709162+0000","last_became_peered":"2024-06-27T21:32:59.709162+0000","last_unstale":"2024-06-27T21:35:57.919481+0000","last_undegraded":"2024-06-27T21:35:57.919481+0000","last_fullsized":"2024-06-27T21:35:57.919481+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.167617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.167617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"606'10954","reported_seq":11320,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.939187+0000","last_change":"2024-06-27T21:30:52.123816+0000","last_active":"2024-06-27T21:35:57.939187+0000","last_peered":"2024-06-27T21:35:57.939187+0000","last_clean":"2024-06-27T21:35:57.939187+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:35:57.939187+0000","last_undegraded":"2024-06-27T21:35:57.939187+0000","last_fullsized":"2024-06-27T21:35:57.939187+0000","mapping_epoch":467,"log_start":"475'900","ondisk_log_start":"475'900","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6360","last_scrub_stamp":"2024-06-27T21:30:52.123515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.123515+0000","objects_scrubbed":6329,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171573248,"num_objects":2618,"num_object_clones":0,"num_object_copies":7854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10954,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":380,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.919617+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:35:58.919617+0000","last_peered":"2024-06-27T21:35:58.919617+0000","last_clean":"2024-06-27T21:35:58.919617+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:35:58.919617+0000","last_undegraded":"2024-06-27T21:35:58.919617+0000","last_fullsized":"2024-06-27T21:35:58.919617+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"606'10932","reported_seq":14389,"reported_epoch":606,"state":"active+remapped+backfilling","last_fresh":"2024-06-27T21:35:58.389694+0000","last_change":"2024-06-27T21:35:48.097443+0000","last_active":"2024-06-27T21:35:58.389694+0000","last_peered":"2024-06-27T21:35:58.389694+0000","last_clean":"2024-06-27T21:35:45.361307+0000","last_became_active":"2024-06-27T21:35:48.041053+0000","last_became_peered":"2024-06-27T21:35:48.041053+0000","last_unstale":"2024-06-27T21:35:58.389694+0000","last_undegraded":"2024-06-27T21:35:58.389694+0000","last_fullsized":"2024-06-27T21:35:58.389694+0000","mapping_epoch":599,"log_start":"475'900","ondisk_log_start":"475'900","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6100","last_scrub_stamp":"2024-06-27T21:30:48.342993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.342993+0000","objects_scrubbed":6058,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171966464,"num_objects":2624,"num_object_clones":0,"num_object_copies":7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1106,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10932,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1544,"num_bytes_recovered":101187584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,1],"avail_no_missing":["2","1","4"],"object_location_counts":[{"shards":"1,2,4","objects":2624}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":221,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.908060+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:35:57.908060+0000","last_peered":"2024-06-27T21:35:57.908060+0000","last_clean":"2024-06-27T21:35:57.908060+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:35:57.908060+0000","last_undegraded":"2024-06-27T21:35:57.908060+0000","last_fullsized":"2024-06-27T21:35:57.908060+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"607'10843","reported_seq":28502,"reported_epoch":607,"state":"active+remapped+backfilling","last_fresh":"2024-06-27T21:35:59.985476+0000","last_change":"2024-06-27T21:35:48.050152+0000","last_active":"2024-06-27T21:35:59.985476+0000","last_peered":"2024-06-27T21:35:59.985476+0000","last_clean":"2024-06-27T21:35:45.667707+0000","last_became_active":"2024-06-27T21:35:47.988203+0000","last_became_peered":"2024-06-27T21:35:47.988203+0000","last_unstale":"2024-06-27T21:35:59.985476+0000","last_undegraded":"2024-06-27T21:35:59.985476+0000","last_fullsized":"2024-06-27T21:35:59.985476+0000","mapping_epoch":599,"log_start":"474'800","ondisk_log_start":"474'800","created":467,"last_epoch_clean":551,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174325784,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10843,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8637,"num_bytes_recovered":456785968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,1],"avail_no_missing":["10","0","1"],"object_location_counts":[{"shards":"0,1,10","objects":2661}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1141,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.991731+0000","last_change":"2024-06-27T21:32:13.809592+0000","last_active":"2024-06-27T21:35:58.991731+0000","last_peered":"2024-06-27T21:35:58.991731+0000","last_clean":"2024-06-27T21:35:58.991731+0000","last_became_active":"2024-06-27T21:32:13.809197+0000","last_became_peered":"2024-06-27T21:32:13.809197+0000","last_unstale":"2024-06-27T21:35:58.991731+0000","last_undegraded":"2024-06-27T21:35:58.991731+0000","last_fullsized":"2024-06-27T21:35:58.991731+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"607'10477","reported_seq":17481,"reported_epoch":607,"state":"active+recovering","last_fresh":"2024-06-27T21:35:59.132725+0000","last_change":"2024-06-27T21:33:26.890778+0000","last_active":"2024-06-27T21:35:59.132725+0000","last_peered":"2024-06-27T21:35:59.132725+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:33:26.864503+0000","last_became_peered":"2024-06-27T21:33:26.864503+0000","last_unstale":"2024-06-27T21:35:59.132725+0000","last_undegraded":"2024-06-27T21:35:59.132725+0000","last_fullsized":"2024-06-27T21:35:59.132725+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":10477,"log_dups_size":0,"ondisk_log_size":10477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87162880,"num_objects":1330,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":499,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":218080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":137756672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"0,7,11","objects":831},{"shards":"6,7,11","objects":499}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":718,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.913742+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:35:58.913742+0000","last_peered":"2024-06-27T21:35:58.913742+0000","last_clean":"2024-06-27T21:35:58.913742+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:35:58.913742+0000","last_undegraded":"2024-06-27T21:35:58.913742+0000","last_fullsized":"2024-06-27T21:35:58.913742+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"606'10169","reported_seq":10587,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.379965+0000","last_change":"2024-06-27T21:33:54.377162+0000","last_active":"2024-06-27T21:35:58.379965+0000","last_peered":"2024-06-27T21:35:58.379965+0000","last_clean":"2024-06-27T21:35:58.379965+0000","last_became_active":"2024-06-27T21:33:25.695260+0000","last_became_peered":"2024-06-27T21:33:25.695260+0000","last_unstale":"2024-06-27T21:35:58.379965+0000","last_undegraded":"2024-06-27T21:35:58.379965+0000","last_fullsized":"2024-06-27T21:35:58.379965+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9736","last_scrub_stamp":"2024-06-27T21:33:54.376905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.376905+0000","objects_scrubbed":1765,"log_size":10169,"log_dups_size":0,"ondisk_log_size":10169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87097344,"num_objects":1329,"num_object_clones":0,"num_object_copies":3987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":436,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":713,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.913422+0000","last_change":"2024-06-27T21:33:01.182699+0000","last_active":"2024-06-27T21:35:58.913422+0000","last_peered":"2024-06-27T21:35:58.913422+0000","last_clean":"2024-06-27T21:35:58.913422+0000","last_became_active":"2024-06-27T21:32:59.711777+0000","last_became_peered":"2024-06-27T21:32:59.711777+0000","last_unstale":"2024-06-27T21:35:58.913422+0000","last_undegraded":"2024-06-27T21:35:58.913422+0000","last_fullsized":"2024-06-27T21:35:58.913422+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"605'9989","reported_seq":21622,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.854527+0000","last_change":"2024-06-27T21:33:48.555029+0000","last_active":"2024-06-27T21:35:54.854527+0000","last_peered":"2024-06-27T21:35:54.854527+0000","last_clean":"2024-06-27T21:35:54.854527+0000","last_became_active":"2024-06-27T21:33:35.694586+0000","last_became_peered":"2024-06-27T21:33:35.694586+0000","last_unstale":"2024-06-27T21:35:54.854527+0000","last_undegraded":"2024-06-27T21:35:54.854527+0000","last_fullsized":"2024-06-27T21:35:54.854527+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'9582","last_scrub_stamp":"2024-06-27T21:33:48.554929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:48.554929+0000","objects_scrubbed":1728,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,7],"acting":[9,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":634,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.913520+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:35:58.913520+0000","last_peered":"2024-06-27T21:35:58.913520+0000","last_clean":"2024-06-27T21:35:58.913520+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:35:58.913520+0000","last_undegraded":"2024-06-27T21:35:58.913520+0000","last_fullsized":"2024-06-27T21:35:58.913520+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"605'10317","reported_seq":24161,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.733808+0000","last_change":"2024-06-27T21:33:54.883116+0000","last_active":"2024-06-27T21:35:54.733808+0000","last_peered":"2024-06-27T21:35:54.733808+0000","last_clean":"2024-06-27T21:35:54.733808+0000","last_became_active":"2024-06-27T21:33:25.693582+0000","last_became_peered":"2024-06-27T21:33:25.693582+0000","last_unstale":"2024-06-27T21:35:54.733808+0000","last_undegraded":"2024-06-27T21:35:54.733808+0000","last_fullsized":"2024-06-27T21:35:54.733808+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":1712,"log_size":10317,"log_dups_size":0,"ondisk_log_size":10317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86704128,"num_objects":1323,"num_object_clones":0,"num_object_copies":3969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":389,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":712,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.913769+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:35:58.913769+0000","last_peered":"2024-06-27T21:35:58.913769+0000","last_clean":"2024-06-27T21:35:58.913769+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:35:58.913769+0000","last_undegraded":"2024-06-27T21:35:58.913769+0000","last_fullsized":"2024-06-27T21:35:58.913769+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"606'10305","reported_seq":24522,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.020300+0000","last_change":"2024-06-27T21:33:50.968993+0000","last_active":"2024-06-27T21:35:58.020300+0000","last_peered":"2024-06-27T21:35:58.020300+0000","last_clean":"2024-06-27T21:35:58.020300+0000","last_became_active":"2024-06-27T21:33:25.692464+0000","last_became_peered":"2024-06-27T21:33:25.692464+0000","last_unstale":"2024-06-27T21:35:58.020300+0000","last_undegraded":"2024-06-27T21:35:58.020300+0000","last_fullsized":"2024-06-27T21:35:58.020300+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'9882","last_scrub_stamp":"2024-06-27T21:33:50.968852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.968852+0000","objects_scrubbed":1716,"log_size":10305,"log_dups_size":0,"ondisk_log_size":10305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84738048,"num_objects":1293,"num_object_clones":0,"num_object_copies":3879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":387,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.919746+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:35:58.919746+0000","last_peered":"2024-06-27T21:35:58.919746+0000","last_clean":"2024-06-27T21:35:58.919746+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:35:58.919746+0000","last_undegraded":"2024-06-27T21:35:58.919746+0000","last_fullsized":"2024-06-27T21:35:58.919746+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"607'10274","reported_seq":20120,"reported_epoch":607,"state":"active+recovering","last_fresh":"2024-06-27T21:35:59.442828+0000","last_change":"2024-06-27T21:33:26.778860+0000","last_active":"2024-06-27T21:35:59.442828+0000","last_peered":"2024-06-27T21:35:59.442828+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:26.702107+0000","last_became_peered":"2024-06-27T21:33:26.702107+0000","last_unstale":"2024-06-27T21:35:59.442828+0000","last_undegraded":"2024-06-27T21:35:59.442828+0000","last_fullsized":"2024-06-27T21:35:59.442828+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":10274,"log_dups_size":0,"ondisk_log_size":10274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79396864,"num_objects":1211,"num_object_clones":0,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3144,"num_bytes_recovered":173604864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,7,9","objects":1211}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":275,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.908027+0000","last_change":"2024-06-27T21:35:46.855308+0000","last_active":"2024-06-27T21:35:57.908027+0000","last_peered":"2024-06-27T21:35:57.908027+0000","last_clean":"2024-06-27T21:35:57.908027+0000","last_became_active":"2024-06-27T21:35:46.854995+0000","last_became_peered":"2024-06-27T21:35:46.854995+0000","last_unstale":"2024-06-27T21:35:57.908027+0000","last_undegraded":"2024-06-27T21:35:57.908027+0000","last_fullsized":"2024-06-27T21:35:57.908027+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"605'10713","reported_seq":13824,"reported_epoch":606,"state":"peering","last_fresh":"2024-06-27T21:35:57.959977+0000","last_change":"2024-06-27T21:35:57.959977+0000","last_active":"2024-06-27T21:35:57.959977+0000","last_peered":"2024-06-27T21:35:57.910802+0000","last_clean":"2024-06-27T21:35:57.910802+0000","last_became_active":"2024-06-27T21:35:47.935572+0000","last_became_peered":"2024-06-27T21:35:47.935572+0000","last_unstale":"2024-06-27T21:35:57.959977+0000","last_undegraded":"2024-06-27T21:35:57.959977+0000","last_fullsized":"2024-06-27T21:35:57.959977+0000","mapping_epoch":606,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'10169","last_scrub_stamp":"2024-06-27T21:33:50.128579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.128579+0000","objects_scrubbed":99,"log_size":10613,"log_dups_size":0,"ondisk_log_size":10613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1354,"num_bytes_recovered":88735744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":335,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.919688+0000","last_change":"2024-06-27T21:35:47.454903+0000","last_active":"2024-06-27T21:35:58.919688+0000","last_peered":"2024-06-27T21:35:58.919688+0000","last_clean":"2024-06-27T21:35:58.919688+0000","last_became_active":"2024-06-27T21:35:46.856358+0000","last_became_peered":"2024-06-27T21:35:46.856358+0000","last_unstale":"2024-06-27T21:35:58.919688+0000","last_undegraded":"2024-06-27T21:35:58.919688+0000","last_fullsized":"2024-06-27T21:35:58.919688+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.454820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.454820+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:10.172593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"605'10189","reported_seq":10644,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.911392+0000","last_change":"2024-06-27T21:33:49.922777+0000","last_active":"2024-06-27T21:35:57.911392+0000","last_peered":"2024-06-27T21:35:57.911392+0000","last_clean":"2024-06-27T21:35:57.911392+0000","last_became_active":"2024-06-27T21:33:25.697304+0000","last_became_peered":"2024-06-27T21:33:25.697304+0000","last_unstale":"2024-06-27T21:35:57.911392+0000","last_undegraded":"2024-06-27T21:35:57.911392+0000","last_fullsized":"2024-06-27T21:35:57.911392+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9759","last_scrub_stamp":"2024-06-27T21:33:49.922636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.922636+0000","objects_scrubbed":1732,"log_size":10189,"log_dups_size":0,"ondisk_log_size":10189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85327872,"num_objects":1302,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":913,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.911290+0000","last_change":"2024-06-27T21:35:47.736735+0000","last_active":"2024-06-27T21:35:57.911290+0000","last_peered":"2024-06-27T21:35:57.911290+0000","last_clean":"2024-06-27T21:35:57.911290+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:35:57.911290+0000","last_undegraded":"2024-06-27T21:35:57.911290+0000","last_fullsized":"2024-06-27T21:35:57.911290+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.736633+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.736633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:07.591052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"607'10396","reported_seq":10807,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:59.207973+0000","last_change":"2024-06-27T21:33:49.851182+0000","last_active":"2024-06-27T21:35:59.207973+0000","last_peered":"2024-06-27T21:35:59.207973+0000","last_clean":"2024-06-27T21:35:59.207973+0000","last_became_active":"2024-06-27T21:33:25.736643+0000","last_became_peered":"2024-06-27T21:33:25.736643+0000","last_unstale":"2024-06-27T21:35:59.207973+0000","last_undegraded":"2024-06-27T21:35:59.207973+0000","last_fullsized":"2024-06-27T21:35:59.207973+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9962","last_scrub_stamp":"2024-06-27T21:33:49.850981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.850981+0000","objects_scrubbed":1778,"log_size":10396,"log_dups_size":0,"ondisk_log_size":10396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":301,"reported_epoch":606,"state":"active+clean","last_fresh":"2024-06-27T21:35:57.908267+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:35:57.908267+0000","last_peered":"2024-06-27T21:35:57.908267+0000","last_clean":"2024-06-27T21:35:57.908267+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:35:57.908267+0000","last_undegraded":"2024-06-27T21:35:57.908267+0000","last_fullsized":"2024-06-27T21:35:57.908267+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"605'10298","reported_seq":17658,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.845142+0000","last_change":"2024-06-27T21:34:00.348871+0000","last_active":"2024-06-27T21:35:54.845142+0000","last_peered":"2024-06-27T21:35:54.845142+0000","last_clean":"2024-06-27T21:35:54.845142+0000","last_became_active":"2024-06-27T21:33:26.705196+0000","last_became_peered":"2024-06-27T21:33:26.705196+0000","last_unstale":"2024-06-27T21:35:54.845142+0000","last_undegraded":"2024-06-27T21:35:54.845142+0000","last_fullsized":"2024-06-27T21:35:54.845142+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9854","last_scrub_stamp":"2024-06-27T21:34:00.348580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:00.348580+0000","objects_scrubbed":1790,"log_size":10298,"log_dups_size":0,"ondisk_log_size":10298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88211456,"num_objects":1346,"num_object_clones":0,"num_object_copies":4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":444,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":301,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.919972+0000","last_change":"2024-06-27T21:35:46.858649+0000","last_active":"2024-06-27T21:35:58.919972+0000","last_peered":"2024-06-27T21:35:58.919972+0000","last_clean":"2024-06-27T21:35:58.919972+0000","last_became_active":"2024-06-27T21:35:46.857022+0000","last_became_peered":"2024-06-27T21:35:46.857022+0000","last_unstale":"2024-06-27T21:35:58.919972+0000","last_undegraded":"2024-06-27T21:35:58.919972+0000","last_fullsized":"2024-06-27T21:35:58.919972+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"605'10818","reported_seq":24591,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.994551+0000","last_change":"2024-06-27T21:33:23.873709+0000","last_active":"2024-06-27T21:35:54.994551+0000","last_peered":"2024-06-27T21:35:54.994551+0000","last_clean":"2024-06-27T21:35:54.994551+0000","last_became_active":"2024-06-27T21:33:23.851489+0000","last_became_peered":"2024-06-27T21:33:23.851489+0000","last_unstale":"2024-06-27T21:35:54.994551+0000","last_undegraded":"2024-06-27T21:35:54.994551+0000","last_fullsized":"2024-06-27T21:35:54.994551+0000","mapping_epoch":545,"log_start":"474'800","ondisk_log_start":"474'800","created":467,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169082880,"num_objects":2580,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10818,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6699,"num_bytes_recovered":340197376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"605'10614","reported_seq":13763,"reported_epoch":605,"state":"active+remapped+backfilling","last_fresh":"2024-06-27T21:35:57.529845+0000","last_change":"2024-06-27T21:35:48.060862+0000","last_active":"2024-06-27T21:35:57.529845+0000","last_peered":"2024-06-27T21:35:57.529845+0000","last_clean":"2024-06-27T21:35:45.849885+0000","last_became_active":"2024-06-27T21:35:48.014781+0000","last_became_peered":"2024-06-27T21:35:48.014781+0000","last_unstale":"2024-06-27T21:35:57.529845+0000","last_undegraded":"2024-06-27T21:35:57.529845+0000","last_fullsized":"2024-06-27T21:35:57.529845+0000","mapping_epoch":599,"log_start":"472'600","ondisk_log_start":"472'600","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'10541","last_scrub_stamp":"2024-06-27T21:35:43.673315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:43.673315+0000","objects_scrubbed":2645,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168427520,"num_objects":2570,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1228,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10614,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1353,"num_bytes_recovered":88670208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":2570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":307,"reported_epoch":605,"state":"active+clean","last_fresh":"2024-06-27T21:35:54.726110+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:35:54.726110+0000","last_peered":"2024-06-27T21:35:54.726110+0000","last_clean":"2024-06-27T21:35:54.726110+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:35:54.726110+0000","last_undegraded":"2024-06-27T21:35:54.726110+0000","last_fullsized":"2024-06-27T21:35:54.726110+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"607'11155","reported_seq":21331,"reported_epoch":607,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:35:59.429406+0000","last_change":"2024-06-27T21:32:36.645550+0000","last_active":"2024-06-27T21:35:59.429406+0000","last_peered":"2024-06-27T21:35:59.429406+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:32:34.904596+0000","last_became_peered":"2024-06-27T21:32:34.904596+0000","last_unstale":"2024-06-27T21:35:59.429406+0000","last_undegraded":"2024-06-27T21:32:22.691354+0000","last_fullsized":"2024-06-27T21:32:22.682941+0000","mapping_epoch":526,"log_start":"475'1100","ondisk_log_start":"475'1100","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174915584,"num_objects":2669,"num_object_clones":0,"num_object_copies":8007,"num_objects_missing_on_primary":4036,"num_objects_missing":4036,"num_objects_degraded":3,"num_objects_misplaced":7995,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11155,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2867,"num_bytes_recovered":187564032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10],"avail_no_missing":["5","6","8"],"object_location_counts":[{"shards":"5,6,7,8","objects":9},{"shards":"5,6,8","objects":2659},{"shards":"7,10","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":309,"reported_epoch":607,"state":"active+clean","last_fresh":"2024-06-27T21:35:58.987086+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:35:58.987086+0000","last_peered":"2024-06-27T21:35:58.987086+0000","last_clean":"2024-06-27T21:35:58.987086+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:35:58.987086+0000","last_undegraded":"2024-06-27T21:35:58.987086+0000","last_fullsized":"2024-06-27T21:35:58.987086+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":2738585624,"num_objects":41789,"num_object_clones":0,"num_object_copies":125367,"num_objects_missing_on_primary":4036,"num_objects_missing":4036,"num_objects_degraded":5633,"num_objects_misplaced":18828,"num_objects_unfound":0,"num_objects_dirty":41789,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":92413,"num_write_kb":3455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31437,"num_bytes_recovered":1754267696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9976561664,"data_stored":9976545376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267745,"ondisk_log_size":267745,"up":78,"acting":74,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253185,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482556,"kb_used_data":861360,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":92262596,"statfs":{"total":95995035648,"available":94476898304,"internally_reserved":0,"allocated":882032640,"data_stored":881447114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":636090821},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514821,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1242368,"kb_used_data":751668,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92502784,"statfs":{"total":95995035648,"available":94722850816,"internally_reserved":0,"allocated":769708032,"data_stored":769121832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":502462912},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743753,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251812,"kb_used_data":791640,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92493340,"statfs":{"total":95995035648,"available":94713180160,"internally_reserved":0,"allocated":810639360,"data_stored":810065663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":471202240},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":266287972685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267023,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1701384,"kb_used_data":1124600,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92043768,"statfs":{"total":95995035648,"available":94252818432,"internally_reserved":0,"allocated":1151590400,"data_stored":1151004720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528663,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1631012,"kb_used_data":1116696,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92114140,"statfs":{"total":95995035648,"available":94324879360,"internally_reserved":0,"allocated":1143496704,"data_stored":1142923007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":526645702},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757597,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994784,"kb_used_data":598488,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92750368,"statfs":{"total":95995035648,"available":94976376832,"internally_reserved":0,"allocated":612851712,"data_stored":612278015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161302,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844848,"kb_used_data":590884,"kb_used_omap":92,"kb_used_meta":253859,"kb_avail":92900304,"statfs":{"total":95995035648,"available":95129911296,"internally_reserved":0,"allocated":605065216,"data_stored":604487314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":259951714},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":801828,"kb_used_data":419416,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92943324,"statfs":{"total":95995035648,"available":95173963776,"internally_reserved":0,"allocated":429481984,"data_stored":428908287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":391576007},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411690,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706616,"kb_used_data":1270060,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92038536,"statfs":{"total":95995035648,"available":94247460864,"internally_reserved":0,"allocated":1300541440,"data_stored":1299959986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575215,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738739,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1553388,"kb_used_data":954396,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92191764,"statfs":{"total":95995035648,"available":94404366336,"internally_reserved":0,"allocated":977301504,"data_stored":976723735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":975310848,"data_stored":975306776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":948768768,"data_stored":948764696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1298530304,"data_stored":1298530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":426704896,"data_stored":426704896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":603193344,"data_stored":603193344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":610992128,"data_stored":610992128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1141571584,"data_stored":1141571584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1148911616,"data_stored":1148911616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":363986944,"data_stored":363986944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":808648704,"data_stored":808648704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":764940288,"data_stored":764936216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":885002240,"data_stored":884998168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:36:11.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg force-backfill 6.18 6.10 2024-06-27T21:36:11.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:11.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:11.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:11.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:11.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:36:11.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:11.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:11.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:11.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:11.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:11.984 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:12.013 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:12.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:12.045 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:12.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:12.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:11 smithi151 ceph-mon[28965]: osdmap e615: 12 total, 12 up, 10 in 2024-06-27T21:36:12.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:11 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:12.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:11 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:12.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:36:12.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:36:12.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:12.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:12.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:12.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:11 smithi190 ceph-mon[32247]: osdmap e615: 12 total, 12 up, 10 in 2024-06-27T21:36:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:11 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:11 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:36:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:36:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:12.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:11 smithi192 ceph-mon[32956]: osdmap e615: 12 total, 12 up, 10 in 2024-06-27T21:36:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:11 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:11 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:36:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:36:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:36:12.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:12.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.426 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:12.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:12.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:12.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:12.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:12.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:12.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:12.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:12.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:12.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:12.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:13.009 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:36:13.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:13.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:13.130 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:13 smithi190 ceph-mon[32247]: osdmap e616: 12 total, 12 up, 10 in 2024-06-27T21:36:13.130 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:13 smithi190 ceph-mon[32247]: pgmap v1579: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 49 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 117/120678 objects degraded (0.097%); 16273/120678 objects misplaced (13.485%) 2024-06-27T21:36:13.130 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1597157729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:36:13.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:13.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:13.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:13 smithi151 ceph-mon[28965]: osdmap e616: 12 total, 12 up, 10 in 2024-06-27T21:36:13.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:13 smithi151 ceph-mon[28965]: pgmap v1579: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 49 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 117/120678 objects degraded (0.097%); 16273/120678 objects misplaced (13.485%) 2024-06-27T21:36:13.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1597157729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:36:13.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:13.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:13 smithi192 ceph-mon[32956]: osdmap e616: 12 total, 12 up, 10 in 2024-06-27T21:36:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:13 smithi192 ceph-mon[32956]: pgmap v1579: 57 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 49 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 117/120678 objects degraded (0.097%); 16273/120678 objects misplaced (13.485%) 2024-06-27T21:36:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1597157729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:36:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:13.398 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:13.419 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:13.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:13.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:13.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:13.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:13.720 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:13.720 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:13.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:13.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:13.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:13.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:13.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:14.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:14.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:14.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:14.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:14.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:14 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:14.224 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1597157729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:36:14.224 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:14 smithi151 ceph-mon[28965]: osdmap e617: 12 total, 12 up, 10 in 2024-06-27T21:36:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:14 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1597157729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:36:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:14 smithi190 ceph-mon[32247]: osdmap e617: 12 total, 12 up, 10 in 2024-06-27T21:36:14.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:14.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:14 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1597157729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:36:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:14 smithi192 ceph-mon[32956]: osdmap e617: 12 total, 12 up, 10 in 2024-06-27T21:36:14.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:14.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:14.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:36:14.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:14.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:14.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:14.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:14.714 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:14.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:14.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:14.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:14.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:14.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:15.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:15.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:15.102 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:15 smithi190 ceph-mon[32247]: pgmap v1581: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 50 active+clean; 2.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 133 op/s; 117/119940 objects degraded (0.098%); 16213/119940 objects misplaced (13.518%); 8.5 MiB/s, 362 objects/s recovering 2024-06-27T21:36:15.102 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:15 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 117/120678 objects degraded (0.097%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:15.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:15.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:15 smithi151 ceph-mon[28965]: pgmap v1581: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 50 active+clean; 2.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 133 op/s; 117/119940 objects degraded (0.098%); 16213/119940 objects misplaced (13.518%); 8.5 MiB/s, 362 objects/s recovering 2024-06-27T21:36:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:15 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 117/120678 objects degraded (0.097%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:15.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:15.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:15 smithi192 ceph-mon[32956]: pgmap v1581: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering, 50 active+clean; 2.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 133 op/s; 117/119940 objects degraded (0.098%); 16213/119940 objects misplaced (13.518%); 8.5 MiB/s, 362 objects/s recovering 2024-06-27T21:36:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:15 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 117/120678 objects degraded (0.097%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:15.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:15.412 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:15.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:15.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:15.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:15.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:15.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:15.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:15.867 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:36:15.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:36:15.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:15.889 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:36:15.889 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:36:15.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:15.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:15.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:36:15.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:36:15.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:36:15.938 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:36:15.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:36:15.962 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:36:15.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:36:15.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:36:16.008 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:36:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:36:16.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:16.030 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:36:16.031 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:36:16.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:16.054 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:36:16.054 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:36:16.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:36:16.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:36:16.098 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:36:16.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:36:16.120 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:36:16.120 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:36:16.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:16.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:16.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:16.176 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3981431022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:16.187 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3981431022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:16.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:16.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:16.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3981431022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:16.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:16.401 INFO:teuthology.orchestra.run.smithi151.stderr:instructing pg(s) [6.10,6.18] on osd.0 to force-backfill; 2024-06-27T21:36:16.401 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-06-27T21:36:16.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:16.425 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:36:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:36:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:36:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:36:16.425 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:36:16.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:16.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:16.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:36:16.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:16.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:16.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:16.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:16.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:16.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:16.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:17.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:17.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:17.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:17.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:17.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:17.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:17.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:17 smithi190 ceph-mon[32247]: pgmap v1582: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 125/118641 objects degraded (0.105%); 15630/118641 objects misplaced (13.174%); 9.1 MiB/s, 420 objects/s recovering 2024-06-27T21:36:17.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:17 smithi190 ceph-mon[32247]: from='client.16773 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["6.18", "6.10"], "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:36:17.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:36:17.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:17 smithi192 ceph-mon[32956]: pgmap v1582: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 125/118641 objects degraded (0.105%); 15630/118641 objects misplaced (13.174%); 9.1 MiB/s, 420 objects/s recovering 2024-06-27T21:36:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:17 smithi192 ceph-mon[32956]: from='client.16773 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["6.18", "6.10"], "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:36:17.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:17.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:17.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:17.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:17.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:17.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:17 smithi151 ceph-mon[28965]: pgmap v1582: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 125/118641 objects degraded (0.105%); 15630/118641 objects misplaced (13.174%); 9.1 MiB/s, 420 objects/s recovering 2024-06-27T21:36:17.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:17 smithi151 ceph-mon[28965]: from='client.16773 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["6.18", "6.10"], "target": ["mon-mgr", ""]}]: dispatch 2024-06-27T21:36:17.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:17.540 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:17.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:17.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:17.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:17.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:17.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:17.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:17.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:17.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:18.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:18.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:18.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:18.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:18 smithi190 ceph-mon[32247]: osdmap e618: 12 total, 12 up, 10 in 2024-06-27T21:36:18.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:18.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:18 smithi192 ceph-mon[32956]: osdmap e618: 12 total, 12 up, 10 in 2024-06-27T21:36:18.296 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:18.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:18.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:18.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:18.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:18 smithi151 ceph-mon[28965]: osdmap e618: 12 total, 12 up, 10 in 2024-06-27T21:36:18.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:18.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:18.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:18.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:18.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:18.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:18.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:18.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:18.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:19.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:19.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:19.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:19.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:19.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:19.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:19.283 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:36:19.283 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:36:19.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:36:19.283 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:36:19.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 531/526 [PushOp(6:3696e1a2:::benchmark_data_smithi192_117994_object45109:head, version: 486'2900, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:3696e1a2:::benchmark_data_smithi192_117994_object45109:head@486'2900, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:37.453743+0000", 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.827346092, 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4397225599999999, 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:37.453743+0000", 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:37.453743+0000", 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:37.453744+0000", 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1299999999999999e-07 2024-06-27T21:36:19.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:37.453771+0000", 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7265000000000001e-05 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:37.453792+0000", 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0865000000000001e-05 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:37.453804+0000", 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2057e-05 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:39.860746+0000", 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4069418919999999 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:39.861502+0000", 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075619100000000002 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:39.893466+0000", 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.031963476999999997 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 532/526 [PushReplyOp(6:3696e1a2:::benchmark_data_smithi192_117994_object45109:head)])", 2024-06-27T21:36:19.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:41.562700+0000", 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 217.71838887300001, 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1343877629999999, 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.562700+0000", 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.562700+0000", 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.562701+0000", 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3499999999999996e-07 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.562705+0000", 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3170000000000003e-06 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.562730+0000", 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5060999999999999e-05 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.562736+0000", 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8170000000000004e-06 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.683822+0000", 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1210860180000002 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.697088+0000", 2024-06-27T21:36:19.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013265915 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 532/526 [PushReplyOp(6:3696e1a2:::benchmark_data_smithi192_117994_object45109:head)])", 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:41.564663+0000", 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 217.71642614000001, 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1666441939999999, 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.564663+0000", 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.564663+0000", 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.564664+0000", 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8400000000000002e-07 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.564670+0000", 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8470000000000001e-06 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.564700+0000", 2024-06-27T21:36:19.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9988000000000002e-05 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:41.564708+0000", 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1640000000000001e-06 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.699600+0000", 2024-06-27T21:36:19.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1348921540000001 2024-06-27T21:36:19.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.731307+0000", 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.031707056999999997 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 533/526 [PushOp(6:3a9dba04:::benchmark_data_smithi192_117994_object45745:head, version: 486'2940, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:3a9dba04:::benchmark_data_smithi192_117994_object45745:head@486'2940, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:44.270717+0000", 2024-06-27T21:36:19.298 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 215.010371875, 2024-06-27T21:36:19.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:19 smithi192 ceph-mon[32956]: osdmap e619: 12 total, 12 up, 10 in 2024-06-27T21:36:19.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:19 smithi192 ceph-mon[32956]: pgmap v1585: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 127 op/s; 125/118374 objects degraded (0.106%); 15630/118374 objects misplaced (13.204%); 4.8 MiB/s, 194 objects/s recovering 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.472291851, 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.270717+0000", 2024-06-27T21:36:19.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.270717+0000", 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.270718+0000", 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0100000000000003e-07 2024-06-27T21:36:19.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.270750+0000", 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1742000000000002e-05 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.270793+0000", 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3149000000000003e-05 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.270812+0000", 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8329999999999999e-05 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:46.710785+0000", 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4399734959999999 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:46.711493+0000", 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070799000000000003 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:46.743009+0000", 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.031516242999999999 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 533/526 [PushReplyOp(6:3a9dba04:::benchmark_data_smithi192_117994_object45745:head)])", 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:48.721664+0000", 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.559425667, 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7337532950000001, 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.721664+0000", 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.721664+0000", 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.721665+0000", 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-06 2024-06-27T21:36:19.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.721669+0000", 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1749999999999998e-06 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.721694+0000", 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4790000000000002e-05 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.721702+0000", 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6119999999999999e-06 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:50.438877+0000", 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.71717465 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:50.455417+0000", 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016539865000000001 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 533/526 [PushReplyOp(6:3a9dba04:::benchmark_data_smithi192_117994_object45745:head)])", 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:48.722862+0000", 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.55822727, 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9834898110000001, 2024-06-27T21:36:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.722862+0000", 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.722862+0000", 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.722863+0000", 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5000000000000003e-07 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.722867+0000", 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0339999999999998e-06 2024-06-27T21:36:19.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.722887+0000", 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0418999999999999e-05 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:48.722894+0000", 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4619999999999999e-06 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:50.629038+0000", 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9061432759999999 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:50.706352+0000", 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.077314069999999999 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 540/526 [PushReplyOp(6:3f8e676a:::benchmark_data_smithi192_117994_object48067:head)])", 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:14.087915+0000", 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 185.193173833, 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8906088240000001, 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.087915+0000", 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.087915+0000", 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.087916+0000", 2024-06-27T21:36:19.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8299999999999996e-07 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.087921+0000", 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1660000000000002e-06 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.087946+0000", 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4698e-05 2024-06-27T21:36:19.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.087955+0000", 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.217e-06 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.965213+0000", 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.87725794 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.978524+0000", 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01331102 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 540/526 [PushReplyOp(6:3f8e676a:::benchmark_data_smithi192_117994_object48067:head)])", 2024-06-27T21:36:19.319 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:14.089632+0000", 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 185.191457138, 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0146005910000002, 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.089632+0000", 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.089632+0000", 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.089633+0000", 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2600000000000002e-07 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.089637+0000", 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5410000000000002e-06 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.089662+0000", 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4975000000000001e-05 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.089668+0000", 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9329999999999998e-06 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.072442+0000", 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.982774179 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.104233+0000", 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.031790337000000002 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.324 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 540/526 [PushReplyOp(6:3b148def:::benchmark_data_smithi192_117994_object48326:head)])", 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:16.268394+0000", 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 183.01269489699999, 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.55937075, 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.268394+0000", 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.268394+0000", 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.268395+0000", 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7499999999999999e-07 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.268401+0000", 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4049999999999999e-06 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.268427+0000", 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6614999999999998e-05 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.268435+0000", 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5950000000000001e-06 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.813567+0000", 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.545132607 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.827765+0000", 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014197652999999999 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 540/526 [PushReplyOp(6:3b148def:::benchmark_data_smithi192_117994_object48326:head)])", 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:16.270717+0000", 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 183.01037208, 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5874785330000001, 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.329 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.270717+0000", 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.270717+0000", 2024-06-27T21:36:19.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.270718+0000", 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4099999999999998e-07 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.270726+0000", 2024-06-27T21:36:19.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0900000000000005e-06 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.270754+0000", 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7937999999999999e-05 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.270761+0000", 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7139999999999996e-06 2024-06-27T21:36:19.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.828364+0000", 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5576036470000001 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.858196+0000", 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.029831502999999999 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.333 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 551/526 [PushReplyOp(6:31fd6786:::benchmark_data_smithi192_117994_object49664:head)])", 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:30.511032+0000", 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 168.77005689999999, 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8243082940000002, 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.334 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.511032+0000", 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.511032+0000", 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.511033+0000", 2024-06-27T21:36:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0800000000000004e-07 2024-06-27T21:36:19.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.511041+0000", 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1440000000000003e-06 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.511058+0000", 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6725e-05 2024-06-27T21:36:19.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:19 smithi190 ceph-mon[32247]: osdmap e619: 12 total, 12 up, 10 in 2024-06-27T21:36:19.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:19 smithi190 ceph-mon[32247]: pgmap v1585: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 127 op/s; 125/118374 objects degraded (0.106%); 15630/118374 objects misplaced (13.204%); 4.8 MiB/s, 194 objects/s recovering 2024-06-27T21:36:19.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:19.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:19.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.511064+0000", 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9660000000000001e-06 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:32.305195+0000", 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7941315740000001 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:32.335341+0000", 2024-06-27T21:36:19.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.030145176999999999 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 562/526 [PushOp(6:3e0eb16b:::benchmark_data_smithi192_117994_object53939:head, version: 486'3456, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:3e0eb16b:::benchmark_data_smithi192_117994_object53939:head@486'3456, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:06.669052+0000", 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 132.612036824, 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.630691863, 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.341 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669052+0000", 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669052+0000", 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669054+0000", 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.062e-06 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669078+0000", 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4839999999999999e-05 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669114+0000", 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5289e-05 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669131+0000", 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7782e-05 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.276632+0000", 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.607500828 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.277207+0000", 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057437199999999997 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.299744+0000", 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.022537689999999999 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.345 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 563/526 [PushOp(6:37131d4f:::benchmark_data_smithi192_117994_object54534:head, version: 486'3492, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:37131d4f:::benchmark_data_smithi192_117994_object54534:head@486'3492, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 127.47613448, 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8262880849999998, 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.805071+0000", 2024-06-27T21:36:19.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011569999999999999 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.805091+0000", 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0188999999999999e-05 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.805103+0000", 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1243999999999999e-05 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.607611+0000", 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.802508198 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.608194+0000", 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058344199999999999 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.631243+0000", 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023048639999999999 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 563/526 [PushReplyOp(6:3fd23acf:::benchmark_data_smithi192_117994_object54296:head)])", 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:12.279687+0000", 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 127.001402103, 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.580862403, 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.350 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279687+0000", 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279687+0000", 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279688+0000", 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6300000000000004e-07 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279693+0000", 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4280000000000004e-06 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.308287+0000", 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.028593997999999999 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.308293+0000", 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6280000000000002e-06 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.833554+0000", 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5252613340000001 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.860550+0000", 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026995252000000001 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 563/526 [PushReplyOp(6:3fd23acf:::benchmark_data_smithi192_117994_object54296:head)])", 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:12.281111+0000", 2024-06-27T21:36:19.354 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 126.999977999, 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5498509820000002, 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281111+0000", 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281111+0000", 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281112+0000", 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281118+0000", 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1510000000000001e-06 2024-06-27T21:36:19.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281134+0000", 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6079e-05 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281140+0000", 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7799999999999997e-06 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.819872+0000", 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.538732019 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.830962+0000", 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011090368999999999 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 564/526 [PushOp(6:3f5493a1:::benchmark_data_smithi192_117994_object54907:head, version: 486'3518, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:3f5493a1:::benchmark_data_smithi192_117994_object54907:head@486'3518, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 123.65265860700001, 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.825308699, 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0800000000000004e-07 2024-06-27T21:36:19.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628460+0000", 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8890999999999998e-05 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628498+0000", 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7774999999999997e-05 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628513+0000", 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4922999999999999e-05 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:17.429599+0000", 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.801085638 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:17.430172+0000", 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057386999999999998 2024-06-27T21:36:19.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:17.453739+0000", 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023566894000000001 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.14 589/550 [PushOp(6:2810808f:::benchmark_data_smithi192_117994_object70270:head, version: 491'4530, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:2810808f:::benchmark_data_smithi192_117994_object70270:head@491'4530, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:19.148860+0000", 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 60.132229166000002, 2024-06-27T21:36:19.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.685495178, 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148860+0000", 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148860+0000", 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148861+0000", 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2099999999999995e-07 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148951+0000", 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0097000000000002e-05 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148986+0000", 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5046000000000003e-05 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148997+0000", 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0667e-05 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:20.834172+0000", 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6851757279999999 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:20.834183+0000", 2024-06-27T21:36:19.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.082e-05 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:20.834355+0000", 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017199899999999999 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 595/526 [PushOp(6:37916176:::benchmark_data_smithi192_117994_object62965:head, version: 489'4052, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:37916176:::benchmark_data_smithi192_117994_object62965:head@489'4052, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.368 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.297408670000003, 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254610199999999, 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7e-07 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983704+0000", 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3281e-05 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983735+0000", 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.082e-05 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983745+0000", 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0020000000000001e-05 2024-06-27T21:36:19.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587861+0000", 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6041156220000001 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.588341+0000", 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048023500000000003 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.609142+0000", 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020800572 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.373 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.14 595/550 [PushOp(6:2828d551:::benchmark_data_smithi192_117994_object77037:head, version: 493'4953, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:2828d551:::benchmark_data_smithi192_117994_object77037:head@493'4953, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.279371531000002, 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5858424599999998, 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:36:19.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8899999999999999e-07 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001810+0000", 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1711000000000003e-05 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001853+0000", 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2876000000000001e-05 2024-06-27T21:36:19.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001869+0000", 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6161999999999999e-05 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587317+0000", 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5854477760000001 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587331+0000", 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4338000000000001e-05 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587560+0000", 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000228908 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.378 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 604/526 [PushReplyOp(6:3c890e83:::benchmark_data_smithi192_117994_object64998:head)])", 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:53.074990+0000", 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.206098953000001, 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6298133859999999, 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:36:19.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074990+0000", 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074990+0000", 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074991+0000", 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1999999999999999e-07 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074995+0000", 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9999999999999998e-06 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.075019+0000", 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4428999999999999e-05 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.075025+0000", 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9669999999999998e-06 2024-06-27T21:36:19.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.678270+0000", 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6032448320000001 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.704804+0000", 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026533537999999999 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.382 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:36:19.383 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:36:19.383 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:36:19.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:19.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:19 smithi151 ceph-mon[28965]: osdmap e619: 12 total, 12 up, 10 in 2024-06-27T21:36:19.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:19 smithi151 ceph-mon[28965]: pgmap v1585: 57 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 2 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 127 op/s; 125/118374 objects degraded (0.106%); 15630/118374 objects misplaced (13.204%); 4.8 MiB/s, 194 objects/s recovering 2024-06-27T21:36:19.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:19.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:19.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:19.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:19.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:19.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:19.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:19.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:19.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:19.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:20.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:20.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:20.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:36:20.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:20.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:20 smithi190 ceph-mon[32247]: osdmap e620: 12 total, 12 up, 10 in 2024-06-27T21:36:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:20 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 125/118374 objects degraded (0.106%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:20 smithi190 ceph-mon[32247]: osdmap e621: 12 total, 12 up, 10 in 2024-06-27T21:36:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:20 smithi192 ceph-mon[32956]: osdmap e620: 12 total, 12 up, 10 in 2024-06-27T21:36:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:20 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 125/118374 objects degraded (0.106%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:20 smithi192 ceph-mon[32956]: osdmap e621: 12 total, 12 up, 10 in 2024-06-27T21:36:20.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:20.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:20.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:20 smithi151 ceph-mon[28965]: osdmap e620: 12 total, 12 up, 10 in 2024-06-27T21:36:20.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:20 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 125/118374 objects degraded (0.106%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:20.430 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:20 smithi151 ceph-mon[28965]: osdmap e621: 12 total, 12 up, 10 in 2024-06-27T21:36:20.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:20.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:20.467 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:20.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:20.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:20.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:20.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:20.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:20.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:20.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:20.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:20.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:20.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:21.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:21.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:21.145 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:21 smithi192 ceph-mon[32956]: pgmap v1588: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 125/116871 objects degraded (0.107%); 14932/116871 objects misplaced (12.776%); 6.7 MiB/s, 107 objects/s recovering 2024-06-27T21:36:21.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:21.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:21.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:21.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:21.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:21.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:21 smithi151 ceph-mon[28965]: pgmap v1588: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 125/116871 objects degraded (0.107%); 14932/116871 objects misplaced (12.776%); 6.7 MiB/s, 107 objects/s recovering 2024-06-27T21:36:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:21.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:21.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:21 smithi190 ceph-mon[32247]: pgmap v1588: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 50 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 147 op/s; 125/116871 objects degraded (0.107%); 14932/116871 objects misplaced (12.776%); 6.7 MiB/s, 107 objects/s recovering 2024-06-27T21:36:21.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:21.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:21.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:21.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:21.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:21.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:21.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:21.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:21.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:22.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:22.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:22.108 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:22.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:22.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:22.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:22.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:22.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:22 smithi151 ceph-mon[28965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:36:22.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:22 smithi151 ceph-mon[28965]: osdmap e622: 12 total, 12 up, 10 in 2024-06-27T21:36:22.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:22 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:22.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:22 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:22.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:22.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:22.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:22 smithi190 ceph-mon[32247]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:36:22.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:22 smithi190 ceph-mon[32247]: osdmap e622: 12 total, 12 up, 10 in 2024-06-27T21:36:22.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:22 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:22.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:22 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:22.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:22.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:22 smithi192 ceph-mon[32956]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:36:22.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:22 smithi192 ceph-mon[32956]: osdmap e622: 12 total, 12 up, 10 in 2024-06-27T21:36:22.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:22 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:22.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:22 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:22.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:22.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:22.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:22.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:22.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:22.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:22.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:22.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:23.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:23.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:23.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:23.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:23.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:23.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:23.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:23 smithi151 ceph-mon[28965]: osdmap e623: 12 total, 12 up, 10 in 2024-06-27T21:36:23.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:23 smithi151 ceph-mon[28965]: pgmap v1591: 57 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 51 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 183 op/s; 126/116169 objects degraded (0.108%); 14536/116169 objects misplaced (12.513%); 13 MiB/s, 288 objects/s recovering 2024-06-27T21:36:23.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:23 smithi151 ceph-mon[28965]: osdmap e624: 12 total, 12 up, 10 in 2024-06-27T21:36:23.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:23.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:23.441 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:23 smithi192 ceph-mon[32956]: osdmap e623: 12 total, 12 up, 10 in 2024-06-27T21:36:23.441 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:23 smithi192 ceph-mon[32956]: pgmap v1591: 57 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 51 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 183 op/s; 126/116169 objects degraded (0.108%); 14536/116169 objects misplaced (12.513%); 13 MiB/s, 288 objects/s recovering 2024-06-27T21:36:23.441 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:23 smithi192 ceph-mon[32956]: osdmap e624: 12 total, 12 up, 10 in 2024-06-27T21:36:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:23 smithi190 ceph-mon[32247]: osdmap e623: 12 total, 12 up, 10 in 2024-06-27T21:36:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:23 smithi190 ceph-mon[32247]: pgmap v1591: 57 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 51 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 183 op/s; 126/116169 objects degraded (0.108%); 14536/116169 objects misplaced (12.513%); 13 MiB/s, 288 objects/s recovering 2024-06-27T21:36:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:23 smithi190 ceph-mon[32247]: osdmap e624: 12 total, 12 up, 10 in 2024-06-27T21:36:23.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:23.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:23.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:23.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:23.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:23.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:23.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:23.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:23.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:24.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:24.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:24.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:36:24.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:24.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:24.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:24.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:24.371 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:24.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:24.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:24.467 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:36:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:36:24.468 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:36:24.468 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:36:24.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:24.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:24.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:24.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:24.732 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:36:24.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:24.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:24.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:24.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:24.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:25.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:25.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:25.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:25.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:25.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:25.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:36:25.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:25.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:25 smithi151 ceph-mon[28965]: pgmap v1593: 57 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 51 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 126/115500 objects degraded (0.109%); 14482/115500 objects misplaced (12.539%); 12 MiB/s, 260 objects/s recovering 2024-06-27T21:36:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3874212202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:25 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 126/115500 objects degraded (0.109%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:25 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3874212202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:25.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:25 smithi151 ceph-mon[28965]: osdmap e625: 12 total, 12 up, 10 in 2024-06-27T21:36:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:25 smithi190 ceph-mon[32247]: pgmap v1593: 57 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 51 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 126/115500 objects degraded (0.109%); 14482/115500 objects misplaced (12.539%); 12 MiB/s, 260 objects/s recovering 2024-06-27T21:36:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3874212202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:25 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 126/115500 objects degraded (0.109%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:25 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3874212202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:25.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:25 smithi190 ceph-mon[32247]: osdmap e625: 12 total, 12 up, 10 in 2024-06-27T21:36:25.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:25.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:25 smithi192 ceph-mon[32956]: pgmap v1593: 57 pgs: 1 active+recovering+undersized+degraded+remapped+forced_backfill, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 51 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 126/115500 objects degraded (0.109%); 14482/115500 objects misplaced (12.539%); 12 MiB/s, 260 objects/s recovering 2024-06-27T21:36:25.539 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3874212202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:25.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:25 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 126/115500 objects degraded (0.109%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:25.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:25 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:25.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3874212202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:25.540 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:25 smithi192 ceph-mon[32956]: osdmap e625: 12 total, 12 up, 10 in 2024-06-27T21:36:25.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:25.640 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:25.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:25.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:25.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:25.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:25.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:25.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:25.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:26.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:26.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:26.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:26.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:26.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:26.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:26.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:26.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:26.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:26.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:26.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:26.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:26.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:26.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:26.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:26.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:26.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:27.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:27.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:27.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:27.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:27.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:36:27.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:36:27.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:27.284 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:36:27.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:36:27.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:27.305 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:36:27.305 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:36:27.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:36:27.326 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:36:27.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:36:27.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:36:27.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:36:27.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:36:27.379 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:27 smithi192 ceph-mon[32956]: pgmap v1595: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 3/114438 objects degraded (0.003%); 13396/114438 objects misplaced (11.706%); 2.2 MiB/s, 71 objects/s recovering 2024-06-27T21:36:27.380 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/666322450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:27.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:27.390 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:36:27.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:36:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:27.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:27.509 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:36:27.510 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:36:27.510 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:27 smithi151 ceph-mon[28965]: pgmap v1595: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 3/114438 objects degraded (0.003%); 13396/114438 objects misplaced (11.706%); 2.2 MiB/s, 71 objects/s recovering 2024-06-27T21:36:27.510 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/666322450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:27.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:27.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:27 smithi190 ceph-mon[32247]: pgmap v1595: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 3/114438 objects degraded (0.003%); 13396/114438 objects misplaced (11.706%); 2.2 MiB/s, 71 objects/s recovering 2024-06-27T21:36:27.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/666322450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:27.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:36:27.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:36:27.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:27.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:36:27.557 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:36:27.582 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:36:27.583 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:36:27.603 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:36:27.603 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:36:27.629 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:27.629 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:27.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:27.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:27.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:27.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:27.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:27.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:27.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:28.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.034 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:28.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:28.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:28.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:28.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.282 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:28.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:28.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:36:28.382 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:36:28.382 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:36:28.382 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-06-27T21:36:28.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:36:28.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:28.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:28.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:28.635 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:28.636 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:28.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:28.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:28.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:28.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:28.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:28.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:28.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:36:29.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:29.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:29.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:29.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:29.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:29 smithi192 ceph-mon[32956]: pgmap v1596: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 108 op/s; 3/114177 objects degraded (0.003%); 13396/114177 objects misplaced (11.733%); 1.8 MiB/s, 60 objects/s recovering 2024-06-27T21:36:29.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:29.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:29.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:29 smithi151 ceph-mon[28965]: pgmap v1596: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 108 op/s; 3/114177 objects degraded (0.003%); 13396/114177 objects misplaced (11.733%); 1.8 MiB/s, 60 objects/s recovering 2024-06-27T21:36:29.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:29 smithi190 ceph-mon[32247]: pgmap v1596: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 108 op/s; 3/114177 objects degraded (0.003%); 13396/114177 objects misplaced (11.733%); 1.8 MiB/s, 60 objects/s recovering 2024-06-27T21:36:29.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:29.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:29.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:29.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:29.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:29.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:29.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:29.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:29.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:30.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:30.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:30.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:30.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:30.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:36:30.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:30.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:30.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:30 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/114177 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:30.366 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:30.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:30.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:30.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:30 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/114177 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:30.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:30.538 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:30 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/114177 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:30.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:30.553 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:30.554 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:30.573 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:36:30.574 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:36:30.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:30.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:30.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:30.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:30.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:30.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:31.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:31.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:31.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:31.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:31.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:36:31.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:31.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:31.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:31.416 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-06-27T21:36:30.164585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2469921336,"num_objects":37683,"num_object_clones":0,"num_object_copies":113049,"num_objects_missing_on_primary":3769,"num_objects_missing":3769,"num_objects_degraded":3,"num_objects_misplaced":13183,"num_objects_unfound":0,"num_objects_dirty":37683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":91124,"num_write_kb":3243713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37975,"num_bytes_recovered":1973752400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273353,"ondisk_log_size":273353,"up":171,"acting":169,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":182,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":12790612,"kb_used_data":8080052,"kb_used_omap":106,"kb_used_meta":4710357,"kb_avail":924660908,"statfs":{"total":959950356480,"available":946852769792,"internally_reserved":0,"allocated":8273973248,"data_stored":8267355656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109491,"internal_metadata":4823405645},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-68157440,"num_objects":-1040,"num_object_clones":0,"num_object_copies":-3120,"num_objects_missing_on_primary":-96,"num_objects_missing":-96,"num_objects_degraded":-123,"num_objects_misplaced":-1353,"num_objects_unfound":0,"num_objects_dirty":-1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":14811136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001873"},"pg_stats":[{"pgid":"6.18","version":"625'10522","reported_seq":27690,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:25.196201+0000","last_change":"2024-06-27T21:36:19.056568+0000","last_active":"2024-06-27T21:36:25.196201+0000","last_peered":"2024-06-27T21:36:25.196201+0000","last_clean":"2024-06-27T21:36:25.196201+0000","last_became_active":"2024-06-27T21:36:19.037779+0000","last_became_peered":"2024-06-27T21:36:19.037779+0000","last_unstale":"2024-06-27T21:36:25.196201+0000","last_undegraded":"2024-06-27T21:36:25.196201+0000","last_fullsized":"2024-06-27T21:36:25.196201+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10353","last_scrub_stamp":"2024-06-27T21:35:42.524197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.524197+0000","objects_scrubbed":1327,"log_size":10522,"log_dups_size":0,"ondisk_log_size":10522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:08.680736+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3341,"num_bytes_recovered":85590016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"625'10399","reported_seq":25206,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.536553+0000","last_change":"2024-06-27T21:36:23.070309+0000","last_active":"2024-06-27T21:36:29.536553+0000","last_peered":"2024-06-27T21:36:29.536553+0000","last_clean":"2024-06-27T21:36:29.536553+0000","last_became_active":"2024-06-27T21:36:23.050064+0000","last_became_peered":"2024-06-27T21:36:23.050064+0000","last_unstale":"2024-06-27T21:36:29.536553+0000","last_undegraded":"2024-06-27T21:36:29.536553+0000","last_fullsized":"2024-06-27T21:36:29.536553+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10195","last_scrub_stamp":"2024-06-27T21:35:42.766433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.766433+0000","objects_scrubbed":1384,"log_size":10399,"log_dups_size":0,"ondisk_log_size":10399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:57.757734+0000","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27000832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":195,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.873512+0000","last_change":"2024-06-27T21:33:02.987696+0000","last_active":"2024-06-27T21:36:24.873512+0000","last_peered":"2024-06-27T21:36:24.873512+0000","last_clean":"2024-06-27T21:36:24.873512+0000","last_became_active":"2024-06-27T21:33:01.707872+0000","last_became_peered":"2024-06-27T21:33:01.707872+0000","last_unstale":"2024-06-27T21:36:24.873512+0000","last_undegraded":"2024-06-27T21:36:24.873512+0000","last_fullsized":"2024-06-27T21:36:24.873512+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,10],"acting":[11,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":201,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.873516+0000","last_change":"2024-06-27T21:35:46.856033+0000","last_active":"2024-06-27T21:36:24.873516+0000","last_peered":"2024-06-27T21:36:24.873516+0000","last_clean":"2024-06-27T21:36:24.873516+0000","last_became_active":"2024-06-27T21:35:46.855054+0000","last_became_peered":"2024-06-27T21:35:46.855054+0000","last_unstale":"2024-06-27T21:36:24.873516+0000","last_undegraded":"2024-06-27T21:36:24.873516+0000","last_fullsized":"2024-06-27T21:36:24.873516+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,10],"acting":[11,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":228,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.873474+0000","last_change":"2024-06-27T21:35:46.857696+0000","last_active":"2024-06-27T21:36:24.873474+0000","last_peered":"2024-06-27T21:36:24.873474+0000","last_clean":"2024-06-27T21:36:24.873474+0000","last_became_active":"2024-06-27T21:35:46.857381+0000","last_became_peered":"2024-06-27T21:35:46.857381+0000","last_unstale":"2024-06-27T21:36:24.873474+0000","last_undegraded":"2024-06-27T21:36:24.873474+0000","last_fullsized":"2024-06-27T21:36:24.873474+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:52.993157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:52.993157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,5],"acting":[11,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":197,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.744837+0000","last_change":"2024-06-27T21:33:02.568824+0000","last_active":"2024-06-27T21:36:24.744837+0000","last_peered":"2024-06-27T21:36:24.744837+0000","last_clean":"2024-06-27T21:36:24.744837+0000","last_became_active":"2024-06-27T21:33:01.773541+0000","last_became_peered":"2024-06-27T21:33:01.773541+0000","last_unstale":"2024-06-27T21:36:24.744837+0000","last_undegraded":"2024-06-27T21:36:24.744837+0000","last_fullsized":"2024-06-27T21:36:24.744837+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":269,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.733454+0000","last_change":"2024-06-27T21:35:46.858389+0000","last_active":"2024-06-27T21:36:24.733454+0000","last_peered":"2024-06-27T21:36:24.733454+0000","last_clean":"2024-06-27T21:36:24.733454+0000","last_became_active":"2024-06-27T21:35:46.858050+0000","last_became_peered":"2024-06-27T21:35:46.858050+0000","last_unstale":"2024-06-27T21:36:24.733454+0000","last_undegraded":"2024-06-27T21:36:24.733454+0000","last_fullsized":"2024-06-27T21:36:24.733454+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":219,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.754118+0000","last_change":"2024-06-27T21:33:50.003594+0000","last_active":"2024-06-27T21:36:24.754118+0000","last_peered":"2024-06-27T21:36:24.754118+0000","last_clean":"2024-06-27T21:36:24.754118+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:36:24.754118+0000","last_undegraded":"2024-06-27T21:36:24.754118+0000","last_fullsized":"2024-06-27T21:36:24.754118+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:50.003522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.003522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":649,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.737479+0000","last_change":"2024-06-27T21:35:46.862167+0000","last_active":"2024-06-27T21:36:24.737479+0000","last_peered":"2024-06-27T21:36:24.737479+0000","last_clean":"2024-06-27T21:36:24.737479+0000","last_became_active":"2024-06-27T21:35:46.859076+0000","last_became_peered":"2024-06-27T21:35:46.859076+0000","last_unstale":"2024-06-27T21:36:24.737479+0000","last_undegraded":"2024-06-27T21:36:24.737479+0000","last_fullsized":"2024-06-27T21:36:24.737479+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.184716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.184716+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,11],"acting":[2,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"625'12072","reported_seq":42831,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.919253+0000","last_change":"2024-06-27T21:36:22.072201+0000","last_active":"2024-06-27T21:36:24.919253+0000","last_peered":"2024-06-27T21:36:24.919253+0000","last_clean":"2024-06-27T21:36:24.919253+0000","last_became_active":"2024-06-27T21:36:22.045968+0000","last_became_peered":"2024-06-27T21:36:22.045968+0000","last_unstale":"2024-06-27T21:36:24.919253+0000","last_undegraded":"2024-06-27T21:36:24.919253+0000","last_fullsized":"2024-06-27T21:36:24.919253+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":5,"last_scrub":"574'11661","last_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_deep_scrub":"574'11661","last_deep_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_clean_scrub_stamp":"2024-06-27T21:34:46.137011+0000","objects_scrubbed":1586,"log_size":12072,"log_dups_size":0,"ondisk_log_size":12072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"queued for scrub","scrub_duration":10216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75759616,"num_objects":1156,"num_object_clones":0,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3365,"num_bytes_recovered":79429632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,9],"acting":[0,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":675,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.754259+0000","last_change":"2024-06-27T21:33:03.234800+0000","last_active":"2024-06-27T21:36:24.754259+0000","last_peered":"2024-06-27T21:36:24.754259+0000","last_clean":"2024-06-27T21:36:24.754259+0000","last_became_active":"2024-06-27T21:33:01.779398+0000","last_became_peered":"2024-06-27T21:33:01.779398+0000","last_unstale":"2024-06-27T21:36:24.754259+0000","last_undegraded":"2024-06-27T21:36:24.754259+0000","last_fullsized":"2024-06-27T21:36:24.754259+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:33:03.234726+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.11","version":"625'10139","reported_seq":30219,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:25.292835+0000","last_change":"2024-06-27T21:34:22.233121+0000","last_active":"2024-06-27T21:36:25.292835+0000","last_peered":"2024-06-27T21:36:25.292835+0000","last_clean":"2024-06-27T21:36:25.292835+0000","last_became_active":"2024-06-27T21:34:21.788051+0000","last_became_peered":"2024-06-27T21:34:21.788051+0000","last_unstale":"2024-06-27T21:36:25.292835+0000","last_undegraded":"2024-06-27T21:36:25.292835+0000","last_fullsized":"2024-06-27T21:36:25.292835+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"569'9717","last_scrub_stamp":"2024-06-27T21:34:22.232987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:22.232987+0000","objects_scrubbed":1552,"log_size":10139,"log_dups_size":0,"ondisk_log_size":10139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73990144,"num_objects":1129,"num_object_clones":0,"num_object_copies":3387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":73,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.745001+0000","last_change":"2024-06-27T21:35:47.430217+0000","last_active":"2024-06-27T21:36:24.745001+0000","last_peered":"2024-06-27T21:36:24.745001+0000","last_clean":"2024-06-27T21:36:24.745001+0000","last_became_active":"2024-06-27T21:35:46.855888+0000","last_became_peered":"2024-06-27T21:35:46.855888+0000","last_unstale":"2024-06-27T21:36:24.745001+0000","last_undegraded":"2024-06-27T21:36:24.745001+0000","last_fullsized":"2024-06-27T21:36:24.745001+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.430103+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.430103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:48.754970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"625'10247","reported_seq":10699,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.551512+0000","last_change":"2024-06-27T21:33:51.180577+0000","last_active":"2024-06-27T21:36:29.551512+0000","last_peered":"2024-06-27T21:36:29.551512+0000","last_clean":"2024-06-27T21:36:29.551512+0000","last_became_active":"2024-06-27T21:33:27.698780+0000","last_became_peered":"2024-06-27T21:33:27.698780+0000","last_unstale":"2024-06-27T21:36:29.551512+0000","last_undegraded":"2024-06-27T21:36:29.551512+0000","last_fullsized":"2024-06-27T21:36:29.551512+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'9712","last_scrub_stamp":"2024-06-27T21:33:51.180247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.180247+0000","objects_scrubbed":1680,"log_size":10247,"log_dups_size":0,"ondisk_log_size":10247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75038720,"num_objects":1145,"num_object_clones":0,"num_object_copies":3435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":535,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2],"acting":[11,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":201,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.736904+0000","last_change":"2024-06-27T21:35:46.859131+0000","last_active":"2024-06-27T21:36:24.736904+0000","last_peered":"2024-06-27T21:36:24.736904+0000","last_clean":"2024-06-27T21:36:24.736904+0000","last_became_active":"2024-06-27T21:35:46.858265+0000","last_became_peered":"2024-06-27T21:35:46.858265+0000","last_unstale":"2024-06-27T21:36:24.736904+0000","last_undegraded":"2024-06-27T21:36:24.736904+0000","last_fullsized":"2024-06-27T21:36:24.736904+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"625'10640","reported_seq":27319,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.194091+0000","last_change":"2024-06-27T21:35:52.894201+0000","last_active":"2024-06-27T21:36:29.194091+0000","last_peered":"2024-06-27T21:36:29.194091+0000","last_clean":"2024-06-27T21:36:29.194091+0000","last_became_active":"2024-06-27T21:35:52.878734+0000","last_became_peered":"2024-06-27T21:35:52.878734+0000","last_unstale":"2024-06-27T21:36:29.194091+0000","last_undegraded":"2024-06-27T21:36:29.194091+0000","last_fullsized":"2024-06-27T21:36:29.194091+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"566'10180","last_scrub_stamp":"2024-06-27T21:34:19.641768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:19.641768+0000","objects_scrubbed":1673,"log_size":10640,"log_dups_size":0,"ondisk_log_size":10640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79233024,"num_objects":1209,"num_object_clones":0,"num_object_copies":3627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":326,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.737719+0000","last_change":"2024-06-27T21:31:19.010738+0000","last_active":"2024-06-27T21:36:24.737719+0000","last_peered":"2024-06-27T21:36:24.737719+0000","last_clean":"2024-06-27T21:36:24.737719+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:36:24.737719+0000","last_undegraded":"2024-06-27T21:36:24.737719+0000","last_fullsized":"2024-06-27T21:36:24.737719+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.010614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.010614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"625'10754","reported_seq":11317,"reported_epoch":625,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:36:27.760166+0000","last_change":"2024-06-27T21:36:17.047592+0000","last_active":"2024-06-27T21:36:27.760166+0000","last_peered":"2024-06-27T21:36:27.760166+0000","last_clean":"2024-06-27T21:33:25.735988+0000","last_became_active":"2024-06-27T21:33:28.958742+0000","last_became_peered":"2024-06-27T21:33:28.958742+0000","last_unstale":"2024-06-27T21:36:27.760166+0000","last_undegraded":"2024-06-27T21:36:27.760166+0000","last_fullsized":"2024-06-27T21:36:27.760166+0000","mapping_epoch":550,"log_start":"470'100","ondisk_log_start":"470'100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":10654,"log_dups_size":0,"ondisk_log_size":10654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78348288,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3501,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5702,"num_write_kb":220704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2228224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[1,5,9],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5,9","objects":1196}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":332,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.873436+0000","last_change":"2024-06-27T21:35:46.853442+0000","last_active":"2024-06-27T21:36:24.873436+0000","last_peered":"2024-06-27T21:36:24.873436+0000","last_clean":"2024-06-27T21:36:24.873436+0000","last_became_active":"2024-06-27T21:35:46.853087+0000","last_became_peered":"2024-06-27T21:35:46.853087+0000","last_unstale":"2024-06-27T21:36:24.873436+0000","last_undegraded":"2024-06-27T21:36:24.873436+0000","last_fullsized":"2024-06-27T21:36:24.873436+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.15","version":"625'10343","reported_seq":19076,"reported_epoch":625,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:36:29.739327+0000","last_change":"2024-06-27T21:36:07.801903+0000","last_active":"2024-06-27T21:36:29.739327+0000","last_peered":"2024-06-27T21:36:29.739327+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:28.917741+0000","last_became_peered":"2024-06-27T21:33:28.917741+0000","last_unstale":"2024-06-27T21:36:29.739327+0000","last_undegraded":"2024-06-27T21:36:29.739327+0000","last_fullsized":"2024-06-27T21:33:28.693311+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":10343,"log_dups_size":0,"ondisk_log_size":10343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74874880,"num_objects":1143,"num_object_clones":0,"num_object_copies":3429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2148,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5476,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":147652608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,10],"acting":[7,4],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"2,6,7,9","objects":74},{"shards":"4,7","objects":69},{"shards":"6,7,9","objects":1000}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":356,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.799885+0000","last_change":"2024-06-27T21:36:06.824164+0000","last_active":"2024-06-27T21:36:24.799885+0000","last_peered":"2024-06-27T21:36:24.799885+0000","last_clean":"2024-06-27T21:36:24.799885+0000","last_became_active":"2024-06-27T21:35:46.862310+0000","last_became_peered":"2024-06-27T21:35:46.862310+0000","last_unstale":"2024-06-27T21:36:24.799885+0000","last_undegraded":"2024-06-27T21:36:24.799885+0000","last_fullsized":"2024-06-27T21:36:24.799885+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:06.824082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:06.824082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:31.578769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"625'10435","reported_seq":31444,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:25.252828+0000","last_change":"2024-06-27T21:34:06.592362+0000","last_active":"2024-06-27T21:36:25.252828+0000","last_peered":"2024-06-27T21:36:25.252828+0000","last_clean":"2024-06-27T21:36:25.252828+0000","last_became_active":"2024-06-27T21:34:05.705454+0000","last_became_peered":"2024-06-27T21:34:05.705454+0000","last_unstale":"2024-06-27T21:36:25.252828+0000","last_undegraded":"2024-06-27T21:36:25.252828+0000","last_fullsized":"2024-06-27T21:36:25.252828+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":5,"last_scrub":"562'9943","last_scrub_stamp":"2024-06-27T21:34:06.592264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:06.592264+0000","objects_scrubbed":1716,"log_size":10435,"log_dups_size":0,"ondisk_log_size":10435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":830,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80084992,"num_objects":1222,"num_object_clones":0,"num_object_copies":3666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":494,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":340,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.744798+0000","last_change":"2024-06-27T21:35:46.855364+0000","last_active":"2024-06-27T21:36:24.744798+0000","last_peered":"2024-06-27T21:36:24.744798+0000","last_clean":"2024-06-27T21:36:24.744798+0000","last_became_active":"2024-06-27T21:35:46.854914+0000","last_became_peered":"2024-06-27T21:35:46.854914+0000","last_unstale":"2024-06-27T21:36:24.744798+0000","last_undegraded":"2024-06-27T21:36:24.744798+0000","last_fullsized":"2024-06-27T21:36:24.744798+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,10],"acting":[0,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"625'10338","reported_seq":24191,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.542319+0000","last_change":"2024-06-27T21:35:41.151637+0000","last_active":"2024-06-27T21:36:29.542319+0000","last_peered":"2024-06-27T21:36:29.542319+0000","last_clean":"2024-06-27T21:36:29.542319+0000","last_became_active":"2024-06-27T21:35:01.964347+0000","last_became_peered":"2024-06-27T21:35:01.964347+0000","last_unstale":"2024-06-27T21:36:29.542319+0000","last_undegraded":"2024-06-27T21:36:29.542319+0000","last_fullsized":"2024-06-27T21:36:29.542319+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10152","last_scrub_stamp":"2024-06-27T21:35:41.151550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:41.151550+0000","objects_scrubbed":1305,"log_size":10338,"log_dups_size":0,"ondisk_log_size":10338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:56.383804+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73334784,"num_objects":1119,"num_object_clones":0,"num_object_copies":3357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":355,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.736532+0000","last_change":"2024-06-27T21:32:13.808094+0000","last_active":"2024-06-27T21:36:24.736532+0000","last_peered":"2024-06-27T21:36:24.736532+0000","last_clean":"2024-06-27T21:36:24.736532+0000","last_became_active":"2024-06-27T21:32:13.807596+0000","last_became_peered":"2024-06-27T21:32:13.807596+0000","last_unstale":"2024-06-27T21:36:24.736532+0000","last_undegraded":"2024-06-27T21:36:24.736532+0000","last_fullsized":"2024-06-27T21:36:24.736532+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":629,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.737585+0000","last_change":"2024-06-27T21:33:00.167722+0000","last_active":"2024-06-27T21:36:24.737585+0000","last_peered":"2024-06-27T21:36:24.737585+0000","last_clean":"2024-06-27T21:36:24.737585+0000","last_became_active":"2024-06-27T21:32:59.709162+0000","last_became_peered":"2024-06-27T21:32:59.709162+0000","last_unstale":"2024-06-27T21:36:24.737585+0000","last_undegraded":"2024-06-27T21:36:24.737585+0000","last_fullsized":"2024-06-27T21:36:24.737585+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.167617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.167617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"625'11196","reported_seq":11601,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:27.910324+0000","last_change":"2024-06-27T21:30:52.123816+0000","last_active":"2024-06-27T21:36:27.910324+0000","last_peered":"2024-06-27T21:36:27.910324+0000","last_clean":"2024-06-27T21:36:27.910324+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:36:27.910324+0000","last_undegraded":"2024-06-27T21:36:27.910324+0000","last_fullsized":"2024-06-27T21:36:27.910324+0000","mapping_epoch":467,"log_start":"475'1100","ondisk_log_start":"475'1100","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6360","last_scrub_stamp":"2024-06-27T21:30:52.123515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:52.123515+0000","objects_scrubbed":6329,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:30.241710+0000","scrub_duration":1322,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155713536,"num_objects":2376,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":417,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.736648+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:36:24.736648+0000","last_peered":"2024-06-27T21:36:24.736648+0000","last_clean":"2024-06-27T21:36:24.736648+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:36:24.736648+0000","last_undegraded":"2024-06-27T21:36:24.736648+0000","last_fullsized":"2024-06-27T21:36:24.736648+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"625'11182","reported_seq":16865,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:27.931864+0000","last_change":"2024-06-27T21:36:08.017526+0000","last_active":"2024-06-27T21:36:27.931864+0000","last_peered":"2024-06-27T21:36:27.931864+0000","last_clean":"2024-06-27T21:36:27.931864+0000","last_became_active":"2024-06-27T21:36:07.970524+0000","last_became_peered":"2024-06-27T21:36:07.970524+0000","last_unstale":"2024-06-27T21:36:27.931864+0000","last_undegraded":"2024-06-27T21:36:27.931864+0000","last_fullsized":"2024-06-27T21:36:27.931864+0000","mapping_epoch":612,"log_start":"475'1100","ondisk_log_start":"475'1100","created":467,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'6100","last_scrub_stamp":"2024-06-27T21:30:48.342993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.342993+0000","objects_scrubbed":6058,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:30.241648+0000","scrub_duration":1488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2633,"num_bytes_recovered":172556288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":259,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.733632+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:36:24.733632+0000","last_peered":"2024-06-27T21:36:24.733632+0000","last_clean":"2024-06-27T21:36:24.733632+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:36:24.733632+0000","last_undegraded":"2024-06-27T21:36:24.733632+0000","last_fullsized":"2024-06-27T21:36:24.733632+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"625'11126","reported_seq":30426,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.730024+0000","last_change":"2024-06-27T21:36:08.329099+0000","last_active":"2024-06-27T21:36:29.730024+0000","last_peered":"2024-06-27T21:36:29.730024+0000","last_clean":"2024-06-27T21:36:29.730024+0000","last_became_active":"2024-06-27T21:36:08.302687+0000","last_became_peered":"2024-06-27T21:36:08.302687+0000","last_unstale":"2024-06-27T21:36:29.730024+0000","last_undegraded":"2024-06-27T21:36:29.730024+0000","last_fullsized":"2024-06-27T21:36:29.730024+0000","mapping_epoch":612,"log_start":"475'1100","ondisk_log_start":"475'1100","created":467,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155779096,"num_objects":2378,"num_object_clones":0,"num_object_copies":7134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11126,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9434,"num_bytes_recovered":509018160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1177,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.873585+0000","last_change":"2024-06-27T21:32:13.809592+0000","last_active":"2024-06-27T21:36:24.873585+0000","last_peered":"2024-06-27T21:36:24.873585+0000","last_clean":"2024-06-27T21:36:24.873585+0000","last_became_active":"2024-06-27T21:32:13.809197+0000","last_became_peered":"2024-06-27T21:32:13.809197+0000","last_unstale":"2024-06-27T21:36:24.873585+0000","last_undegraded":"2024-06-27T21:36:24.873585+0000","last_fullsized":"2024-06-27T21:36:24.873585+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"625'10619","reported_seq":18001,"reported_epoch":625,"state":"active+recovering","last_fresh":"2024-06-27T21:36:29.536263+0000","last_change":"2024-06-27T21:33:26.890778+0000","last_active":"2024-06-27T21:36:29.536263+0000","last_peered":"2024-06-27T21:36:29.536263+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:33:26.864503+0000","last_became_peered":"2024-06-27T21:33:26.864503+0000","last_unstale":"2024-06-27T21:36:29.536263+0000","last_undegraded":"2024-06-27T21:36:29.536263+0000","last_fullsized":"2024-06-27T21:36:29.536263+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":10619,"log_dups_size":0,"ondisk_log_size":10619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77856768,"num_objects":1188,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":328,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5627,"num_write_kb":218080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2273,"num_bytes_recovered":148963328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"0,7,11","objects":860},{"shards":"6,7,11","objects":328}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":754,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.754242+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:36:24.754242+0000","last_peered":"2024-06-27T21:36:24.754242+0000","last_clean":"2024-06-27T21:36:24.754242+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:36:24.754242+0000","last_undegraded":"2024-06-27T21:36:24.754242+0000","last_fullsized":"2024-06-27T21:36:24.754242+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"625'10297","reported_seq":10753,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:27.927192+0000","last_change":"2024-06-27T21:33:54.377162+0000","last_active":"2024-06-27T21:36:27.927192+0000","last_peered":"2024-06-27T21:36:27.927192+0000","last_clean":"2024-06-27T21:36:27.927192+0000","last_became_active":"2024-06-27T21:33:25.695260+0000","last_became_peered":"2024-06-27T21:33:25.695260+0000","last_unstale":"2024-06-27T21:36:27.927192+0000","last_undegraded":"2024-06-27T21:36:27.927192+0000","last_fullsized":"2024-06-27T21:36:27.927192+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9736","last_scrub_stamp":"2024-06-27T21:33:54.376905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.376905+0000","objects_scrubbed":1765,"log_size":10297,"log_dups_size":0,"ondisk_log_size":10297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:30.241841+0000","scrub_duration":1024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78708736,"num_objects":1201,"num_object_clones":0,"num_object_copies":3603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":750,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.754261+0000","last_change":"2024-06-27T21:33:01.182699+0000","last_active":"2024-06-27T21:36:24.754261+0000","last_peered":"2024-06-27T21:36:24.754261+0000","last_clean":"2024-06-27T21:36:24.754261+0000","last_became_active":"2024-06-27T21:32:59.711777+0000","last_became_peered":"2024-06-27T21:32:59.711777+0000","last_unstale":"2024-06-27T21:36:24.754261+0000","last_undegraded":"2024-06-27T21:36:24.754261+0000","last_fullsized":"2024-06-27T21:36:24.754261+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"625'10112","reported_seq":21786,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:25.320875+0000","last_change":"2024-06-27T21:33:48.555029+0000","last_active":"2024-06-27T21:36:25.320875+0000","last_peered":"2024-06-27T21:36:25.320875+0000","last_clean":"2024-06-27T21:36:25.320875+0000","last_became_active":"2024-06-27T21:33:35.694586+0000","last_became_peered":"2024-06-27T21:33:35.694586+0000","last_unstale":"2024-06-27T21:36:25.320875+0000","last_undegraded":"2024-06-27T21:36:25.320875+0000","last_fullsized":"2024-06-27T21:36:25.320875+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'9582","last_scrub_stamp":"2024-06-27T21:33:48.554929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:48.554929+0000","objects_scrubbed":1728,"log_size":10112,"log_dups_size":0,"ondisk_log_size":10112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:26.893256+0000","scrub_duration":232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,7],"acting":[9,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":670,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.754376+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:36:24.754376+0000","last_peered":"2024-06-27T21:36:24.754376+0000","last_clean":"2024-06-27T21:36:24.754376+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:36:24.754376+0000","last_undegraded":"2024-06-27T21:36:24.754376+0000","last_fullsized":"2024-06-27T21:36:24.754376+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"625'10438","reported_seq":24322,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.744864+0000","last_change":"2024-06-27T21:33:54.883116+0000","last_active":"2024-06-27T21:36:24.744864+0000","last_peered":"2024-06-27T21:36:24.744864+0000","last_clean":"2024-06-27T21:36:24.744864+0000","last_became_active":"2024-06-27T21:33:25.693582+0000","last_became_peered":"2024-06-27T21:33:25.693582+0000","last_unstale":"2024-06-27T21:36:24.744864+0000","last_undegraded":"2024-06-27T21:36:24.744864+0000","last_fullsized":"2024-06-27T21:36:24.744864+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":1712,"log_size":10438,"log_dups_size":0,"ondisk_log_size":10438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78774272,"num_objects":1202,"num_object_clones":0,"num_object_copies":3606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":510,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":748,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.754260+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:36:24.754260+0000","last_peered":"2024-06-27T21:36:24.754260+0000","last_clean":"2024-06-27T21:36:24.754260+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:36:24.754260+0000","last_undegraded":"2024-06-27T21:36:24.754260+0000","last_fullsized":"2024-06-27T21:36:24.754260+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"625'10441","reported_seq":24696,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:28.181407+0000","last_change":"2024-06-27T21:33:50.968993+0000","last_active":"2024-06-27T21:36:28.181407+0000","last_peered":"2024-06-27T21:36:28.181407+0000","last_clean":"2024-06-27T21:36:28.181407+0000","last_became_active":"2024-06-27T21:33:25.692464+0000","last_became_peered":"2024-06-27T21:33:25.692464+0000","last_unstale":"2024-06-27T21:36:28.181407+0000","last_undegraded":"2024-06-27T21:36:28.181407+0000","last_fullsized":"2024-06-27T21:36:28.181407+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'9882","last_scrub_stamp":"2024-06-27T21:33:50.968852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.968852+0000","objects_scrubbed":1716,"log_size":10441,"log_dups_size":0,"ondisk_log_size":10441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75825152,"num_objects":1157,"num_object_clones":0,"num_object_copies":3471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":559,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":423,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.736705+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:36:24.736705+0000","last_peered":"2024-06-27T21:36:24.736705+0000","last_clean":"2024-06-27T21:36:24.736705+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:36:24.736705+0000","last_undegraded":"2024-06-27T21:36:24.736705+0000","last_fullsized":"2024-06-27T21:36:24.736705+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"625'10405","reported_seq":20648,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.498247+0000","last_change":"2024-06-27T21:36:08.622320+0000","last_active":"2024-06-27T21:36:29.498247+0000","last_peered":"2024-06-27T21:36:29.498247+0000","last_clean":"2024-06-27T21:36:29.498247+0000","last_became_active":"2024-06-27T21:33:26.702107+0000","last_became_peered":"2024-06-27T21:33:26.702107+0000","last_unstale":"2024-06-27T21:36:29.498247+0000","last_undegraded":"2024-06-27T21:36:29.498247+0000","last_fullsized":"2024-06-27T21:36:29.498247+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'10314","last_scrub_stamp":"2024-06-27T21:36:08.622226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:08.622226+0000","objects_scrubbed":1220,"log_size":10405,"log_dups_size":0,"ondisk_log_size":10405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:16.776796+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73924608,"num_objects":1128,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":313,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.733501+0000","last_change":"2024-06-27T21:35:46.855308+0000","last_active":"2024-06-27T21:36:24.733501+0000","last_peered":"2024-06-27T21:36:24.733501+0000","last_clean":"2024-06-27T21:36:24.733501+0000","last_became_active":"2024-06-27T21:35:46.854995+0000","last_became_peered":"2024-06-27T21:35:46.854995+0000","last_unstale":"2024-06-27T21:36:24.733501+0000","last_undegraded":"2024-06-27T21:36:24.733501+0000","last_fullsized":"2024-06-27T21:36:24.733501+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"625'10855","reported_seq":14106,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:26.172597+0000","last_change":"2024-06-27T21:36:00.782219+0000","last_active":"2024-06-27T21:36:26.172597+0000","last_peered":"2024-06-27T21:36:26.172597+0000","last_clean":"2024-06-27T21:36:26.172597+0000","last_became_active":"2024-06-27T21:35:58.992302+0000","last_became_peered":"2024-06-27T21:35:58.992302+0000","last_unstale":"2024-06-27T21:36:26.172597+0000","last_undegraded":"2024-06-27T21:36:26.172597+0000","last_fullsized":"2024-06-27T21:36:26.172597+0000","mapping_epoch":606,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10731","last_scrub_stamp":"2024-06-27T21:36:00.782086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:00.782086+0000","objects_scrubbed":1308,"log_size":10755,"log_dups_size":0,"ondisk_log_size":10755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:13.438162+0000","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":88735744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":371,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.736639+0000","last_change":"2024-06-27T21:35:47.454903+0000","last_active":"2024-06-27T21:36:24.736639+0000","last_peered":"2024-06-27T21:36:24.736639+0000","last_clean":"2024-06-27T21:36:24.736639+0000","last_became_active":"2024-06-27T21:35:46.856358+0000","last_became_peered":"2024-06-27T21:35:46.856358+0000","last_unstale":"2024-06-27T21:36:24.736639+0000","last_undegraded":"2024-06-27T21:36:24.736639+0000","last_fullsized":"2024-06-27T21:36:24.736639+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.454820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.454820+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:10.172593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"625'10321","reported_seq":10834,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:27.513193+0000","last_change":"2024-06-27T21:36:05.559816+0000","last_active":"2024-06-27T21:36:27.513193+0000","last_peered":"2024-06-27T21:36:27.513193+0000","last_clean":"2024-06-27T21:36:27.513193+0000","last_became_active":"2024-06-27T21:33:25.697304+0000","last_became_peered":"2024-06-27T21:33:25.697304+0000","last_unstale":"2024-06-27T21:36:27.513193+0000","last_undegraded":"2024-06-27T21:36:27.513193+0000","last_fullsized":"2024-06-27T21:36:27.513193+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'10215","last_scrub_stamp":"2024-06-27T21:36:05.559677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:05.559677+0000","objects_scrubbed":1276,"log_size":10321,"log_dups_size":0,"ondisk_log_size":10321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:21.964359+0000","scrub_duration":728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":562,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":951,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.800005+0000","last_change":"2024-06-27T21:35:47.736735+0000","last_active":"2024-06-27T21:36:24.800005+0000","last_peered":"2024-06-27T21:36:24.800005+0000","last_clean":"2024-06-27T21:36:24.800005+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:36:24.800005+0000","last_undegraded":"2024-06-27T21:36:24.800005+0000","last_fullsized":"2024-06-27T21:36:24.800005+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.736633+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.736633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:07.591052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"625'10526","reported_seq":10974,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:29.743070+0000","last_change":"2024-06-27T21:33:49.851182+0000","last_active":"2024-06-27T21:36:29.743070+0000","last_peered":"2024-06-27T21:36:29.743070+0000","last_clean":"2024-06-27T21:36:29.743070+0000","last_became_active":"2024-06-27T21:33:25.736643+0000","last_became_peered":"2024-06-27T21:33:25.736643+0000","last_unstale":"2024-06-27T21:36:29.743070+0000","last_undegraded":"2024-06-27T21:36:29.743070+0000","last_fullsized":"2024-06-27T21:36:29.743070+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9962","last_scrub_stamp":"2024-06-27T21:33:49.850981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.850981+0000","objects_scrubbed":1778,"log_size":10526,"log_dups_size":0,"ondisk_log_size":10526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79560704,"num_objects":1214,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":564,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":339,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.733676+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:36:24.733676+0000","last_peered":"2024-06-27T21:36:24.733676+0000","last_clean":"2024-06-27T21:36:24.733676+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:36:24.733676+0000","last_undegraded":"2024-06-27T21:36:24.733676+0000","last_fullsized":"2024-06-27T21:36:24.733676+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"625'10439","reported_seq":17839,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.745142+0000","last_change":"2024-06-27T21:34:00.348871+0000","last_active":"2024-06-27T21:36:24.745142+0000","last_peered":"2024-06-27T21:36:24.745142+0000","last_clean":"2024-06-27T21:36:24.745142+0000","last_became_active":"2024-06-27T21:33:26.705196+0000","last_became_peered":"2024-06-27T21:33:26.705196+0000","last_unstale":"2024-06-27T21:36:24.745142+0000","last_undegraded":"2024-06-27T21:36:24.745142+0000","last_fullsized":"2024-06-27T21:36:24.745142+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9854","last_scrub_stamp":"2024-06-27T21:34:00.348580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:00.348580+0000","objects_scrubbed":1790,"log_size":10439,"log_dups_size":0,"ondisk_log_size":10439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78970880,"num_objects":1205,"num_object_clones":0,"num_object_copies":3615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,2],"acting":[0,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":337,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.737001+0000","last_change":"2024-06-27T21:35:46.858649+0000","last_active":"2024-06-27T21:36:24.737001+0000","last_peered":"2024-06-27T21:36:24.737001+0000","last_clean":"2024-06-27T21:36:24.737001+0000","last_became_active":"2024-06-27T21:35:46.857022+0000","last_became_peered":"2024-06-27T21:35:46.857022+0000","last_unstale":"2024-06-27T21:36:24.737001+0000","last_undegraded":"2024-06-27T21:36:24.737001+0000","last_fullsized":"2024-06-27T21:36:24.737001+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,11],"acting":[7,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"625'11068","reported_seq":24882,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:25.337034+0000","last_change":"2024-06-27T21:33:23.873709+0000","last_active":"2024-06-27T21:36:25.337034+0000","last_peered":"2024-06-27T21:36:25.337034+0000","last_clean":"2024-06-27T21:36:25.337034+0000","last_became_active":"2024-06-27T21:33:23.851489+0000","last_became_peered":"2024-06-27T21:33:23.851489+0000","last_unstale":"2024-06-27T21:36:25.337034+0000","last_undegraded":"2024-06-27T21:36:25.337034+0000","last_fullsized":"2024-06-27T21:36:25.337034+0000","mapping_epoch":545,"log_start":"475'1000","ondisk_log_start":"475'1000","created":467,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152698880,"num_objects":2330,"num_object_clones":0,"num_object_copies":6990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11068,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6699,"num_bytes_recovered":340197376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"625'10870","reported_seq":16474,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:27.509771+0000","last_change":"2024-06-27T21:36:08.015340+0000","last_active":"2024-06-27T21:36:27.509771+0000","last_peered":"2024-06-27T21:36:27.509771+0000","last_clean":"2024-06-27T21:36:27.509771+0000","last_became_active":"2024-06-27T21:36:07.973506+0000","last_became_peered":"2024-06-27T21:36:07.973506+0000","last_unstale":"2024-06-27T21:36:27.509771+0000","last_undegraded":"2024-06-27T21:36:27.509771+0000","last_fullsized":"2024-06-27T21:36:27.509771+0000","mapping_epoch":612,"log_start":"473'800","ondisk_log_start":"473'800","created":467,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'10541","last_scrub_stamp":"2024-06-27T21:35:43.673315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:43.673315+0000","objects_scrubbed":2645,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:36:44.360020+0000","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650304,"num_objects":2314,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2556,"num_bytes_recovered":167510016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":347,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.744931+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:36:24.744931+0000","last_peered":"2024-06-27T21:36:24.744931+0000","last_clean":"2024-06-27T21:36:24.744931+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:36:24.744931+0000","last_undegraded":"2024-06-27T21:36:24.744931+0000","last_fullsized":"2024-06-27T21:36:24.744931+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"625'11417","reported_seq":22679,"reported_epoch":625,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:36:29.767325+0000","last_change":"2024-06-27T21:32:36.645550+0000","last_active":"2024-06-27T21:36:29.767325+0000","last_peered":"2024-06-27T21:36:29.767325+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:32:34.904596+0000","last_became_peered":"2024-06-27T21:32:34.904596+0000","last_unstale":"2024-06-27T21:36:29.767325+0000","last_undegraded":"2024-06-27T21:32:22.691354+0000","last_fullsized":"2024-06-27T21:32:22.682941+0000","mapping_epoch":526,"log_start":"475'1400","ondisk_log_start":"475'1400","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157745152,"num_objects":2407,"num_object_clones":0,"num_object_copies":7221,"num_objects_missing_on_primary":3769,"num_objects_missing":3769,"num_objects_degraded":3,"num_objects_misplaced":7206,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3128,"num_bytes_recovered":204668928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,4],"acting":[7,10],"avail_no_missing":["5","6","8"],"object_location_counts":[{"shards":"5,6,7,8","objects":15},{"shards":"5,6,8","objects":2392},{"shards":"7,10","objects":-3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":345,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-06-27T21:36:24.873332+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:36:24.873332+0000","last_peered":"2024-06-27T21:36:24.873332+0000","last_clean":"2024-06-27T21:36:24.873332+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:36:24.873332+0000","last_undegraded":"2024-06-27T21:36:24.873332+0000","last_fullsized":"2024-06-27T21:36:24.873332+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":2469330968,"num_objects":37681,"num_object_clones":0,"num_object_copies":113043,"num_objects_missing_on_primary":3769,"num_objects_missing":3769,"num_objects_degraded":3,"num_objects_misplaced":13183,"num_objects_unfound":0,"num_objects_dirty":37681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":91124,"num_write_kb":3243713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37974,"num_bytes_recovered":1973551152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8675536896,"data_stored":8675524680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270501,"ondisk_log_size":270501,"up":78,"acting":76,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253191,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1441116,"kb_used_data":824084,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92304036,"statfs":{"total":95995035648,"available":94519332864,"internally_reserved":0,"allocated":843862016,"data_stored":843194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514827,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161620,"kb_used_data":679244,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92583532,"statfs":{"total":95995035648,"available":94805536768,"internally_reserved":0,"allocated":695545856,"data_stored":694882312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":493943232},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743759,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266388,"kb_used_data":802056,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92478764,"statfs":{"total":95995035648,"available":94698254336,"internally_reserved":0,"allocated":821305344,"data_stored":820645872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":475462080},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":0,"seq":266287972691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267029,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663768,"kb_used_data":1086992,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92081384,"statfs":{"total":95995035648,"available":94291337216,"internally_reserved":0,"allocated":1113079808,"data_stored":1112416784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528669,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1664788,"kb_used_data":1137992,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92080364,"statfs":{"total":95995035648,"available":94290292736,"internally_reserved":0,"allocated":1165303808,"data_stored":1164644336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757603,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973264,"kb_used_data":572808,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92771888,"statfs":{"total":95995035648,"available":94998413312,"internally_reserved":0,"allocated":586555392,"data_stored":585895920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161308,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781136,"kb_used_data":535496,"kb_used_omap":92,"kb_used_meta":245539,"kb_avail":92964016,"statfs":{"total":95995035648,"available":95195152384,"internally_reserved":0,"allocated":548347904,"data_stored":547688432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":251432034},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182763,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747664,"kb_used_data":377736,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92997488,"statfs":{"total":95995035648,"available":95229427712,"internally_reserved":0,"allocated":386801664,"data_stored":386142192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411696,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1606808,"kb_used_data":1174416,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92138344,"statfs":{"total":95995035648,"available":94349664256,"internally_reserved":0,"allocated":1202601984,"data_stored":1201938960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575221,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738745,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1484060,"kb_used_data":889228,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92261092,"statfs":{"total":95995035648,"available":94475358208,"internally_reserved":0,"allocated":910569472,"data_stored":909905928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":907743232,"data_stored":907739160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":78970880,"data_stored":78970880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1200095232,"data_stored":1200095232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383647744,"data_stored":383647744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":545849344,"data_stored":545849344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":584515584,"data_stored":584515584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1162608640,"data_stored":1162608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1109983232,"data_stored":1109983232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":352321536,"data_stored":352321536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":816578560,"data_stored":816578560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":691474432,"data_stored":691470360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":841748480,"data_stored":841744408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:36:31.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_4 pgp_num 26 2024-06-27T21:36:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:31.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:31.510 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:31.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:31.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:31.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:31.669 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:31.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:31.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:31.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:31.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:31.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:31.955 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:31 smithi192 ceph-mon[32956]: pgmap v1597: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 3/113049 objects degraded (0.003%); 13183/113049 objects misplaced (11.661%); 1.8 MiB/s, 51 objects/s recovering 2024-06-27T21:36:31.955 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:31 smithi192 ceph-mon[32956]: from='client.16791 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:36:31.964 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:32.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:31 smithi190 ceph-mon[32247]: pgmap v1597: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 3/113049 objects degraded (0.003%); 13183/113049 objects misplaced (11.661%); 1.8 MiB/s, 51 objects/s recovering 2024-06-27T21:36:32.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:31 smithi190 ceph-mon[32247]: from='client.16791 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:36:32.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:32.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:32.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:31 smithi151 ceph-mon[28965]: pgmap v1597: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 3/113049 objects degraded (0.003%); 13183/113049 objects misplaced (11.661%); 1.8 MiB/s, 51 objects/s recovering 2024-06-27T21:36:32.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:31 smithi151 ceph-mon[28965]: from='client.16791 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:36:32.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:32.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:32.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:32.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:32.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:32.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:32.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:32.536 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:32.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:32.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:32.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:32.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:32.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:32.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:32.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:33.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:33.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:33.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:33.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:33.222 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:33.222 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:33.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:33.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:33.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:33.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:33.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:33.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:33.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:33.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:33.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:33.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:33.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:33 smithi192 ceph-mon[32956]: pgmap v1598: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 3/112536 objects degraded (0.003%); 13183/112536 objects misplaced (11.714%); 1.5 MiB/s, 45 objects/s recovering 2024-06-27T21:36:33.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:33.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:33.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:34.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:34.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:33 smithi190 ceph-mon[32247]: pgmap v1598: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 3/112536 objects degraded (0.003%); 13183/112536 objects misplaced (11.714%); 1.5 MiB/s, 45 objects/s recovering 2024-06-27T21:36:34.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:34.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:36:34.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:33 smithi151 ceph-mon[28965]: pgmap v1598: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 3/112536 objects degraded (0.003%); 13183/112536 objects misplaced (11.714%); 1.5 MiB/s, 45 objects/s recovering 2024-06-27T21:36:34.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:34.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:34.241 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:34.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:34.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:34.330 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:34.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:34.394 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:34.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:34.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:34.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:34.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:34.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:34.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:34.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:34.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:34.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: osdmap e626: 12 total, 12 up, 10 in 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/112536 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:34 smithi190 ceph-mon[32247]: osdmap e627: 12 total, 12 up, 10 in 2024-06-27T21:36:35.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:35.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:35.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: osdmap e626: 12 total, 12 up, 10 in 2024-06-27T21:36:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:36:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:36:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/112536 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:34 smithi192 ceph-mon[32956]: osdmap e627: 12 total, 12 up, 10 in 2024-06-27T21:36:35.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: osdmap e626: 12 total, 12 up, 10 in 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/112536 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:34 smithi151 ceph-mon[28965]: osdmap e627: 12 total, 12 up, 10 in 2024-06-27T21:36:35.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:35.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:35.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:35.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:35.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:35.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:35.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:35.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:35.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:35.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:35.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:35.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:35.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:35.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:36.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:35 smithi190 ceph-mon[32247]: pgmap v1600: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 3/111867 objects degraded (0.003%); 13099/111867 objects misplaced (11.709%); 1.5 MiB/s, 43 objects/s recovering 2024-06-27T21:36:36.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1179118427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:36:36.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:35 smithi151 ceph-mon[28965]: pgmap v1600: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 3/111867 objects degraded (0.003%); 13099/111867 objects misplaced (11.709%); 1.5 MiB/s, 43 objects/s recovering 2024-06-27T21:36:36.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1179118427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:36:36.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:35 smithi192 ceph-mon[32956]: pgmap v1600: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 128 op/s; 3/111867 objects degraded (0.003%); 13099/111867 objects misplaced (11.709%); 1.5 MiB/s, 43 objects/s recovering 2024-06-27T21:36:36.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1179118427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:36:36.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:36.071 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:36:36.071 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:36:36.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:36.073 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:36.073 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:36.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:36:36.104 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:36:36.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:36:36.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:36:36.162 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:36:36.162 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:36:36.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:36.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:36:36.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:36:36.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:36.207 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:36:36.207 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:36:36.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:36:36.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:36:36.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:36:36.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:36:36.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:36.303 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:36:36.303 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:36:36.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:36:36.326 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:36:36.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:36.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:36:36.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:36:36.371 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:36:36.372 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:36:36.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:36.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:36.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:36.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:36.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:36.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:36.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:36:36.762 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 6 pgp_num to 26 2024-06-27T21:36:36.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:36.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:36.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:36.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:36.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:36.988 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1179118427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: osdmap e628: 12 total, 12 up, 10 in 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/920962120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: 6.e scrub ok 2024-06-27T21:36:37.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/458721965' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1179118427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: osdmap e628: 12 total, 12 up, 10 in 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/920962120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: 6.e scrub ok 2024-06-27T21:36:37.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/458721965' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-06-27T21:36:37.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:37.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:37.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:37.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1179118427' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: osdmap e628: 12 total, 12 up, 10 in 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/920962120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: 6.e scrub ok 2024-06-27T21:36:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/458721965' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]: dispatch 2024-06-27T21:36:37.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:37.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:37.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:37.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:37.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:37.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:37.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:37.653 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:36:37.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:37.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:37.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:37.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:37.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:37 smithi151 ceph-mon[28965]: pgmap v1603: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 3/110481 objects degraded (0.003%); 12940/110481 objects misplaced (11.712%); 651 KiB/s, 10 objects/s recovering 2024-06-27T21:36:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/458721965' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-06-27T21:36:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:37 smithi151 ceph-mon[28965]: osdmap e629: 12 total, 12 up, 10 in 2024-06-27T21:36:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:37 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:36:37.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:37 smithi151 ceph-mon[28965]: 6.f scrub ok 2024-06-27T21:36:37.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:37.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:38.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:37 smithi190 ceph-mon[32247]: pgmap v1603: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 3/110481 objects degraded (0.003%); 12940/110481 objects misplaced (11.712%); 651 KiB/s, 10 objects/s recovering 2024-06-27T21:36:38.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/458721965' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-06-27T21:36:38.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:37 smithi190 ceph-mon[32247]: osdmap e629: 12 total, 12 up, 10 in 2024-06-27T21:36:38.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:37 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:36:38.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:37 smithi190 ceph-mon[32247]: 6.f scrub ok 2024-06-27T21:36:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:37 smithi192 ceph-mon[32956]: pgmap v1603: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 3/110481 objects degraded (0.003%); 12940/110481 objects misplaced (11.712%); 651 KiB/s, 10 objects/s recovering 2024-06-27T21:36:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/458721965' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "26"}]': finished 2024-06-27T21:36:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:37 smithi192 ceph-mon[32956]: osdmap e629: 12 total, 12 up, 10 in 2024-06-27T21:36:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:37 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:36:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:37 smithi192 ceph-mon[32956]: 6.f scrub ok 2024-06-27T21:36:38.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:38.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.082 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:38.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:38.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:38.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:38.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:38.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:38.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:38.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:38.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:38.602 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.25615.0:163152 5.e 5:71dd832e:::benchmark_data_smithi192_104167_object57111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:45.864530+0000", 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 532.73562523800001, 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50746658499999997, 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.603 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.25615", 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/670929887", 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 163152 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864530+0000", 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864530+0000", 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864531+0000", 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.342e-06 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864536+0000", 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0889999999999999e-06 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864549+0000", 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2221e-05 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864554+0000", 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.242e-06 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864614+0000", 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9647e-05 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864719+0000", 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105387 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,8", 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.864797+0000", 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7749000000000004e-05 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868345+0000", 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035478649999999999 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868453+0000", 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010888899999999999 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371912+0000", 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50345891799999998 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371953+0000", 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0469000000000002e-05 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371997+0000", 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3767000000000003e-05 2024-06-27T21:36:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.25615.0:163152 5.e e444/382 ondisk, result = 0)", 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 532.73134769599994, 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50319430700000001, 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868808+0000", 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3200000000000005e-07 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:45.868815+0000", 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3790000000000003e-06 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371852+0000", 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50303705799999998 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371859+0000", 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9489999999999999e-06 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371903+0000", 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4549000000000003e-05 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.371909+0000", 2024-06-27T21:36:38.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4020000000000001e-06 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:27:46.372002+0000", 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3338000000000007e-05 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(5.1 e463,441 5:8240b269:::benchmark_data_smithi192_104167_object9873:head,419'7188)", 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:28:56.068059+0000", 2024-06-27T21:36:38.614 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 462.53209630999999, 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47462944000000001, 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068059+0000", 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068059+0000", 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068060+0000", 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-07 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.068064+0000", 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9899999999999999e-06 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542461+0000", 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47439695399999998 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542475+0000", 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4139e-05 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542597+0000", 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012245100000000001 2024-06-27T21:36:38.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542615+0000", 2024-06-27T21:36:38.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7456999999999999e-05 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:28:56.542688+0000", 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3833999999999998e-05 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(5.1 e463,441 5:8c81c6a2:::benchmark_data_smithi192_104167_object16628:head,420'7624)", 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:29:00.278266+0000", 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 458.32188926100002, 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51104331599999997, 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278266+0000", 2024-06-27T21:36:38.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278266+0000", 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278267+0000", 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.466e-06 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.278277+0000", 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7200000000000001e-06 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789046+0000", 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51076909400000003 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789062+0000", 2024-06-27T21:36:38.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5509999999999999e-05 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789183+0000", 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120825 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789200+0000", 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7036000000000001e-05 2024-06-27T21:36:38.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:29:00.789309+0000", 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109665 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:131672 6.4 e515/467 6:2c24218f:::benchmark_data_smithi192_117994_object24078:head v 515'8484, mlcod=514'8481)", 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:31:31.348825+0000", 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 307.25133051199998, 2024-06-27T21:36:38.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57843952099999996, 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.348825+0000", 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.348825+0000", 2024-06-27T21:36:38.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.348826+0000", 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2500000000000004e-07 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.348831+0000", 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9119999999999997e-06 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.924040+0000", 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57520938499999996 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.924048+0000", 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620000000000008e-06 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.924102+0000", 2024-06-27T21:36:38.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4317000000000002e-05 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.924122+0000", 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9128999999999999e-05 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.927226+0000", 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031044100000000002 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:31:31.927264+0000", 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8281000000000002e-05 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:153642 6.1 6:883c986a:::benchmark_data_smithi192_117994_object46048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:48.606121+0000", 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 229.99403393599999, 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50147799100000001, 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.629 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 153642 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606121+0000", 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606121+0000", 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606124+0000", 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.537e-06 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606134+0000", 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0131e-05 2024-06-27T21:36:38.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606153+0000", 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8749e-05 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606162+0000", 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2830000000000007e-06 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606636+0000", 2024-06-27T21:36:38.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047422799999999999 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606750+0000", 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011377700000000001 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,10", 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606848+0000", 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8141999999999996e-05 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:36:38.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.610478+0000", 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036293060000000001 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.615637+0000", 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0051595449999999998 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.106267+0000", 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49063005700000001 2024-06-27T21:36:38.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.106309+0000", 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1702000000000002e-05 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.107599+0000", 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012905340000000001 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.635 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:156111 6.f e540/467 6:fcae7150:::benchmark_data_smithi192_117994_object48517:head v 540'9808, mlcod=540'9805)", 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:13.813813+0000", 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 204.78634262899999, 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64154621700000003, 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:13.813813+0000", 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:13.813813+0000", 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:13.813813+0000", 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0100000000000004e-07 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:13.813822+0000", 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6510000000000007e-06 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:14.451661+0000", 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63783920800000005 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:14.451670+0000", 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0909999999999999e-06 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:14.451731+0000", 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0470000000000002e-05 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:14.451747+0000", 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6586999999999999e-05 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:14.455322+0000", 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035747600000000002 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:14.455359+0000", 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6649000000000001e-05 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:157011 6.11 6.bc73c51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e548)", 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:25.909324+0000", 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 192.69083080999999, 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78573860799999995, 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157011 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909324+0000", 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909324+0000", 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909327+0000", 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.103e-06 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909336+0000", 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5799999999999998e-06 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909357+0000", 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0435e-05 2024-06-27T21:36:38.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909367+0000", 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0389e-05 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909427+0000", 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0062999999999997e-05 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.909435+0000", 2024-06-27T21:36:38.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9289999999999998e-06 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:26.695038+0000", 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78560294100000005 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:26.695063+0000", 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5168e-05 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.1 e551,547 6:84da3139:::benchmark_data_smithi192_117994_object31167:head,518'8357)", 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:30.594656+0000", 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 188.00549927099999, 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.616738746, 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.594656+0000", 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.594656+0000", 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.594657+0000", 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.61e-07 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.594663+0000", 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.998e-06 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.211037+0000", 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.61637392599999996 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.211045+0000", 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5560000000000008e-06 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.211108+0000", 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2475000000000001e-05 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.211395+0000", 2024-06-27T21:36:38.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028713000000000002 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(6.11 551/549 [PushReplyOp(6:8899ddc3:::benchmark_data_smithi192_117994_object57376:head)])", 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:33.876843+0000", 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 184.723312634, 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53499281700000001, 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.650 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.876843+0000", 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.876843+0000", 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.876844+0000", 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.023e-06 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.876850+0000", 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5370000000000003e-06 2024-06-27T21:36:38.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:34.409713+0000", 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53286297400000004 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:34.409728+0000", 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4973e-05 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:34.409870+0000", 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000141734 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:34.411835+0000", 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019655760000000001 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.654 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.11 e562,549 6:8a16cbd6:::benchmark_data_smithi192_117994_object12213:head,509'7238)", 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:07.463338+0000", 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 151.13681695700001, 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56443222599999998, 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:07.463338+0000", 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:07.463338+0000", 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:07.463339+0000", 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6000000000000003e-07 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:07.463345+0000", 2024-06-27T21:36:38.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9549999999999997e-06 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.027274+0000", 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.56392932299999998 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.027282+0000", 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7759999999999997e-06 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.027331+0000", 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9021999999999999e-05 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.027771+0000", 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00043949000000000001 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:161672 6.1 6:8725b172:::benchmark_data_smithi192_117994_object54078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-06-27T21:36:38.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:08.249070+0000", 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 150.35108485399999, 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.469699378, 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161672 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249070+0000", 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249070+0000", 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249072+0000", 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4890000000000001e-06 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249077+0000", 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5210000000000002e-06 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249089+0000", 2024-06-27T21:36:38.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1430000000000001e-05 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249094+0000", 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0919999999999997e-06 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249132+0000", 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7876000000000001e-05 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249216+0000", 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3782000000000005e-05 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7,10", 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.249283+0000", 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7736000000000003e-05 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.251330+0000", 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020467369999999999 2024-06-27T21:36:38.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.251364+0000", 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3976999999999997e-05 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.718700+0000", 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.46733594000000001 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.718733+0000", 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2956000000000001e-05 2024-06-27T21:36:38.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.718770+0000", 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6841999999999997e-05 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:161673 6.11 6:89485f00:::benchmark_data_smithi192_117994_object54079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e562)", 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:08.440301+0000", 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 150.15985462, 2024-06-27T21:36:38.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.413039675, 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 161673 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440301+0000", 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440301+0000", 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440302+0000", 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4699999999999999e-06 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440313+0000", 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0387999999999999e-05 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440325+0000", 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2038000000000001e-05 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440332+0000", 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4200000000000001e-06 2024-06-27T21:36:38.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440825+0000", 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049311500000000002 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.440932+0000", 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107297 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,10,11", 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.441031+0000", 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8425999999999996e-05 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.442428+0000", 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001397492 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.444772+0000", 2024-06-27T21:36:38.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023440779999999999 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.446404+0000", 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016312029999999999 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.853127+0000", 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40672335300000001 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.853167+0000", 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0371999999999999e-05 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:08.853340+0000", 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017302299999999999 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.672 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.11 e563,549 6:8d50637f:::benchmark_data_smithi192_117994_object25594:head,517'8008)", 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:12.408876+0000", 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 146.19127928200001, 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36957279300000001, 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.408876+0000", 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.408876+0000", 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.408877+0000", 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6300000000000005e-07 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.408880+0000", 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4829999999999999e-06 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.777961+0000", 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36908063400000002 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.777966+0000", 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6969999999999998e-06 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.778051+0000", 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5002999999999999e-05 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.778449+0000", 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000397413 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.11 e564,549 6:8be33d1f:::benchmark_data_smithi192_117994_object34223:head,519'8550)", 2024-06-27T21:36:38.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:16.095787+0000", 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 142.50436865399999, 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49779157000000002, 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.095787+0000", 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.095787+0000", 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.095787+0000", 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3900000000000004e-07 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.095792+0000", 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2579999999999997e-06 2024-06-27T21:36:38.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.593153+0000", 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49736171099999998 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.593158+0000", 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1819999999999997e-06 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.593199+0000", 2024-06-27T21:36:38.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0330000000000002e-05 2024-06-27T21:36:38.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.593578+0000", 2024-06-27T21:36:38.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:38.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037944999999999998 2024-06-27T21:36:38.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.11 e564,549 6:88d3b2ac:::benchmark_data_smithi192_117994_object39078:head,519'8855)", 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:17.333202+0000", 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 141.266952968, 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.443180238, 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.333202+0000", 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.333202+0000", 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.333203+0000", 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2300000000000001e-07 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.333206+0000", 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1930000000000002e-06 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.776103+0000", 2024-06-27T21:36:38.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44289715699999999 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.776109+0000", 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4840000000000003e-06 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.776178+0000", 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9558e-05 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:17.776383+0000", 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020432299999999999 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:170693 6.f e595/467 6:ff66131a:::benchmark_data_smithi192_117994_object63099:head v 595'10776, mlcod=595'10774)", 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:34.707305+0000", 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 63.892850441, 2024-06-27T21:36:38.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.480114031, 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:34.707305+0000", 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:34.707305+0000", 2024-06-27T21:36:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:34.707306+0000", 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8100000000000002e-07 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:34.707311+0000", 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5679999999999999e-06 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:35.183750+0000", 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.47643845699999998 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:35.183758+0000", 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.858e-06 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:35.183819+0000", 2024-06-27T21:36:38.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1904999999999998e-05 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:35.183834+0000", 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4812e-05 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:35.187381+0000", 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035463270000000002 2024-06-27T21:36:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:35.187419+0000", 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8322999999999999e-05 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.26020.0:175945 6.1 e617/552 ondisk, result = 0)", 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:16.057770+0000", 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.542385569, 2024-06-27T21:36:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66511086900000005, 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057770+0000", 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057770+0000", 2024-06-27T21:36:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057771+0000", 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7899999999999997e-07 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057773+0000", 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.694e-06 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722691+0000", 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66491738499999997 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722698+0000", 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0709999999999997e-06 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722766+0000", 2024-06-27T21:36:38.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8103999999999997e-05 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722777+0000", 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1148000000000001e-05 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722881+0000", 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103588 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:176869 6.14 e623/550 6:2d3ec838:::benchmark_data_smithi192_117994_object69275:head v 623'10725, mlcod=622'10721)", 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:22.341090+0000", 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 16.259065553999999, 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42259212899999998, 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.341090+0000", 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.341090+0000", 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.341090+0000", 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6700000000000003e-07 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.341096+0000", 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3569999999999997e-06 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.761868+0000", 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42077194600000001 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.761877+0000", 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8710000000000003e-06 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.761932+0000", 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5526999999999999e-05 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.761954+0000", 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1594999999999999e-05 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.763642+0000", 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001688206 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:22.763682+0000", 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9959999999999997e-05 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:177152 6.4 e624/606 6:228c94ea:::benchmark_data_smithi192_117994_object69558:head v 624'10842, mlcod=624'10840)", 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:24.041473+0000", 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.558682081000001, 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78381791700000003, 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041473+0000", 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041473+0000", 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041474+0000", 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:38.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041479+0000", 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.361e-06 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.822955+0000", 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78147614200000004 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.822971+0000", 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5420000000000001e-05 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.823065+0000", 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4021e-05 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.823084+0000", 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8881000000000001e-05 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.825254+0000", 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021705190000000001 2024-06-27T21:36:38.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.825291+0000", 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6813000000000001e-05 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:38.705 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:36:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:38.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:38.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:38.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:38.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:38.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:38.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:39.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:39.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:39.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:39.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:39.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:39.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:39.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:39.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:36:39.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:39.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:39.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:39.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:39.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:39.683 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:39.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:39.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:39.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:39.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:39.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:39.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:40.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:39 smithi190 ceph-mon[32247]: pgmap v1605: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 3/110211 objects degraded (0.003%); 12940/110211 objects misplaced (11.741%); 883 KiB/s, 13 objects/s recovering 2024-06-27T21:36:40.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:39 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:36:40.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:39 smithi190 ceph-mon[32247]: 6.1 scrub ok 2024-06-27T21:36:40.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/110211 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:40.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:40.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:39 smithi192 ceph-mon[32956]: pgmap v1605: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 3/110211 objects degraded (0.003%); 12940/110211 objects misplaced (11.741%); 883 KiB/s, 13 objects/s recovering 2024-06-27T21:36:40.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:39 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:36:40.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:39 smithi192 ceph-mon[32956]: 6.1 scrub ok 2024-06-27T21:36:40.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/110211 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:40.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:40.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:40.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:39 smithi151 ceph-mon[28965]: pgmap v1605: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 124 op/s; 3/110211 objects degraded (0.003%); 12940/110211 objects misplaced (11.741%); 883 KiB/s, 13 objects/s recovering 2024-06-27T21:36:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:39 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:36:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:39 smithi151 ceph-mon[28965]: 6.1 scrub ok 2024-06-27T21:36:40.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/110211 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:40.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:40.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:40.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:40.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:40.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:40.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:40.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:40.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:40.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:40.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:40.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:36:40.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:40.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:40.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:41.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:41.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:41.132 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:41.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:41.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:41.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:41.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:41.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:41.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:41.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:179617 6.d 6:b0073b82:::benchmark_data_smithi192_117994_object72023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:41.448045+0000", 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.022410771999999999, 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022434777999999999, 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:36:41.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 179617 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448045+0000", 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:41.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448045+0000", 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448046+0000", 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1200000000000001e-06 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448051+0000", 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.728e-06 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448062+0000", 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0349999999999999e-05 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448067+0000", 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5119999999999999e-06 2024-06-27T21:36:41.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448123+0000", 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5982e-05 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448255+0000", 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013176799999999999 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,11", 2024-06-27T21:36:41.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.448384+0000", 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000128959 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.452142+0000", 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037576049999999998 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.452267+0000", 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012587500000000001 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:179623 6.f e629/467 6:f01725ba:::benchmark_data_smithi192_117994_object72029:head v 629'11340, mlcod=629'11338)", 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:41.469466+0000", 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00098894400000000006, 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001059087, 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:41.478 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469466+0000", 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469466+0000", 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469467+0000", 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0550000000000001e-06 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469474+0000", 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7499999999999997e-06 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469506+0000", 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1588999999999999e-05 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469515+0000", 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3749999999999992e-06 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469570+0000", 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4811000000000001e-05 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:41.469586+0000", 2024-06-27T21:36:41.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5988e-05 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-06-27T21:36:41.482 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:36:41.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:41.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:41.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:41.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:41.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:41.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:41.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:41.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:41.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:41 smithi151 ceph-mon[28965]: pgmap v1606: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 3/108909 objects degraded (0.003%); 12702/108909 objects misplaced (11.663%); 1.4 MiB/s, 22 objects/s recovering 2024-06-27T21:36:41.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:41 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:36:42.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:42.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:41 smithi192 ceph-mon[32956]: pgmap v1606: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 3/108909 objects degraded (0.003%); 12702/108909 objects misplaced (11.663%); 1.4 MiB/s, 22 objects/s recovering 2024-06-27T21:36:42.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:41 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:36:42.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:42.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:36:42.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:42.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:41 smithi190 ceph-mon[32247]: pgmap v1606: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 3/108909 objects degraded (0.003%); 12702/108909 objects misplaced (11.663%); 1.4 MiB/s, 22 objects/s recovering 2024-06-27T21:36:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:41 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:36:42.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:42.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:42.407 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:42.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:42.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:42.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:42.574 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7] out_osds: [8, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:36:42.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:36:42.575 INFO:tasks.thrashosds.thrasher:Adding osd 8 2024-06-27T21:36:42.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 8 2024-06-27T21:36:42.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:42.586 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:42.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:42.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:42.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:42 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:36:42.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:42 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:36:42.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:42 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:36:42.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:42.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:42.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:42.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:42.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:43.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:43.076 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:42 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:36:43.076 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:42 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:36:43.076 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:42 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:36:43.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:43.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:43.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:42 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:36:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:42 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:36:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:42 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:36:43.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:43.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:43.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:43.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:43.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:43.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:43.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:43.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:43.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:43.773 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:36:43.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:43.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:43.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:43 smithi151 ceph-mon[28965]: pgmap v1607: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 3/108171 objects degraded (0.003%); 12702/108171 objects misplaced (11.743%); 1.2 MiB/s, 18 objects/s recovering 2024-06-27T21:36:43.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1956546446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:43.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:43.942 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:43.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:44.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:44.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:43 smithi192 ceph-mon[32956]: pgmap v1607: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 3/108171 objects degraded (0.003%); 12702/108171 objects misplaced (11.743%); 1.2 MiB/s, 18 objects/s recovering 2024-06-27T21:36:44.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1956546446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:44.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:44.087 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:36:44.087 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:36:44.087 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:36:44.087 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:36:44.087 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:36:44.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:44.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:43 smithi190 ceph-mon[32247]: pgmap v1607: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 165 op/s; 3/108171 objects degraded (0.003%); 12702/108171 objects misplaced (11.743%); 1.2 MiB/s, 18 objects/s recovering 2024-06-27T21:36:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1956546446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:44.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:44.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:44.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:44.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:44.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:44.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:44.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:44.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:44.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:44.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:44.912 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:44 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:36:44.912 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1956546446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:44.913 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:44 smithi192 ceph-mon[32956]: osdmap e630: 12 total, 12 up, 10 in 2024-06-27T21:36:44.913 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/108171 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:44.913 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:44 smithi192 ceph-mon[32956]: osdmap e631: 12 total, 12 up, 10 in 2024-06-27T21:36:44.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:44.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:44.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:44.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:36:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:44 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:36:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1956546446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:44 smithi151 ceph-mon[28965]: osdmap e630: 12 total, 12 up, 10 in 2024-06-27T21:36:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/108171 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:44.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:44 smithi151 ceph-mon[28965]: osdmap e631: 12 total, 12 up, 10 in 2024-06-27T21:36:45.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:45.059 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:45.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:45.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:45.250 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:44 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1956546446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:44 smithi190 ceph-mon[32247]: osdmap e630: 12 total, 12 up, 10 in 2024-06-27T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/108171 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:44 smithi190 ceph-mon[32247]: osdmap e631: 12 total, 12 up, 10 in 2024-06-27T21:36:45.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:45.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:45.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:45.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:45.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:45.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:45.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:45.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:45.736 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:36:45.736 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:36:45.762 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:45.771 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:36:45.772 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:36:45.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:45.799 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:36:45.799 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:36:45.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:45.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:36:45.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:36:45.847 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:36:45.847 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:36:45.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:36:45.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:36:45.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:36:45.896 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:36:45.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:45.920 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:36:45.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:36:45.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:45.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:45.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:36:45.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:36:45.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:36:45.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:36:45.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:36:45.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:36:45.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:45 smithi151 ceph-mon[28965]: pgmap v1609: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 3/107109 objects degraded (0.003%); 12600/107109 objects misplaced (11.764%); 616 KiB/s, 9 objects/s recovering 2024-06-27T21:36:45.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/952407051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:46.009 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:36:46.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:36:46.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:45 smithi190 ceph-mon[32247]: pgmap v1609: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 3/107109 objects degraded (0.003%); 12600/107109 objects misplaced (11.764%); 616 KiB/s, 9 objects/s recovering 2024-06-27T21:36:46.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/952407051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:46.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:46.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:45 smithi192 ceph-mon[32956]: pgmap v1609: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.2 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 140 op/s; 3/107109 objects degraded (0.003%); 12600/107109 objects misplaced (11.764%); 616 KiB/s, 9 objects/s recovering 2024-06-27T21:36:46.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/952407051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:46.053 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:46.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:46.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:46.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:46.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:46.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:46.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:46.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:46.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:46.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:36:46.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:46.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:46.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:46.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:46.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:46.926 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:46.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:46.978 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:46.978 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:47.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:47.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:47.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:47.176 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:47.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:47.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:47.358 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:47.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:47.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:47.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:47.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:47.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:36:47.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:47.668 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:47.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:47.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:47.849 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:47.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:47.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:47.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:47.970 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:47 smithi192 ceph-mon[32956]: pgmap v1611: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 3/105321 objects degraded (0.003%); 12383/105321 objects misplaced (11.757%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:36:48.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:48.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:48.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:48.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:48.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:47 smithi151 ceph-mon[28965]: pgmap v1611: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 3/105321 objects degraded (0.003%); 12383/105321 objects misplaced (11.757%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:36:48.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:47 smithi190 ceph-mon[32247]: pgmap v1611: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 3/105321 objects degraded (0.003%); 12383/105321 objects misplaced (11.757%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:36:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:48.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:48.389 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:48.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:48.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:48.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:48.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:48.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:48.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:48.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:48.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:36:48.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:48.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:48.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:49.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:49.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:49.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:49.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:49.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:49.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:49.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:49.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:49.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:49.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:49.645 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:49.645 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:49.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:49.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:49.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:49.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:49.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:49.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:50.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:50.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:50.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:49 smithi192 ceph-mon[32956]: pgmap v1612: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 3/105066 objects degraded (0.003%); 12383/105066 objects misplaced (11.786%); 728 KiB/s, 11 objects/s recovering 2024-06-27T21:36:50.035 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/105066 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:50.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:49 smithi151 ceph-mon[28965]: pgmap v1612: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 3/105066 objects degraded (0.003%); 12383/105066 objects misplaced (11.786%); 728 KiB/s, 11 objects/s recovering 2024-06-27T21:36:50.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/105066 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:50.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:49 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:36:49.988+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e631 definitely_dead 0 2024-06-27T21:36:50.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:50.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:50.198 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:50.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:49 smithi190 ceph-mon[32247]: pgmap v1612: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 3/105066 objects degraded (0.003%); 12383/105066 objects misplaced (11.786%); 728 KiB/s, 11 objects/s recovering 2024-06-27T21:36:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/105066 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:50.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:50.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:36:50.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:50.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:50.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:50.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:50.610 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:50.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.646 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:50.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:50.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:50.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:50.796 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:36:50.796 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.8. 2024-06-27T21:36:50.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:50.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:50.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:50.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/444739856' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-06-27T21:36:50.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1900529781' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:36:50.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:51.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:51.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/444739856' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-06-27T21:36:51.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1900529781' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:36:51.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:51.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:51.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:51.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/444739856' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["8"]}]: dispatch 2024-06-27T21:36:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1900529781' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:36:51.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:51.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:51.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:51.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:51.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:51.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:51.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:51.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:51.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:51.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:51.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:51.989 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:51 smithi192 ceph-mon[32956]: pgmap v1613: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 175 op/s; 3/103962 objects degraded (0.003%); 12129/103962 objects misplaced (11.667%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:36:51.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:51 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:51.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/444739856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-06-27T21:36:51.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1900529781' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:36:51.990 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:51 smithi192 ceph-mon[32956]: osdmap e632: 12 total, 12 up, 11 in 2024-06-27T21:36:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:52.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:52.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:51 smithi151 ceph-mon[28965]: pgmap v1613: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 175 op/s; 3/103962 objects degraded (0.003%); 12129/103962 objects misplaced (11.667%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:36:52.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:51 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:52.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/444739856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-06-27T21:36:52.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1900529781' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:36:52.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:51 smithi151 ceph-mon[28965]: osdmap e632: 12 total, 12 up, 11 in 2024-06-27T21:36:52.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:52.119 INFO:tasks.thrashosds.thrasher:Added osd 8 2024-06-27T21:36:52.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:52.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:52.248 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:36:52.248 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:36:52.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:51 smithi190 ceph-mon[32247]: pgmap v1613: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 175 op/s; 3/103962 objects degraded (0.003%); 12129/103962 objects misplaced (11.667%); 1.3 MiB/s, 20 objects/s recovering 2024-06-27T21:36:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:51 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/444739856' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["8"]}]': finished 2024-06-27T21:36:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1900529781' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:36:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:51 smithi190 ceph-mon[32247]: osdmap e632: 12 total, 12 up, 11 in 2024-06-27T21:36:52.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:52.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:52.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:52.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:52.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:52.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:52.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:52.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:52.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:52.844 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:52 smithi192 ceph-mon[32956]: osdmap e633: 12 total, 12 up, 11 in 2024-06-27T21:36:52.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:52.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:52.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:52.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:36:52.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:52.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:53.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:52 smithi151 ceph-mon[28965]: osdmap e633: 12 total, 12 up, 11 in 2024-06-27T21:36:53.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:53.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:53.212 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:36:53.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:53.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:52 smithi190 ceph-mon[32247]: osdmap e633: 12 total, 12 up, 11 in 2024-06-27T21:36:53.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:36:53.288 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:36:53.316 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:36:53.316 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:36:53.342 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:36:53.342 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:36:53.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:53.365 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:36:53.366 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:36:53.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:53.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:36:53.394 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:36:53.416 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:36:53.417 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:36:53.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:36:53.439 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:36:53.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:36:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:36:53.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:36:53.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:53.483 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:36:53.483 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:36:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:53.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:53.522 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:36:53.523 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:36:53.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:36:53.560 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:36:53.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:36:53.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:36:53.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:53.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:53.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:53.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:53.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:53.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:53.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:53.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:53 smithi151 ceph-mon[28965]: pgmap v1616: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 3/103443 objects degraded (0.003%); 12129/103443 objects misplaced (11.725%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:36:53.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:53 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:36:53.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:53 smithi151 ceph-mon[28965]: osdmap e634: 12 total, 12 up, 11 in 2024-06-27T21:36:53.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:53 smithi151 ceph-mon[28965]: 6.4 starting backfill to osd.8 from (0'0,0'0] MAX to 631'10954 2024-06-27T21:36:53.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1623093674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:53.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:54.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:54.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:53 smithi192 ceph-mon[32956]: pgmap v1616: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 3/103443 objects degraded (0.003%); 12129/103443 objects misplaced (11.725%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:36:54.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:53 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:36:54.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:53 smithi192 ceph-mon[32956]: osdmap e634: 12 total, 12 up, 11 in 2024-06-27T21:36:54.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:53 smithi192 ceph-mon[32956]: 6.4 starting backfill to osd.8 from (0'0,0'0] MAX to 631'10954 2024-06-27T21:36:54.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1623093674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:54.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:36:54.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:36:54.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:54.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:53 smithi190 ceph-mon[32247]: pgmap v1616: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 53 active+clean; 2.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 3/103443 objects degraded (0.003%); 12129/103443 objects misplaced (11.725%); 1.4 MiB/s, 21 objects/s recovering 2024-06-27T21:36:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:53 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:36:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:53 smithi190 ceph-mon[32247]: osdmap e634: 12 total, 12 up, 11 in 2024-06-27T21:36:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:53 smithi190 ceph-mon[32247]: 6.4 starting backfill to osd.8 from (0'0,0'0] MAX to 631'10954 2024-06-27T21:36:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1623093674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:36:54.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:54.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:54.428 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:54.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:54.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:54.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:54.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:54.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:54.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:54.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:54.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:54.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3/103443 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:54.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:54 smithi151 ceph-mon[28965]: osdmap e635: 12 total, 12 up, 11 in 2024-06-27T21:36:54.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:54.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:54.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3/103443 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:55.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:54 smithi192 ceph-mon[32956]: osdmap e635: 12 total, 12 up, 11 in 2024-06-27T21:36:55.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:55.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:55.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:55.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3/103443 objects degraded (0.003%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:36:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:54 smithi190 ceph-mon[32247]: osdmap e635: 12 total, 12 up, 11 in 2024-06-27T21:36:55.304 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:36:55.304 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:36:55.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.6 530/515 [PushOp(6:6fcb7a05:::benchmark_data_smithi192_117994_object61457:head, version: 488'3822, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:6fcb7a05:::benchmark_data_smithi192_117994_object61457:head@488'3822, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:28.754260+0000", 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 266.54633149400001, 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39696379599999998, 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:28.754260+0000", 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:28.754260+0000", 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:28.754261+0000", 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4789999999999999e-06 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:28.754306+0000", 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4591e-05 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:29.148852+0000", 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39454622900000003 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:29.148866+0000", 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3585e-05 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:29.148944+0000", 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8214999999999996e-05 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:29.148953+0000", 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5550000000000008e-06 2024-06-27T21:36:55.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:29.151223+0000", 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022701420000000002 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.6 530/515 [PushOp(6:622d1bda:::benchmark_data_smithi192_117994_object64764:head, version: 489'4018, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:622d1bda:::benchmark_data_smithi192_117994_object64764:head@489'4018, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:31.971619+0000", 2024-06-27T21:36:55.309 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.32897203700003, 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67493599599999998, 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:31.971619+0000", 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:31.971619+0000", 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:31.971620+0000", 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3069999999999999e-06 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:31.971657+0000", 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6180000000000003e-05 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:32.644080+0000", 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.672423726 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:32.644110+0000", 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9346999999999999e-05 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:32.644316+0000", 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020633899999999999 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:32.644332+0000", 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6073999999999999e-05 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:32.646555+0000", 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022230230000000002 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.6 530/515 [PushOp(6:63bc5487:::benchmark_data_smithi192_117994_object66512:head, version: 490'4143, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:63bc5487:::benchmark_data_smithi192_117994_object66512:head@490'4143, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:34.323519+0000", 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 260.97707241400002, 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42225131399999999, 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.323519+0000", 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.323519+0000", 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.323521+0000", 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0099999999999998e-06 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.323568+0000", 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7122e-05 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.743163+0000", 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41959465000000001 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.743192+0000", 2024-06-27T21:36:55.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9179e-05 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.743403+0000", 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00021166800000000001 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.743413+0000", 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9709999999999999e-06 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:34.745770+0000", 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023567140000000002 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.6 531/515 [PushOp(6:66a8fa5e:::benchmark_data_smithi192_117994_object68536:head, version: 490'4262, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:66a8fa5e:::benchmark_data_smithi192_117994_object68536:head@490'4262, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:36.683571+0000", 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 258.61702034799998, 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42881646800000001, 2024-06-27T21:36:55.318 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:36.683571+0000", 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:36.683571+0000", 2024-06-27T21:36:55.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:36.683572+0000", 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.176e-06 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:36.683611+0000", 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8543e-05 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:37.110043+0000", 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42643282799999999 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:37.110055+0000", 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2058e-05 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:37.110130+0000", 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4800000000000002e-05 2024-06-27T21:36:55.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:37.110140+0000", 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6460000000000005e-06 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:37.112387+0000", 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022474169999999998 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.322 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.6 532/515 [PushOp(6:6e359d88:::benchmark_data_smithi192_117994_object74523:head, version: 492'4617, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:6e359d88:::benchmark_data_smithi192_117994_object74523:head@492'4617, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:42.327467+0000", 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 252.97312404100001, 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53478681100000003, 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.327467+0000", 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.327467+0000", 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.327469+0000", 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9010000000000001e-06 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.327522+0000", 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2772999999999998e-05 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.859893+0000", 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.53237089999999998 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.859911+0000", 2024-06-27T21:36:55.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7819e-05 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.860034+0000", 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012385099999999999 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.860045+0000", 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0356999999999999e-05 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:42.862254+0000", 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022092100000000001 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.6 533/515 [PushOp(6:69536772:::benchmark_data_smithi192_117994_object79987:head, version: 494'4950, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:69536772:::benchmark_data_smithi192_117994_object79987:head@494'4950, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:48.108106+0000", 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 247.19248476000001, 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44360082499999998, 2024-06-27T21:36:55.327 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.108106+0000", 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.108106+0000", 2024-06-27T21:36:55.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.108107+0000", 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0610000000000001e-06 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.108145+0000", 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759000000000002e-05 2024-06-27T21:36:55.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.549147+0000", 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44100162500000001 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.549159+0000", 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2449999999999999e-05 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.549229+0000", 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0042000000000005e-05 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.549240+0000", 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0210999999999999e-05 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.551707+0000", 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024676770000000002 2024-06-27T21:36:55.331 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.11 551/549 [PushOp(6:88b01f5b:::benchmark_data_smithi192_117994_object52897:head, version: 486'3213, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:88b01f5b:::benchmark_data_smithi192_117994_object52897:head@486'3213, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:30.373375+0000", 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 204.92721632000001, 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.688589954, 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.373375+0000", 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.373375+0000", 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.373376+0000", 2024-06-27T21:36:55.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.187e-06 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.373404+0000", 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7696999999999999e-05 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.060066+0000", 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.68666268600000002 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.060078+0000", 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1138000000000001e-05 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.060186+0000", 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010812899999999999 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.060195+0000", 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4530000000000008e-06 2024-06-27T21:36:55.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.061965+0000", 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001769664 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(6.13 551/550 [PushReplyOp(6:ca935286:::benchmark_data_smithi192_117994_object51457:head)])", 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:30.779528+0000", 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 204.52106358500001, 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70503941599999997, 2024-06-27T21:36:55.336 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.779528+0000", 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.779528+0000", 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.779528+0000", 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:30.779533+0000", 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.482309+0000", 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.70277645300000002 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.482320+0000", 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0224000000000001e-05 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.482392+0000", 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1920000000000003e-05 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.484567+0000", 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021753969999999999 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(6.13 551/550 [PushReplyOp(6:cc2c0cb8:::benchmark_data_smithi192_117994_object51757:head)])", 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:31.833423+0000", 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 203.467168292, 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44743042599999999, 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.340 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.833423+0000", 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.833423+0000", 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.833423+0000", 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6000000000000004e-07 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.833428+0000", 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0849999999999999e-06 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.278809+0000", 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44538160999999998 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.278815+0000", 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9549999999999997e-06 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.278864+0000", 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8834e-05 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.280853+0000", 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001989382 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.344 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(6.13 551/550 [PushReplyOp(6:ccc26b67:::benchmark_data_smithi192_117994_object52806:head)])", 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:33.009210+0000", 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 202.29138130300001, 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45820343000000002, 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.009210+0000", 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.009210+0000", 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.009211+0000", 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6000000000000002e-07 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.009216+0000", 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4620000000000004e-06 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.465120+0000", 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45590362299999998 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.465127+0000", 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1389999999999998e-06 2024-06-27T21:36:55.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.465186+0000", 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8564999999999999e-05 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:33.467413+0000", 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022277809999999999 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.348 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.17 559/550 [PushOp(6:ec288195:::benchmark_data_smithi192_117994_object53539:head, version: 486'3309, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:ec288195:::benchmark_data_smithi192_117994_object53539:head@486'3309, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:59.344475+0000", 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 175.95611636300001, 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91184076400000003, 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:59.344475+0000", 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:59.344475+0000", 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:59.344476+0000", 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8299999999999996e-07 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:59.344525+0000", 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9625e-05 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:00.255001+0000", 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91047613199999999 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:00.255012+0000", 2024-06-27T21:36:55.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0842999999999999e-05 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:00.255078+0000", 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6216000000000004e-05 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:00.255087+0000", 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:00.256316+0000", 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012287070000000001 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.17 561/550 [PushOp(6:e815fb3d:::benchmark_data_smithi192_117994_object59141:head, version: 487'3675, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e815fb3d:::benchmark_data_smithi192_117994_object59141:head@487'3675, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:05.159231+0000", 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 170.14135990700001, 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43609203800000002, 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.353 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.159231+0000", 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.159231+0000", 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.159233+0000", 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3960000000000001e-06 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.159301+0000", 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889e-05 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.593109+0000", 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43380892900000001 2024-06-27T21:36:55.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.593118+0000", 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6819999999999992e-06 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.593206+0000", 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7851999999999998e-05 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.593219+0000", 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3251999999999999e-05 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:05.595323+0000", 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021040379999999999 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.17 563/550 [PushOp(6:ebf17411:::benchmark_data_smithi192_117994_object66286:head, version: 490'4106, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:ebf17411:::benchmark_data_smithi192_117994_object66286:head@490'4106, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:11.448497+0000", 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 163.85209445000001, 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.553820282, 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:11.448497+0000", 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:11.448497+0000", 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:11.448498+0000", 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0299999999999998e-07 2024-06-27T21:36:55.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:11.448525+0000", 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7891000000000001e-05 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.000836+0000", 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55231102200000004 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.000852+0000", 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.524e-05 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.000934+0000", 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2694999999999993e-05 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.000945+0000", 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0203e-05 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.002317+0000", 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001372428 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.13 e564,550 6:c92c687f:::benchmark_data_smithi192_117994_object44319:head,524'9667)", 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:15.712628+0000", 2024-06-27T21:36:55.362 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 159.58796283300001, 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44675125300000001, 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:15.712628+0000", 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:15.712628+0000", 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:15.712629+0000", 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7299999999999995e-07 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:15.712634+0000", 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6990000000000002e-06 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.158929+0000", 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44629511500000002 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.158934+0000", 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8640000000000004e-06 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.159025+0000", 2024-06-27T21:36:55.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1470000000000003e-05 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:16.159380+0000", 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035443200000000001 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:172058 6.12 6:4f3e26e3:::benchmark_data_smithi192_117994_object64464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:45.865754+0000", 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.434836877999999, 2024-06-27T21:36:55.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55146667599999999, 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172058 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865754+0000", 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865754+0000", 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865755+0000", 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1489999999999999e-06 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865760+0000", 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7820000000000001e-06 2024-06-27T21:36:55.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865773+0000", 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2860999999999999e-05 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865782+0000", 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7490000000000004e-06 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865842+0000", 2024-06-27T21:36:55.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0189999999999998e-05 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.865935+0000", 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3374000000000004e-05 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,4", 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.866023+0000", 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7326999999999999e-05 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:36:55.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.867244+0000", 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012217630000000001 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.868111+0000", 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00086669799999999995 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417146+0000", 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54903487500000003 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417188+0000", 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1458999999999997e-05 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417221+0000", 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3448999999999998e-05 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.26020.0:172058 6.12 e598/549 ondisk, result = 0)", 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:45.867758+0000", 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.432833044999995, 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54946827300000001, 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.867758+0000", 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.867758+0000", 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.867760+0000", 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5460000000000001e-06 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:45.867774+0000", 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4347e-05 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417061+0000", 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54928668199999997 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417068+0000", 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7000000000000008e-06 2024-06-27T21:36:55.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417134+0000", 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5524000000000005e-05 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417142+0000", 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9389999999999997e-06 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417226+0000", 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4535000000000003e-05 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:172059 6.13 6.bad0cdd3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:46.003955+0000", 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.296636352999997, 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.84500614500000004, 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.377 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172059 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.003955+0000", 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.003955+0000", 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.003956+0000", 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4929999999999999e-06 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.003963+0000", 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2840000000000003e-06 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.003988+0000", 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5483999999999999e-05 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.003996+0000", 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7000000000000008e-06 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.004052+0000", 2024-06-27T21:36:55.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6264000000000002e-05 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.004061+0000", 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.848934+0000", 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.84487298200000005 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.848961+0000", 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6723000000000001e-05 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:172062 6.13 6.20b7a8d3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:46.112478+0000", 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.188112942000004, 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.73668481600000002, 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.382 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172062 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112478+0000", 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112478+0000", 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112480+0000", 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.73e-06 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112487+0000", 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7680000000000003e-06 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112501+0000", 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4433999999999999e-05 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112508+0000", 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2540000000000003e-06 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112570+0000", 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2008999999999995e-05 2024-06-27T21:36:55.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.112580+0000", 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1859999999999998e-06 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.849154+0000", 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.73657412899999997 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.849163+0000", 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3060000000000003e-06 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:172068 6.13 6.e0a28eb3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:46.417371+0000", 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.883219730999997, 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43182478000000002, 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.387 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172068 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417371+0000", 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417371+0000", 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417372+0000", 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9600000000000008e-07 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417376+0000", 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7129999999999999e-06 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417386+0000", 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5990000000000008e-06 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417393+0000", 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5100000000000001e-06 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417443+0000", 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9441999999999999e-05 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.417450+0000", 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5599999999999996e-06 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.849191+0000", 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.431740548 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.849196+0000", 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.412e-06 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:176664 6.4 e621/606 6:20a3f22e:::benchmark_data_smithi192_117994_object69070:head v 621'10824, mlcod=620'10822)", 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:19.739334+0000", 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 35.561256962000002, 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41467594000000002, 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:36:55.392 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:19.739334+0000", 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:19.739334+0000", 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:19.739334+0000", 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.73e-07 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:19.739339+0000", 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.138e-06 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:20.151991+0000", 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41265206399999999 2024-06-27T21:36:55.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:20.152000+0000", 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.02e-06 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:20.152085+0000", 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5698000000000005e-05 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:36:55.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:20.152101+0000", 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5945000000000001e-05 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:20.153957+0000", 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001855385 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:20.154010+0000", 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3517000000000003e-05 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.396 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.397 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:36:55.397 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:36:55.397 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:36:55.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:55.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:55.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:55.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:55.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:55.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:55.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:55.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:55.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:55.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:55.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:56.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:56.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:55 smithi151 ceph-mon[28965]: pgmap v1618: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 52 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 3/103005 objects degraded (0.003%); 13174/103005 objects misplaced (12.790%); 874 KiB/s, 13 objects/s recovering 2024-06-27T21:36:56.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:56.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:56.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:56.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:55 smithi192 ceph-mon[32956]: pgmap v1618: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 52 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 3/103005 objects degraded (0.003%); 13174/103005 objects misplaced (12.790%); 874 KiB/s, 13 objects/s recovering 2024-06-27T21:36:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:55 smithi190 ceph-mon[32247]: pgmap v1618: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 52 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 114 op/s; 3/103005 objects degraded (0.003%); 13174/103005 objects misplaced (12.790%); 874 KiB/s, 13 objects/s recovering 2024-06-27T21:36:56.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:36:56.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:56.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:56.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:56.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:36:56.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:56.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:56.578 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:56.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:56.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:56.697 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:36:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:56.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:56.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:36:56.815 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:36:56.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:36:56.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:56.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:56.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:56.982 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4271292126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:56.983 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:56 smithi190 ceph-mon[32247]: from='client.16845 172.21.15.151:0/4271292126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:57.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:57.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:57.120 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7, 8] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:36:57.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:36:57.120 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-27T21:36:57.121 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.902373 2024-06-27T21:36:57.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 5 0.9023726381772217 2024-06-27T21:36:57.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4271292126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:57.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:56 smithi151 ceph-mon[28965]: from='client.16845 172.21.15.151:0/4271292126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:57.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:57.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4271292126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:56 smithi192 ceph-mon[32956]: from='client.16845 172.21.15.151:0/4271292126' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:36:57.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:57.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:57.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:57.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:57.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:57.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:57.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:57.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:57.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:57.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:57.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:57.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:57.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:58.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:36:58.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:36:58.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:57 smithi190 ceph-mon[32247]: pgmap v1620: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 50 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 3368/102798 objects degraded (3.276%); 14005/102798 objects misplaced (13.624%); 3.0 MiB/s, 48 objects/s recovering 2024-06-27T21:36:58.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:57 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:58.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:57 smithi190 ceph-mon[32247]: from='client.16845 172.21.15.151:0/4271292126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:58.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:57 smithi190 ceph-mon[32247]: osdmap e636: 12 total, 12 up, 11 in 2024-06-27T21:36:58.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:58.196 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:57 smithi192 ceph-mon[32956]: pgmap v1620: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 50 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 3368/102798 objects degraded (3.276%); 14005/102798 objects misplaced (13.624%); 3.0 MiB/s, 48 objects/s recovering 2024-06-27T21:36:58.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:57 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:58.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:57 smithi192 ceph-mon[32956]: from='client.16845 172.21.15.151:0/4271292126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:58.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:57 smithi192 ceph-mon[32956]: osdmap e636: 12 total, 12 up, 11 in 2024-06-27T21:36:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:57 smithi151 ceph-mon[28965]: pgmap v1620: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 1 active+recovering, 50 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 40 op/s; 3368/102798 objects degraded (3.276%); 14005/102798 objects misplaced (13.624%); 3.0 MiB/s, 48 objects/s recovering 2024-06-27T21:36:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:57 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:36:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:57 smithi151 ceph-mon[28965]: from='client.16845 172.21.15.151:0/4271292126' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:36:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:57 smithi151 ceph-mon[28965]: osdmap e636: 12 total, 12 up, 11 in 2024-06-27T21:36:58.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:58.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:58.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:58.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:36:58.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:36:58.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:58.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:58.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:58.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:58.726 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:36:58.726 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:36:58.726 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:36:58.726 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:36:58.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:36:58.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:36:58.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:58 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:36:58.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:58.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:58.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:36:58.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:58 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:36:59.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:36:59.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:36:59.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:59.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:58 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:36:59.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:36:59.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:36:59.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:36:59.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:36:59.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:36:59.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:36:59.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:36:59.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:36:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:36:59.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:36:59.775 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:36:59.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:36:59.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:36:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:36:59.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:36:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:36:59.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:36:59.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:00.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:59 smithi190 ceph-mon[32247]: pgmap v1622: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 42 op/s; 12519/102675 objects degraded (12.193%); 7672/102675 objects misplaced (7.472%); 15 MiB/s, 242 objects/s recovering 2024-06-27T21:37:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 12519/102675 objects degraded (12.193%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:36:59 smithi190 ceph-mon[32247]: osdmap e637: 12 total, 12 up, 11 in 2024-06-27T21:37:00.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:00.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:59 smithi151 ceph-mon[28965]: pgmap v1622: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 42 op/s; 12519/102675 objects degraded (12.193%); 7672/102675 objects misplaced (7.472%); 15 MiB/s, 242 objects/s recovering 2024-06-27T21:37:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:59 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 12519/102675 objects degraded (12.193%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:36:59 smithi151 ceph-mon[28965]: osdmap e637: 12 total, 12 up, 11 in 2024-06-27T21:37:00.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:59 smithi192 ceph-mon[32956]: pgmap v1622: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 42 op/s; 12519/102675 objects degraded (12.193%); 7672/102675 objects misplaced (7.472%); 15 MiB/s, 242 objects/s recovering 2024-06-27T21:37:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 12519/102675 objects degraded (12.193%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:36:59 smithi192 ceph-mon[32956]: osdmap e637: 12 total, 12 up, 11 in 2024-06-27T21:37:00.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:00.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:00.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:00.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:00.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:00.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:00.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:00.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:00.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:00.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:00.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:01.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:01.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:01.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:01.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:01.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:01.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:01.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:01.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:01.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:01.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:01.593 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:37:01.593 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:37:01.594 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:37:01.594 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:37:01.594 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:37:01.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:01.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:01.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:01.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:01.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:01.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:01.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:01.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:01.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:37:01.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:37:01.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:37:01.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:37:01.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:37:01.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:37:02.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:02.006 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:37:02.007 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:37:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:01 smithi190 ceph-mon[32247]: pgmap v1624: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 19 op/s; 12519/102645 objects degraded (12.196%); 7663/102645 objects misplaced (7.466%); 15 MiB/s, 241 objects/s recovering 2024-06-27T21:37:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:02.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:01 smithi190 ceph-mon[32247]: osdmap e638: 12 total, 12 up, 11 in 2024-06-27T21:37:02.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:02.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:37:02.029 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:37:02.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:37:02.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:37:02.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:37:02.073 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:37:02.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:01 smithi151 ceph-mon[28965]: pgmap v1624: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 19 op/s; 12519/102645 objects degraded (12.196%); 7663/102645 objects misplaced (7.466%); 15 MiB/s, 241 objects/s recovering 2024-06-27T21:37:02.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:02.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:02.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:01 smithi151 ceph-mon[28965]: osdmap e638: 12 total, 12 up, 11 in 2024-06-27T21:37:02.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:02.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:02.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:37:02.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:37:02.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:37:02.212 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:37:02.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:02.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:02.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:37:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:01 smithi192 ceph-mon[32956]: pgmap v1624: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 19 op/s; 12519/102645 objects degraded (12.196%); 7663/102645 objects misplaced (7.466%); 15 MiB/s, 241 objects/s recovering 2024-06-27T21:37:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:01 smithi192 ceph-mon[32956]: osdmap e638: 12 total, 12 up, 11 in 2024-06-27T21:37:02.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:02.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:02.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:37:02.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:37:02.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:37:02.459 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:37:02.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:37:02.481 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:37:02.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:02.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:02.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:02.629 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:37:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:02.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:02.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:02.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:02.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:02.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:02.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:02.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3009608226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:02.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:03.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:03.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3009608226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:03.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:03.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:03.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:03.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3009608226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:03.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:03.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:03.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:37:03.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:03.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:03.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:03.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:03.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:03.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:03.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:03.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:03.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:03.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:04.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:04.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:04.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:04.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:03 smithi151 ceph-mon[28965]: pgmap v1626: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 8 op/s; 12519/102642 objects degraded (12.197%); 7175/102642 objects misplaced (6.990%); 18 MiB/s, 281 objects/s recovering 2024-06-27T21:37:04.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:04.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:04.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:03 smithi190 ceph-mon[32247]: pgmap v1626: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 8 op/s; 12519/102642 objects degraded (12.197%); 7175/102642 objects misplaced (6.990%); 18 MiB/s, 281 objects/s recovering 2024-06-27T21:37:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:03 smithi192 ceph-mon[32956]: pgmap v1626: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 8 op/s; 12519/102642 objects degraded (12.197%); 7175/102642 objects misplaced (6.990%); 18 MiB/s, 281 objects/s recovering 2024-06-27T21:37:04.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:04.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:04.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:04.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:04.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:04.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:04.634 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:04.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:04.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:04.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:04.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:04.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:04.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:05.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:05.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:04 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:37:05.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 12519/102642 objects degraded (12.197%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:05.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:04 smithi151 ceph-mon[28965]: osdmap e639: 12 total, 12 up, 11 in 2024-06-27T21:37:05.027 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:04 smithi151 ceph-mon[28965]: from='client.16857 172.21.15.151:0/4220748595' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9023726381772217}]: dispatch 2024-06-27T21:37:05.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:05.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:05.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:05.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:04 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:37:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 12519/102642 objects degraded (12.197%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:04 smithi190 ceph-mon[32247]: osdmap e639: 12 total, 12 up, 11 in 2024-06-27T21:37:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:04 smithi190 ceph-mon[32247]: from='client.16857 172.21.15.151:0/4220748595' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9023726381772217}]: dispatch 2024-06-27T21:37:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:04 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:37:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 12519/102642 objects degraded (12.197%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:04 smithi192 ceph-mon[32956]: osdmap e639: 12 total, 12 up, 11 in 2024-06-27T21:37:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:04 smithi192 ceph-mon[32956]: from='client.16857 172.21.15.151:0/4220748595' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9023726381772217}]: dispatch 2024-06-27T21:37:05.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:05.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:05.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:37:05.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:05.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:05.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:05.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:05.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:05.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:05.743 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.5 primary-affinity to 0.902373 (8591372) 2024-06-27T21:37:05.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:05.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:05.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:05 smithi190 ceph-mon[32247]: pgmap v1627: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 12519/102642 objects degraded (12.197%); 7175/102642 objects misplaced (6.990%); 14 MiB/s, 229 objects/s recovering 2024-06-27T21:37:05.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:05 smithi190 ceph-mon[32247]: from='client.16857 172.21.15.151:0/4220748595' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9023726381772217}]': finished 2024-06-27T21:37:05.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:05 smithi190 ceph-mon[32247]: osdmap e640: 12 total, 12 up, 11 in 2024-06-27T21:37:05.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:05.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:05.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:06.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:06.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:05 smithi151 ceph-mon[28965]: pgmap v1627: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 12519/102642 objects degraded (12.197%); 7175/102642 objects misplaced (6.990%); 14 MiB/s, 229 objects/s recovering 2024-06-27T21:37:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:05 smithi151 ceph-mon[28965]: from='client.16857 172.21.15.151:0/4220748595' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9023726381772217}]': finished 2024-06-27T21:37:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:05 smithi151 ceph-mon[28965]: osdmap e640: 12 total, 12 up, 11 in 2024-06-27T21:37:06.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:06.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:05 smithi192 ceph-mon[32956]: pgmap v1627: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 7 op/s; 12519/102642 objects degraded (12.197%); 7175/102642 objects misplaced (6.990%); 14 MiB/s, 229 objects/s recovering 2024-06-27T21:37:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:05 smithi192 ceph-mon[32956]: from='client.16857 172.21.15.151:0/4220748595' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9023726381772217}]': finished 2024-06-27T21:37:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:05 smithi192 ceph-mon[32956]: osdmap e640: 12 total, 12 up, 11 in 2024-06-27T21:37:06.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:06.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:06.407 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:06.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:06.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:06.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:06.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:06.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:06.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:06.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:06.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:06.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:06.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:06.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:07.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:07.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:07.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:07.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:07.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:07.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:07.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:07.488 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:07.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:07.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:07.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:07.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:07.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:07.843 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:37:07.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:07.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:07.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:07.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:08.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:08.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:08.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:07 smithi151 ceph-mon[28965]: pgmap v1630: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 12519/102642 objects degraded (12.197%); 6832/102642 objects misplaced (6.656%); 8.7 MiB/s, 141 objects/s recovering 2024-06-27T21:37:08.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/611677210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:37:08.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:08.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:07 smithi190 ceph-mon[32247]: pgmap v1630: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 12519/102642 objects degraded (12.197%); 6832/102642 objects misplaced (6.656%); 8.7 MiB/s, 141 objects/s recovering 2024-06-27T21:37:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/611677210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:37:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:07 smithi192 ceph-mon[32956]: pgmap v1630: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 0 op/s; 12519/102642 objects degraded (12.197%); 6832/102642 objects misplaced (6.656%); 8.7 MiB/s, 141 objects/s recovering 2024-06-27T21:37:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/611677210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:37:08.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:08.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:08.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:08.486 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:08.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:08.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:08.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:08.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:08.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:08.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:08 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:08.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:08.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:08.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:08.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:08.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:09.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:09.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:09.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/611677210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:37:09.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:08 smithi151 ceph-mon[28965]: osdmap e641: 12 total, 12 up, 11 in 2024-06-27T21:37:09.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:09.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:08 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/611677210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:37:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:08 smithi190 ceph-mon[32247]: osdmap e641: 12 total, 12 up, 11 in 2024-06-27T21:37:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:08 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/611677210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:37:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:08 smithi192 ceph-mon[32956]: osdmap e641: 12 total, 12 up, 11 in 2024-06-27T21:37:09.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:09.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:09.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:09.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:09.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:37:09.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:37:09.552 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:37:09.552 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:37:09.573 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:37:09.573 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:37:09.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:09.593 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:37:09.594 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:37:09.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:09.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:37:09.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:37:09.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:37:09.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:37:09.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:37:09.661 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:37:09.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:37:09.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:37:09.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:09.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:09.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:09.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:09.908 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:37:09.908 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:37:09.930 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:37:09.930 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:37:09.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:09 smithi151 ceph-mon[28965]: pgmap v1632: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 12519/102642 objects degraded (12.197%); 6832/102642 objects misplaced (6.656%); 3.6 MiB/s, 60 objects/s recovering 2024-06-27T21:37:09.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:09 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:37:09.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:09 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:37:09.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:09 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:37:09.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:09 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:37:09.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2070852529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:09.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:37:09.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:37:09.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:09.975 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:37:09.976 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:37:09.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:09.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:10.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:10.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:10.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:10.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:09 smithi190 ceph-mon[32247]: pgmap v1632: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 12519/102642 objects degraded (12.197%); 6832/102642 objects misplaced (6.656%); 3.6 MiB/s, 60 objects/s recovering 2024-06-27T21:37:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:09 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:37:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:09 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:37:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:09 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:37:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:09 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:37:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2070852529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:09 smithi192 ceph-mon[32956]: pgmap v1632: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 12519/102642 objects degraded (12.197%); 6832/102642 objects misplaced (6.656%); 3.6 MiB/s, 60 objects/s recovering 2024-06-27T21:37:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:09 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:37:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:09 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:37:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:09 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:37:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:09 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:37:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2070852529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:10.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:10.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:10.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:10.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:10.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:37:10.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:10.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:10.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:10.728 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:10.783 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:10.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:10.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:10.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:10.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:10.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:11.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:11.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:11.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:11.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:10 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 12519/102564 objects degraded (12.206%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:11.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:10 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 12519/102564 objects degraded (12.206%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:10 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 12519/102564 objects degraded (12.206%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:11.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:11.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:11.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:11.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:11.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:11.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:11.591 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:11.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:11.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:11.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:11.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:11.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:11.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:11.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:11.982 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7, 8] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:37:11.983 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:37:11.983 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-06-27T21:37:11.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:37:12.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:11 smithi190 ceph-mon[32247]: pgmap v1633: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 4 op/s; 12519/102564 objects degraded (12.206%); 4288/102564 objects misplaced (4.181%); 24 MiB/s, 556 objects/s recovering 2024-06-27T21:37:12.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:37:12.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:12.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:12.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:11 smithi151 ceph-mon[28965]: pgmap v1633: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 4 op/s; 12519/102564 objects degraded (12.206%); 4288/102564 objects misplaced (4.181%); 24 MiB/s, 556 objects/s recovering 2024-06-27T21:37:12.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:37:12.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:11 smithi192 ceph-mon[32956]: pgmap v1633: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 4 op/s; 12519/102564 objects degraded (12.206%); 4288/102564 objects misplaced (4.181%); 24 MiB/s, 556 objects/s recovering 2024-06-27T21:37:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:37:12.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:12.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:12.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:12.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:12.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:12.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:12.619 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:12.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:37:12.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:12.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:12.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:12.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:12.920 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:37:12.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:12.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:12.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:13.007 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:12 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:13.007 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:12 smithi151 ceph-mon[28965]: 6.2 scrub ok 2024-06-27T21:37:13.008 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1461105044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:13.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:13.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:13.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:13.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:12 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:12 smithi190 ceph-mon[32247]: 6.2 scrub ok 2024-06-27T21:37:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1461105044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:12 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:12 smithi192 ceph-mon[32956]: 6.2 scrub ok 2024-06-27T21:37:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1461105044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:13.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:13.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:13.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:13.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:13.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:13.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:13.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:13.713 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:13.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:13.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:13.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:13.935 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:37:13.935 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:37:13.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:37:13.935 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:37:13.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.935 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25615.0:163151 5.c e444/382 5:3b58818c:::benchmark_data_smithi192_104167_object57110:head v 444'14197, mlcod=444'14194)", 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:27:45.859743+0000", 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 568.072667787, 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.478047059, 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859743+0000", 2024-06-27T21:37:13.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859743+0000", 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859744+0000", 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:45.859752+0000", 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6149999999999997e-06 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333734+0000", 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.473982142 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333749+0000", 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.505e-05 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333923+0000", 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017347200000000001 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.333988+0000", 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4951999999999997e-05 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.337749+0000", 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037617269999999999 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:27:46.337790+0000", 2024-06-27T21:37:13.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0340999999999999e-05 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(5.1 e462,441 5:8856e036:::benchmark_data_smithi192_104167_object2374:head,418'6731)", 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:52.695520+0000", 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 501.23689013500001, 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.824091307, 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.941 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695520+0000", 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695520+0000", 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695521+0000", 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0200000000000001e-07 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:52.695527+0000", 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5729999999999998e-06 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519007+0000", 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82348054000000004 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519017+0000", 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4229999999999994e-06 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519104+0000", 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7374999999999996e-05 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:53.519612+0000", 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050769400000000005 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(5.1 e463,441 5:81f76185:::benchmark_data_smithi192_104167_object9863:head,419'7187)", 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 497.86244575799998, 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.469556784, 2024-06-27T21:37:13.945 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:37:13.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069965+0000", 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4600000000000005e-07 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.069969+0000", 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5659999999999998e-06 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539033+0000", 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46906430100000002 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539041+0000", 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.858e-06 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539101+0000", 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9648000000000002e-05 2024-06-27T21:37:13.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:56.539522+0000", 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042086499999999998 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(5.1 e463,441 5:83820c48:::benchmark_data_smithi192_104167_object14200:head,419'7457)", 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:28:58.249046+0000", 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 495.68336483399997, 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.656992821, 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.949 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249046+0000", 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249046+0000", 2024-06-27T21:37:13.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249052+0000", 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.46e-06 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.249063+0000", 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.044e-05 2024-06-27T21:37:13.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.905492+0000", 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.65642939600000005 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.905501+0000", 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0850000000000003e-06 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.905598+0000", 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7083000000000005e-05 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:28:58.906039+0000", 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000440357 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:103715 6.6 6:657c8769:::benchmark_data_smithi192_117994_object103714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:55.451659+0000", 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 378.48075174899998, 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37733334299999999, 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 103715 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451659+0000", 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451659+0000", 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451660+0000", 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.378e-06 2024-06-27T21:37:13.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451702+0000", 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1346e-05 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451718+0000", 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6872999999999999e-05 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451726+0000", 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5989999999999996e-06 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.451950+0000", 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022395299999999999 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.452040+0000", 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0006999999999994e-05 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.452167+0000", 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000127155 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.455909+0000", 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037418730000000002 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:37:13.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.455944+0000", 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5447000000000001e-05 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828876+0000", 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37293180799999998 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828944+0000", 2024-06-27T21:37:13.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7915000000000003e-05 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828992+0000", 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7988999999999997e-05 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.26020.0:103715 6.6 e503/467 ondisk, result = 0)", 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:55.456069+0000", 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 378.47634175600001, 2024-06-27T21:37:13.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37293120099999999, 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456069+0000", 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456069+0000", 2024-06-27T21:37:13.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456070+0000", 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1600000000000001e-07 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.456074+0000", 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1239999999999998e-06 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828792+0000", 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37271848800000001 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828800+0000", 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5769999999999996e-06 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828857+0000", 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7281999999999998e-05 2024-06-27T21:37:13.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.828868+0000", 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1195e-05 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.829000+0000", 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013181900000000001 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.965 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:153161 6.6 6:6be1abb8:::benchmark_data_smithi192_117994_object45567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:42.313083+0000", 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 271.61932750699998, 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34669207800000001, 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:37:13.966 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 153161 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313083+0000", 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313083+0000", 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313084+0000", 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1349999999999999e-06 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313090+0000", 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.66e-06 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313100+0000", 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6819999999999998e-06 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313105+0000", 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1900000000000003e-06 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313155+0000", 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0491999999999997e-05 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313260+0000", 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0001052 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.313352+0000", 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1137999999999994e-05 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.317409+0000", 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004057326 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.317494+0000", 2024-06-27T21:37:13.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5440999999999997e-05 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.657992+0000", 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34049808999999998 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.658060+0000", 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7937999999999999e-05 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:42.659775+0000", 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001714786 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.6 532/515 [PushReplyOp(6:6854d079:::benchmark_data_smithi192_117994_object75012:head)])", 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 270.56077593800001, 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.60063333399999996, 2024-06-27T21:37:13.973 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3799999999999998e-07 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371638+0000", 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.326e-06 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.969696+0000", 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.59805774700000003 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.969702+0000", 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7169999999999996e-06 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.969760+0000", 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8294999999999997e-05 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.972268+0000", 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002507811 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.6 533/515 [PushReplyOp(6:64c151bb:::benchmark_data_smithi192_117994_object75565:head)])", 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:44.654815+0000", 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 269.27759583599999, 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36155716799999998, 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.977 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.654815+0000", 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.654815+0000", 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.654816+0000", 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4449999999999999e-06 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:44.654821+0000", 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5950000000000001e-06 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:45.013770+0000", 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35894970999999998 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:45.013779+0000", 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3629999999999994e-06 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:45.013842+0000", 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3425999999999999e-05 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:45.016372+0000", 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025296289999999998 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.6 533/515 [PushReplyOp(6:6292732d:::benchmark_data_smithi192_117994_object82946:head)])", 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:51.047303+0000", 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 262.88510718200001, 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42797745300000001, 2024-06-27T21:37:13.981 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047303+0000", 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047303+0000", 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047304+0000", 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047309+0000", 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7899999999999999e-06 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.473245+0000", 2024-06-27T21:37:13.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42593658000000001 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.473256+0000", 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0553999999999999e-05 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.473309+0000", 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3248999999999998e-05 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.475281+0000", 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001971549 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e540,515 6:6fcfe693:::benchmark_data_smithi192_117994_object14778:head,510'7668)", 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:14.560235+0000", 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 239.37217533, 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.59406511399999995, 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.985 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560235+0000", 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560235+0000", 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560236+0000", 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6100000000000001e-07 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560240+0000", 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0929999999999996e-06 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.153309+0000", 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.593068924 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.153316+0000", 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.678e-06 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.153975+0000", 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065804500000000001 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.154300+0000", 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032581299999999999 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:156223 6.e e540/467 6:74cd8b3d:::benchmark_data_smithi192_117994_object48629:head v 540'9858, mlcod=540'9856)", 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:15.195227+0000", 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 238.73718319299999, 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43734508599999999, 2024-06-27T21:37:13.989 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195227+0000", 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195227+0000", 2024-06-27T21:37:13.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195228+0000", 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0419999999999999e-06 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195237+0000", 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6179999999999995e-06 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628577+0000", 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43333962999999998 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628584+0000", 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4240000000000004e-06 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628646+0000", 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2119999999999995e-05 2024-06-27T21:37:13.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628671+0000", 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5228e-05 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.632531+0000", 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038593740000000001 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.632572+0000", 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1650000000000003e-05 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e540,515 6:6157eff0:::benchmark_data_smithi192_117994_object18148:head,511'7888)", 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:17.212981+0000", 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 236.71942980399999, 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49524637900000001, 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.994 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212981+0000", 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212981+0000", 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212982+0000", 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8000000000000005e-07 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212987+0000", 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1019999999999996e-06 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.707161+0000", 2024-06-27T21:37:13.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49417447399999997 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.707170+0000", 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2019999999999993e-06 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.707758+0000", 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058814200000000005 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:13.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.708227+0000", 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000468679 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e540,515 6:6fc38716:::benchmark_data_smithi192_117994_object18210:head,511'7892)", 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:18.026250+0000", 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 235.90616009799999, 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37025059999999999, 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:13.998 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026250+0000", 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026250+0000", 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026251+0000", 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7400000000000002e-07 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026258+0000", 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6479999999999997e-06 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396215+0000", 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36995685299999997 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396223+0000", 2024-06-27T21:37:14.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1159999999999999e-06 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396271+0000", 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8344999999999999e-05 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396501+0000", 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000229764 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.16 551/550 [PushReplyOp(6:6955813a:::benchmark_data_smithi192_117994_object59286:head)])", 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:33.187949+0000", 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.74446131400001, 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.447962637, 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187949+0000", 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187949+0000", 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187951+0000", 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3960000000000001e-06 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187956+0000", 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3580000000000002e-06 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.633549+0000", 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44559270699999998 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.633554+0000", 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2039999999999996e-06 2024-06-27T21:37:14.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.633613+0000", 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8850000000000001e-05 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.635912+0000", 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022991219999999998 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.16 551/550 [PushReplyOp(6:6ba9cbd0:::benchmark_data_smithi192_117994_object60030:head)])", 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:34.122133+0000", 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 219.810278006, 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58028879200000005, 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122133+0000", 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122133+0000", 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122134+0000", 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1289999999999999e-06 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122138+0000", 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6829999999999999e-06 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.700499+0000", 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.57836112900000003 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.700512+0000", 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2306999999999999e-05 2024-06-27T21:37:14.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.700595+0000", 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3567999999999997e-05 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.702421+0000", 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001825976 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "replica_scrub(pg: 6.16,from:0'0,to:560'9928,epoch:562/561,start:6:6d94347c:::benchmark_data_smithi192_117994_object81448:0,end:6:6e05da67:::benchmark_data_smithi192_117994_object101194:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:06.073868+0000", 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 187.858542403, 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44751875699999999, 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073868+0000", 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073868+0000", 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073869+0000", 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073875+0000", 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1979999999999998e-06 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521287+0000", 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44741202499999999 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521298+0000", 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0920999999999999e-05 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521349+0000", 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1128999999999998e-05 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521358+0000", 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5820000000000002e-06 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521387+0000", 2024-06-27T21:37:14.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9125999999999999e-05 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:161547 6.16 e562/561 6:6fc2e18a:::benchmark_data_smithi192_117994_object53953:head v 562'9945, mlcod=562'9943)", 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:06.982216+0000", 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 186.950194138, 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58225666300000001, 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:37:14.016 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982216+0000", 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982216+0000", 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982217+0000", 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1999999999999998e-07 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982225+0000", 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7689999999999996e-06 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562719+0000", 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58049409299999999 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562728+0000", 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6270000000000006e-06 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562780+0000", 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1952e-05 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562797+0000", 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7552000000000001e-05 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.564443+0000", 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001645229 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.564473+0000", 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0521000000000001e-05 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:172635 6.12 e602/549 6:4a785fe0:::benchmark_data_smithi192_117994_object65041:head v 602'10089, mlcod=601'10086)", 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:51.028940+0000", 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 82.903470103000004, 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.344152707, 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:37:14.023 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.028940+0000", 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.028940+0000", 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.028941+0000", 2024-06-27T21:37:14.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4300000000000003e-07 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.028947+0000", 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.446e-06 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.371535+0000", 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.342587998 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.371541+0000", 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5749999999999997e-06 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.371597+0000", 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5479999999999997e-05 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.371615+0000", 2024-06-27T21:37:14.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8734000000000001e-05 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.373049+0000", 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014334980000000001 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:51.373093+0000", 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4332999999999997e-05 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.15 636/550 [PushOp(6:adc7d4ea:::benchmark_data_smithi192_117994_object88705:head, version: 496'5501, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:adc7d4ea:::benchmark_data_smithi192_117994_object88705:head@496'5501, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.858006629, 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.545111978, 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:37:14.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074406+0000", 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7090000000000001e-06 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074452+0000", 2024-06-27T21:37:14.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5967999999999998e-05 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617875+0000", 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54342337500000004 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617887+0000", 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.236e-05 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:37:14.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617955+0000", 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7934000000000004e-05 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617964+0000", 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8130000000000002e-06 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.619516+0000", 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015518190000000001 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.031 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.032 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:37:14.032 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:37:14.032 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:37:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:14.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:14.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:14.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:13 smithi151 ceph-mon[28965]: pgmap v1634: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 11 op/s; 12519/102390 objects degraded (12.227%); 4288/102390 objects misplaced (4.188%); 19 MiB/s, 512 objects/s recovering 2024-06-27T21:37:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:13 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:13 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:13 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:37:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1461105044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:37:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:13 smithi151 ceph-mon[28965]: osdmap e642: 12 total, 12 up, 11 in 2024-06-27T21:37:14.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:13 smithi190 ceph-mon[32247]: pgmap v1634: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 11 op/s; 12519/102390 objects degraded (12.227%); 4288/102390 objects misplaced (4.188%); 19 MiB/s, 512 objects/s recovering 2024-06-27T21:37:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:13 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:13 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:13 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:37:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1461105044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:37:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:13 smithi190 ceph-mon[32247]: osdmap e642: 12 total, 12 up, 11 in 2024-06-27T21:37:14.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:13 smithi192 ceph-mon[32956]: pgmap v1634: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 11 op/s; 12519/102390 objects degraded (12.227%); 4288/102390 objects misplaced (4.188%); 19 MiB/s, 512 objects/s recovering 2024-06-27T21:37:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:13 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:13 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:13 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:37:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1461105044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:37:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:13 smithi192 ceph-mon[32956]: osdmap e642: 12 total, 12 up, 11 in 2024-06-27T21:37:14.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:14.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:14.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:14.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:14.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:14.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:14.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:14.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:14.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:14.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:14.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:15.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:15.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:15.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:15.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:15.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:15.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:15.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:15.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:15.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:15.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:15.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:15.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:15.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:15.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:15.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:15.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:15.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:15.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:16.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:15 smithi190 ceph-mon[32247]: pgmap v1636: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 12406/101058 objects degraded (12.276%); 4113/101058 objects misplaced (4.070%); 16 MiB/s, 479 objects/s recovering 2024-06-27T21:37:16.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:15 smithi190 ceph-mon[32247]: osdmap e643: 12 total, 12 up, 11 in 2024-06-27T21:37:16.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:16.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:15 smithi192 ceph-mon[32956]: pgmap v1636: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 12406/101058 objects degraded (12.276%); 4113/101058 objects misplaced (4.070%); 16 MiB/s, 479 objects/s recovering 2024-06-27T21:37:16.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:15 smithi192 ceph-mon[32956]: osdmap e643: 12 total, 12 up, 11 in 2024-06-27T21:37:16.079 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:37:16.080 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:15 smithi151 ceph-mon[28965]: pgmap v1636: 57 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 65 op/s; 12406/101058 objects degraded (12.276%); 4113/101058 objects misplaced (4.070%); 16 MiB/s, 479 objects/s recovering 2024-06-27T21:37:16.080 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:15 smithi151 ceph-mon[28965]: osdmap e643: 12 total, 12 up, 11 in 2024-06-27T21:37:16.081 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:37:16.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:16.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:16.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:37:16.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:16.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:16.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:16.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:16.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:16.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:16.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:16.645 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:16.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:16.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:16.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:16 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 12406/99816 objects degraded (12.429%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:16.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:16.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:16.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:16.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:37:16.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:17.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:17.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:17.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:17.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:16 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 12406/99816 objects degraded (12.429%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:17.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:16 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 12406/99816 objects degraded (12.429%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:17.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:17.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:17.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:17.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:17.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:17.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:17.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:17.768 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:17.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:17.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:17.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:17.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:17 smithi151 ceph-mon[28965]: from='client.16881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:37:17.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:17 smithi151 ceph-mon[28965]: pgmap v1638: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 117 op/s; 12406/99816 objects degraded (12.429%); 4113/99816 objects misplaced (4.121%); 16 MiB/s, 569 objects/s recovering 2024-06-27T21:37:17.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:18.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:18.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:18.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:18.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:17 smithi190 ceph-mon[32247]: from='client.16881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:37:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:17 smithi190 ceph-mon[32247]: pgmap v1638: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 117 op/s; 12406/99816 objects degraded (12.429%); 4113/99816 objects misplaced (4.121%); 16 MiB/s, 569 objects/s recovering 2024-06-27T21:37:18.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:17 smithi192 ceph-mon[32956]: from='client.16881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:37:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:17 smithi192 ceph-mon[32956]: pgmap v1638: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 117 op/s; 12406/99816 objects degraded (12.429%); 4113/99816 objects misplaced (4.121%); 16 MiB/s, 569 objects/s recovering 2024-06-27T21:37:18.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:18.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:18.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:18.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:18.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:18.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:18.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:18.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:18.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:18.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:18.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:18.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:19.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:19.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:19.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:19.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:19.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:19.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:19.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:19.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:19.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:19.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:19.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:19.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:37:19.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:37:19.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:37:19.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:37:19.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:37:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:37:19.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:37:19.660 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:37:19.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:19.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:19.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:19.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:19.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:19.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:19 smithi151 ceph-mon[28965]: pgmap v1639: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 12406/99471 objects degraded (12.472%); 4074/99471 objects misplaced (4.096%); 656 KiB/s, 197 objects/s recovering 2024-06-27T21:37:19.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2739291301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:19.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:20.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:20.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:20.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:20.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:20.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:19 smithi190 ceph-mon[32247]: pgmap v1639: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 12406/99471 objects degraded (12.472%); 4074/99471 objects misplaced (4.096%); 656 KiB/s, 197 objects/s recovering 2024-06-27T21:37:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2739291301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:19 smithi192 ceph-mon[32956]: pgmap v1639: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 128 op/s; 12406/99471 objects degraded (12.472%); 4074/99471 objects misplaced (4.096%); 656 KiB/s, 197 objects/s recovering 2024-06-27T21:37:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2739291301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:20.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:20.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:20.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:20.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:20.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:20.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:20.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:20.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:20.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:20.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:20.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:20.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:21.003 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:21.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:21.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:21.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:21.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:21.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:21.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:21.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:21.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:21.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:21.647 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-06-27T21:37:14.744503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2207973944,"num_objects":33686,"num_object_clones":0,"num_object_copies":101058,"num_objects_missing_on_primary":3508,"num_objects_missing":3508,"num_objects_degraded":12406,"num_objects_misplaced":4113,"num_objects_unfound":0,"num_objects_dirty":33686,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95121,"num_write_kb":3243713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44423,"num_bytes_recovered":2275480144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275850,"ondisk_log_size":275850,"up":171,"acting":166,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":12746700,"kb_used_data":7705056,"kb_used_omap":108,"kb_used_meta":5041427,"kb_avail":1018449972,"statfs":{"total":1055945392128,"available":1042892771328,"internally_reserved":0,"allocated":7889977344,"data_stored":7881937153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111086,"internal_metadata":5162421778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,0,0,0,0,0,0,16],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-34603008,"num_objects":-528,"num_object_clones":0,"num_object_copies":-1584,"num_objects_missing_on_primary":-62,"num_objects_missing":-62,"num_objects_degraded":-113,"num_objects_misplaced":-2719,"num_objects_unfound":0,"num_objects_dirty":-528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3839,"num_bytes_recovered":132055040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.901298"},"pg_stats":[{"pgid":"6.18","version":"641'10635","reported_seq":27836,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:10.247934+0000","last_change":"2024-06-27T21:36:19.056568+0000","last_active":"2024-06-27T21:37:10.247934+0000","last_peered":"2024-06-27T21:37:10.247934+0000","last_clean":"2024-06-27T21:37:10.247934+0000","last_became_active":"2024-06-27T21:36:19.037779+0000","last_became_peered":"2024-06-27T21:36:19.037779+0000","last_unstale":"2024-06-27T21:37:10.247934+0000","last_undegraded":"2024-06-27T21:37:10.247934+0000","last_fullsized":"2024-06-27T21:37:10.247934+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10353","last_scrub_stamp":"2024-06-27T21:35:42.524197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.524197+0000","objects_scrubbed":1327,"log_size":10635,"log_dups_size":0,"ondisk_log_size":10635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68485120,"num_objects":1045,"num_object_clones":0,"num_object_copies":3135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3341,"num_bytes_recovered":85590016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"641'10517","reported_seq":25357,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:09.140775+0000","last_change":"2024-06-27T21:36:23.070309+0000","last_active":"2024-06-27T21:37:09.140775+0000","last_peered":"2024-06-27T21:37:09.140775+0000","last_clean":"2024-06-27T21:37:09.140775+0000","last_became_active":"2024-06-27T21:36:23.050064+0000","last_became_peered":"2024-06-27T21:36:23.050064+0000","last_unstale":"2024-06-27T21:37:09.140775+0000","last_undegraded":"2024-06-27T21:37:09.140775+0000","last_fullsized":"2024-06-27T21:37:09.140775+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10195","last_scrub_stamp":"2024-06-27T21:35:42.766433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.766433+0000","objects_scrubbed":1384,"log_size":10517,"log_dups_size":0,"ondisk_log_size":10517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69599232,"num_objects":1062,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":322,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27000832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":234,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.403581+0000","last_change":"2024-06-27T21:36:51.809004+0000","last_active":"2024-06-27T21:37:08.403581+0000","last_peered":"2024-06-27T21:37:08.403581+0000","last_clean":"2024-06-27T21:37:08.403581+0000","last_became_active":"2024-06-27T21:36:51.808540+0000","last_became_peered":"2024-06-27T21:36:51.808540+0000","last_unstale":"2024-06-27T21:37:08.403581+0000","last_undegraded":"2024-06-27T21:37:08.403581+0000","last_fullsized":"2024-06-27T21:37:08.403581+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":233,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.403564+0000","last_change":"2024-06-27T21:35:46.856033+0000","last_active":"2024-06-27T21:37:08.403564+0000","last_peered":"2024-06-27T21:37:08.403564+0000","last_clean":"2024-06-27T21:37:08.403564+0000","last_became_active":"2024-06-27T21:35:46.855054+0000","last_became_peered":"2024-06-27T21:35:46.855054+0000","last_unstale":"2024-06-27T21:37:08.403564+0000","last_undegraded":"2024-06-27T21:37:08.403564+0000","last_fullsized":"2024-06-27T21:37:08.403564+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,10],"acting":[11,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":27,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.856616+0000","last_change":"2024-06-27T21:36:51.806395+0000","last_active":"2024-06-27T21:37:07.856616+0000","last_peered":"2024-06-27T21:37:07.856616+0000","last_clean":"2024-06-27T21:37:07.856616+0000","last_became_active":"2024-06-27T21:36:51.805865+0000","last_became_peered":"2024-06-27T21:36:51.805865+0000","last_unstale":"2024-06-27T21:37:07.856616+0000","last_undegraded":"2024-06-27T21:37:07.856616+0000","last_fullsized":"2024-06-27T21:37:07.856616+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:52.993157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:52.993157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,11],"acting":[8,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":27,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.856494+0000","last_change":"2024-06-27T21:36:51.803807+0000","last_active":"2024-06-27T21:37:07.856494+0000","last_peered":"2024-06-27T21:37:07.856494+0000","last_clean":"2024-06-27T21:37:07.856494+0000","last_became_active":"2024-06-27T21:36:51.803452+0000","last_became_peered":"2024-06-27T21:36:51.803452+0000","last_unstale":"2024-06-27T21:37:07.856494+0000","last_undegraded":"2024-06-27T21:37:07.856494+0000","last_fullsized":"2024-06-27T21:37:07.856494+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":303,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.919670+0000","last_change":"2024-06-27T21:35:46.858389+0000","last_active":"2024-06-27T21:37:12.919670+0000","last_peered":"2024-06-27T21:37:12.919670+0000","last_clean":"2024-06-27T21:37:12.919670+0000","last_became_active":"2024-06-27T21:35:46.858050+0000","last_became_peered":"2024-06-27T21:35:46.858050+0000","last_unstale":"2024-06-27T21:37:12.919670+0000","last_undegraded":"2024-06-27T21:37:12.919670+0000","last_fullsized":"2024-06-27T21:37:12.919670+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":251,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.844110+0000","last_change":"2024-06-27T21:33:50.003594+0000","last_active":"2024-06-27T21:37:07.844110+0000","last_peered":"2024-06-27T21:37:07.844110+0000","last_clean":"2024-06-27T21:37:07.844110+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:37:07.844110+0000","last_undegraded":"2024-06-27T21:37:07.844110+0000","last_fullsized":"2024-06-27T21:37:07.844110+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:50.003522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.003522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":691,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.923362+0000","last_change":"2024-06-27T21:36:42.000057+0000","last_active":"2024-06-27T21:37:12.923362+0000","last_peered":"2024-06-27T21:37:12.923362+0000","last_clean":"2024-06-27T21:37:12.923362+0000","last_became_active":"2024-06-27T21:35:46.859076+0000","last_became_peered":"2024-06-27T21:35:46.859076+0000","last_unstale":"2024-06-27T21:37:12.923362+0000","last_undegraded":"2024-06-27T21:37:12.923362+0000","last_fullsized":"2024-06-27T21:37:12.923362+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:41.999953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:41.999953+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:45.149689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,11],"acting":[2,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"641'12180","reported_seq":42976,"reported_epoch":641,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:37:10.183123+0000","last_change":"2024-06-27T21:36:53.037460+0000","last_active":"2024-06-27T21:37:10.183123+0000","last_peered":"2024-06-27T21:37:10.183123+0000","last_clean":"2024-06-27T21:36:50.727975+0000","last_became_active":"2024-06-27T21:36:53.012253+0000","last_became_peered":"2024-06-27T21:36:53.012253+0000","last_unstale":"2024-06-27T21:37:10.183123+0000","last_undegraded":"2024-06-27T21:36:52.832282+0000","last_fullsized":"2024-06-27T21:36:52.805187+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":5,"last_scrub":"574'11661","last_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_deep_scrub":"574'11661","last_deep_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_clean_scrub_stamp":"2024-06-27T21:34:46.137011+0000","objects_scrubbed":1586,"log_size":12180,"log_dups_size":0,"ondisk_log_size":12180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"no scrub is scheduled","scrub_duration":10216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68681728,"num_objects":1048,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":538,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3365,"num_bytes_recovered":79429632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,3],"avail_no_missing":["0","3","9"],"object_location_counts":[{"shards":"0,3","objects":-10},{"shards":"0,3,9","objects":1048}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":44,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.639479+0000","last_change":"2024-06-27T21:37:08.639499+0000","last_active":"2024-06-27T21:37:08.639479+0000","last_peered":"2024-06-27T21:37:08.639479+0000","last_clean":"2024-06-27T21:37:08.639479+0000","last_became_active":"2024-06-27T21:36:51.808305+0000","last_became_peered":"2024-06-27T21:36:51.808305+0000","last_unstale":"2024-06-27T21:37:08.639479+0000","last_undegraded":"2024-06-27T21:37:08.639479+0000","last_fullsized":"2024-06-27T21:37:08.639479+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:37:08.639432+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:37:08.639432+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:12.523411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.11","version":"641'10247","reported_seq":30361,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:10.063039+0000","last_change":"2024-06-27T21:34:22.233121+0000","last_active":"2024-06-27T21:37:10.063039+0000","last_peered":"2024-06-27T21:37:10.063039+0000","last_clean":"2024-06-27T21:37:10.063039+0000","last_became_active":"2024-06-27T21:34:21.788051+0000","last_became_peered":"2024-06-27T21:34:21.788051+0000","last_unstale":"2024-06-27T21:37:10.063039+0000","last_undegraded":"2024-06-27T21:37:10.063039+0000","last_fullsized":"2024-06-27T21:37:10.063039+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"569'9717","last_scrub_stamp":"2024-06-27T21:34:22.232987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:22.232987+0000","objects_scrubbed":1552,"log_size":10247,"log_dups_size":0,"ondisk_log_size":10247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66912256,"num_objects":1021,"num_object_clones":0,"num_object_copies":3063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":105,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.919563+0000","last_change":"2024-06-27T21:35:47.430217+0000","last_active":"2024-06-27T21:37:07.919563+0000","last_peered":"2024-06-27T21:37:07.919563+0000","last_clean":"2024-06-27T21:37:07.919563+0000","last_became_active":"2024-06-27T21:35:46.855888+0000","last_became_peered":"2024-06-27T21:35:46.855888+0000","last_unstale":"2024-06-27T21:37:07.919563+0000","last_undegraded":"2024-06-27T21:37:07.919563+0000","last_fullsized":"2024-06-27T21:37:07.919563+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.430103+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.430103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"641'10346","reported_seq":10831,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:09.053786+0000","last_change":"2024-06-27T21:33:51.180577+0000","last_active":"2024-06-27T21:37:09.053786+0000","last_peered":"2024-06-27T21:37:09.053786+0000","last_clean":"2024-06-27T21:37:09.053786+0000","last_became_active":"2024-06-27T21:33:27.698780+0000","last_became_peered":"2024-06-27T21:33:27.698780+0000","last_unstale":"2024-06-27T21:37:09.053786+0000","last_undegraded":"2024-06-27T21:37:09.053786+0000","last_fullsized":"2024-06-27T21:37:09.053786+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'9712","last_scrub_stamp":"2024-06-27T21:33:51.180247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.180247+0000","objects_scrubbed":1680,"log_size":10346,"log_dups_size":0,"ondisk_log_size":10346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68550656,"num_objects":1046,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":634,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2],"acting":[11,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":241,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.921315+0000","last_change":"2024-06-27T21:36:51.821200+0000","last_active":"2024-06-27T21:37:12.921315+0000","last_peered":"2024-06-27T21:37:12.921315+0000","last_clean":"2024-06-27T21:37:12.921315+0000","last_became_active":"2024-06-27T21:36:51.820468+0000","last_became_peered":"2024-06-27T21:36:51.820468+0000","last_unstale":"2024-06-27T21:37:12.921315+0000","last_undegraded":"2024-06-27T21:37:12.921315+0000","last_fullsized":"2024-06-27T21:37:12.921315+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"641'10762","reported_seq":27474,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:09.653422+0000","last_change":"2024-06-27T21:35:52.894201+0000","last_active":"2024-06-27T21:37:09.653422+0000","last_peered":"2024-06-27T21:37:09.653422+0000","last_clean":"2024-06-27T21:37:09.653422+0000","last_became_active":"2024-06-27T21:35:52.878734+0000","last_became_peered":"2024-06-27T21:35:52.878734+0000","last_unstale":"2024-06-27T21:37:09.653422+0000","last_undegraded":"2024-06-27T21:37:09.653422+0000","last_fullsized":"2024-06-27T21:37:09.653422+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"566'10180","last_scrub_stamp":"2024-06-27T21:34:19.641768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:19.641768+0000","objects_scrubbed":1673,"log_size":10762,"log_dups_size":0,"ondisk_log_size":10762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71237632,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":586,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":369,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.923834+0000","last_change":"2024-06-27T21:36:42.332358+0000","last_active":"2024-06-27T21:37:12.923834+0000","last_peered":"2024-06-27T21:37:12.923834+0000","last_clean":"2024-06-27T21:37:12.923834+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:37:12.923834+0000","last_undegraded":"2024-06-27T21:37:12.923834+0000","last_fullsized":"2024-06-27T21:37:12.923834+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:42.332234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:42.332234+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:44.683852+0000","scrub_duration":1328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"642'10901","reported_seq":11503,"reported_epoch":642,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:37:13.433578+0000","last_change":"2024-06-27T21:36:47.048801+0000","last_active":"2024-06-27T21:37:13.433578+0000","last_peered":"2024-06-27T21:37:13.433578+0000","last_clean":"2024-06-27T21:33:25.735988+0000","last_became_active":"2024-06-27T21:33:28.958742+0000","last_became_peered":"2024-06-27T21:33:28.958742+0000","last_unstale":"2024-06-27T21:37:13.433578+0000","last_undegraded":"2024-06-27T21:37:13.433578+0000","last_fullsized":"2024-06-27T21:37:13.433578+0000","mapping_epoch":550,"log_start":"470'100","ondisk_log_start":"470'100","created":547,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":6198,"log_size":10801,"log_dups_size":0,"ondisk_log_size":10801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68714496,"num_objects":1049,"num_object_clones":0,"num_object_copies":3147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3075,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":220704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2228224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[1,5,9],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5,9","objects":1049}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":365,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.403462+0000","last_change":"2024-06-27T21:35:46.853442+0000","last_active":"2024-06-27T21:37:08.403462+0000","last_peered":"2024-06-27T21:37:08.403462+0000","last_clean":"2024-06-27T21:37:08.403462+0000","last_became_active":"2024-06-27T21:35:46.853087+0000","last_became_peered":"2024-06-27T21:35:46.853087+0000","last_unstale":"2024-06-27T21:37:08.403462+0000","last_undegraded":"2024-06-27T21:37:08.403462+0000","last_fullsized":"2024-06-27T21:37:08.403462+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.15","version":"642'10463","reported_seq":22693,"reported_epoch":642,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:37:13.377577+0000","last_change":"2024-06-27T21:37:13.377723+0000","last_active":"2024-06-27T21:37:13.377577+0000","last_peered":"2024-06-27T21:37:13.377577+0000","last_clean":"2024-06-27T21:31:21.926059+0000","last_became_active":"2024-06-27T21:33:28.917741+0000","last_became_peered":"2024-06-27T21:33:28.917741+0000","last_unstale":"2024-06-27T21:37:13.377577+0000","last_undegraded":"2024-06-27T21:37:13.377577+0000","last_fullsized":"2024-06-27T21:33:28.693311+0000","mapping_epoch":550,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":5975,"log_size":10463,"log_dups_size":0,"ondisk_log_size":10463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67010560,"num_objects":1023,"num_object_clones":0,"num_object_copies":3069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":218038272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,10],"acting":[7,4],"avail_no_missing":["7","6","9"],"object_location_counts":[{"shards":"4,7","objects":1023}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":391,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.963415+0000","last_change":"2024-06-27T21:36:06.824164+0000","last_active":"2024-06-27T21:37:12.963415+0000","last_peered":"2024-06-27T21:37:12.963415+0000","last_clean":"2024-06-27T21:37:12.963415+0000","last_became_active":"2024-06-27T21:35:46.862310+0000","last_became_peered":"2024-06-27T21:35:46.862310+0000","last_unstale":"2024-06-27T21:37:12.963415+0000","last_undegraded":"2024-06-27T21:37:12.963415+0000","last_fullsized":"2024-06-27T21:37:12.963415+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:06.824082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:06.824082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:31.578769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"641'10556","reported_seq":31599,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:10.092872+0000","last_change":"2024-06-27T21:34:06.592362+0000","last_active":"2024-06-27T21:37:10.092872+0000","last_peered":"2024-06-27T21:37:10.092872+0000","last_clean":"2024-06-27T21:37:10.092872+0000","last_became_active":"2024-06-27T21:34:05.705454+0000","last_became_peered":"2024-06-27T21:34:05.705454+0000","last_unstale":"2024-06-27T21:37:10.092872+0000","last_undegraded":"2024-06-27T21:37:10.092872+0000","last_fullsized":"2024-06-27T21:37:10.092872+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":5,"last_scrub":"562'9943","last_scrub_stamp":"2024-06-27T21:34:06.592264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:06.592264+0000","objects_scrubbed":1716,"log_size":10556,"log_dups_size":0,"ondisk_log_size":10556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":830,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72155136,"num_objects":1101,"num_object_clones":0,"num_object_copies":3303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":615,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":379,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.919422+0000","last_change":"2024-06-27T21:36:51.811442+0000","last_active":"2024-06-27T21:37:07.919422+0000","last_peered":"2024-06-27T21:37:07.919422+0000","last_clean":"2024-06-27T21:37:07.919422+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:37:07.919422+0000","last_undegraded":"2024-06-27T21:37:07.919422+0000","last_fullsized":"2024-06-27T21:37:07.919422+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"641'10439","reported_seq":24325,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.951131+0000","last_change":"2024-06-27T21:35:41.151637+0000","last_active":"2024-06-27T21:37:08.951131+0000","last_peered":"2024-06-27T21:37:08.951131+0000","last_clean":"2024-06-27T21:37:08.951131+0000","last_became_active":"2024-06-27T21:35:01.964347+0000","last_became_peered":"2024-06-27T21:35:01.964347+0000","last_unstale":"2024-06-27T21:37:08.951131+0000","last_undegraded":"2024-06-27T21:37:08.951131+0000","last_fullsized":"2024-06-27T21:37:08.951131+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10152","last_scrub_stamp":"2024-06-27T21:35:41.151550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:41.151550+0000","objects_scrubbed":1305,"log_size":10439,"log_dups_size":0,"ondisk_log_size":10439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66715648,"num_objects":1018,"num_object_clones":0,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":396,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.920766+0000","last_change":"2024-06-27T21:36:51.811300+0000","last_active":"2024-06-27T21:37:12.920766+0000","last_peered":"2024-06-27T21:37:12.920766+0000","last_clean":"2024-06-27T21:37:12.920766+0000","last_became_active":"2024-06-27T21:36:51.810932+0000","last_became_peered":"2024-06-27T21:36:51.810932+0000","last_unstale":"2024-06-27T21:37:12.920766+0000","last_undegraded":"2024-06-27T21:37:12.920766+0000","last_fullsized":"2024-06-27T21:37:12.920766+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":677,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.923492+0000","last_change":"2024-06-27T21:37:09.129218+0000","last_active":"2024-06-27T21:37:12.923492+0000","last_peered":"2024-06-27T21:37:12.923492+0000","last_clean":"2024-06-27T21:37:12.923492+0000","last_became_active":"2024-06-27T21:36:51.804872+0000","last_became_peered":"2024-06-27T21:36:51.804872+0000","last_unstale":"2024-06-27T21:37:12.923492+0000","last_undegraded":"2024-06-27T21:37:12.923492+0000","last_fullsized":"2024-06-27T21:37:12.923492+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:37:09.129137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:37:09.129137+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:27.953864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,4],"acting":[2,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"642'11476","reported_seq":11946,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.923968+0000","last_change":"2024-06-27T21:36:37.373142+0000","last_active":"2024-06-27T21:37:12.923968+0000","last_peered":"2024-06-27T21:37:12.923968+0000","last_clean":"2024-06-27T21:37:12.923968+0000","last_became_active":"2024-06-27T21:29:19.890509+0000","last_became_peered":"2024-06-27T21:29:19.890509+0000","last_unstale":"2024-06-27T21:37:12.923968+0000","last_undegraded":"2024-06-27T21:37:12.923968+0000","last_fullsized":"2024-06-27T21:37:12.923968+0000","mapping_epoch":467,"log_start":"475'1400","ondisk_log_start":"475'1400","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'11283","last_scrub_stamp":"2024-06-27T21:36:37.373008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:37.373008+0000","objects_scrubbed":2292,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:51.597422+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11476,"num_write_kb":434304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":451,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.921274+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:37:12.921274+0000","last_peered":"2024-06-27T21:37:12.921274+0000","last_clean":"2024-06-27T21:37:12.921274+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:37:12.921274+0000","last_undegraded":"2024-06-27T21:37:12.921274+0000","last_fullsized":"2024-06-27T21:37:12.921274+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"642'11467","reported_seq":17216,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:13.182242+0000","last_change":"2024-06-27T21:36:36.394240+0000","last_active":"2024-06-27T21:37:13.182242+0000","last_peered":"2024-06-27T21:37:13.182242+0000","last_clean":"2024-06-27T21:37:13.182242+0000","last_became_active":"2024-06-27T21:36:07.970524+0000","last_became_peered":"2024-06-27T21:36:07.970524+0000","last_unstale":"2024-06-27T21:37:13.182242+0000","last_undegraded":"2024-06-27T21:37:13.182242+0000","last_fullsized":"2024-06-27T21:37:13.182242+0000","mapping_epoch":612,"log_start":"475'1400","ondisk_log_start":"475'1400","created":467,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"628'11257","last_scrub_stamp":"2024-06-27T21:36:36.394150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:36.394150+0000","objects_scrubbed":2300,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:00.155595+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136904704,"num_objects":2089,"num_object_clones":0,"num_object_copies":6267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11467,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2633,"num_bytes_recovered":172556288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":293,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.919813+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:37:12.919813+0000","last_peered":"2024-06-27T21:37:12.919813+0000","last_clean":"2024-06-27T21:37:12.919813+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:37:12.919813+0000","last_undegraded":"2024-06-27T21:37:12.919813+0000","last_fullsized":"2024-06-27T21:37:12.919813+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"641'11330","reported_seq":30663,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:09.663349+0000","last_change":"2024-06-27T21:36:08.329099+0000","last_active":"2024-06-27T21:37:09.663349+0000","last_peered":"2024-06-27T21:37:09.663349+0000","last_clean":"2024-06-27T21:37:09.663349+0000","last_became_active":"2024-06-27T21:36:08.302687+0000","last_became_peered":"2024-06-27T21:36:08.302687+0000","last_unstale":"2024-06-27T21:37:09.663349+0000","last_undegraded":"2024-06-27T21:37:09.663349+0000","last_fullsized":"2024-06-27T21:37:09.663349+0000","mapping_epoch":612,"log_start":"475'1300","ondisk_log_start":"475'1300","created":467,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142409752,"num_objects":2174,"num_object_clones":0,"num_object_copies":6522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11330,"num_write_kb":432065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9434,"num_bytes_recovered":509018160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1215,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.403525+0000","last_change":"2024-06-27T21:36:51.809130+0000","last_active":"2024-06-27T21:37:08.403525+0000","last_peered":"2024-06-27T21:37:08.403525+0000","last_clean":"2024-06-27T21:37:08.403525+0000","last_became_active":"2024-06-27T21:36:51.808538+0000","last_became_peered":"2024-06-27T21:36:51.808538+0000","last_unstale":"2024-06-27T21:37:08.403525+0000","last_undegraded":"2024-06-27T21:37:08.403525+0000","last_fullsized":"2024-06-27T21:37:08.403525+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"642'10736","reported_seq":21788,"reported_epoch":642,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:37:13.443822+0000","last_change":"2024-06-27T21:36:52.924234+0000","last_active":"2024-06-27T21:37:13.443822+0000","last_peered":"2024-06-27T21:37:13.443822+0000","last_clean":"2024-06-27T21:31:31.245570+0000","last_became_active":"2024-06-27T21:36:52.878015+0000","last_became_peered":"2024-06-27T21:36:52.878015+0000","last_unstale":"2024-06-27T21:37:13.443822+0000","last_undegraded":"2024-06-27T21:36:52.815425+0000","last_fullsized":"2024-06-27T21:36:52.805581+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":6302,"log_size":10736,"log_dups_size":0,"ondisk_log_size":10736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70189056,"num_objects":1071,"num_object_clones":0,"num_object_copies":3213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":218080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4088,"num_bytes_recovered":220528640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"7,11","objects":1067}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":786,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.844381+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:37:07.844381+0000","last_peered":"2024-06-27T21:37:07.844381+0000","last_clean":"2024-06-27T21:37:07.844381+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:37:07.844381+0000","last_undegraded":"2024-06-27T21:37:07.844381+0000","last_fullsized":"2024-06-27T21:37:07.844381+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"642'10434","reported_seq":10943,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:13.223474+0000","last_change":"2024-06-27T21:37:12.279586+0000","last_active":"2024-06-27T21:37:13.223474+0000","last_peered":"2024-06-27T21:37:13.223474+0000","last_clean":"2024-06-27T21:37:13.223474+0000","last_became_active":"2024-06-27T21:33:25.695260+0000","last_became_peered":"2024-06-27T21:33:25.695260+0000","last_unstale":"2024-06-27T21:37:13.223474+0000","last_undegraded":"2024-06-27T21:37:13.223474+0000","last_fullsized":"2024-06-27T21:37:13.223474+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'10427","last_scrub_stamp":"2024-06-27T21:37:12.279210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:37:12.279210+0000","objects_scrubbed":1071,"log_size":10434,"log_dups_size":0,"ondisk_log_size":10434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:24.258138+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69730304,"num_objects":1064,"num_object_clones":0,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":701,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":788,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.844129+0000","last_change":"2024-06-27T21:36:51.810036+0000","last_active":"2024-06-27T21:37:07.844129+0000","last_peered":"2024-06-27T21:37:07.844129+0000","last_clean":"2024-06-27T21:37:07.844129+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:37:07.844129+0000","last_undegraded":"2024-06-27T21:37:07.844129+0000","last_fullsized":"2024-06-27T21:37:07.844129+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"641'10254","reported_seq":25254,"reported_epoch":642,"state":"active+recovering+remapped","last_fresh":"2024-06-27T21:37:13.363817+0000","last_change":"2024-06-27T21:36:53.020941+0000","last_active":"2024-06-27T21:37:13.363817+0000","last_peered":"2024-06-27T21:37:13.363817+0000","last_clean":"2024-06-27T21:36:50.265928+0000","last_became_active":"2024-06-27T21:36:52.987862+0000","last_became_peered":"2024-06-27T21:36:52.987862+0000","last_unstale":"2024-06-27T21:37:13.363817+0000","last_undegraded":"2024-06-27T21:37:13.363817+0000","last_fullsized":"2024-06-27T21:37:13.363817+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'10170","last_scrub_stamp":"2024-06-27T21:36:39.166411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:39.166411+0000","objects_scrubbed":1138,"log_size":10254,"log_dups_size":0,"ondisk_log_size":10254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074944,"num_objects":1054,"num_object_clones":0,"num_object_copies":3162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":674,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1634,"num_bytes_recovered":70713344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[7,9,10],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"7,9,10","objects":1053}],"blocked_by":[],"up_primary":8,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":703,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.844381+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:37:07.844381+0000","last_peered":"2024-06-27T21:37:07.844381+0000","last_clean":"2024-06-27T21:37:07.844381+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:37:07.844381+0000","last_undegraded":"2024-06-27T21:37:07.844381+0000","last_fullsized":"2024-06-27T21:37:07.844381+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"641'10563","reported_seq":24480,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:10.257722+0000","last_change":"2024-06-27T21:33:54.883116+0000","last_active":"2024-06-27T21:37:10.257722+0000","last_peered":"2024-06-27T21:37:10.257722+0000","last_clean":"2024-06-27T21:37:10.257722+0000","last_became_active":"2024-06-27T21:33:25.693582+0000","last_became_peered":"2024-06-27T21:33:25.693582+0000","last_unstale":"2024-06-27T21:37:10.257722+0000","last_undegraded":"2024-06-27T21:37:10.257722+0000","last_fullsized":"2024-06-27T21:37:10.257722+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":1712,"log_size":10563,"log_dups_size":0,"ondisk_log_size":10563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70582272,"num_objects":1077,"num_object_clones":0,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":635,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":780,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.844213+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:37:07.844213+0000","last_peered":"2024-06-27T21:37:07.844213+0000","last_clean":"2024-06-27T21:37:07.844213+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:37:07.844213+0000","last_undegraded":"2024-06-27T21:37:07.844213+0000","last_fullsized":"2024-06-27T21:37:07.844213+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.6","version":"642'10563","reported_seq":24853,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:13.190920+0000","last_change":"2024-06-27T21:33:50.968993+0000","last_active":"2024-06-27T21:37:13.190920+0000","last_peered":"2024-06-27T21:37:13.190920+0000","last_clean":"2024-06-27T21:37:13.190920+0000","last_became_active":"2024-06-27T21:33:25.692464+0000","last_became_peered":"2024-06-27T21:33:25.692464+0000","last_unstale":"2024-06-27T21:37:13.190920+0000","last_undegraded":"2024-06-27T21:37:13.190920+0000","last_fullsized":"2024-06-27T21:37:13.190920+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'9882","last_scrub_stamp":"2024-06-27T21:33:50.968852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.968852+0000","objects_scrubbed":1716,"log_size":10563,"log_dups_size":0,"ondisk_log_size":10563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67829760,"num_objects":1035,"num_object_clones":0,"num_object_copies":3105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":457,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.920918+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:37:12.920918+0000","last_peered":"2024-06-27T21:37:12.920918+0000","last_clean":"2024-06-27T21:37:12.920918+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:37:12.920918+0000","last_undegraded":"2024-06-27T21:37:12.920918+0000","last_fullsized":"2024-06-27T21:37:12.920918+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"642'10529","reported_seq":20807,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:13.553277+0000","last_change":"2024-06-27T21:36:08.622320+0000","last_active":"2024-06-27T21:37:13.553277+0000","last_peered":"2024-06-27T21:37:13.553277+0000","last_clean":"2024-06-27T21:37:13.553277+0000","last_became_active":"2024-06-27T21:33:26.702107+0000","last_became_peered":"2024-06-27T21:33:26.702107+0000","last_unstale":"2024-06-27T21:37:13.553277+0000","last_undegraded":"2024-06-27T21:37:13.553277+0000","last_fullsized":"2024-06-27T21:37:13.553277+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'10314","last_scrub_stamp":"2024-06-27T21:36:08.622226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:08.622226+0000","objects_scrubbed":1220,"log_size":10529,"log_dups_size":0,"ondisk_log_size":10529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:36.026947+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65798144,"num_objects":1004,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":354,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.919711+0000","last_change":"2024-06-27T21:36:51.808599+0000","last_active":"2024-06-27T21:37:12.919711+0000","last_peered":"2024-06-27T21:37:12.919711+0000","last_clean":"2024-06-27T21:37:12.919711+0000","last_became_active":"2024-06-27T21:36:51.808187+0000","last_became_peered":"2024-06-27T21:36:51.808187+0000","last_unstale":"2024-06-27T21:37:12.919711+0000","last_undegraded":"2024-06-27T21:37:12.919711+0000","last_fullsized":"2024-06-27T21:37:12.919711+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10],"acting":[4,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"641'10980","reported_seq":14322,"reported_epoch":642,"state":"active+remapped+backfill_wait","last_fresh":"2024-06-27T21:37:12.963548+0000","last_change":"2024-06-27T21:36:53.019174+0000","last_active":"2024-06-27T21:37:12.963548+0000","last_peered":"2024-06-27T21:37:12.963548+0000","last_clean":"2024-06-27T21:36:50.700367+0000","last_became_active":"2024-06-27T21:36:52.921052+0000","last_became_peered":"2024-06-27T21:36:52.921052+0000","last_unstale":"2024-06-27T21:37:12.963548+0000","last_undegraded":"2024-06-27T21:37:12.963548+0000","last_fullsized":"2024-06-27T21:37:12.963548+0000","mapping_epoch":633,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10731","last_scrub_stamp":"2024-06-27T21:36:00.782086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:00.782086+0000","objects_scrubbed":1308,"log_size":10880,"log_dups_size":0,"ondisk_log_size":10880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69402624,"num_objects":1059,"num_object_clones":0,"num_object_copies":3177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1037,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":90308608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,11],"avail_no_missing":["5","9","11"],"object_location_counts":[{"shards":"5,9,11","objects":1059}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":405,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.920882+0000","last_change":"2024-06-27T21:35:47.454903+0000","last_active":"2024-06-27T21:37:12.920882+0000","last_peered":"2024-06-27T21:37:12.920882+0000","last_clean":"2024-06-27T21:37:12.920882+0000","last_became_active":"2024-06-27T21:35:46.856358+0000","last_became_peered":"2024-06-27T21:35:46.856358+0000","last_unstale":"2024-06-27T21:37:12.920882+0000","last_undegraded":"2024-06-27T21:37:12.920882+0000","last_fullsized":"2024-06-27T21:37:12.920882+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.454820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.454820+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"641'10459","reported_seq":11006,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.963627+0000","last_change":"2024-06-27T21:36:05.559816+0000","last_active":"2024-06-27T21:37:12.963627+0000","last_peered":"2024-06-27T21:37:12.963627+0000","last_clean":"2024-06-27T21:37:12.963627+0000","last_became_active":"2024-06-27T21:33:25.697304+0000","last_became_peered":"2024-06-27T21:33:25.697304+0000","last_unstale":"2024-06-27T21:37:12.963627+0000","last_undegraded":"2024-06-27T21:37:12.963627+0000","last_fullsized":"2024-06-27T21:37:12.963627+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'10215","last_scrub_stamp":"2024-06-27T21:36:05.559677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:05.559677+0000","objects_scrubbed":1276,"log_size":10459,"log_dups_size":0,"ondisk_log_size":10459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:37:13.224040+0000","scrub_duration":728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":700,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":985,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.963551+0000","last_change":"2024-06-27T21:35:47.736735+0000","last_active":"2024-06-27T21:37:12.963551+0000","last_peered":"2024-06-27T21:37:12.963551+0000","last_clean":"2024-06-27T21:37:12.963551+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:37:12.963551+0000","last_undegraded":"2024-06-27T21:37:12.963551+0000","last_fullsized":"2024-06-27T21:37:12.963551+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.736633+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.736633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"642'10662","reported_seq":11145,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:13.573744+0000","last_change":"2024-06-27T21:33:49.851182+0000","last_active":"2024-06-27T21:37:13.573744+0000","last_peered":"2024-06-27T21:37:13.573744+0000","last_clean":"2024-06-27T21:37:13.573744+0000","last_became_active":"2024-06-27T21:33:25.736643+0000","last_became_peered":"2024-06-27T21:33:25.736643+0000","last_unstale":"2024-06-27T21:37:13.573744+0000","last_undegraded":"2024-06-27T21:37:13.573744+0000","last_fullsized":"2024-06-27T21:37:13.573744+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9962","last_scrub_stamp":"2024-06-27T21:33:49.850981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.850981+0000","objects_scrubbed":1778,"log_size":10662,"log_dups_size":0,"ondisk_log_size":10662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70647808,"num_objects":1078,"num_object_clones":0,"num_object_copies":3234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":700,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":374,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.919884+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:37:12.919884+0000","last_peered":"2024-06-27T21:37:12.919884+0000","last_clean":"2024-06-27T21:37:12.919884+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:37:12.919884+0000","last_undegraded":"2024-06-27T21:37:12.919884+0000","last_fullsized":"2024-06-27T21:37:12.919884+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"641'10576","reported_seq":21141,"reported_epoch":641,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:37:10.340424+0000","last_change":"2024-06-27T21:36:53.024152+0000","last_active":"2024-06-27T21:37:10.340424+0000","last_peered":"2024-06-27T21:37:10.340424+0000","last_clean":"2024-06-27T21:36:50.720915+0000","last_became_active":"2024-06-27T21:36:52.989773+0000","last_became_peered":"2024-06-27T21:36:52.989773+0000","last_unstale":"2024-06-27T21:37:10.340424+0000","last_undegraded":"2024-06-27T21:37:10.340424+0000","last_fullsized":"2024-06-27T21:36:52.805462+0000","mapping_epoch":633,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9854","last_scrub_stamp":"2024-06-27T21:34:00.348580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:00.348580+0000","objects_scrubbed":1790,"log_size":10576,"log_dups_size":0,"ondisk_log_size":10576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69992448,"num_objects":1068,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":722,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1564,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9],"avail_no_missing":["0","2","9"],"object_location_counts":[{"shards":"0,9","objects":1068}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":378,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.921348+0000","last_change":"2024-06-27T21:36:51.821471+0000","last_active":"2024-06-27T21:37:12.921348+0000","last_peered":"2024-06-27T21:37:12.921348+0000","last_clean":"2024-06-27T21:37:12.921348+0000","last_became_active":"2024-06-27T21:36:51.820882+0000","last_became_peered":"2024-06-27T21:36:51.820882+0000","last_unstale":"2024-06-27T21:37:12.921348+0000","last_undegraded":"2024-06-27T21:37:12.921348+0000","last_fullsized":"2024-06-27T21:37:12.921348+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"641'11341","reported_seq":25188,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:10.209240+0000","last_change":"2024-06-27T21:33:23.873709+0000","last_active":"2024-06-27T21:37:10.209240+0000","last_peered":"2024-06-27T21:37:10.209240+0000","last_clean":"2024-06-27T21:37:10.209240+0000","last_became_active":"2024-06-27T21:33:23.851489+0000","last_became_peered":"2024-06-27T21:33:23.851489+0000","last_unstale":"2024-06-27T21:37:10.209240+0000","last_undegraded":"2024-06-27T21:37:10.209240+0000","last_fullsized":"2024-06-27T21:37:10.209240+0000","mapping_epoch":545,"log_start":"475'1300","ondisk_log_start":"475'1300","created":467,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134807552,"num_objects":2057,"num_object_clones":0,"num_object_copies":6171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11341,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6699,"num_bytes_recovered":340197376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"641'11100","reported_seq":16737,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:12.030550+0000","last_change":"2024-06-27T21:36:08.015340+0000","last_active":"2024-06-27T21:37:12.030550+0000","last_peered":"2024-06-27T21:37:12.030550+0000","last_clean":"2024-06-27T21:37:12.030550+0000","last_became_active":"2024-06-27T21:36:07.973506+0000","last_became_peered":"2024-06-27T21:36:07.973506+0000","last_unstale":"2024-06-27T21:37:12.030550+0000","last_undegraded":"2024-06-27T21:37:12.030550+0000","last_fullsized":"2024-06-27T21:37:12.030550+0000","mapping_epoch":612,"log_start":"475'1000","ondisk_log_start":"475'1000","created":467,"last_epoch_clean":613,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'10541","last_scrub_stamp":"2024-06-27T21:35:43.673315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:43.673315+0000","objects_scrubbed":2645,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136577024,"num_objects":2084,"num_object_clones":0,"num_object_copies":6252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2556,"num_bytes_recovered":167510016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":380,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:07.919439+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:37:07.919439+0000","last_peered":"2024-06-27T21:37:07.919439+0000","last_clean":"2024-06-27T21:37:07.919439+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:37:07.919439+0000","last_undegraded":"2024-06-27T21:37:07.919439+0000","last_fullsized":"2024-06-27T21:37:07.919439+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"642'11682","reported_seq":23963,"reported_epoch":642,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:37:13.646091+0000","last_change":"2024-06-27T21:37:08.607085+0000","last_active":"2024-06-27T21:37:13.646091+0000","last_peered":"2024-06-27T21:37:13.646091+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:37:00.083010+0000","last_became_peered":"2024-06-27T21:37:00.083010+0000","last_unstale":"2024-06-27T21:37:13.646091+0000","last_undegraded":"2024-06-27T21:36:57.662667+0000","last_fullsized":"2024-06-27T21:36:52.805596+0000","mapping_epoch":633,"log_start":"475'1600","ondisk_log_start":"475'1600","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140378112,"num_objects":2142,"num_object_clones":0,"num_object_copies":6426,"num_objects_missing_on_primary":3508,"num_objects_missing":3508,"num_objects_degraded":9037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3385,"num_bytes_recovered":221511680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[7,8],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6,7,8","objects":11},{"shards":"5,6,8","objects":2131},{"shards":"7,8","objects":-8}],"blocked_by":[],"up_primary":8,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":378,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-06-27T21:37:08.403454+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:37:08.403454+0000","last_peered":"2024-06-27T21:37:08.403454+0000","last_clean":"2024-06-27T21:37:08.403454+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:37:08.403454+0000","last_undegraded":"2024-06-27T21:37:08.403454+0000","last_fullsized":"2024-06-27T21:37:08.403454+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":2207383576,"num_objects":33684,"num_object_clones":0,"num_object_copies":101052,"num_objects_missing_on_primary":3508,"num_objects_missing":3508,"num_objects_degraded":12406,"num_objects_misplaced":4113,"num_objects_unfound":0,"num_objects_dirty":33684,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95121,"num_write_kb":3243713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44422,"num_bytes_recovered":2275278896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7923838976,"data_stored":7923826760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":272998,"ondisk_log_size":272998,"up":78,"acting":73,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253199,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1357536,"kb_used_data":752980,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92387616,"statfs":{"total":95995035648,"available":94604918784,"internally_reserved":0,"allocated":771051520,"data_stored":770316200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514835,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1157400,"kb_used_data":679180,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92587752,"statfs":{"total":95995035648,"available":94809858048,"internally_reserved":0,"allocated":695480320,"data_stored":694748552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":489683392},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743768,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1123728,"kb_used_data":655240,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92621424,"statfs":{"total":95995035648,"available":94844338176,"internally_reserved":0,"allocated":670965760,"data_stored":670238064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":479721920},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972700,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796752,"kb_used_data":462920,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92948400,"statfs":{"total":95995035648,"available":95179161600,"internally_reserved":0,"allocated":474030080,"data_stored":473302384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267036,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459492,"kb_used_data":868828,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92285660,"statfs":{"total":95995035648,"available":94500515840,"internally_reserved":0,"allocated":889679872,"data_stored":888944403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":604830154},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,16],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528678,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1533648,"kb_used_data":1019336,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92211504,"statfs":{"total":95995035648,"available":94424580096,"internally_reserved":0,"allocated":1043800064,"data_stored":1043072368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":526645702},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757612,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947664,"kb_used_data":547208,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92797488,"statfs":{"total":95995035648,"available":95024627712,"internally_reserved":0,"allocated":560340992,"data_stored":559613296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161317,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793824,"kb_used_data":535700,"kb_used_omap":92,"kb_used_meta":258019,"kb_avail":92951328,"statfs":{"total":95995035648,"available":95182159872,"internally_reserved":0,"allocated":548556800,"data_stored":547824883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":264211554},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182772,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708944,"kb_used_data":339016,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93036208,"statfs":{"total":95995035648,"available":95269076992,"internally_reserved":0,"allocated":347152384,"data_stored":346424688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411705,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1470696,"kb_used_data":1046620,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92274456,"statfs":{"total":95995035648,"available":94489042944,"internally_reserved":0,"allocated":1071738880,"data_stored":1071003411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738754,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1397016,"kb_used_data":798028,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92348136,"statfs":{"total":95995035648,"available":94564491264,"internally_reserved":0,"allocated":817180672,"data_stored":816448904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":813371392,"data_stored":813367320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69337088,"data_stored":69337088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1068302336,"data_stored":1068302336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343408640,"data_stored":343408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":545652736,"data_stored":545652736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":557907968,"data_stored":557907968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1041039360,"data_stored":1041039360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":886964224,"data_stored":886964224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":468975616,"data_stored":468975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":668467200,"data_stored":668467200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":692129792,"data_stored":692125720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":768282624,"data_stored":768278552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:37:21.650 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-27T21:37:21.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 36 2024-06-27T21:37:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:21.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:21.699 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:21.699 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:21.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:21.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:21.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:21.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:21.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: pgmap v1640: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 198 op/s; 12282/97623 objects degraded (12.581%); 3981/97623 objects misplaced (4.078%); 1.1 MiB/s, 176 objects/s recovering 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:21 smithi151 ceph-mon[28965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:37:22.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:22.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: pgmap v1640: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 198 op/s; 12282/97623 objects degraded (12.581%); 3981/97623 objects misplaced (4.078%); 1.1 MiB/s, 176 objects/s recovering 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:22.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:21 smithi192 ceph-mon[32956]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:37:22.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:22.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:22.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: pgmap v1640: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 198 op/s; 12282/97623 objects degraded (12.581%); 3981/97623 objects misplaced (4.078%); 1.1 MiB/s, 176 objects/s recovering 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:37:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:21 smithi190 ceph-mon[32247]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:37:22.274 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:22.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:37:22.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:22.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:22.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:22.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:22.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:22.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:22.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:22.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:22.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:22.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:22.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:22.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:22 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 12282/96984 objects degraded (12.664%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:22.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:23.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:23.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:23.139 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:23.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:23.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:22 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 12282/96984 objects degraded (12.664%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:22 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 12282/96984 objects degraded (12.664%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:23.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:23.386 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:23.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:23.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:23.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:23.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:23.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:23.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:23.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:23.859 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:23.881 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:23.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:23.899 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:23.901 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:23.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:24.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:24.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:24.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:24.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:23 smithi151 ceph-mon[28965]: pgmap v1641: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 190 op/s; 12282/96984 objects degraded (12.664%); 3981/96984 objects misplaced (4.105%); 1003 KiB/s, 206 objects/s recovering 2024-06-27T21:37:24.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:23 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:24.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:23 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:24.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:23 smithi151 ceph-mon[28965]: osdmap e644: 12 total, 12 up, 11 in 2024-06-27T21:37:24.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:24.252 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:23 smithi190 ceph-mon[32247]: pgmap v1641: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 190 op/s; 12282/96984 objects degraded (12.664%); 3981/96984 objects misplaced (4.105%); 1003 KiB/s, 206 objects/s recovering 2024-06-27T21:37:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:23 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:23 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:23 smithi190 ceph-mon[32247]: osdmap e644: 12 total, 12 up, 11 in 2024-06-27T21:37:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:23 smithi192 ceph-mon[32956]: pgmap v1641: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 190 op/s; 12282/96984 objects degraded (12.664%); 3981/96984 objects misplaced (4.105%); 1003 KiB/s, 206 objects/s recovering 2024-06-27T21:37:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:23 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:23 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:23 smithi192 ceph-mon[32956]: osdmap e644: 12 total, 12 up, 11 in 2024-06-27T21:37:24.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:24.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:24.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:24.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:24.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:24.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:24.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:24.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:24.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:24.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:24.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:24.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:25.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:25.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:25.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:25.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:25.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:25.371 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:25.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:25.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:25.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:25.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:25.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:25.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:25.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:25.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:25.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:25.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:25.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:25.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:25 smithi151 ceph-mon[28965]: pgmap v1643: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 198 op/s; 12152/95451 objects degraded (12.731%); 3849/95451 objects misplaced (4.032%); 909 KiB/s, 183 objects/s recovering 2024-06-27T21:37:25.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:25 smithi151 ceph-mon[28965]: osdmap e645: 12 total, 12 up, 11 in 2024-06-27T21:37:26.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:25 smithi190 ceph-mon[32247]: pgmap v1643: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 198 op/s; 12152/95451 objects degraded (12.731%); 3849/95451 objects misplaced (4.032%); 909 KiB/s, 183 objects/s recovering 2024-06-27T21:37:26.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:25 smithi190 ceph-mon[32247]: osdmap e645: 12 total, 12 up, 11 in 2024-06-27T21:37:26.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:25 smithi192 ceph-mon[32956]: pgmap v1643: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 198 op/s; 12152/95451 objects degraded (12.731%); 3849/95451 objects misplaced (4.032%); 909 KiB/s, 183 objects/s recovering 2024-06-27T21:37:26.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:25 smithi192 ceph-mon[32956]: osdmap e645: 12 total, 12 up, 11 in 2024-06-27T21:37:26.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:26.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:26.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:26.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:26.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:26.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:26.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:26.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:26.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:26.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:26.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:26.734 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:26.764 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:37:26.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:26.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:26.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:26.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:26.981 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:26 smithi192 ceph-mon[32956]: osdmap e646: 12 total, 12 up, 11 in 2024-06-27T21:37:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/111796368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:37:27.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:27.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:26 smithi151 ceph-mon[28965]: osdmap e646: 12 total, 12 up, 11 in 2024-06-27T21:37:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/111796368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:37:27.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:27.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:26 smithi190 ceph-mon[32247]: osdmap e646: 12 total, 12 up, 11 in 2024-06-27T21:37:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/111796368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:37:27.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:27.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:27.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:27.475 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:27.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:27.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:27.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:27.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:27.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:27.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:27.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:27.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:27.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:28.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:27 smithi151 ceph-mon[28965]: pgmap v1646: 57 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 178 op/s; 8882/94401 objects degraded (9.409%); 4812/94401 objects misplaced (5.097%); 757 KiB/s, 242 objects/s recovering 2024-06-27T21:37:28.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:27 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:28.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/111796368' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:37:28.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:27 smithi151 ceph-mon[28965]: osdmap e647: 12 total, 12 up, 11 in 2024-06-27T21:37:28.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:28.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:27 smithi192 ceph-mon[32956]: pgmap v1646: 57 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 178 op/s; 8882/94401 objects degraded (9.409%); 4812/94401 objects misplaced (5.097%); 757 KiB/s, 242 objects/s recovering 2024-06-27T21:37:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:27 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/111796368' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:37:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:27 smithi192 ceph-mon[32956]: osdmap e647: 12 total, 12 up, 11 in 2024-06-27T21:37:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:28.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:28.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:28.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:28.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:28.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:27 smithi190 ceph-mon[32247]: pgmap v1646: 57 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 178 op/s; 8882/94401 objects degraded (9.409%); 4812/94401 objects misplaced (5.097%); 757 KiB/s, 242 objects/s recovering 2024-06-27T21:37:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:27 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/111796368' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:37:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:27 smithi190 ceph-mon[32247]: osdmap e647: 12 total, 12 up, 11 in 2024-06-27T21:37:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:28.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:28.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:28.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:28.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:28.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:28.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:28.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:28.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:28.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:28.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:28.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:28.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:28.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:29.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:28 smithi192 ceph-mon[32956]: osdmap e648: 12 total, 12 up, 11 in 2024-06-27T21:37:29.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:29.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:29.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:29.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:28 smithi151 ceph-mon[28965]: osdmap e648: 12 total, 12 up, 11 in 2024-06-27T21:37:29.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:28 smithi190 ceph-mon[32247]: osdmap e648: 12 total, 12 up, 11 in 2024-06-27T21:37:29.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:29.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:29.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:29.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:29.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:29.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:29.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:29.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:29.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:29.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:29.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:29.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:29.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:30.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:29 smithi192 ceph-mon[32956]: pgmap v1649: 57 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 111 op/s; 8882/94113 objects degraded (9.438%); 4793/94113 objects misplaced (5.093%); 0 B/s, 183 objects/s recovering 2024-06-27T21:37:30.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:29 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 8882/94113 objects degraded (9.438%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:30.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:29 smithi192 ceph-mon[32956]: osdmap e649: 12 total, 12 up, 11 in 2024-06-27T21:37:30.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:30.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:30.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:30.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:29 smithi151 ceph-mon[28965]: pgmap v1649: 57 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 111 op/s; 8882/94113 objects degraded (9.438%); 4793/94113 objects misplaced (5.093%); 0 B/s, 183 objects/s recovering 2024-06-27T21:37:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:29 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 8882/94113 objects degraded (9.438%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:29 smithi151 ceph-mon[28965]: osdmap e649: 12 total, 12 up, 11 in 2024-06-27T21:37:30.202 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:29 smithi190 ceph-mon[32247]: pgmap v1649: 57 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 111 op/s; 8882/94113 objects degraded (9.438%); 4793/94113 objects misplaced (5.093%); 0 B/s, 183 objects/s recovering 2024-06-27T21:37:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:29 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 8882/94113 objects degraded (9.438%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:29 smithi190 ceph-mon[32247]: osdmap e649: 12 total, 12 up, 11 in 2024-06-27T21:37:30.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:30.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:30.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:30.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:30.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:30.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:30.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:30.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:30.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:30.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:30.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:30.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:30.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:31.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:31.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:31.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:31.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:31.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:31.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:31.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:31.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:31.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:31.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:31.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:31.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:31.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:31.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:31.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:31.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:31.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:31.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:31.865 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 6 pg_num to 36 2024-06-27T21:37:31.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:31.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:31.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:32.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:32.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:31 smithi192 ceph-mon[32956]: pgmap v1651: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 154 op/s; 8757/92352 objects degraded (9.482%); 4700/92352 objects misplaced (5.089%); 1.0 MiB/s, 176 objects/s recovering 2024-06-27T21:37:32.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1073240842' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]: dispatch 2024-06-27T21:37:32.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:32.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:32.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:32.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:32.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:31 smithi151 ceph-mon[28965]: pgmap v1651: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 154 op/s; 8757/92352 objects degraded (9.482%); 4700/92352 objects misplaced (5.089%); 1.0 MiB/s, 176 objects/s recovering 2024-06-27T21:37:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1073240842' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]: dispatch 2024-06-27T21:37:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:31 smithi190 ceph-mon[32247]: pgmap v1651: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 49 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 154 op/s; 8757/92352 objects degraded (9.482%); 4700/92352 objects misplaced (5.089%); 1.0 MiB/s, 176 objects/s recovering 2024-06-27T21:37:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1073240842' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]: dispatch 2024-06-27T21:37:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:32.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:32.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:32.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:32.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:32.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:32.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:32.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:32.681 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:32.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:32.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:32.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:32.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:33.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:33.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:33.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1073240842' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]': finished 2024-06-27T21:37:33.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:32 smithi151 ceph-mon[28965]: osdmap e650: 12 total, 12 up, 11 in 2024-06-27T21:37:33.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-06-27T21:37:33.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:33.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1073240842' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]': finished 2024-06-27T21:37:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:32 smithi190 ceph-mon[32247]: osdmap e650: 12 total, 12 up, 11 in 2024-06-27T21:37:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-06-27T21:37:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1073240842' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "36"}]': finished 2024-06-27T21:37:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:32 smithi192 ceph-mon[32956]: osdmap e650: 12 total, 12 up, 11 in 2024-06-27T21:37:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-06-27T21:37:33.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:33.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:33.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:33.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:33.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:33.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:33.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:33.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:33.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:33.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:33.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:33.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:33.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:34.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:34.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:34.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:33 smithi190 ceph-mon[32247]: pgmap v1653: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 168 op/s; 8777/91668 objects degraded (9.575%); 4680/91668 objects misplaced (5.105%); 863 KiB/s, 201 objects/s recovering 2024-06-27T21:37:34.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]': finished 2024-06-27T21:37:34.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:33 smithi190 ceph-mon[32247]: osdmap e651: 12 total, 12 up, 11 in 2024-06-27T21:37:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:33 smithi151 ceph-mon[28965]: pgmap v1653: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 168 op/s; 8777/91668 objects degraded (9.575%); 4680/91668 objects misplaced (5.105%); 863 KiB/s, 201 objects/s recovering 2024-06-27T21:37:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]': finished 2024-06-27T21:37:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:33 smithi151 ceph-mon[28965]: osdmap e651: 12 total, 12 up, 11 in 2024-06-27T21:37:34.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:34.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:33 smithi192 ceph-mon[32956]: pgmap v1653: 57 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 168 op/s; 8777/91668 objects degraded (9.575%); 4680/91668 objects misplaced (5.105%); 863 KiB/s, 201 objects/s recovering 2024-06-27T21:37:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "36"}]': finished 2024-06-27T21:37:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:33 smithi192 ceph-mon[32956]: osdmap e651: 12 total, 12 up, 11 in 2024-06-27T21:37:34.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:34.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:34.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:34.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:34.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:34.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:34.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:34.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:34.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:34.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:34.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:34.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:35.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:35.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:35.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: osdmap e652: 12 total, 12 up, 11 in 2024-06-27T21:37:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-06-27T21:37:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:37:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 8777/91668 objects degraded (9.575%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]': finished 2024-06-27T21:37:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:34 smithi151 ceph-mon[28965]: osdmap e653: 12 total, 12 up, 11 in 2024-06-27T21:37:35.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:35.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: osdmap e652: 12 total, 12 up, 11 in 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 8777/91668 objects degraded (9.575%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]': finished 2024-06-27T21:37:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:34 smithi190 ceph-mon[32247]: osdmap e653: 12 total, 12 up, 11 in 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: osdmap e652: 12 total, 12 up, 11 in 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 8777/91668 objects degraded (9.575%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "36"}]': finished 2024-06-27T21:37:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:34 smithi192 ceph-mon[32956]: osdmap e653: 12 total, 12 up, 11 in 2024-06-27T21:37:35.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:35.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:35.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:35.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:35.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:35.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:35.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:35.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:35.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:35.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:35.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:35.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:35.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:36.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:36.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:36.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:35 smithi151 ceph-mon[28965]: pgmap v1656: 67 pgs: 1 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 115/86058 objects degraded (0.134%); 4207/86058 objects misplaced (4.889%) 2024-06-27T21:37:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:35 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:37:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:35 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:37:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:35 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:37:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:35 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:37:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:35 smithi151 ceph-mon[28965]: osdmap e654: 12 total, 12 up, 11 in 2024-06-27T21:37:36.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:36.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:35 smithi190 ceph-mon[32247]: pgmap v1656: 67 pgs: 1 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 115/86058 objects degraded (0.134%); 4207/86058 objects misplaced (4.889%) 2024-06-27T21:37:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:35 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:37:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:35 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:37:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:35 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:37:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:35 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:37:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:35 smithi190 ceph-mon[32247]: osdmap e654: 12 total, 12 up, 11 in 2024-06-27T21:37:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:35 smithi192 ceph-mon[32956]: pgmap v1656: 67 pgs: 1 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 115/86058 objects degraded (0.134%); 4207/86058 objects misplaced (4.889%) 2024-06-27T21:37:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:35 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:37:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:35 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:37:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:35 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:37:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:35 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:37:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:35 smithi192 ceph-mon[32956]: osdmap e654: 12 total, 12 up, 11 in 2024-06-27T21:37:36.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:36.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:36.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:36.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:36.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:36.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:36.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:36.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:36.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:36.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:36 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:37:36.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:36 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:37:36.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:36 smithi151 ceph-mon[28965]: 6.1c starting backfill to osd.11 from (0'0,0'0] MAX to 650'11926 2024-06-27T21:37:36.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:36 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:36.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:36 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:37:36.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:36 smithi151 ceph-mon[28965]: osdmap e655: 12 total, 12 up, 11 in 2024-06-27T21:37:36.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:36.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:37.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:37.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:37.184 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:37:37.184 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:37:37.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:37.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:36 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:37:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:36 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:37:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:36 smithi190 ceph-mon[32247]: 6.1c starting backfill to osd.11 from (0'0,0'0] MAX to 650'11926 2024-06-27T21:37:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:36 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:36 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:37:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:36 smithi190 ceph-mon[32247]: osdmap e655: 12 total, 12 up, 11 in 2024-06-27T21:37:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:36 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:37:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:36 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:37:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:36 smithi192 ceph-mon[32956]: 6.1c starting backfill to osd.11 from (0'0,0'0] MAX to 650'11926 2024-06-27T21:37:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:36 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:36 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:37:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:36 smithi192 ceph-mon[32956]: osdmap e655: 12 total, 12 up, 11 in 2024-06-27T21:37:37.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:37.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:37.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:37.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:37.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:37.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:37.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:37.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:37.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:37.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:37.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:37.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:37.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:38.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:37:38.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:38.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: pgmap v1659: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 46 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 136/78381 objects degraded (0.174%); 4186/78381 objects misplaced (5.341%) 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1f starting backfill to osd.3 from (0'0,0'0] MAX to 652'11713 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1e starting backfill to osd.5 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1a starting backfill to osd.4 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1d starting backfill to osd.2 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1b starting backfill to osd.8 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1b starting backfill to osd.10 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1b starting backfill to osd.11 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1d starting backfill to osd.8 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1a starting backfill to osd.8 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1f starting backfill to osd.8 from (0'0,0'0] MAX to 652'11713 2024-06-27T21:37:38.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1e starting backfill to osd.8 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1d starting backfill to osd.9 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1a starting backfill to osd.10 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.1e starting backfill to osd.10 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:37 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:37:38.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:38.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:37:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: pgmap v1659: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 46 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 136/78381 objects degraded (0.174%); 4186/78381 objects misplaced (5.341%) 2024-06-27T21:37:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1f starting backfill to osd.3 from (0'0,0'0] MAX to 652'11713 2024-06-27T21:37:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1e starting backfill to osd.5 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1a starting backfill to osd.4 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1d starting backfill to osd.2 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1b starting backfill to osd.8 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1b starting backfill to osd.10 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1b starting backfill to osd.11 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1d starting backfill to osd.8 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1a starting backfill to osd.8 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1f starting backfill to osd.8 from (0'0,0'0] MAX to 652'11713 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1e starting backfill to osd.8 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1d starting backfill to osd.9 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1a starting backfill to osd.10 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.1e starting backfill to osd.10 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:37 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:37:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:37:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: pgmap v1659: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 10 unknown, 2 active+remapped+backfilling, 46 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 136/78381 objects degraded (0.174%); 4186/78381 objects misplaced (5.341%) 2024-06-27T21:37:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1f starting backfill to osd.3 from (0'0,0'0] MAX to 652'11713 2024-06-27T21:37:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1e starting backfill to osd.5 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1a starting backfill to osd.4 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1d starting backfill to osd.2 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1b starting backfill to osd.8 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1b starting backfill to osd.10 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1b starting backfill to osd.11 from (0'0,0'0] MAX to 652'11309 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1d starting backfill to osd.8 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1a starting backfill to osd.8 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1f starting backfill to osd.8 from (0'0,0'0] MAX to 652'11713 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1e starting backfill to osd.8 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1d starting backfill to osd.9 from (0'0,0'0] MAX to 652'11598 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1a starting backfill to osd.10 from (0'0,0'0] MAX to 652'11552 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.1e starting backfill to osd.10 from (0'0,0'0] MAX to 652'11701 2024-06-27T21:37:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:37 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:37:38.347 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:38.373 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:38.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:38.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:38.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:38.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:38.725 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:38.730 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:37:38.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:37:38.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:38.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:37:38.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:37:38.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:38.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:37:38.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:37:38.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:37:38.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:37:38.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:37:38.845 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:37:38.867 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:37:38.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:37:38.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:38.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:37:38.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:37:38.891 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:37:38.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:38.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:38.923 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:37:38.923 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:37:38.959 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:37:38.960 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:37:38.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:37:38.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:37:38.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:39.030 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:37:39.030 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:37:39.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:39.065 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:37:39.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:37:39.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:38 smithi151 ceph-mon[28965]: osdmap e656: 12 total, 12 up, 11 in 2024-06-27T21:37:39.128 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:38 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:39.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:38 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:39.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2176119681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:39.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:39.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:38 smithi190 ceph-mon[32247]: osdmap e656: 12 total, 12 up, 11 in 2024-06-27T21:37:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:38 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:38 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2176119681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:39.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:39.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:38 smithi192 ceph-mon[32956]: osdmap e656: 12 total, 12 up, 11 in 2024-06-27T21:37:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:38 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:38 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2176119681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:37:39.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:39.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:39.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:39.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:39.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:39.674 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:39.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:39.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:39.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:39.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:39.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:40.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:40.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:40.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:39 smithi151 ceph-mon[28965]: pgmap v1662: 67 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 7 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 46 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 3651/81180 objects degraded (4.497%); 7985/81180 objects misplaced (9.836%) 2024-06-27T21:37:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:39 smithi151 ceph-mon[28965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:37:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:39 smithi151 ceph-mon[28965]: osdmap e657: 12 total, 12 up, 11 in 2024-06-27T21:37:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:39 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3651/81180 objects degraded (4.497%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:40.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:39 smithi190 ceph-mon[32247]: pgmap v1662: 67 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 7 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 46 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 3651/81180 objects degraded (4.497%); 7985/81180 objects misplaced (9.836%) 2024-06-27T21:37:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:39 smithi190 ceph-mon[32247]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:37:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:39 smithi190 ceph-mon[32247]: osdmap e657: 12 total, 12 up, 11 in 2024-06-27T21:37:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:39 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3651/81180 objects degraded (4.497%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:39 smithi192 ceph-mon[32956]: pgmap v1662: 67 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 3 peering, 3 remapped+peering, 7 unknown, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 46 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 3651/81180 objects degraded (4.497%); 7985/81180 objects misplaced (9.836%) 2024-06-27T21:37:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:39 smithi192 ceph-mon[32956]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:37:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:39 smithi192 ceph-mon[32956]: osdmap e657: 12 total, 12 up, 11 in 2024-06-27T21:37:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:39 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3651/81180 objects degraded (4.497%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:40.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:40.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:40.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:40.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:40.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:40.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:40.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:40.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:40.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:40.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:40.863 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:40.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:40.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:40.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:41.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:41.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:41.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:40 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:37:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:40 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:37:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:40 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:37:41.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:41.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:40 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:37:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:40 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:37:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:40 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:37:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:40 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:37:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:40 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:37:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:40 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:37:41.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:41.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:41.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:41.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:41.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:41.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:41.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:41.721 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7, 8] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:37:41.746 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:37:41.746 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-27T21:37:41.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:37:41.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:41.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:41.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:41.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:41.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:42.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:42.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:42.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:41 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:37:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:41 smithi151 ceph-mon[28965]: pgmap v1664: 67 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 1 remapped+peering, 3 unknown, 3 active+remapped+backfill_toofull, 50 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 49 MiB/s wr, 1.45k op/s; 3655/82416 objects degraded (4.435%); 15893/82416 objects misplaced (19.284%); 31 MiB/s, 729 objects/s recovering 2024-06-27T21:37:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:41 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:41 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:37:42.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:41 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:37:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:41 smithi190 ceph-mon[32247]: pgmap v1664: 67 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 1 remapped+peering, 3 unknown, 3 active+remapped+backfill_toofull, 50 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 49 MiB/s wr, 1.45k op/s; 3655/82416 objects degraded (4.435%); 15893/82416 objects misplaced (19.284%); 31 MiB/s, 729 objects/s recovering 2024-06-27T21:37:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:41 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:41 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:37:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:41 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:37:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:41 smithi192 ceph-mon[32956]: pgmap v1664: 67 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 1 remapped+peering, 3 unknown, 3 active+remapped+backfill_toofull, 50 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 49 MiB/s wr, 1.45k op/s; 3655/82416 objects degraded (4.435%); 15893/82416 objects misplaced (19.284%); 31 MiB/s, 729 objects/s recovering 2024-06-27T21:37:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:41 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:41 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:37:42.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:42.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:42.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:42.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:42.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:42.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:42.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:42.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:42.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:42.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:42.870 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:42.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:42.906 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:37:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:37:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:37:42.907 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:37:42.907 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:37:42.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.003 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:43.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:43.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:37:43.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:43.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:43.157 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:42 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:37:43.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:42 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:37:43.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:42 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:37:43.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:43.260 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:42 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:37:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:42 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:37:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:42 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:37:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:42 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:37:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:42 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:37:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:42 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:37:43.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:43.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:43.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:43.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:43.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:43.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:43.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:43.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:43.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:43.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:43.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:43.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:44.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:44.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:37:44.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:44.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:43 smithi151 ceph-mon[28965]: pgmap v1665: 67 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 remapped+peering, 4 active+remapped+backfill_toofull, 51 active+clean; 1.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 170 B/s rd, 106 MiB/s wr, 3.09k op/s; 3669/89076 objects degraded (4.119%); 21804/89076 objects misplaced (24.478%); 99 MiB/s, 2.16k objects/s recovering 2024-06-27T21:37:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:43 smithi151 ceph-mon[28965]: osdmap e658: 12 total, 12 up, 11 in 2024-06-27T21:37:44.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:43 smithi190 ceph-mon[32247]: pgmap v1665: 67 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 remapped+peering, 4 active+remapped+backfill_toofull, 51 active+clean; 1.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 170 B/s rd, 106 MiB/s wr, 3.09k op/s; 3669/89076 objects degraded (4.119%); 21804/89076 objects misplaced (24.478%); 99 MiB/s, 2.16k objects/s recovering 2024-06-27T21:37:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:43 smithi190 ceph-mon[32247]: osdmap e658: 12 total, 12 up, 11 in 2024-06-27T21:37:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:43 smithi192 ceph-mon[32956]: pgmap v1665: 67 pgs: 1 active+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 remapped+peering, 4 active+remapped+backfill_toofull, 51 active+clean; 1.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 170 B/s rd, 106 MiB/s wr, 3.09k op/s; 3669/89076 objects degraded (4.119%); 21804/89076 objects misplaced (24.478%); 99 MiB/s, 2.16k objects/s recovering 2024-06-27T21:37:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:43 smithi192 ceph-mon[32956]: osdmap e658: 12 total, 12 up, 11 in 2024-06-27T21:37:44.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:44.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:44.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:44.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:44.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:44.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:44.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:44.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:44.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:44.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:44.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:44.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:45.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:45.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:44 smithi151 ceph-mon[28965]: osdmap e659: 12 total, 12 up, 11 in 2024-06-27T21:37:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:44 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:37:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:44 smithi151 ceph-mon[28965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:37:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 3669/89076 objects degraded (4.119%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:45.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:45.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:44 smithi190 ceph-mon[32247]: osdmap e659: 12 total, 12 up, 11 in 2024-06-27T21:37:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:44 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:37:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:44 smithi190 ceph-mon[32247]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:37:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 3669/89076 objects degraded (4.119%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:44 smithi192 ceph-mon[32956]: osdmap e659: 12 total, 12 up, 11 in 2024-06-27T21:37:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:44 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:37:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:44 smithi192 ceph-mon[32956]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:37:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 3669/89076 objects degraded (4.119%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:37:45.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:45.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:45.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:45.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:45.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:45.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:45.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:45.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:45.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:45.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:45.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:45.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:46.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:46.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:46.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:46.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:45 smithi151 ceph-mon[28965]: pgmap v1668: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 170 B/s rd, 70 MiB/s wr, 2.27k op/s; 157/85809 objects degraded (0.183%); 19532/85809 objects misplaced (22.762%); 95 MiB/s, 2.22k objects/s recovering 2024-06-27T21:37:46.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:45 smithi190 ceph-mon[32247]: pgmap v1668: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 170 B/s rd, 70 MiB/s wr, 2.27k op/s; 157/85809 objects degraded (0.183%); 19532/85809 objects misplaced (22.762%); 95 MiB/s, 2.22k objects/s recovering 2024-06-27T21:37:46.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:45 smithi192 ceph-mon[32956]: pgmap v1668: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 170 B/s rd, 70 MiB/s wr, 2.27k op/s; 157/85809 objects degraded (0.183%); 19532/85809 objects misplaced (22.762%); 95 MiB/s, 2.22k objects/s recovering 2024-06-27T21:37:46.383 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:46.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:46.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:46.545 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:46.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:46.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:46.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:46.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:46.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:46.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:46.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:46.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:46.963 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:46 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:37:47.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:47.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:47.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:47.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:46 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:37:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:46 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:37:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:46 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:37:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:46 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:37:47.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:46 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:37:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:46 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:37:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:46 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:37:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:46 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:37:47.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:47.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:47.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:46 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:37:47.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:46 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:37:47.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:46 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:37:47.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:47.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:47.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:47.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:47.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.699 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:47.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:47.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:47.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:47.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:47.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:47.964 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:37:48.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:48.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:48.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:48.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:47 smithi190 ceph-mon[32247]: pgmap v1669: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 140 B/s rd, 58 MiB/s wr, 1.98k op/s; 247/82233 objects degraded (0.300%); 18533/82233 objects misplaced (22.537%); 77 MiB/s, 1.76k objects/s recovering 2024-06-27T21:37:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:47 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:37:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2628479548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:47 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:47 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:47 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:37:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:47 smithi192 ceph-mon[32956]: pgmap v1669: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 140 B/s rd, 58 MiB/s wr, 1.98k op/s; 247/82233 objects degraded (0.300%); 18533/82233 objects misplaced (22.537%); 77 MiB/s, 1.76k objects/s recovering 2024-06-27T21:37:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:47 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:37:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2628479548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:47 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:47 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:47 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:37:48.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:48.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:48.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:47 smithi151 ceph-mon[28965]: pgmap v1669: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 140 B/s rd, 58 MiB/s wr, 1.98k op/s; 247/82233 objects degraded (0.300%); 18533/82233 objects misplaced (22.537%); 77 MiB/s, 1.76k objects/s recovering 2024-06-27T21:37:48.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:47 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:37:48.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2628479548' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:48.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:47 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:37:48.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:47 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:48.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:47 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:37:48.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:48.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:48.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:48.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:48.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:48.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:48.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:48.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:48.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:48.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:48.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:49.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:49.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:48 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:37:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:48 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:48 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:37:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:48 smithi151 ceph-mon[28965]: osdmap e660: 12 total, 12 up, 11 in 2024-06-27T21:37:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:48 smithi151 ceph-mon[28965]: pgmap v1671: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 247/80688 objects degraded (0.306%); 17954/80688 objects misplaced (22.251%); 18 MiB/s, 614 objects/s recovering 2024-06-27T21:37:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:48 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:37:49.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:49.253 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:48 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:37:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:48 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:48 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:37:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:48 smithi190 ceph-mon[32247]: osdmap e660: 12 total, 12 up, 11 in 2024-06-27T21:37:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:48 smithi190 ceph-mon[32247]: pgmap v1671: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 247/80688 objects degraded (0.306%); 17954/80688 objects misplaced (22.251%); 18 MiB/s, 614 objects/s recovering 2024-06-27T21:37:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:48 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:37:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:48 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:37:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:48 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:37:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:48 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:37:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:48 smithi192 ceph-mon[32956]: osdmap e660: 12 total, 12 up, 11 in 2024-06-27T21:37:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:48 smithi192 ceph-mon[32956]: pgmap v1671: 67 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 247/80688 objects degraded (0.306%); 17954/80688 objects misplaced (22.251%); 18 MiB/s, 614 objects/s recovering 2024-06-27T21:37:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:48 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:37:49.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:49.383 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:49.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:49.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:49.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:49.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:49.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:49.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:49.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:49.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:49.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:50.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:50.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:50.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:50.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:50.181 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:37:50.181 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:37:50.181 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 661, 2024-06-27T21:37:50.181 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:37:50.181 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:37:49.752696+0000", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:23:43.806967+0000", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:36:50.790038+0000", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5806080, 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:37:50.182 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 27, 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 6, 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:37:50.183 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:37:50.184 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:37:50.185 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "595", 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "535", 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:37:50.186 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:37:50.187 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:37:50.188 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:37:50.189 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.3199999332427979, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.3199999332427979, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.130000114440918, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.130000114440918, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.79288095235824585, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.79000002145767212, 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 6, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_4", 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:29:17.908064+0000", 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:37:50.191 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 36, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 36, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 36, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 36, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 36, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:37:50.192 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "653", 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "651", 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:37:50.193 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:37:50.194 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:37:50.195 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.9900000095367432, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.6500000953674316, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.75, 2024-06-27T21:37:50.196 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.440000057220459, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.79288095235824585, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.79000002145767212, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.197 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 654, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:37:50.199 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:37:50.200 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.201 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:37:50.202 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-06-27T21:37:50.203 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.1994171142578125, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 593, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:37:50.205 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.206 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.207 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:37:50.208 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.209 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 654, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:37:50.210 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.211 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.212 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.213 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:37:50.214 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.3845977783203125, 2024-06-27T21:37:50.215 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 654, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:37:50.218 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:37:50.219 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:37:50.220 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.3238067626953125, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 31, 2024-06-27T21:37:50.221 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 347, 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 362, 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 632, 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 348, 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.222 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.224 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.225 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/3587059633", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/3587059633", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/3587059633", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/3587059633", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.226 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.5222625732421875, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.9023590087890625, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 633, 2024-06-27T21:37:50.227 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.228 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:37:50.230 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:37:50.231 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:37:50.232 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6059112548828125, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.0889892578125, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 479, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.233 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:37:50.238 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.239 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 658, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.240 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:37:50.241 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.242 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.244 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:37:50.245 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 656, 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.247 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.248 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:37:50.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:37:50.250 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6501922607421875, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 654, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.251 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.252 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:37:50.253 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.255 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 651, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.256 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.258 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:37:50.260 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:37:50.261 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 653, 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.262 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:37:50.266 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:37.950983+0000", 2024-06-27T21:37:50.267 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:27.346003+0000", 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:37:50.268 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:36:34.345386+0000", 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.269 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:00.754770+0000", 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:36:33.146211+0000", 2024-06-27T21:37:50.270 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 347 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:29.406065+0000", 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.271 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:32.144402+0000", 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:42.860745+0000", 2024-06-27T21:37:50.272 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:12.509249+0000", 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.273 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:31.202479+0000", 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.274 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:22.814876+0000", 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:37:38.285403+0000", 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:37:50.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.0", 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: 10 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:37:50.276 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.4", 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: 5, 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: 11 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.c", 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: 8 2024-06-27T21:37:50.277 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.10", 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: 3 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.14", 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: 5, 2024-06-27T21:37:50.278 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.1a", 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: 2, 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: 6 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.1b", 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: 3, 2024-06-27T21:37:50.279 INFO:teuthology.orchestra.run.smithi151.stdout: 6, 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.1c", 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: 8, 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.1d", 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-06-27T21:37:50.280 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: 11 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.1e", 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: 2, 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.1f", 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.281 INFO:teuthology.orchestra.run.smithi151.stdout: 2, 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.20", 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: 2, 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: 6 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.21", 2024-06-27T21:37:50.282 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.22", 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: 2, 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: 3 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "6.23", 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: 8, 2024-06-27T21:37:50.283 INFO:teuthology.orchestra.run.smithi151.stdout: 3 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:37:50.284 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:37:50.285 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:37:50.286 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:37:50.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:50.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 247/80688 objects degraded (0.306%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:37:50.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:49 smithi190 ceph-mon[32247]: osdmap e661: 12 total, 12 up, 11 in 2024-06-27T21:37:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 247/80688 objects degraded (0.306%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:37:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:49 smithi192 ceph-mon[32956]: osdmap e661: 12 total, 12 up, 11 in 2024-06-27T21:37:50.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:50.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:50.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:50.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 247/80688 objects degraded (0.306%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:37:50.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:49 smithi151 ceph-mon[28965]: osdmap e661: 12 total, 12 up, 11 in 2024-06-27T21:37:50.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:50.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:50.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:50.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:50.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:50.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:50.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:50.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:50.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:51.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:51.075 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:51.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:51.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/268650502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:37:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:50 smithi190 ceph-mon[32247]: pgmap v1673: 67 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 530 op/s; 247/75861 objects degraded (0.326%); 16329/75861 objects misplaced (21.525%); 7.3 MiB/s, 343 objects/s recovering 2024-06-27T21:37:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:50 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:37:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/268650502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:37:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:50 smithi192 ceph-mon[32956]: pgmap v1673: 67 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 530 op/s; 247/75861 objects degraded (0.326%); 16329/75861 objects misplaced (21.525%); 7.3 MiB/s, 343 objects/s recovering 2024-06-27T21:37:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:50 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:37:51.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:51.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:51.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/268650502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:37:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:50 smithi151 ceph-mon[28965]: pgmap v1673: 67 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 530 op/s; 247/75861 objects degraded (0.326%); 16329/75861 objects misplaced (21.525%); 7.3 MiB/s, 343 objects/s recovering 2024-06-27T21:37:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:50 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:37:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:51.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:51.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:51.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:51.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:51.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:51.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:51.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:51.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:51.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:51.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:52.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:52.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:52.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:52.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:51 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:37:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:37:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:37:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:51 smithi190 ceph-mon[32247]: osdmap e662: 12 total, 12 up, 11 in 2024-06-27T21:37:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:51 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:37:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:51 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:37:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:51 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:37:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:37:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:37:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:51 smithi192 ceph-mon[32956]: osdmap e662: 12 total, 12 up, 11 in 2024-06-27T21:37:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:51 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:37:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:51 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:37:52.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:52.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:51 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:37:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:37:52.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:37:52.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:51 smithi151 ceph-mon[28965]: osdmap e662: 12 total, 12 up, 11 in 2024-06-27T21:37:52.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:52 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:37:52.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:52 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:37:52.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:52.491 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:52.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:52.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:52.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:52.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:52.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:52.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:52.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:52.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:53.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:53.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:53.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:53.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:52 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:37:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:52 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:37:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:52 smithi190 ceph-mon[32247]: pgmap v1675: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 6 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 494 op/s; 346/73329 objects degraded (0.472%); 15391/73329 objects misplaced (20.989%); 8.5 MiB/s, 371 objects/s recovering 2024-06-27T21:37:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:52 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:53.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:52 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:37:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:52 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:37:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:52 smithi192 ceph-mon[32956]: pgmap v1675: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 6 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 494 op/s; 346/73329 objects degraded (0.472%); 15391/73329 objects misplaced (20.989%); 8.5 MiB/s, 371 objects/s recovering 2024-06-27T21:37:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:52 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:53.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:53.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:53.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:52 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:37:53.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:37:53.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: pgmap v1675: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 6 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 494 op/s; 346/73329 objects degraded (0.472%); 15391/73329 objects misplaced (20.989%); 8.5 MiB/s, 371 objects/s recovering 2024-06-27T21:37:53.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:53.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:53.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:53.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:53.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:53.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:53.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:53.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:53.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:53.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:54.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:54.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:54.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:54.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:54.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:53 smithi190 ceph-mon[32247]: osdmap e663: 12 total, 12 up, 11 in 2024-06-27T21:37:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:53 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:37:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:53 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:37:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:53 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:37:54.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:53 smithi192 ceph-mon[32956]: osdmap e663: 12 total, 12 up, 11 in 2024-06-27T21:37:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:53 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:37:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:53 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:37:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:53 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:37:54.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:54.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: osdmap e663: 12 total, 12 up, 11 in 2024-06-27T21:37:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:37:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:37:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:53 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:37:54.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:54.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:54.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:54.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:54.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:54.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:54.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:37:54.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:54.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:37:55.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:55.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:55.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:55.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:54 smithi190 ceph-mon[32247]: osdmap e664: 12 total, 12 up, 11 in 2024-06-27T21:37:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:54 smithi190 ceph-mon[32247]: pgmap v1678: 67 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 348/68451 objects degraded (0.508%); 13786/68451 objects misplaced (20.140%); 8.6 MiB/s, 499 objects/s recovering 2024-06-27T21:37:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:54 smithi190 ceph-mon[32247]: 6.14 scrub starts 2024-06-27T21:37:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 346/73329 objects degraded (0.472%), 6 pgs degraded (PG_DEGRADED) 2024-06-27T21:37:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:54 smithi190 ceph-mon[32247]: osdmap e665: 12 total, 12 up, 11 in 2024-06-27T21:37:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:54 smithi192 ceph-mon[32956]: osdmap e664: 12 total, 12 up, 11 in 2024-06-27T21:37:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:54 smithi192 ceph-mon[32956]: pgmap v1678: 67 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 348/68451 objects degraded (0.508%); 13786/68451 objects misplaced (20.140%); 8.6 MiB/s, 499 objects/s recovering 2024-06-27T21:37:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:54 smithi192 ceph-mon[32956]: 6.14 scrub starts 2024-06-27T21:37:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 346/73329 objects degraded (0.472%), 6 pgs degraded (PG_DEGRADED) 2024-06-27T21:37:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:54 smithi192 ceph-mon[32956]: osdmap e665: 12 total, 12 up, 11 in 2024-06-27T21:37:55.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:55.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:55.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:54 smithi151 ceph-mon[28965]: osdmap e664: 12 total, 12 up, 11 in 2024-06-27T21:37:55.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:55 smithi151 ceph-mon[28965]: pgmap v1678: 67 pgs: 1 peering, 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 348/68451 objects degraded (0.508%); 13786/68451 objects misplaced (20.140%); 8.6 MiB/s, 499 objects/s recovering 2024-06-27T21:37:55.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:55 smithi151 ceph-mon[28965]: 6.14 scrub starts 2024-06-27T21:37:55.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:55 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 346/73329 objects degraded (0.472%), 6 pgs degraded (PG_DEGRADED) 2024-06-27T21:37:55.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:55 smithi151 ceph-mon[28965]: osdmap e665: 12 total, 12 up, 11 in 2024-06-27T21:37:55.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:55.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:55.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:55.602 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:55.611 INFO:tasks.thrashosds.thrasher:j is {'epoch': 661, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:37:49.752696+0000', 'last_up_change': '2024-06-27T21:23:43.806967+0000', 'last_in_change': '2024-06-27T21:36:50.790038+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '595', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '535', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.7928809523582458, 'average_primary_affinity': 0.7900000214576721, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-06-27T21:29:17.908064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '653', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '651', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.6500000953674316, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.75, 'raw_score_stable': 2.440000057220459, 'primary_affinity_weighted': 0.7928809523582458, 'average_primary_affinity': 0.7900000214576721, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 654, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.1994171142578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 593, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 654, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 0.3845977783203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 654, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3238067626953125, 'last_clean_begin': 31, 'last_clean_end': 347, 'up_from': 362, 'up_thru': 632, 'down_at': 348, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 3587059633}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 3587059633}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 3587059633}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 3587059633}]}, 'public_addr': '172.21.15.190:6801/3587059633', 'cluster_addr': '172.21.15.190:6803/3587059633', 'heartbeat_back_addr': '172.21.15.190:6807/3587059633', 'heartbeat_front_addr': '172.21.15.190:6805/3587059633', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 0.5222625732421875, 'primary_affinity': 0.9023590087890625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 633, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 0.6059112548828125, 'primary_affinity': 0.0889892578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 479, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 658, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 656, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 0.6501922607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 654, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 651, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 653, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:37.950983+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-27T21:37:27.346003+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:36:34.345386+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:00.754770+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:36:33.146211+0000', 'dead_epoch': 347}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:29.406065+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:32.144402+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:42.860745+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:12.509249+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:31.202479+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:22.814876+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:37:38.285403+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 9, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.4', 'osds': [5, 9, 11]}, {'pgid': '6.c', 'osds': [7, 8]}, {'pgid': '6.10', 'osds': [0, 3]}, {'pgid': '6.14', 'osds': [1, 5, 9]}, {'pgid': '6.1a', 'osds': [0, 2, 6]}, {'pgid': '6.1b', 'osds': [3, 6, 7]}, {'pgid': '6.1c', 'osds': [8, 7]}, {'pgid': '6.1d', 'osds': [0, 10, 11]}, {'pgid': '6.1e', 'osds': [2, 4, 7]}, {'pgid': '6.1f', 'osds': [2, 10, 9]}, {'pgid': '6.20', 'osds': [2, 6]}, {'pgid': '6.21', 'osds': [9, 2]}, {'pgid': '6.22', 'osds': [2, 3]}, {'pgid': '6.23', 'osds': [8, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:37:55.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:37:55.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:55.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.732 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:55.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:37:55.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:37:55.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:55.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:55.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:55.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:56.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:56.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:56.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:56.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:56.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:56.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:56.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:56.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:56.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:56.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:56.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:56.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:56.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:56.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:56.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:56.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:37:57.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:37:57.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:57.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:57.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:56 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:37:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:56 smithi190 ceph-mon[32247]: osdmap e666: 12 total, 12 up, 11 in 2024-06-27T21:37:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:56 smithi190 ceph-mon[32247]: pgmap v1681: 67 pgs: 1 active+undersized+remapped, 1 peering, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 743 op/s; 371/64401 objects degraded (0.576%); 12790/64401 objects misplaced (19.860%); 12 MiB/s, 697 objects/s recovering 2024-06-27T21:37:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:56 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:37:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:56 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:37:57.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:56 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:37:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:56 smithi192 ceph-mon[32956]: osdmap e666: 12 total, 12 up, 11 in 2024-06-27T21:37:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:56 smithi192 ceph-mon[32956]: pgmap v1681: 67 pgs: 1 active+undersized+remapped, 1 peering, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 743 op/s; 371/64401 objects degraded (0.576%); 12790/64401 objects misplaced (19.860%); 12 MiB/s, 697 objects/s recovering 2024-06-27T21:37:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:56 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:37:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:56 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:37:57.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:57.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:56 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:37:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:56 smithi151 ceph-mon[28965]: osdmap e666: 12 total, 12 up, 11 in 2024-06-27T21:37:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:56 smithi151 ceph-mon[28965]: pgmap v1681: 67 pgs: 1 active+undersized+remapped, 1 peering, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 743 op/s; 371/64401 objects degraded (0.576%); 12790/64401 objects misplaced (19.860%); 12 MiB/s, 697 objects/s recovering 2024-06-27T21:37:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:56 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:37:57.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:56 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:37:57.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:57.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:57.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:57.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:57.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:57.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:57.821 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:57.821 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:37:57.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:57.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:57.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:57.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:58.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:58.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:58.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:58.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:57 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:37:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:57 smithi190 ceph-mon[32247]: osdmap e667: 12 total, 12 up, 11 in 2024-06-27T21:37:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:57 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:37:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:57 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:37:58.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:37:58.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:57 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:37:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:57 smithi192 ceph-mon[32956]: osdmap e667: 12 total, 12 up, 11 in 2024-06-27T21:37:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:57 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:37:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:57 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:37:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:37:58.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:58.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:57 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:37:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:57 smithi151 ceph-mon[28965]: osdmap e667: 12 total, 12 up, 11 in 2024-06-27T21:37:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:57 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:37:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:57 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:37:58.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:58.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:58.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:37:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.686 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:58.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:37:58.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:58.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:58.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:37:58.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:58.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:37:59.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:37:59.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:37:59.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:37:59.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:37:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:58 smithi190 ceph-mon[32247]: 6.21 scrub starts 2024-06-27T21:37:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:58 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:37:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:58 smithi190 ceph-mon[32247]: pgmap v1683: 67 pgs: 1 active+undersized+remapped, 1 peering, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 444 op/s; 371/62829 objects degraded (0.590%); 12283/62829 objects misplaced (19.550%); 7.5 MiB/s, 478 objects/s recovering 2024-06-27T21:37:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:58 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:37:59.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:37:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:58 smithi192 ceph-mon[32956]: 6.21 scrub starts 2024-06-27T21:37:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:58 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:37:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:58 smithi192 ceph-mon[32956]: pgmap v1683: 67 pgs: 1 active+undersized+remapped, 1 peering, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 444 op/s; 371/62829 objects degraded (0.590%); 12283/62829 objects misplaced (19.550%); 7.5 MiB/s, 478 objects/s recovering 2024-06-27T21:37:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:58 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:37:59.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:37:59.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:58 smithi151 ceph-mon[28965]: 6.21 scrub starts 2024-06-27T21:37:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:58 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:37:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:58 smithi151 ceph-mon[28965]: pgmap v1683: 67 pgs: 1 active+undersized+remapped, 1 peering, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 444 op/s; 371/62829 objects degraded (0.590%); 12283/62829 objects misplaced (19.550%); 7.5 MiB/s, 478 objects/s recovering 2024-06-27T21:37:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:58 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:37:59.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:59.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:37:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:37:59.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:37:59.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:37:59.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:37:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:37:59.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:59.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:59.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:37:59.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:37:59.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:37:59.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:37:59.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:00.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:00.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:00.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:00.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:59 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:38:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:59 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:38:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:59 smithi190 ceph-mon[32247]: 6.14 scrub starts 2024-06-27T21:38:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:59 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 371/62829 objects degraded (0.590%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:38:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:37:59 smithi190 ceph-mon[32247]: osdmap e668: 12 total, 12 up, 11 in 2024-06-27T21:38:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:59 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:38:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:59 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:38:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:59 smithi192 ceph-mon[32956]: 6.14 scrub starts 2024-06-27T21:38:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:59 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 371/62829 objects degraded (0.590%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:38:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:37:59 smithi192 ceph-mon[32956]: osdmap e668: 12 total, 12 up, 11 in 2024-06-27T21:38:00.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:00.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:00.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:37:59 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:38:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:00 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:38:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:00 smithi151 ceph-mon[28965]: 6.14 scrub starts 2024-06-27T21:38:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:00 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 371/62829 objects degraded (0.590%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:38:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:00 smithi151 ceph-mon[28965]: osdmap e668: 12 total, 12 up, 11 in 2024-06-27T21:38:00.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:00.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:00.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:00.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:00.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:00.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:00.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:00.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:00.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:01.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:01.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:01.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:01.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:01.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:01.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:01.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:01.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:01.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:01.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:01.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:01.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:01.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:01.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:01.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:01.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:01.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:01.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:01 smithi192 ceph-mon[32956]: pgmap v1685: 67 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 1.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 371/58071 objects degraded (0.639%); 11633/58071 objects misplaced (20.032%); 7.1 MiB/s, 542 objects/s recovering 2024-06-27T21:38:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:01 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:38:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:01 smithi192 ceph-mon[32956]: osdmap e669: 12 total, 12 up, 11 in 2024-06-27T21:38:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:01 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:38:02.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:01 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:38:02.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:02.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:01 smithi151 ceph-mon[28965]: pgmap v1685: 67 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 1.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 371/58071 objects degraded (0.639%); 11633/58071 objects misplaced (20.032%); 7.1 MiB/s, 542 objects/s recovering 2024-06-27T21:38:02.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:01 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:38:02.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:01 smithi151 ceph-mon[28965]: osdmap e669: 12 total, 12 up, 11 in 2024-06-27T21:38:02.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:01 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:38:02.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:01 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:38:02.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:02.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:01 smithi190 ceph-mon[32247]: pgmap v1685: 67 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 1.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 371/58071 objects degraded (0.639%); 11633/58071 objects misplaced (20.032%); 7.1 MiB/s, 542 objects/s recovering 2024-06-27T21:38:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:01 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:38:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:01 smithi190 ceph-mon[32247]: osdmap e669: 12 total, 12 up, 11 in 2024-06-27T21:38:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:01 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:38:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:01 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:38:02.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:02.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:02.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:02.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:02.523 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:02.625 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:02.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:02.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:02.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:02.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:02.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:02.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:03.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:03.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:03.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:02 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:38:03.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:02 smithi151 ceph-mon[28965]: osdmap e670: 12 total, 12 up, 11 in 2024-06-27T21:38:03.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:02 smithi151 ceph-mon[28965]: 6.23 scrub starts 2024-06-27T21:38:03.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:02 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:38:03.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:02 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:38:03.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:03.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:03.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:02 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:38:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:02 smithi190 ceph-mon[32247]: osdmap e670: 12 total, 12 up, 11 in 2024-06-27T21:38:03.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:02 smithi190 ceph-mon[32247]: 6.23 scrub starts 2024-06-27T21:38:03.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:02 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:38:03.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:02 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:38:03.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:02 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:38:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:02 smithi192 ceph-mon[32956]: osdmap e670: 12 total, 12 up, 11 in 2024-06-27T21:38:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:02 smithi192 ceph-mon[32956]: 6.23 scrub starts 2024-06-27T21:38:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:02 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:38:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:02 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:38:03.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:03.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:03.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:03.529 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:03.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:03.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:03.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:03.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:03.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:03.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:03.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:04.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:04.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:04.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:04.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:03 smithi190 ceph-mon[32247]: pgmap v1688: 67 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 55 active+clean; 1.1 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 144/55662 objects degraded (0.259%); 11132/55662 objects misplaced (19.999%); 2.4 MiB/s, 435 objects/s recovering 2024-06-27T21:38:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:03 smithi190 ceph-mon[32247]: 6.21 scrub starts 2024-06-27T21:38:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:03 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:38:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:03 smithi190 ceph-mon[32247]: osdmap e671: 12 total, 12 up, 11 in 2024-06-27T21:38:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:03 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:38:04.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:03 smithi151 ceph-mon[28965]: pgmap v1688: 67 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 55 active+clean; 1.1 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 144/55662 objects degraded (0.259%); 11132/55662 objects misplaced (19.999%); 2.4 MiB/s, 435 objects/s recovering 2024-06-27T21:38:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:03 smithi151 ceph-mon[28965]: 6.21 scrub starts 2024-06-27T21:38:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:03 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:38:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:03 smithi151 ceph-mon[28965]: osdmap e671: 12 total, 12 up, 11 in 2024-06-27T21:38:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:03 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:38:04.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:03 smithi192 ceph-mon[32956]: pgmap v1688: 67 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 55 active+clean; 1.1 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 144/55662 objects degraded (0.259%); 11132/55662 objects misplaced (19.999%); 2.4 MiB/s, 435 objects/s recovering 2024-06-27T21:38:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:03 smithi192 ceph-mon[32956]: 6.21 scrub starts 2024-06-27T21:38:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:03 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:38:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:03 smithi192 ceph-mon[32956]: osdmap e671: 12 total, 12 up, 11 in 2024-06-27T21:38:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:03 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:38:04.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:04.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:04.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:04.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:04.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:04.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:04.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:04.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:04.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:04.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:04.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:04.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:05.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:05.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:04 smithi151 ceph-mon[28965]: 6.22 scrub starts 2024-06-27T21:38:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:04 smithi151 ceph-mon[28965]: osdmap e672: 12 total, 12 up, 11 in 2024-06-27T21:38:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:04 smithi151 ceph-mon[28965]: 6.20 scrub starts 2024-06-27T21:38:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:04 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:38:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 144/55662 objects degraded (0.259%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:38:05.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:04 smithi151 ceph-mon[28965]: osdmap e673: 12 total, 12 up, 11 in 2024-06-27T21:38:05.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:38:05.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:05.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:04 smithi190 ceph-mon[32247]: 6.22 scrub starts 2024-06-27T21:38:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:04 smithi190 ceph-mon[32247]: osdmap e672: 12 total, 12 up, 11 in 2024-06-27T21:38:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:04 smithi190 ceph-mon[32247]: 6.20 scrub starts 2024-06-27T21:38:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:04 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:38:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 144/55662 objects degraded (0.259%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:38:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:04 smithi190 ceph-mon[32247]: osdmap e673: 12 total, 12 up, 11 in 2024-06-27T21:38:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:04 smithi192 ceph-mon[32956]: 6.22 scrub starts 2024-06-27T21:38:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:04 smithi192 ceph-mon[32956]: osdmap e672: 12 total, 12 up, 11 in 2024-06-27T21:38:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:04 smithi192 ceph-mon[32956]: 6.20 scrub starts 2024-06-27T21:38:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:04 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:38:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 144/55662 objects degraded (0.259%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:38:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:04 smithi192 ceph-mon[32956]: osdmap e673: 12 total, 12 up, 11 in 2024-06-27T21:38:05.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:05.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:05.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:05.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:05.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:05.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:05.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:05.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:05.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:05.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:05.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:05.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:05.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:05 smithi151 ceph-mon[28965]: pgmap v1691: 67 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 57 active+clean; 1.0 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 142/51027 objects degraded (0.278%); 9520/51027 objects misplaced (18.657%); 5.5 MiB/s, 483 objects/s recovering 2024-06-27T21:38:05.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:05 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:38:05.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:05 smithi151 ceph-mon[28965]: 6.14 scrub starts 2024-06-27T21:38:05.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:05.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:05.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:06.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:06.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:06.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:06.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:05 smithi190 ceph-mon[32247]: pgmap v1691: 67 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 57 active+clean; 1.0 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 142/51027 objects degraded (0.278%); 9520/51027 objects misplaced (18.657%); 5.5 MiB/s, 483 objects/s recovering 2024-06-27T21:38:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:05 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:38:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:05 smithi190 ceph-mon[32247]: 6.14 scrub starts 2024-06-27T21:38:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:05 smithi192 ceph-mon[32956]: pgmap v1691: 67 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 57 active+clean; 1.0 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 142/51027 objects degraded (0.278%); 9520/51027 objects misplaced (18.657%); 5.5 MiB/s, 483 objects/s recovering 2024-06-27T21:38:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:05 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:38:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:05 smithi192 ceph-mon[32956]: 6.14 scrub starts 2024-06-27T21:38:06.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:06.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:06.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:06.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:06.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:06.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:06.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:06.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:06.752 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:06.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:06.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:06.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:06.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:07.017 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:07.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:07.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:06 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:38:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:06 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:38:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:06 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:38:07.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:06 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:38:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:06 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:38:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:06 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:38:07.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:06 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:38:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:06 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:38:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:06 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:38:07.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:07.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:07.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:07.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:07.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:07.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:07.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:07.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:07.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:07.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:07.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:07 smithi151 ceph-mon[28965]: pgmap v1693: 67 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 993 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 7/47661 objects degraded (0.015%); 8968/47661 objects misplaced (18.816%); 5.1 MiB/s, 328 objects/s recovering 2024-06-27T21:38:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:07 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:38:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:07 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:38:07.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:07 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:38:08.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:08.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:08.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:08.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:07 smithi190 ceph-mon[32247]: pgmap v1693: 67 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 993 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 7/47661 objects degraded (0.015%); 8968/47661 objects misplaced (18.816%); 5.1 MiB/s, 328 objects/s recovering 2024-06-27T21:38:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:07 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:38:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:07 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:38:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:07 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:38:08.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:07 smithi192 ceph-mon[32956]: pgmap v1693: 67 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 993 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 609 op/s; 7/47661 objects degraded (0.015%); 8968/47661 objects misplaced (18.816%); 5.1 MiB/s, 328 objects/s recovering 2024-06-27T21:38:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:07 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:38:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:07 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:38:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:07 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:38:08.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:08.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:08.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:08.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:08.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:08.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:08.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:08.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:08.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:08.848 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:08.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:08.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:08.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:08.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:08 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:38:08.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:08 smithi151 ceph-mon[28965]: osdmap e674: 12 total, 12 up, 11 in 2024-06-27T21:38:08.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:08 smithi151 ceph-mon[28965]: 6.21 scrub starts 2024-06-27T21:38:09.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:09.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:09.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:09.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:09.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:08 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:38:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:08 smithi190 ceph-mon[32247]: osdmap e674: 12 total, 12 up, 11 in 2024-06-27T21:38:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:08 smithi190 ceph-mon[32247]: 6.21 scrub starts 2024-06-27T21:38:09.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:08 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:38:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:08 smithi192 ceph-mon[32956]: osdmap e674: 12 total, 12 up, 11 in 2024-06-27T21:38:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:08 smithi192 ceph-mon[32956]: 6.21 scrub starts 2024-06-27T21:38:09.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:09.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:09.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:09.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:09.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:09.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:09.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:09.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:09.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:09.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:38:09.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:38:09.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:38:09.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:38:09.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:38:09.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:38:09.873 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:38:09.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:09.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:09.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:10.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:10.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:10.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:09 smithi151 ceph-mon[28965]: pgmap v1695: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 943 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 7/45261 objects degraded (0.015%); 7910/45261 objects misplaced (17.476%); 6.7 MiB/s, 191 objects/s recovering 2024-06-27T21:38:10.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:09 smithi151 ceph-mon[28965]: 6.23 scrub starts 2024-06-27T21:38:10.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:09 smithi151 ceph-mon[28965]: osdmap e675: 12 total, 12 up, 11 in 2024-06-27T21:38:10.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4261461232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:10.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:09 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7/45261 objects degraded (0.015%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:10.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:10.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:09 smithi190 ceph-mon[32247]: pgmap v1695: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 943 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 7/45261 objects degraded (0.015%); 7910/45261 objects misplaced (17.476%); 6.7 MiB/s, 191 objects/s recovering 2024-06-27T21:38:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:09 smithi190 ceph-mon[32247]: 6.23 scrub starts 2024-06-27T21:38:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:09 smithi190 ceph-mon[32247]: osdmap e675: 12 total, 12 up, 11 in 2024-06-27T21:38:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4261461232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:09 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7/45261 objects degraded (0.015%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:09 smithi192 ceph-mon[32956]: pgmap v1695: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 943 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 7/45261 objects degraded (0.015%); 7910/45261 objects misplaced (17.476%); 6.7 MiB/s, 191 objects/s recovering 2024-06-27T21:38:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:09 smithi192 ceph-mon[32956]: 6.23 scrub starts 2024-06-27T21:38:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:09 smithi192 ceph-mon[32956]: osdmap e675: 12 total, 12 up, 11 in 2024-06-27T21:38:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4261461232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:09 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7/45261 objects degraded (0.015%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:10.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:10.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:10.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:10.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:10.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:10.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:10.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:10.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:10.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:10.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:10.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:10 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:38:10.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:10 smithi151 ceph-mon[28965]: 6.22 scrub starts 2024-06-27T21:38:10.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:10 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:38:10.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:10 smithi151 ceph-mon[28965]: 6.14 scrub starts 2024-06-27T21:38:10.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:10.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:10.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:11.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:11.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:11.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:11.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:10 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:38:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:10 smithi190 ceph-mon[32247]: 6.22 scrub starts 2024-06-27T21:38:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:10 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:38:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:10 smithi190 ceph-mon[32247]: 6.14 scrub starts 2024-06-27T21:38:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:10 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:38:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:10 smithi192 ceph-mon[32956]: 6.22 scrub starts 2024-06-27T21:38:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:10 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:38:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:10 smithi192 ceph-mon[32956]: 6.14 scrub starts 2024-06-27T21:38:11.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:11.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:11.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:11.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:11.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:11.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:11.722 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:38:11.722 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:38:11.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:11.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:11.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:11.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:11.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:11.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:12.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:12.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:12.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:11 smithi151 ceph-mon[28965]: pgmap v1697: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 810 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 7/38883 objects degraded (0.018%); 7312/38883 objects misplaced (18.805%); 4.8 MiB/s, 111 objects/s recovering 2024-06-27T21:38:12.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:11 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:38:12.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:11 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:38:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:11 smithi151 ceph-mon[28965]: 6.1e scrub starts 2024-06-27T21:38:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:11 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:38:12.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:12.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:11 smithi190 ceph-mon[32247]: pgmap v1697: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 810 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 7/38883 objects degraded (0.018%); 7312/38883 objects misplaced (18.805%); 4.8 MiB/s, 111 objects/s recovering 2024-06-27T21:38:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:11 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:38:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:11 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:38:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:11 smithi190 ceph-mon[32247]: 6.1e scrub starts 2024-06-27T21:38:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:11 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:38:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:11 smithi192 ceph-mon[32956]: pgmap v1697: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 810 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 7/38883 objects degraded (0.018%); 7312/38883 objects misplaced (18.805%); 4.8 MiB/s, 111 objects/s recovering 2024-06-27T21:38:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:11 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:38:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:11 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:38:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:11 smithi192 ceph-mon[32956]: 6.1e scrub starts 2024-06-27T21:38:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:11 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:38:12.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:12.345 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:12.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:12.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:12.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:12.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:12.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:38:12.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:12.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:12.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:12.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:12.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:12 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:38:12.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:12 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:38:12.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:12 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:38:12.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:12.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:12.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:13.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:13.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:13.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:13.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:12 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:38:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:12 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:38:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:12 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:38:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:12 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:38:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:12 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:38:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:12 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:38:13.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:13.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:13.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:13.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:13.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:13.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:13.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:13.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:13.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:13.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:13.885 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:38:13.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:13.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:13.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:14.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:13 smithi151 ceph-mon[28965]: pgmap v1698: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 728 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 721 op/s; 7/34947 objects degraded (0.020%); 5989/34947 objects misplaced (17.137%); 5.4 MiB/s, 114 objects/s recovering 2024-06-27T21:38:14.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3919314626' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:38:14.031 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:13 smithi151 ceph-mon[28965]: 6.20 scrub starts 2024-06-27T21:38:14.033 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:38:14.033 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:38:14.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:14.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:14.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:14.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:13 smithi190 ceph-mon[32247]: pgmap v1698: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 728 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 721 op/s; 7/34947 objects degraded (0.020%); 5989/34947 objects misplaced (17.137%); 5.4 MiB/s, 114 objects/s recovering 2024-06-27T21:38:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3919314626' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:38:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:13 smithi190 ceph-mon[32247]: 6.20 scrub starts 2024-06-27T21:38:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:13 smithi192 ceph-mon[32956]: pgmap v1698: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 59 active+clean; 728 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 721 op/s; 7/34947 objects degraded (0.020%); 5989/34947 objects misplaced (17.137%); 5.4 MiB/s, 114 objects/s recovering 2024-06-27T21:38:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3919314626' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:38:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:13 smithi192 ceph-mon[32956]: 6.20 scrub starts 2024-06-27T21:38:14.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:14.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:14.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:14.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:14.466 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:14.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:14.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:14.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:14.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:14.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:14.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:14.875 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1699,"stamp":"2024-06-27T21:38:13.884170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":677249592,"num_objects":10329,"num_object_clones":0,"num_object_copies":30987,"num_objects_missing_on_primary":728,"num_objects_missing":728,"num_objects_degraded":7,"num_objects_misplaced":3596,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118478,"num_write_kb":3243713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64483,"num_bytes_recovered":2684195408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400442,"ondisk_log_size":400442,"up":201,"acting":199,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":230,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":8622068,"kb_used_data":3175108,"kb_used_omap":108,"kb_used_meta":5446739,"kb_avail":1022574604,"statfs":{"total":1055945392128,"available":1047116394496,"internally_reserved":0,"allocated":3251310592,"data_stored":3241740776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111086,"internal_metadata":5577461266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,2,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-364249088,"num_objects":-5558,"num_object_clones":0,"num_object_copies":-16674,"num_objects_missing_on_primary":-418,"num_objects_missing":-418,"num_objects_degraded":0,"num_objects_misplaced":-5372,"num_objects_unfound":0,"num_objects_dirty":-5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1594,"num_bytes_recovered":90308608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.696666"},"pg_stats":[{"pgid":"6.23","version":"675'11138","reported_seq":28521,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.553184+0000","last_change":"2024-06-27T21:38:08.870930+0000","last_active":"2024-06-27T21:38:10.553184+0000","last_peered":"2024-06-27T21:38:10.553184+0000","last_clean":"2024-06-27T21:38:10.553184+0000","last_became_active":"2024-06-27T21:38:00.770709+0000","last_became_peered":"2024-06-27T21:38:00.770709+0000","last_unstale":"2024-06-27T21:38:10.553184+0000","last_undegraded":"2024-06-27T21:38:10.553184+0000","last_fullsized":"2024-06-27T21:38:10.553184+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":6,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":0,"log_size":11138,"log_dups_size":0,"ondisk_log_size":11138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.766342+0000","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":109040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4210,"num_bytes_recovered":140476416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,9],"acting":[8,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.22","version":"675'10850","reported_seq":18109,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.572301+0000","last_change":"2024-06-27T21:38:09.798549+0000","last_active":"2024-06-27T21:38:10.572301+0000","last_peered":"2024-06-27T21:38:10.572301+0000","last_clean":"2024-06-27T21:38:10.572301+0000","last_became_active":"2024-06-27T21:38:02.828098+0000","last_became_peered":"2024-06-27T21:38:02.828098+0000","last_unstale":"2024-06-27T21:38:10.572301+0000","last_undegraded":"2024-06-27T21:38:10.572301+0000","last_fullsized":"2024-06-27T21:38:10.572301+0000","mapping_epoch":670,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":671,"parent":"0.0","parent_split_bits":6,"last_scrub":"641'10427","last_scrub_stamp":"2024-06-27T21:37:12.279210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:37:12.279210+0000","objects_scrubbed":0,"log_size":10850,"log_dups_size":0,"ondisk_log_size":10850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:14.767633+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10911744,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1664,"num_bytes_recovered":30736384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.21","version":"675'10701","reported_seq":31559,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.309718+0000","last_change":"2024-06-27T21:38:08.661413+0000","last_active":"2024-06-27T21:38:10.309718+0000","last_peered":"2024-06-27T21:38:10.309718+0000","last_clean":"2024-06-27T21:38:10.309718+0000","last_became_active":"2024-06-27T21:37:56.997169+0000","last_became_peered":"2024-06-27T21:37:56.997169+0000","last_unstale":"2024-06-27T21:38:10.309718+0000","last_undegraded":"2024-06-27T21:38:10.309718+0000","last_fullsized":"2024-06-27T21:38:10.309718+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":6,"last_scrub":"629'10170","last_scrub_stamp":"2024-06-27T21:36:39.166411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:39.166411+0000","objects_scrubbed":0,"log_size":10701,"log_dups_size":0,"ondisk_log_size":10701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.394191+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437184,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2667,"num_bytes_recovered":66289664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.20","version":"675'11022","reported_seq":30120,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.874684+0000","last_change":"2024-06-27T21:38:04.541780+0000","last_active":"2024-06-27T21:38:10.874684+0000","last_peered":"2024-06-27T21:38:10.874684+0000","last_clean":"2024-06-27T21:38:10.874684+0000","last_became_active":"2024-06-27T21:38:01.822089+0000","last_became_peered":"2024-06-27T21:38:01.822089+0000","last_unstale":"2024-06-27T21:38:10.874684+0000","last_undegraded":"2024-06-27T21:38:10.874684+0000","last_fullsized":"2024-06-27T21:38:10.874684+0000","mapping_epoch":669,"log_start":"0'0","ondisk_log_start":"0'0","created":651,"last_epoch_clean":670,"parent":"0.0","parent_split_bits":6,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":0,"log_size":11022,"log_dups_size":0,"ondisk_log_size":11022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9338880,"num_objects":143,"num_object_clones":0,"num_object_copies":429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":714,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1697,"num_bytes_recovered":32178176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,6],"acting":[2,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":446,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.879866+0000","last_change":"2024-06-27T21:32:59.918488+0000","last_active":"2024-06-27T21:38:08.879866+0000","last_peered":"2024-06-27T21:38:08.879866+0000","last_clean":"2024-06-27T21:38:08.879866+0000","last_became_active":"2024-06-27T21:32:59.704835+0000","last_became_peered":"2024-06-27T21:32:59.704835+0000","last_unstale":"2024-06-27T21:38:08.879866+0000","last_undegraded":"2024-06-27T21:38:08.879866+0000","last_fullsized":"2024-06-27T21:38:08.879866+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:32:59.918366+0000","last_clean_scrub_stamp":"2024-06-27T21:32:59.918366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.c","version":"675'12574","reported_seq":28387,"reported_epoch":675,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:38:13.782342+0000","last_change":"2024-06-27T21:37:38.338435+0000","last_active":"2024-06-27T21:38:13.782342+0000","last_peered":"2024-06-27T21:38:13.782342+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:37:35.494414+0000","last_became_peered":"2024-06-27T21:37:35.494414+0000","last_unstale":"2024-06-27T21:38:13.782342+0000","last_undegraded":"2024-06-27T21:37:34.978548+0000","last_fullsized":"2024-06-27T21:37:33.889065+0000","mapping_epoch":633,"log_start":"478'1900","ondisk_log_start":"478'1900","created":467,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10674,"log_dups_size":0,"ondisk_log_size":10674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726336,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":728,"num_objects_missing":728,"num_objects_degraded":4,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6611,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":178356224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[7,8],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6,7,8","objects":1},{"shards":"5,6,8","objects":51},{"shards":"7,8","objects":245}],"blocked_by":[],"up_primary":8,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":448,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.911212+0000","last_change":"2024-06-27T21:33:00.619610+0000","last_active":"2024-06-27T21:38:08.911212+0000","last_peered":"2024-06-27T21:38:08.911212+0000","last_clean":"2024-06-27T21:38:08.911212+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:38:08.911212+0000","last_undegraded":"2024-06-27T21:38:08.911212+0000","last_fullsized":"2024-06-27T21:38:08.911212+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.619537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.619537+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"675'11951","reported_seq":17690,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.856969+0000","last_change":"2024-06-27T21:38:10.277211+0000","last_active":"2024-06-27T21:38:10.856969+0000","last_peered":"2024-06-27T21:38:10.856969+0000","last_clean":"2024-06-27T21:38:10.856969+0000","last_became_active":"2024-06-27T21:37:33.902153+0000","last_became_peered":"2024-06-27T21:37:33.902153+0000","last_unstale":"2024-06-27T21:38:10.856969+0000","last_undegraded":"2024-06-27T21:38:10.856969+0000","last_fullsized":"2024-06-27T21:38:10.856969+0000","mapping_epoch":612,"log_start":"475'1300","ondisk_log_start":"475'1300","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"596'10541","last_scrub_stamp":"2024-06-27T21:35:43.673315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:43.673315+0000","objects_scrubbed":0,"log_size":10651,"log_dups_size":0,"ondisk_log_size":10651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:15.204078+0000","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6297,"num_write_kb":210944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1278,"num_bytes_recovered":83755008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"675'12154","reported_seq":26102,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.451810+0000","last_change":"2024-06-27T21:38:10.411714+0000","last_active":"2024-06-27T21:38:10.451810+0000","last_peered":"2024-06-27T21:38:10.451810+0000","last_clean":"2024-06-27T21:38:10.451810+0000","last_became_active":"2024-06-27T21:37:33.895558+0000","last_became_peered":"2024-06-27T21:37:33.895558+0000","last_unstale":"2024-06-27T21:38:10.451810+0000","last_undegraded":"2024-06-27T21:38:10.451810+0000","last_fullsized":"2024-06-27T21:38:10.451810+0000","mapping_epoch":545,"log_start":"475'1500","ondisk_log_start":"475'1500","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":0,"log_size":10654,"log_dups_size":0,"ondisk_log_size":10654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:15.410180+0000","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21004288,"num_objects":320,"num_object_clones":0,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6378,"num_write_kb":214368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3349,"num_bytes_recovered":170098688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":444,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.878193+0000","last_change":"2024-06-27T21:36:51.821471+0000","last_active":"2024-06-27T21:38:08.878193+0000","last_peered":"2024-06-27T21:38:08.878193+0000","last_clean":"2024-06-27T21:38:08.878193+0000","last_became_active":"2024-06-27T21:36:51.820882+0000","last_became_peered":"2024-06-27T21:36:51.820882+0000","last_unstale":"2024-06-27T21:38:08.878193+0000","last_undegraded":"2024-06-27T21:38:08.878193+0000","last_fullsized":"2024-06-27T21:38:08.878193+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.399195+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.399195+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,8],"acting":[7,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"675'11319","reported_seq":25691,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.394950+0000","last_change":"2024-06-27T21:37:25.782477+0000","last_active":"2024-06-27T21:38:10.394950+0000","last_peered":"2024-06-27T21:38:10.394950+0000","last_clean":"2024-06-27T21:38:10.394950+0000","last_became_active":"2024-06-27T21:37:25.766164+0000","last_became_peered":"2024-06-27T21:37:25.766164+0000","last_unstale":"2024-06-27T21:38:10.394950+0000","last_undegraded":"2024-06-27T21:38:10.394950+0000","last_fullsized":"2024-06-27T21:38:10.394950+0000","mapping_epoch":645,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9854","last_scrub_stamp":"2024-06-27T21:34:00.348580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:00.348580+0000","objects_scrubbed":1790,"log_size":11319,"log_dups_size":0,"ondisk_log_size":11319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3428,"num_bytes_recovered":70647808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":440,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.915871+0000","last_change":"2024-06-27T21:33:01.270489+0000","last_active":"2024-06-27T21:38:08.915871+0000","last_peered":"2024-06-27T21:38:08.915871+0000","last_clean":"2024-06-27T21:38:08.915871+0000","last_became_active":"2024-06-27T21:32:59.706135+0000","last_became_peered":"2024-06-27T21:32:59.706135+0000","last_unstale":"2024-06-27T21:38:08.915871+0000","last_undegraded":"2024-06-27T21:38:08.915871+0000","last_fullsized":"2024-06-27T21:38:08.915871+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.270391+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.270391+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"675'11424","reported_seq":11977,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.529162+0000","last_change":"2024-06-27T21:33:49.851182+0000","last_active":"2024-06-27T21:38:10.529162+0000","last_peered":"2024-06-27T21:38:10.529162+0000","last_clean":"2024-06-27T21:38:10.529162+0000","last_became_active":"2024-06-27T21:33:25.736643+0000","last_became_peered":"2024-06-27T21:33:25.736643+0000","last_unstale":"2024-06-27T21:38:10.529162+0000","last_undegraded":"2024-06-27T21:38:10.529162+0000","last_fullsized":"2024-06-27T21:38:10.529162+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"557'9962","last_scrub_stamp":"2024-06-27T21:33:49.850981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:49.850981+0000","objects_scrubbed":1778,"log_size":11424,"log_dups_size":0,"ondisk_log_size":11424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1462,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1052,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.872553+0000","last_change":"2024-06-27T21:35:47.736735+0000","last_active":"2024-06-27T21:38:08.872553+0000","last_peered":"2024-06-27T21:38:08.872553+0000","last_clean":"2024-06-27T21:38:08.872553+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:38:08.872553+0000","last_undegraded":"2024-06-27T21:38:08.872553+0000","last_fullsized":"2024-06-27T21:38:08.872553+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.736633+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.736633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.422437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"675'11149","reported_seq":11767,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.544992+0000","last_change":"2024-06-27T21:36:05.559816+0000","last_active":"2024-06-27T21:38:10.544992+0000","last_peered":"2024-06-27T21:38:10.544992+0000","last_clean":"2024-06-27T21:38:10.544992+0000","last_became_active":"2024-06-27T21:33:25.697304+0000","last_became_peered":"2024-06-27T21:33:25.697304+0000","last_unstale":"2024-06-27T21:38:10.544992+0000","last_undegraded":"2024-06-27T21:38:10.544992+0000","last_fullsized":"2024-06-27T21:38:10.544992+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'10215","last_scrub_stamp":"2024-06-27T21:36:05.559677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:05.559677+0000","objects_scrubbed":1276,"log_size":11149,"log_dups_size":0,"ondisk_log_size":11149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.422462+0000","scrub_duration":728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1390,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":472,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.877952+0000","last_change":"2024-06-27T21:35:47.454903+0000","last_active":"2024-06-27T21:38:08.877952+0000","last_peered":"2024-06-27T21:38:08.877952+0000","last_clean":"2024-06-27T21:38:08.877952+0000","last_became_active":"2024-06-27T21:35:46.856358+0000","last_became_peered":"2024-06-27T21:35:46.856358+0000","last_unstale":"2024-06-27T21:38:08.877952+0000","last_undegraded":"2024-06-27T21:38:08.877952+0000","last_fullsized":"2024-06-27T21:38:08.877952+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.454820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.454820+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"675'11732","reported_seq":15189,"reported_epoch":675,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:38:10.840418+0000","last_change":"2024-06-27T21:38:03.987479+0000","last_active":"2024-06-27T21:38:10.840418+0000","last_peered":"2024-06-27T21:38:10.840418+0000","last_clean":"2024-06-27T21:36:50.700367+0000","last_became_active":"2024-06-27T21:36:52.921052+0000","last_became_peered":"2024-06-27T21:36:52.921052+0000","last_unstale":"2024-06-27T21:38:10.840418+0000","last_undegraded":"2024-06-27T21:38:10.840418+0000","last_fullsized":"2024-06-27T21:38:10.840418+0000","mapping_epoch":633,"log_start":"470'100","ondisk_log_start":"470'100","created":467,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10731","last_scrub_stamp":"2024-06-27T21:36:00.782086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:00.782086+0000","objects_scrubbed":1308,"log_size":11632,"log_dups_size":0,"ondisk_log_size":11632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20119552,"num_objects":307,"num_object_clones":0,"num_object_copies":921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":91488256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,11],"avail_no_missing":["5","9","11"],"object_location_counts":[{"shards":"5,9,11","objects":307}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":420,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.915042+0000","last_change":"2024-06-27T21:36:51.808599+0000","last_active":"2024-06-27T21:38:08.915042+0000","last_peered":"2024-06-27T21:38:08.915042+0000","last_clean":"2024-06-27T21:38:08.915042+0000","last_became_active":"2024-06-27T21:36:51.808187+0000","last_became_peered":"2024-06-27T21:36:51.808187+0000","last_unstale":"2024-06-27T21:38:08.915042+0000","last_undegraded":"2024-06-27T21:38:08.915042+0000","last_fullsized":"2024-06-27T21:38:08.915042+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:30:43.212538+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:30:43.212538+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10],"acting":[4,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"675'11251","reported_seq":21599,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.590493+0000","last_change":"2024-06-27T21:36:08.622320+0000","last_active":"2024-06-27T21:38:10.590493+0000","last_peered":"2024-06-27T21:38:10.590493+0000","last_clean":"2024-06-27T21:38:10.590493+0000","last_became_active":"2024-06-27T21:33:26.702107+0000","last_became_peered":"2024-06-27T21:33:26.702107+0000","last_unstale":"2024-06-27T21:38:10.590493+0000","last_undegraded":"2024-06-27T21:38:10.590493+0000","last_fullsized":"2024-06-27T21:38:10.590493+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":612,"parent":"0.0","parent_split_bits":0,"last_scrub":"613'10314","last_scrub_stamp":"2024-06-27T21:36:08.622226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:08.622226+0000","objects_scrubbed":1220,"log_size":11251,"log_dups_size":0,"ondisk_log_size":11251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":938,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":524,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.877972+0000","last_change":"2024-06-27T21:31:23.047316+0000","last_active":"2024-06-27T21:38:08.877972+0000","last_peered":"2024-06-27T21:38:08.877972+0000","last_clean":"2024-06-27T21:38:08.877972+0000","last_became_active":"2024-06-27T21:29:49.667973+0000","last_became_peered":"2024-06-27T21:29:49.667973+0000","last_unstale":"2024-06-27T21:38:08.877972+0000","last_undegraded":"2024-06-27T21:38:08.877972+0000","last_fullsized":"2024-06-27T21:38:08.877972+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:23.047226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:23.047226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"675'11291","reported_seq":25651,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.664054+0000","last_change":"2024-06-27T21:33:50.968993+0000","last_active":"2024-06-27T21:38:10.664054+0000","last_peered":"2024-06-27T21:38:10.664054+0000","last_clean":"2024-06-27T21:38:10.664054+0000","last_became_active":"2024-06-27T21:33:25.692464+0000","last_became_peered":"2024-06-27T21:33:25.692464+0000","last_unstale":"2024-06-27T21:38:10.664054+0000","last_undegraded":"2024-06-27T21:38:10.664054+0000","last_fullsized":"2024-06-27T21:38:10.664054+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'9882","last_scrub_stamp":"2024-06-27T21:33:50.968852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.968852+0000","objects_scrubbed":1716,"log_size":11291,"log_dups_size":0,"ondisk_log_size":11291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20119552,"num_objects":307,"num_object_clones":0,"num_object_copies":921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":849,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.872685+0000","last_change":"2024-06-27T21:31:22.940108+0000","last_active":"2024-06-27T21:38:08.872685+0000","last_peered":"2024-06-27T21:38:08.872685+0000","last_clean":"2024-06-27T21:38:08.872685+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:38:08.872685+0000","last_undegraded":"2024-06-27T21:38:08.872685+0000","last_fullsized":"2024-06-27T21:38:08.872685+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:21.108620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:21.108620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"675'10982","reported_seq":25001,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.436528+0000","last_change":"2024-06-27T21:38:06.299038+0000","last_active":"2024-06-27T21:38:10.436528+0000","last_peered":"2024-06-27T21:38:10.436528+0000","last_clean":"2024-06-27T21:38:10.436528+0000","last_became_active":"2024-06-27T21:37:33.899373+0000","last_became_peered":"2024-06-27T21:37:33.899373+0000","last_unstale":"2024-06-27T21:38:10.436528+0000","last_undegraded":"2024-06-27T21:38:10.436528+0000","last_fullsized":"2024-06-27T21:38:10.436528+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'9930","last_scrub_stamp":"2024-06-27T21:33:54.882983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:54.882983+0000","objects_scrubbed":0,"log_size":10982,"log_dups_size":0,"ondisk_log_size":10982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:11.275013+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,10],"acting":[0,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":771,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.873127+0000","last_change":"2024-06-27T21:33:02.446000+0000","last_active":"2024-06-27T21:38:08.873127+0000","last_peered":"2024-06-27T21:38:08.873127+0000","last_clean":"2024-06-27T21:38:08.873127+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:38:08.873127+0000","last_undegraded":"2024-06-27T21:38:08.873127+0000","last_fullsized":"2024-06-27T21:38:08.873127+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.445914+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.445914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"675'10669","reported_seq":26658,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.558770+0000","last_change":"2024-06-27T21:38:07.823120+0000","last_active":"2024-06-27T21:38:10.558770+0000","last_peered":"2024-06-27T21:38:10.558770+0000","last_clean":"2024-06-27T21:38:10.558770+0000","last_became_active":"2024-06-27T21:37:38.921376+0000","last_became_peered":"2024-06-27T21:37:38.921376+0000","last_unstale":"2024-06-27T21:38:10.558770+0000","last_undegraded":"2024-06-27T21:38:10.558770+0000","last_fullsized":"2024-06-27T21:38:10.558770+0000","mapping_epoch":656,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":657,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'10170","last_scrub_stamp":"2024-06-27T21:36:39.166411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:39.166411+0000","objects_scrubbed":0,"log_size":10669,"log_dups_size":0,"ondisk_log_size":10669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:12.804851+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11534336,"num_objects":176,"num_object_clones":0,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1582,"num_bytes_recovered":35356672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":856,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.872914+0000","last_change":"2024-06-27T21:36:51.810036+0000","last_active":"2024-06-27T21:38:08.872914+0000","last_peered":"2024-06-27T21:38:08.872914+0000","last_clean":"2024-06-27T21:38:08.872914+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:38:08.872914+0000","last_undegraded":"2024-06-27T21:38:08.872914+0000","last_fullsized":"2024-06-27T21:38:08.872914+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.182586+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.182586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.2","version":"675'10878","reported_seq":11490,"reported_epoch":675,"state":"active+clean+scrubbing","last_fresh":"2024-06-27T21:38:12.528168+0000","last_change":"2024-06-27T21:38:12.527052+0000","last_active":"2024-06-27T21:38:12.528168+0000","last_peered":"2024-06-27T21:38:12.528168+0000","last_clean":"2024-06-27T21:38:12.528168+0000","last_became_active":"2024-06-27T21:37:33.895202+0000","last_became_peered":"2024-06-27T21:37:33.895202+0000","last_unstale":"2024-06-27T21:38:12.528168+0000","last_undegraded":"2024-06-27T21:38:12.528168+0000","last_fullsized":"2024-06-27T21:38:12.528168+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"641'10427","last_scrub_stamp":"2024-06-27T21:37:12.279210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:37:12.279210+0000","objects_scrubbed":0,"log_size":10878,"log_dups_size":0,"ondisk_log_size":10878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9076736,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":744,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,11],"acting":[2,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":855,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.872973+0000","last_change":"2024-06-27T21:33:00.224160+0000","last_active":"2024-06-27T21:38:08.872973+0000","last_peered":"2024-06-27T21:38:08.872973+0000","last_clean":"2024-06-27T21:38:08.872973+0000","last_became_active":"2024-06-27T21:32:59.706075+0000","last_became_peered":"2024-06-27T21:32:59.706075+0000","last_unstale":"2024-06-27T21:38:08.872973+0000","last_undegraded":"2024-06-27T21:38:08.872973+0000","last_fullsized":"2024-06-27T21:38:08.872973+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.224085+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.3","version":"675'11136","reported_seq":23693,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.392922+0000","last_change":"2024-06-27T21:38:10.451021+0000","last_active":"2024-06-27T21:38:10.392922+0000","last_peered":"2024-06-27T21:38:10.392922+0000","last_clean":"2024-06-27T21:38:10.392922+0000","last_became_active":"2024-06-27T21:37:34.762003+0000","last_became_peered":"2024-06-27T21:37:34.762003+0000","last_unstale":"2024-06-27T21:38:10.392922+0000","last_undegraded":"2024-06-27T21:38:10.392922+0000","last_fullsized":"2024-06-27T21:38:10.392922+0000","mapping_epoch":652,"log_start":"0'0","ondisk_log_start":"0'0","created":467,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'6352","last_scrub_stamp":"2024-06-27T21:30:51.726885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:51.726885+0000","objects_scrubbed":0,"log_size":11136,"log_dups_size":0,"ondisk_log_size":11136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:15.393003+0000","scrub_duration":1524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12124160,"num_objects":185,"num_object_clones":0,"num_object_copies":555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3222,"num_write_kb":109040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2504,"num_bytes_recovered":110264320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":1284,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.880883+0000","last_change":"2024-06-27T21:36:51.809130+0000","last_active":"2024-06-27T21:38:08.880883+0000","last_peered":"2024-06-27T21:38:08.880883+0000","last_clean":"2024-06-27T21:38:08.880883+0000","last_became_active":"2024-06-27T21:36:51.808538+0000","last_became_peered":"2024-06-27T21:36:51.808538+0000","last_unstale":"2024-06-27T21:38:08.880883+0000","last_undegraded":"2024-06-27T21:38:08.880883+0000","last_fullsized":"2024-06-27T21:38:08.880883+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.992880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.992880+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.d","version":"675'12199","reported_seq":31632,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:09.980450+0000","last_change":"2024-06-27T21:38:07.251827+0000","last_active":"2024-06-27T21:38:09.980450+0000","last_peered":"2024-06-27T21:38:09.980450+0000","last_clean":"2024-06-27T21:38:09.980450+0000","last_became_active":"2024-06-27T21:37:33.901599+0000","last_became_peered":"2024-06-27T21:37:33.901599+0000","last_unstale":"2024-06-27T21:38:09.980450+0000","last_undegraded":"2024-06-27T21:38:09.980450+0000","last_fullsized":"2024-06-27T21:38:09.980450+0000","mapping_epoch":612,"log_start":"475'1500","ondisk_log_start":"475'1500","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":0,"log_size":10699,"log_dups_size":0,"ondisk_log_size":10699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:12.236300+0000","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003148,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":216032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4717,"num_bytes_recovered":254509080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":360,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.915519+0000","last_change":"2024-06-27T21:33:00.278303+0000","last_active":"2024-06-27T21:38:08.915519+0000","last_peered":"2024-06-27T21:38:08.915519+0000","last_clean":"2024-06-27T21:38:08.915519+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:38:08.915519+0000","last_undegraded":"2024-06-27T21:38:08.915519+0000","last_fullsized":"2024-06-27T21:38:08.915519+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.278200+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.278200+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"675'12325","reported_seq":18173,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:11.535857+0000","last_change":"2024-06-27T21:38:11.730204+0000","last_active":"2024-06-27T21:38:11.535857+0000","last_peered":"2024-06-27T21:38:11.535857+0000","last_clean":"2024-06-27T21:38:11.535857+0000","last_became_active":"2024-06-27T21:37:33.897376+0000","last_became_peered":"2024-06-27T21:37:33.897376+0000","last_unstale":"2024-06-27T21:38:11.535857+0000","last_undegraded":"2024-06-27T21:38:11.535857+0000","last_fullsized":"2024-06-27T21:38:11.535857+0000","mapping_epoch":612,"log_start":"475'1600","ondisk_log_start":"475'1600","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"628'11257","last_scrub_stamp":"2024-06-27T21:36:36.394150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:36.394150+0000","objects_scrubbed":0,"log_size":10725,"log_dups_size":0,"ondisk_log_size":10725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:16.535929+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6475,"num_write_kb":216896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1316,"num_bytes_recovered":86278144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":517,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.877798+0000","last_change":"2024-06-27T21:33:00.406870+0000","last_active":"2024-06-27T21:38:08.877798+0000","last_peered":"2024-06-27T21:38:08.877798+0000","last_clean":"2024-06-27T21:38:08.877798+0000","last_became_active":"2024-06-27T21:32:59.708225+0000","last_became_peered":"2024-06-27T21:32:59.708225+0000","last_unstale":"2024-06-27T21:38:08.877798+0000","last_undegraded":"2024-06-27T21:38:08.877798+0000","last_fullsized":"2024-06-27T21:38:08.877798+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:00.406719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:00.406719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"675'12352","reported_seq":12917,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.519730+0000","last_change":"2024-06-27T21:38:09.530900+0000","last_active":"2024-06-27T21:38:10.519730+0000","last_peered":"2024-06-27T21:38:10.519730+0000","last_clean":"2024-06-27T21:38:10.519730+0000","last_became_active":"2024-06-27T21:37:33.920352+0000","last_became_peered":"2024-06-27T21:37:33.920352+0000","last_unstale":"2024-06-27T21:38:10.519730+0000","last_undegraded":"2024-06-27T21:38:10.519730+0000","last_fullsized":"2024-06-27T21:38:10.519730+0000","mapping_epoch":467,"log_start":"477'1700","ondisk_log_start":"477'1700","created":467,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":0,"last_scrub":"629'11283","last_scrub_stamp":"2024-06-27T21:36:37.373008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:37.373008+0000","objects_scrubbed":0,"log_size":10652,"log_dups_size":0,"ondisk_log_size":10652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:14.516466+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6496,"num_write_kb":217152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":744,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.873560+0000","last_change":"2024-06-27T21:37:09.129218+0000","last_active":"2024-06-27T21:38:08.873560+0000","last_peered":"2024-06-27T21:38:08.873560+0000","last_clean":"2024-06-27T21:38:08.873560+0000","last_became_active":"2024-06-27T21:36:51.804872+0000","last_became_peered":"2024-06-27T21:36:51.804872+0000","last_unstale":"2024-06-27T21:38:08.873560+0000","last_undegraded":"2024-06-27T21:38:08.873560+0000","last_fullsized":"2024-06-27T21:38:08.873560+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:37:09.129137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:37:09.129137+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:27.953864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,4],"acting":[2,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":462,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.877846+0000","last_change":"2024-06-27T21:36:51.811300+0000","last_active":"2024-06-27T21:38:08.877846+0000","last_peered":"2024-06-27T21:38:08.877846+0000","last_clean":"2024-06-27T21:38:08.877846+0000","last_became_active":"2024-06-27T21:36:51.810932+0000","last_became_peered":"2024-06-27T21:36:51.810932+0000","last_unstale":"2024-06-27T21:38:08.877846+0000","last_undegraded":"2024-06-27T21:38:08.877846+0000","last_fullsized":"2024-06-27T21:38:08.877846+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:31:32.931223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"675'11147","reported_seq":25105,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:09.951877+0000","last_change":"2024-06-27T21:35:41.151637+0000","last_active":"2024-06-27T21:38:09.951877+0000","last_peered":"2024-06-27T21:38:09.951877+0000","last_clean":"2024-06-27T21:38:09.951877+0000","last_became_active":"2024-06-27T21:35:01.964347+0000","last_became_peered":"2024-06-27T21:35:01.964347+0000","last_unstale":"2024-06-27T21:38:09.951877+0000","last_undegraded":"2024-06-27T21:38:09.951877+0000","last_fullsized":"2024-06-27T21:38:09.951877+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10152","last_scrub_stamp":"2024-06-27T21:35:41.151550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:41.151550+0000","objects_scrubbed":1305,"log_size":11147,"log_dups_size":0,"ondisk_log_size":11147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":995,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":447,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.911010+0000","last_change":"2024-06-27T21:36:51.811442+0000","last_active":"2024-06-27T21:38:08.911010+0000","last_peered":"2024-06-27T21:38:08.911010+0000","last_clean":"2024-06-27T21:38:08.911010+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:38:08.911010+0000","last_undegraded":"2024-06-27T21:38:08.911010+0000","last_fullsized":"2024-06-27T21:38:08.911010+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.245511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.245511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"675'11312","reported_seq":32425,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.445553+0000","last_change":"2024-06-27T21:34:06.592362+0000","last_active":"2024-06-27T21:38:10.445553+0000","last_peered":"2024-06-27T21:38:10.445553+0000","last_clean":"2024-06-27T21:38:10.445553+0000","last_became_active":"2024-06-27T21:34:05.705454+0000","last_became_peered":"2024-06-27T21:34:05.705454+0000","last_unstale":"2024-06-27T21:38:10.445553+0000","last_undegraded":"2024-06-27T21:38:10.445553+0000","last_fullsized":"2024-06-27T21:38:10.445553+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":5,"last_scrub":"562'9943","last_scrub_stamp":"2024-06-27T21:34:06.592264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:06.592264+0000","objects_scrubbed":1716,"log_size":11312,"log_dups_size":0,"ondisk_log_size":11312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":830,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1371,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":457,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.872474+0000","last_change":"2024-06-27T21:36:06.824164+0000","last_active":"2024-06-27T21:38:08.872474+0000","last_peered":"2024-06-27T21:38:08.872474+0000","last_clean":"2024-06-27T21:38:08.872474+0000","last_became_active":"2024-06-27T21:35:46.862310+0000","last_became_peered":"2024-06-27T21:35:46.862310+0000","last_unstale":"2024-06-27T21:38:08.872474+0000","last_undegraded":"2024-06-27T21:38:08.872474+0000","last_fullsized":"2024-06-27T21:38:08.872474+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:06.824082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:06.824082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.422482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"675'11142","reported_seq":27951,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:12.364667+0000","last_change":"2024-06-27T21:38:13.790244+0000","last_active":"2024-06-27T21:38:12.364667+0000","last_peered":"2024-06-27T21:38:12.364667+0000","last_clean":"2024-06-27T21:38:12.364667+0000","last_became_active":"2024-06-27T21:37:43.935841+0000","last_became_peered":"2024-06-27T21:37:43.935841+0000","last_unstale":"2024-06-27T21:38:12.364667+0000","last_undegraded":"2024-06-27T21:38:12.364667+0000","last_fullsized":"2024-06-27T21:38:12.364667+0000","mapping_epoch":658,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":659,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6006","last_scrub_stamp":"2024-06-27T21:30:48.365392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.365392+0000","objects_scrubbed":0,"log_size":11142,"log_dups_size":0,"ondisk_log_size":11142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:17.364712+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22511616,"num_objects":344,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":211776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4903,"num_bytes_recovered":218038272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,10],"acting":[7,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":433,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.879950+0000","last_change":"2024-06-27T21:35:46.853442+0000","last_active":"2024-06-27T21:38:08.879950+0000","last_peered":"2024-06-27T21:38:08.879950+0000","last_clean":"2024-06-27T21:38:08.879950+0000","last_became_active":"2024-06-27T21:35:46.853087+0000","last_became_peered":"2024-06-27T21:35:46.853087+0000","last_unstale":"2024-06-27T21:38:08.879950+0000","last_undegraded":"2024-06-27T21:38:08.879950+0000","last_fullsized":"2024-06-27T21:38:08.879950+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:31:19.025649+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:31:19.025649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.14","version":"675'11637","reported_seq":15739,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.590195+0000","last_change":"2024-06-27T21:38:10.844141+0000","last_active":"2024-06-27T21:38:10.590195+0000","last_peered":"2024-06-27T21:38:10.590195+0000","last_clean":"2024-06-27T21:38:10.590195+0000","last_became_active":"2024-06-27T21:37:53.989723+0000","last_became_peered":"2024-06-27T21:37:53.989723+0000","last_unstale":"2024-06-27T21:38:10.590195+0000","last_undegraded":"2024-06-27T21:38:10.590195+0000","last_fullsized":"2024-06-27T21:38:10.590195+0000","mapping_epoch":663,"log_start":"470'100","ondisk_log_start":"470'100","created":547,"last_epoch_clean":664,"parent":"0.0","parent_split_bits":5,"last_scrub":"498'6246","last_scrub_stamp":"2024-06-27T21:30:48.475608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:48.475608+0000","objects_scrubbed":0,"log_size":11537,"log_dups_size":0,"ondisk_log_size":11537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:15.590262+0000","scrub_duration":1596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20480000,"num_objects":313,"num_object_clones":0,"num_object_copies":939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6585,"num_write_kb":220704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":58064896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":435,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.873867+0000","last_change":"2024-06-27T21:36:42.332358+0000","last_active":"2024-06-27T21:38:08.873867+0000","last_peered":"2024-06-27T21:38:08.873867+0000","last_clean":"2024-06-27T21:38:08.873867+0000","last_became_active":"2024-06-27T21:29:51.673658+0000","last_became_peered":"2024-06-27T21:29:51.673658+0000","last_unstale":"2024-06-27T21:38:08.873867+0000","last_undegraded":"2024-06-27T21:38:08.873867+0000","last_fullsized":"2024-06-27T21:38:08.873867+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:42.332234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:42.332234+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.472281+0000","scrub_duration":1328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"675'11510","reported_seq":28293,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:09.821484+0000","last_change":"2024-06-27T21:35:52.894201+0000","last_active":"2024-06-27T21:38:09.821484+0000","last_peered":"2024-06-27T21:38:09.821484+0000","last_clean":"2024-06-27T21:38:09.821484+0000","last_became_active":"2024-06-27T21:35:52.878734+0000","last_became_peered":"2024-06-27T21:35:52.878734+0000","last_unstale":"2024-06-27T21:38:09.821484+0000","last_undegraded":"2024-06-27T21:38:09.821484+0000","last_fullsized":"2024-06-27T21:38:09.821484+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"566'10180","last_scrub_stamp":"2024-06-27T21:34:19.641768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:19.641768+0000","objects_scrubbed":1673,"log_size":11510,"log_dups_size":0,"ondisk_log_size":11510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":467,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":308,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.878080+0000","last_change":"2024-06-27T21:36:51.821200+0000","last_active":"2024-06-27T21:38:08.878080+0000","last_peered":"2024-06-27T21:38:08.878080+0000","last_clean":"2024-06-27T21:38:08.878080+0000","last_became_active":"2024-06-27T21:36:51.820468+0000","last_became_peered":"2024-06-27T21:36:51.820468+0000","last_unstale":"2024-06-27T21:38:08.878080+0000","last_undegraded":"2024-06-27T21:38:08.878080+0000","last_fullsized":"2024-06-27T21:38:08.878080+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.531009+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.531009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"675'11083","reported_seq":11640,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:09.779387+0000","last_change":"2024-06-27T21:33:51.180577+0000","last_active":"2024-06-27T21:38:09.779387+0000","last_peered":"2024-06-27T21:38:09.779387+0000","last_clean":"2024-06-27T21:38:09.779387+0000","last_became_active":"2024-06-27T21:33:27.698780+0000","last_became_peered":"2024-06-27T21:33:27.698780+0000","last_unstale":"2024-06-27T21:38:09.779387+0000","last_undegraded":"2024-06-27T21:38:09.779387+0000","last_fullsized":"2024-06-27T21:38:09.779387+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'9712","last_scrub_stamp":"2024-06-27T21:33:51.180247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:33:51.180247+0000","objects_scrubbed":1680,"log_size":11083,"log_dups_size":0,"ondisk_log_size":11083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1371,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,2],"acting":[11,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":173,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.911305+0000","last_change":"2024-06-27T21:35:47.430217+0000","last_active":"2024-06-27T21:38:08.911305+0000","last_peered":"2024-06-27T21:38:08.911305+0000","last_clean":"2024-06-27T21:38:08.911305+0000","last_became_active":"2024-06-27T21:35:46.855888+0000","last_became_peered":"2024-06-27T21:35:46.855888+0000","last_unstale":"2024-06-27T21:38:08.911305+0000","last_undegraded":"2024-06-27T21:38:08.911305+0000","last_fullsized":"2024-06-27T21:38:08.911305+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:35:47.430103+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:35:47.430103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"675'10944","reported_seq":31129,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.370457+0000","last_change":"2024-06-27T21:34:22.233121+0000","last_active":"2024-06-27T21:38:10.370457+0000","last_peered":"2024-06-27T21:38:10.370457+0000","last_clean":"2024-06-27T21:38:10.370457+0000","last_became_active":"2024-06-27T21:34:21.788051+0000","last_became_peered":"2024-06-27T21:34:21.788051+0000","last_unstale":"2024-06-27T21:38:10.370457+0000","last_undegraded":"2024-06-27T21:38:10.370457+0000","last_fullsized":"2024-06-27T21:38:10.370457+0000","mapping_epoch":568,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":569,"parent":"0.0","parent_split_bits":5,"last_scrub":"569'9717","last_scrub_stamp":"2024-06-27T21:34:22.232987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:34:22.232987+0000","objects_scrubbed":1552,"log_size":10944,"log_dups_size":0,"ondisk_log_size":10944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21233664,"num_objects":324,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1228,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":112,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.871222+0000","last_change":"2024-06-27T21:37:08.639499+0000","last_active":"2024-06-27T21:38:08.871222+0000","last_peered":"2024-06-27T21:38:08.871222+0000","last_clean":"2024-06-27T21:38:08.871222+0000","last_became_active":"2024-06-27T21:36:51.808305+0000","last_became_peered":"2024-06-27T21:36:51.808305+0000","last_unstale":"2024-06-27T21:38:08.871222+0000","last_undegraded":"2024-06-27T21:38:08.871222+0000","last_fullsized":"2024-06-27T21:38:08.871222+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:37:08.639432+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:37:08.639432+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:12.523411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.10","version":"675'12901","reported_seq":50506,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.400258+0000","last_change":"2024-06-27T21:38:03.876881+0000","last_active":"2024-06-27T21:38:10.400258+0000","last_peered":"2024-06-27T21:38:10.400258+0000","last_clean":"2024-06-27T21:38:10.400258+0000","last_became_active":"2024-06-27T21:38:03.858322+0000","last_became_peered":"2024-06-27T21:38:03.858322+0000","last_unstale":"2024-06-27T21:38:10.400258+0000","last_undegraded":"2024-06-27T21:38:10.400258+0000","last_fullsized":"2024-06-27T21:38:10.400258+0000","mapping_epoch":671,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":672,"parent":"0.0","parent_split_bits":5,"last_scrub":"574'11661","last_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_deep_scrub":"574'11661","last_deep_scrub_stamp":"2024-06-27T21:34:46.137011+0000","last_clean_scrub_stamp":"2024-06-27T21:34:46.137011+0000","objects_scrubbed":1586,"log_size":12901,"log_dups_size":0,"ondisk_log_size":12901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":11,"scrub_schedule":"queued for scrub","scrub_duration":10216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430272,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1259,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6730,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"675'12310","reported_seq":12868,"reported_epoch":675,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:38:10.508574+0000","last_change":"2024-06-27T21:38:07.059199+0000","last_active":"2024-06-27T21:38:10.508574+0000","last_peered":"2024-06-27T21:38:10.508574+0000","last_clean":"2024-06-27T21:37:34.161033+0000","last_became_active":"2024-06-27T21:37:37.024360+0000","last_became_peered":"2024-06-27T21:37:37.024360+0000","last_unstale":"2024-06-27T21:38:10.508574+0000","last_undegraded":"2024-06-27T21:38:10.508574+0000","last_fullsized":"2024-06-27T21:38:10.508574+0000","mapping_epoch":654,"log_start":"477'1700","ondisk_log_start":"477'1700","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"629'11283","last_scrub_stamp":"2024-06-27T21:36:37.373008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:37.373008+0000","objects_scrubbed":2292,"log_size":10610,"log_dups_size":0,"ondisk_log_size":10610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757952,"num_objects":332,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":664,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6454,"num_write_kb":217152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,8],"acting":[2,10,9],"avail_no_missing":["2","9","10"],"object_location_counts":[{"shards":"2,9,10","objects":332}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":758,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.873427+0000","last_change":"2024-06-27T21:36:42.000057+0000","last_active":"2024-06-27T21:38:08.873427+0000","last_peered":"2024-06-27T21:38:08.873427+0000","last_clean":"2024-06-27T21:38:08.873427+0000","last_became_active":"2024-06-27T21:35:46.859076+0000","last_became_peered":"2024-06-27T21:35:46.859076+0000","last_unstale":"2024-06-27T21:38:08.873427+0000","last_undegraded":"2024-06-27T21:38:08.873427+0000","last_fullsized":"2024-06-27T21:38:08.873427+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:36:41.999953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:36:41.999953+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:13.472310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,11],"acting":[2,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"675'12287","reported_seq":20215,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:11.497151+0000","last_change":"2024-06-27T21:38:11.497151+0000","last_active":"2024-06-27T21:38:11.497151+0000","last_peered":"2024-06-27T21:38:11.497151+0000","last_clean":"2024-06-27T21:38:11.497151+0000","last_became_active":"2024-06-27T21:38:08.873404+0000","last_became_peered":"2024-06-27T21:38:08.873404+0000","last_unstale":"2024-06-27T21:38:11.497151+0000","last_undegraded":"2024-06-27T21:38:11.497151+0000","last_fullsized":"2024-06-27T21:38:11.497151+0000","mapping_epoch":674,"log_start":"475'1600","ondisk_log_start":"475'1600","created":651,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"628'11257","last_scrub_stamp":"2024-06-27T21:36:36.394150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:36:36.394150+0000","objects_scrubbed":0,"log_size":10687,"log_dups_size":0,"ondisk_log_size":10687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:38:15.713841+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22347776,"num_objects":341,"num_object_clones":0,"num_object_copies":1023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6437,"num_write_kb":216896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1838,"num_bytes_recovered":120422400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,10],"acting":[8,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":320,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.872519+0000","last_change":"2024-06-27T21:33:50.003594+0000","last_active":"2024-06-27T21:38:08.872519+0000","last_peered":"2024-06-27T21:38:08.872519+0000","last_clean":"2024-06-27T21:38:08.872519+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:38:08.872519+0000","last_undegraded":"2024-06-27T21:38:08.872519+0000","last_fullsized":"2024-06-27T21:38:08.872519+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:50.003522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:50.003522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1d","version":"675'12218","reported_seq":31638,"reported_epoch":675,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:38:10.461781+0000","last_change":"2024-06-27T21:38:06.979869+0000","last_active":"2024-06-27T21:38:10.461781+0000","last_peered":"2024-06-27T21:38:10.461781+0000","last_clean":"2024-06-27T21:37:33.937322+0000","last_became_active":"2024-06-27T21:37:36.955270+0000","last_became_peered":"2024-06-27T21:37:36.955270+0000","last_unstale":"2024-06-27T21:38:10.461781+0000","last_undegraded":"2024-06-27T21:38:10.461781+0000","last_fullsized":"2024-06-27T21:38:10.461781+0000","mapping_epoch":654,"log_start":"475'1500","ondisk_log_start":"475'1500","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"503'6553","last_scrub_stamp":"2024-06-27T21:30:56.681455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.681455+0000","objects_scrubbed":6485,"log_size":10718,"log_dups_size":0,"ondisk_log_size":10718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21757964,"num_objects":333,"num_object_clones":0,"num_object_copies":999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":999,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6420,"num_write_kb":216033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4717,"num_bytes_recovered":254509080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[0,10,11],"avail_no_missing":["0","10","11"],"object_location_counts":[{"shards":"0,10,11","objects":333}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":370,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.914859+0000","last_change":"2024-06-27T21:35:46.858389+0000","last_active":"2024-06-27T21:38:08.914859+0000","last_peered":"2024-06-27T21:38:08.914859+0000","last_clean":"2024-06-27T21:38:08.914859+0000","last_became_active":"2024-06-27T21:35:46.858050+0000","last_became_peered":"2024-06-27T21:35:46.858050+0000","last_unstale":"2024-06-27T21:38:08.914859+0000","last_undegraded":"2024-06-27T21:38:08.914859+0000","last_fullsized":"2024-06-27T21:38:08.914859+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.266564+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.266564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,11],"acting":[4,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"675'12530","reported_seq":30704,"reported_epoch":675,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:38:12.440043+0000","last_change":"2024-06-27T21:37:39.748366+0000","last_active":"2024-06-27T21:38:12.440043+0000","last_peered":"2024-06-27T21:38:12.440043+0000","last_clean":"2024-06-27T21:31:21.892290+0000","last_became_active":"2024-06-27T21:37:37.585901+0000","last_became_peered":"2024-06-27T21:37:37.585901+0000","last_unstale":"2024-06-27T21:38:12.440043+0000","last_undegraded":"2024-06-27T21:37:39.723398+0000","last_fullsized":"2024-06-27T21:37:35.753031+0000","mapping_epoch":653,"log_start":"478'1900","ondisk_log_start":"478'1900","created":651,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":5,"last_scrub":"503'6713","last_scrub_stamp":"2024-06-27T21:30:56.574897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:56.574897+0000","objects_scrubbed":6674,"log_size":10630,"log_dups_size":0,"ondisk_log_size":10630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6567,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3825,"num_bytes_recovered":179634176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,7],"avail_no_missing":["8","5","6"],"object_location_counts":[{"shards":"7,8","objects":342}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":95,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.871039+0000","last_change":"2024-06-27T21:36:51.803807+0000","last_active":"2024-06-27T21:38:08.871039+0000","last_peered":"2024-06-27T21:38:08.871039+0000","last_clean":"2024-06-27T21:38:08.871039+0000","last_became_active":"2024-06-27T21:36:51.803452+0000","last_became_peered":"2024-06-27T21:36:51.803452+0000","last_unstale":"2024-06-27T21:38:08.871039+0000","last_undegraded":"2024-06-27T21:38:08.871039+0000","last_fullsized":"2024-06-27T21:38:08.871039+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.568751+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.568751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":93,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.871575+0000","last_change":"2024-06-27T21:36:51.806395+0000","last_active":"2024-06-27T21:38:08.871575+0000","last_peered":"2024-06-27T21:38:08.871575+0000","last_clean":"2024-06-27T21:38:08.871575+0000","last_became_active":"2024-06-27T21:36:51.805865+0000","last_became_peered":"2024-06-27T21:36:51.805865+0000","last_unstale":"2024-06-27T21:38:08.871575+0000","last_undegraded":"2024-06-27T21:38:08.871575+0000","last_fullsized":"2024-06-27T21:38:08.871575+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:52.993157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:33:52.993157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,11],"acting":[8,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.1b","version":"675'11910","reported_seq":17635,"reported_epoch":675,"state":"active+remapped+backfill_toofull","last_fresh":"2024-06-27T21:38:10.408176+0000","last_change":"2024-06-27T21:38:07.431667+0000","last_active":"2024-06-27T21:38:10.408176+0000","last_peered":"2024-06-27T21:38:10.408176+0000","last_clean":"2024-06-27T21:37:33.947340+0000","last_became_active":"2024-06-27T21:37:37.026501+0000","last_became_peered":"2024-06-27T21:37:37.026501+0000","last_unstale":"2024-06-27T21:38:10.408176+0000","last_undegraded":"2024-06-27T21:38:10.408176+0000","last_fullsized":"2024-06-27T21:38:10.408176+0000","mapping_epoch":654,"log_start":"475'1300","ondisk_log_start":"475'1300","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10541","last_scrub_stamp":"2024-06-27T21:35:43.673315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:43.673315+0000","objects_scrubbed":2645,"log_size":10610,"log_dups_size":0,"ondisk_log_size":10610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22020096,"num_objects":336,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1008,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6256,"num_write_kb":210944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1278,"num_bytes_recovered":83755008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,8],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":336}],"blocked_by":[],"up_primary":11,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":302,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.879603+0000","last_change":"2024-06-27T21:35:46.856033+0000","last_active":"2024-06-27T21:38:08.879603+0000","last_peered":"2024-06-27T21:38:08.879603+0000","last_clean":"2024-06-27T21:38:08.879603+0000","last_became_active":"2024-06-27T21:35:46.855054+0000","last_became_peered":"2024-06-27T21:35:46.855054+0000","last_unstale":"2024-06-27T21:38:08.879603+0000","last_undegraded":"2024-06-27T21:38:08.879603+0000","last_fullsized":"2024-06-27T21:38:08.879603+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:01.956253+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:33:01.956253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,10],"acting":[11,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.1a","version":"675'12148","reported_seq":26792,"reported_epoch":675,"state":"active+remapped+backfilling","last_fresh":"2024-06-27T21:38:10.464958+0000","last_change":"2024-06-27T21:38:07.437436+0000","last_active":"2024-06-27T21:38:10.464958+0000","last_peered":"2024-06-27T21:38:10.464958+0000","last_clean":"2024-06-27T21:37:33.961826+0000","last_became_active":"2024-06-27T21:37:37.031867+0000","last_became_peered":"2024-06-27T21:37:37.031867+0000","last_unstale":"2024-06-27T21:38:10.464958+0000","last_undegraded":"2024-06-27T21:38:10.464958+0000","last_fullsized":"2024-06-27T21:38:10.464958+0000","mapping_epoch":654,"log_start":"475'1500","ondisk_log_start":"475'1500","created":651,"last_epoch_clean":652,"parent":"0.0","parent_split_bits":5,"last_scrub":"502'6380","last_scrub_stamp":"2024-06-27T21:30:54.172506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:30:54.172506+0000","objects_scrubbed":6328,"log_size":10648,"log_dups_size":0,"ondisk_log_size":10648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21397504,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":525,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6373,"num_write_kb":214368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3528,"num_bytes_recovered":181764096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,10],"acting":[0,2,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":326},{"shards":"4","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":302,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:08.879514+0000","last_change":"2024-06-27T21:36:51.809004+0000","last_active":"2024-06-27T21:38:08.879514+0000","last_peered":"2024-06-27T21:38:08.879514+0000","last_clean":"2024-06-27T21:38:08.879514+0000","last_became_active":"2024-06-27T21:36:51.808540+0000","last_became_peered":"2024-06-27T21:36:51.808540+0000","last_unstale":"2024-06-27T21:38:08.879514+0000","last_undegraded":"2024-06-27T21:38:08.879514+0000","last_fullsized":"2024-06-27T21:38:08.879514+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:33:02.987590+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:33:02.987590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.19","version":"675'11250","reported_seq":26161,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.004682+0000","last_change":"2024-06-27T21:36:23.070309+0000","last_active":"2024-06-27T21:38:10.004682+0000","last_peered":"2024-06-27T21:38:10.004682+0000","last_clean":"2024-06-27T21:38:10.004682+0000","last_became_active":"2024-06-27T21:36:23.050064+0000","last_became_peered":"2024-06-27T21:36:23.050064+0000","last_unstale":"2024-06-27T21:38:10.004682+0000","last_undegraded":"2024-06-27T21:38:10.004682+0000","last_fullsized":"2024-06-27T21:38:10.004682+0000","mapping_epoch":623,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":624,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10195","last_scrub_stamp":"2024-06-27T21:35:42.766433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.766433+0000","objects_scrubbed":1384,"log_size":11250,"log_dups_size":0,"ondisk_log_size":11250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1055,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":27000832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.18","version":"675'11364","reported_seq":28637,"reported_epoch":675,"state":"active+clean","last_fresh":"2024-06-27T21:38:10.362990+0000","last_change":"2024-06-27T21:36:19.056568+0000","last_active":"2024-06-27T21:38:10.362990+0000","last_peered":"2024-06-27T21:38:10.362990+0000","last_clean":"2024-06-27T21:38:10.362990+0000","last_became_active":"2024-06-27T21:36:19.037779+0000","last_became_peered":"2024-06-27T21:36:19.037779+0000","last_unstale":"2024-06-27T21:38:10.362990+0000","last_undegraded":"2024-06-27T21:38:10.362990+0000","last_fullsized":"2024-06-27T21:38:10.362990+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":547,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":5,"last_scrub":"596'10353","last_scrub_stamp":"2024-06-27T21:35:42.524197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:29:18.797481+0000","last_clean_scrub_stamp":"2024-06-27T21:35:42.524197+0000","objects_scrubbed":1327,"log_size":11364,"log_dups_size":0,"ondisk_log_size":11364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3341,"num_bytes_recovered":85590016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":36,"stat_sum":{"num_bytes":676659224,"num_objects":10327,"num_object_clones":0,"num_object_copies":30981,"num_objects_missing_on_primary":728,"num_objects_missing":728,"num_objects_degraded":7,"num_objects_misplaced":3596,"num_objects_unfound":0,"num_objects_dirty":10327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118478,"num_write_kb":3243713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64482,"num_bytes_recovered":2683994160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3227004928,"data_stored":3226992712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397590,"ondisk_log_size":397590,"up":108,"acting":106,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253211,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881976,"kb_used_data":246892,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92863176,"statfs":{"total":95995035648,"available":95091892224,"internally_reserved":0,"allocated":252817408,"data_stored":251941880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514847,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":813172,"kb_used_data":268324,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92931980,"statfs":{"total":95995035648,"available":95162347520,"internally_reserved":0,"allocated":274763776,"data_stored":273891800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743780,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749292,"kb_used_data":258592,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92995860,"statfs":{"total":95995035648,"available":95227760640,"internally_reserved":0,"allocated":264798208,"data_stored":263930304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":502462912},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972712,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618664,"kb_used_data":218208,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93126488,"statfs":{"total":95995035648,"available":95361523712,"internally_reserved":0,"allocated":223444992,"data_stored":222577088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267048,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887536,"kb_used_data":270504,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92857616,"statfs":{"total":95995035648,"available":95086198784,"internally_reserved":0,"allocated":276996096,"data_stored":276124640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":631830986},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528690,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1199080,"kb_used_data":662560,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92546072,"statfs":{"total":95995035648,"available":94767177728,"internally_reserved":0,"allocated":678461440,"data_stored":677593536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757624,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826984,"kb_used_data":408480,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92918168,"statfs":{"total":95995035648,"available":95148204032,"internally_reserved":0,"allocated":418283520,"data_stored":417415616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161329,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":466664,"kb_used_data":176608,"kb_used_omap":92,"kb_used_meta":289955,"kb_avail":93278488,"statfs":{"total":95995035648,"available":95517171712,"internally_reserved":0,"allocated":180846592,"data_stored":179978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":296914018},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182784,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":544232,"kb_used_data":125728,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93200920,"statfs":{"total":95995035648,"available":95437742080,"internally_reserved":0,"allocated":128745472,"data_stored":127877568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411717,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775664,"kb_used_data":297448,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92969488,"statfs":{"total":95995035648,"available":95200755712,"internally_reserved":0,"allocated":304586752,"data_stored":303715296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575243,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738766,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858804,"kb_used_data":241764,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92886348,"statfs":{"total":95995035648,"available":95115620352,"internally_reserved":0,"allocated":247566336,"data_stored":246694360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":242946048,"data_stored":242941976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45416448,"data_stored":45416448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":299958272,"data_stored":299958272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126287872,"data_stored":126287872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":178454528,"data_stored":178454528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":415825920,"data_stored":415825920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":675348480,"data_stored":675348480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":274595840,"data_stored":274595840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":220856320,"data_stored":220856320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":260505600,"data_stored":260505600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":254939136,"data_stored":254935064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":231870464,"data_stored":231866392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:38:14.879 INFO:tasks.thrashosds.thrasher:Setting 6.7 to [4, 10, 6] 2024-06-27T21:38:14.880 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.7', '4', '10', '6'] 2024-06-27T21:38:14.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap 6.7 4 10 6 2024-06-27T21:38:14.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:14.961 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:14.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:15.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:15.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:15.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: 6.1 scrub starts 2024-06-27T21:38:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:38:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3919314626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:38:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: osdmap e676: 12 total, 12 up, 11 in 2024-06-27T21:38:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: 6.21 scrub starts 2024-06-27T21:38:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: 6.21 scrub ok 2024-06-27T21:38:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: 6.f scrub starts 2024-06-27T21:38:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: 6.f scrub ok 2024-06-27T21:38:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7/34947 objects degraded (0.020%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:14 smithi151 ceph-mon[28965]: osdmap e677: 12 total, 12 up, 11 in 2024-06-27T21:38:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:15.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: 6.1 scrub starts 2024-06-27T21:38:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:38:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3919314626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:38:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: osdmap e676: 12 total, 12 up, 11 in 2024-06-27T21:38:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: 6.21 scrub starts 2024-06-27T21:38:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: 6.21 scrub ok 2024-06-27T21:38:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: 6.f scrub starts 2024-06-27T21:38:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: 6.f scrub ok 2024-06-27T21:38:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7/34947 objects degraded (0.020%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:14 smithi190 ceph-mon[32247]: osdmap e677: 12 total, 12 up, 11 in 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: 6.1 scrub starts 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3919314626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: osdmap e676: 12 total, 12 up, 11 in 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: 6.21 scrub starts 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: 6.21 scrub ok 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: 6.f scrub starts 2024-06-27T21:38:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: 6.f scrub ok 2024-06-27T21:38:15.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7/34947 objects degraded (0.020%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:15.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:14 smithi192 ceph-mon[32956]: osdmap e677: 12 total, 12 up, 11 in 2024-06-27T21:38:15.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:15.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:15.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:15.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:15.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:15.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:15.680 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:15.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:15.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:38:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:15.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:15.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:15.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:15.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:15.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:15 smithi151 ceph-mon[28965]: from='client.16935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:38:15.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:15 smithi151 ceph-mon[28965]: pgmap v1700: 67 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 59 active+clean; 646 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 7/30987 objects degraded (0.023%); 3596/30987 objects misplaced (11.605%); 11 MiB/s, 193 objects/s recovering 2024-06-27T21:38:15.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:15 smithi151 ceph-mon[28965]: 6.23 scrub starts 2024-06-27T21:38:15.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:15 smithi151 ceph-mon[28965]: 6.23 scrub ok 2024-06-27T21:38:15.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:15.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:16.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:16.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:16.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:16.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:16.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:15 smithi190 ceph-mon[32247]: from='client.16935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:38:16.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:15 smithi190 ceph-mon[32247]: pgmap v1700: 67 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 59 active+clean; 646 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 7/30987 objects degraded (0.023%); 3596/30987 objects misplaced (11.605%); 11 MiB/s, 193 objects/s recovering 2024-06-27T21:38:16.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:15 smithi190 ceph-mon[32247]: 6.23 scrub starts 2024-06-27T21:38:16.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:15 smithi190 ceph-mon[32247]: 6.23 scrub ok 2024-06-27T21:38:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:15 smithi192 ceph-mon[32956]: from='client.16935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:38:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:15 smithi192 ceph-mon[32956]: pgmap v1700: 67 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 59 active+clean; 646 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 7/30987 objects degraded (0.023%); 3596/30987 objects misplaced (11.605%); 11 MiB/s, 193 objects/s recovering 2024-06-27T21:38:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:15 smithi192 ceph-mon[32956]: 6.23 scrub starts 2024-06-27T21:38:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:15 smithi192 ceph-mon[32956]: 6.23 scrub ok 2024-06-27T21:38:16.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:16.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:16.387 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:16.387 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:16.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:16.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:16.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:16.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:16.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:16.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:16.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:16.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:16.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:16.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:16.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:16.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:17.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:17.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:17.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:17.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.b scrub starts 2024-06-27T21:38:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.b scrub ok 2024-06-27T21:38:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.22 scrub starts 2024-06-27T21:38:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.22 scrub ok 2024-06-27T21:38:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.3 scrub starts 2024-06-27T21:38:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.3 scrub ok 2024-06-27T21:38:17.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.e scrub starts 2024-06-27T21:38:17.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:16 smithi151 ceph-mon[28965]: 6.e scrub ok 2024-06-27T21:38:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.b scrub starts 2024-06-27T21:38:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.b scrub ok 2024-06-27T21:38:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.22 scrub starts 2024-06-27T21:38:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.22 scrub ok 2024-06-27T21:38:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.3 scrub starts 2024-06-27T21:38:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.3 scrub ok 2024-06-27T21:38:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.e scrub starts 2024-06-27T21:38:17.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:16 smithi190 ceph-mon[32247]: 6.e scrub ok 2024-06-27T21:38:17.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.b scrub starts 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.b scrub ok 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.22 scrub starts 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.22 scrub ok 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.3 scrub starts 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.3 scrub ok 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.e scrub starts 2024-06-27T21:38:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:16 smithi192 ceph-mon[32956]: 6.e scrub ok 2024-06-27T21:38:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:17.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:17.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:38:17.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:17.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:17.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:17.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:17.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:17.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:17.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:17.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:17.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:17.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:17.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:18.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:18.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:18.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: pgmap v1702: 67 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 638 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 7/30621 objects degraded (0.023%); 3179/30621 objects misplaced (10.382%); 319 KiB/s, 0 objects/s recovering 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 6.1e scrub starts 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 6.1e scrub ok 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: osdmap e678: 12 total, 12 up, 11 in 2024-06-27T21:38:18.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:38:18.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:38:18.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 6.d scrub starts 2024-06-27T21:38:18.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 6.d scrub ok 2024-06-27T21:38:18.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 6.2 scrub starts 2024-06-27T21:38:18.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:17 smithi151 ceph-mon[28965]: 6.2 scrub ok 2024-06-27T21:38:18.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: pgmap v1702: 67 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 638 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 7/30621 objects degraded (0.023%); 3179/30621 objects misplaced (10.382%); 319 KiB/s, 0 objects/s recovering 2024-06-27T21:38:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:38:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 6.1e scrub starts 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 6.1e scrub ok 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: osdmap e678: 12 total, 12 up, 11 in 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 6.d scrub starts 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 6.d scrub ok 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 6.2 scrub starts 2024-06-27T21:38:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:17 smithi190 ceph-mon[32247]: 6.2 scrub ok 2024-06-27T21:38:18.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: pgmap v1702: 67 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 638 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 7/30621 objects degraded (0.023%); 3179/30621 objects misplaced (10.382%); 319 KiB/s, 0 objects/s recovering 2024-06-27T21:38:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:38:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:38:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 6.1e scrub starts 2024-06-27T21:38:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 6.1e scrub ok 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: osdmap e678: 12 total, 12 up, 11 in 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 6.d scrub starts 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 6.d scrub ok 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 6.2 scrub starts 2024-06-27T21:38:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:17 smithi192 ceph-mon[32956]: 6.2 scrub ok 2024-06-27T21:38:18.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:18.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:18.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:18.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:18.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:18.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:18.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:18.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:18.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:18.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:18.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:18.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:19.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:19.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:19.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:19.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:18 smithi151 ceph-mon[28965]: osdmap e679: 12 total, 12 up, 11 in 2024-06-27T21:38:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:18 smithi151 ceph-mon[28965]: 6.15 scrub starts 2024-06-27T21:38:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:18 smithi151 ceph-mon[28965]: 6.15 scrub ok 2024-06-27T21:38:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:18 smithi151 ceph-mon[28965]: 6.14 scrub starts 2024-06-27T21:38:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:18 smithi190 ceph-mon[32247]: osdmap e679: 12 total, 12 up, 11 in 2024-06-27T21:38:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:18 smithi190 ceph-mon[32247]: 6.15 scrub starts 2024-06-27T21:38:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:18 smithi190 ceph-mon[32247]: 6.15 scrub ok 2024-06-27T21:38:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:18 smithi190 ceph-mon[32247]: 6.14 scrub starts 2024-06-27T21:38:19.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:19.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:18 smithi192 ceph-mon[32956]: osdmap e679: 12 total, 12 up, 11 in 2024-06-27T21:38:19.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:18 smithi192 ceph-mon[32956]: 6.15 scrub starts 2024-06-27T21:38:19.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:18 smithi192 ceph-mon[32956]: 6.15 scrub ok 2024-06-27T21:38:19.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:18 smithi192 ceph-mon[32956]: 6.14 scrub starts 2024-06-27T21:38:19.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:19.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:19.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:19.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:19.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:19.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:19.669 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:19.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:19.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:19.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:19.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:19.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: pgmap v1705: 67 pgs: 1 peering, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 636 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30492 objects degraded (0.023%); 3059/30492 objects misplaced (10.032%) 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: 6.14 scrub ok 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: 6.1 scrub ok 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: osdmap e680: 12 total, 12 up, 11 in 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: 6.20 scrub starts 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: 6.20 scrub ok 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7/30492 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:19 smithi151 ceph-mon[28965]: osdmap e681: 12 total, 12 up, 11 in 2024-06-27T21:38:20.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:20.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:20.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:20.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: pgmap v1705: 67 pgs: 1 peering, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 636 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30492 objects degraded (0.023%); 3059/30492 objects misplaced (10.032%) 2024-06-27T21:38:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:38:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:38:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: 6.14 scrub ok 2024-06-27T21:38:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: 6.1 scrub ok 2024-06-27T21:38:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: osdmap e680: 12 total, 12 up, 11 in 2024-06-27T21:38:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: 6.20 scrub starts 2024-06-27T21:38:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: 6.20 scrub ok 2024-06-27T21:38:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7/30492 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:19 smithi190 ceph-mon[32247]: osdmap e681: 12 total, 12 up, 11 in 2024-06-27T21:38:20.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:20.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: pgmap v1705: 67 pgs: 1 peering, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 636 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30492 objects degraded (0.023%); 3059/30492 objects misplaced (10.032%) 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: 6.14 scrub ok 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: 6.1 scrub ok 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: osdmap e680: 12 total, 12 up, 11 in 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: 6.20 scrub starts 2024-06-27T21:38:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: 6.20 scrub ok 2024-06-27T21:38:20.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7/30492 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:20.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:19 smithi192 ceph-mon[32956]: osdmap e681: 12 total, 12 up, 11 in 2024-06-27T21:38:20.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:20.402 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:20.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:20.442 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:38:20.442 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:38:20.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:20.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:20.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:20.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:20.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:20.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:20.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:20.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:20.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:20.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:20.921 INFO:teuthology.orchestra.run.smithi151.stderr:set 6.7 pg_upmap mapping to [4,10,6] 2024-06-27T21:38:21.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: 6.7 scrub starts 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: 6.1a scrub starts 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: 6.1c continuing backfill to osd.11 from (478'1900,675'12530] 6:38ba2fde:::benchmark_data_smithi192_117994_object101787:head to 675'12530 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: 6.1a scrub ok 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2382357250' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [4, 10, 6]}]: dispatch 2024-06-27T21:38:21.028 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:38:21.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:21.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:21.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:21.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: 6.7 scrub starts 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: 6.1a scrub starts 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: 6.1c continuing backfill to osd.11 from (478'1900,675'12530] 6:38ba2fde:::benchmark_data_smithi192_117994_object101787:head to 675'12530 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: 6.1a scrub ok 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2382357250' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [4, 10, 6]}]: dispatch 2024-06-27T21:38:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:38:21.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: 6.7 scrub starts 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: 6.1a scrub starts 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: 6.1c continuing backfill to osd.11 from (478'1900,675'12530] 6:38ba2fde:::benchmark_data_smithi192_117994_object101787:head to 675'12530 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: 6.1a scrub ok 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2382357250' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [4, 10, 6]}]: dispatch 2024-06-27T21:38:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:38:21.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:21.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:21.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:21.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:21.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:21.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:21.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:21.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:21.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:21.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:21.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: 6.7 scrub ok 2024-06-27T21:38:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: pgmap v1708: 67 pgs: 1 peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 59 active+clean; 632 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30318 objects degraded (0.023%); 2956/30318 objects misplaced (9.750%) 2024-06-27T21:38:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:38:21.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:38:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2382357250' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [4, 10, 6]}]': finished 2024-06-27T21:38:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: osdmap e682: 12 total, 12 up, 11 in 2024-06-27T21:38:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:38:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:21 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:38:21.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: 6.7 scrub ok 2024-06-27T21:38:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: pgmap v1708: 67 pgs: 1 peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 59 active+clean; 632 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30318 objects degraded (0.023%); 2956/30318 objects misplaced (9.750%) 2024-06-27T21:38:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:38:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:38:22.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2382357250' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [4, 10, 6]}]': finished 2024-06-27T21:38:22.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: osdmap e682: 12 total, 12 up, 11 in 2024-06-27T21:38:22.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:38:22.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:21 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:38:22.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:22.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:22.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:22.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:22.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:38:22.189 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:38:22.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:38:22.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:38:22.232 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:38:22.232 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:38:22.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:22.253 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:38:22.253 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:38:22.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:22.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:38:22.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: 6.7 scrub ok 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: pgmap v1708: 67 pgs: 1 peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 59 active+clean; 632 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30318 objects degraded (0.023%); 2956/30318 objects misplaced (9.750%) 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2382357250' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.7", "id": [4, 10, 6]}]': finished 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: osdmap e682: 12 total, 12 up, 11 in 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:38:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:21 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:38:22.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:38:22.297 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:38:22.319 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:38:22.319 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:38:22.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:38:22.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:38:22.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:38:22.362 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:38:22.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:22.385 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:38:22.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:38:22.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:22.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:38:22.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:38:22.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:38:22.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:38:22.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:22.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:22.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:22.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:22.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:22.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:22.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:22.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:22.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:22.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:23.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:23.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:22 smithi190 ceph-mon[32247]: osdmap e683: 12 total, 12 up, 11 in 2024-06-27T21:38:23.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:22 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:38:23.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:22 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:38:23.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1028315829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:23.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:22 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:38:23.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:22 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:38:23.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:23.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:22 smithi151 ceph-mon[28965]: osdmap e683: 12 total, 12 up, 11 in 2024-06-27T21:38:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:22 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:38:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:22 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:38:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1028315829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:22 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:38:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:22 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:38:23.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:23.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:22 smithi192 ceph-mon[32956]: osdmap e683: 12 total, 12 up, 11 in 2024-06-27T21:38:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:22 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:38:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:22 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:38:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1028315829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:22 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:38:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:22 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:38:23.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:23.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:23.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:23.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:23.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:23.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:23.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:38:23.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:23.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:23.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:23.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:23.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:23.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:23.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:24.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:24.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:24.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:23 smithi190 ceph-mon[32247]: pgmap v1711: 67 pgs: 2 active+clean+scrubbing, 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 632 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30318 objects degraded (0.023%); 2956/30318 objects misplaced (9.750%) 2024-06-27T21:38:24.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:23 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:38:24.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:23 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:38:24.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:23 smithi190 ceph-mon[32247]: 6.1c scrub starts 2024-06-27T21:38:24.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:23 smithi190 ceph-mon[32247]: 6.1c scrub ok 2024-06-27T21:38:24.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:23 smithi190 ceph-mon[32247]: osdmap e684: 12 total, 12 up, 11 in 2024-06-27T21:38:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:23 smithi151 ceph-mon[28965]: pgmap v1711: 67 pgs: 2 active+clean+scrubbing, 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 632 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30318 objects degraded (0.023%); 2956/30318 objects misplaced (9.750%) 2024-06-27T21:38:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:23 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:38:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:23 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:38:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:23 smithi151 ceph-mon[28965]: 6.1c scrub starts 2024-06-27T21:38:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:23 smithi151 ceph-mon[28965]: 6.1c scrub ok 2024-06-27T21:38:24.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:23 smithi151 ceph-mon[28965]: osdmap e684: 12 total, 12 up, 11 in 2024-06-27T21:38:24.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:24.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:23 smithi192 ceph-mon[32956]: pgmap v1711: 67 pgs: 2 active+clean+scrubbing, 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 58 active+clean; 632 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 7/30318 objects degraded (0.023%); 2956/30318 objects misplaced (9.750%) 2024-06-27T21:38:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:23 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:38:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:23 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:38:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:23 smithi192 ceph-mon[32956]: 6.1c scrub starts 2024-06-27T21:38:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:23 smithi192 ceph-mon[32956]: 6.1c scrub ok 2024-06-27T21:38:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:23 smithi192 ceph-mon[32956]: osdmap e684: 12 total, 12 up, 11 in 2024-06-27T21:38:24.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:24.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:24.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:24.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:24.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:24.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:24.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:24.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:24.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:24.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:24.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:24.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:24.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:24.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:25.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:25.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:25.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:24 smithi151 ceph-mon[28965]: osdmap e685: 12 total, 12 up, 11 in 2024-06-27T21:38:25.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:24 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:38:25.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:24 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:38:25.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:24 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 7/30318 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:25.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:25.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:24 smithi190 ceph-mon[32247]: osdmap e685: 12 total, 12 up, 11 in 2024-06-27T21:38:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:24 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:38:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:24 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:38:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:24 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 7/30318 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:24 smithi192 ceph-mon[32956]: osdmap e685: 12 total, 12 up, 11 in 2024-06-27T21:38:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:24 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:38:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:24 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:38:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:24 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 7/30318 objects degraded (0.023%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:38:25.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:25.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:25.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:25.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:25.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:25.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:25.649 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:25.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:25.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:25.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:25.884 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:25.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:25.959 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:38:25.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:26.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:26.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:25 smithi151 ceph-mon[28965]: pgmap v1714: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 1 remapped+peering, 2 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 634 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 4/30411 objects degraded (0.013%); 2953/30411 objects misplaced (9.710%) 2024-06-27T21:38:26.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:25 smithi151 ceph-mon[28965]: 6.9 scrub starts 2024-06-27T21:38:26.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:25 smithi151 ceph-mon[28965]: 6.9 scrub ok 2024-06-27T21:38:26.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/36430320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:38:26.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:26.077 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:25 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:26.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:26.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:26.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:25 smithi190 ceph-mon[32247]: pgmap v1714: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 1 remapped+peering, 2 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 634 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 4/30411 objects degraded (0.013%); 2953/30411 objects misplaced (9.710%) 2024-06-27T21:38:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:25 smithi190 ceph-mon[32247]: 6.9 scrub starts 2024-06-27T21:38:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:25 smithi190 ceph-mon[32247]: 6.9 scrub ok 2024-06-27T21:38:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/36430320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:38:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:25 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:26.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:25 smithi192 ceph-mon[32956]: pgmap v1714: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 1 remapped+peering, 2 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 634 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 4/30411 objects degraded (0.013%); 2953/30411 objects misplaced (9.710%) 2024-06-27T21:38:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:25 smithi192 ceph-mon[32956]: 6.9 scrub starts 2024-06-27T21:38:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:25 smithi192 ceph-mon[32956]: 6.9 scrub ok 2024-06-27T21:38:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/36430320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:38:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:25 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:26.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:38:26.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:26.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:26.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:26.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:26.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:26.653 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:26.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:26.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:26.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:26.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:26.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:27.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:27.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:27.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:27.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: 6.a scrub starts 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: 6.0 scrub starts 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: 6.0 scrub ok 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: 6.a scrub ok 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/36430320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: osdmap e686: 12 total, 12 up, 11 in 2024-06-27T21:38:27.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:26 smithi151 ceph-mon[28965]: 1.1e deep-scrub starts 2024-06-27T21:38:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: 6.a scrub starts 2024-06-27T21:38:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: 6.0 scrub starts 2024-06-27T21:38:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: 6.0 scrub ok 2024-06-27T21:38:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: 6.a scrub ok 2024-06-27T21:38:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:38:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/36430320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:38:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: osdmap e686: 12 total, 12 up, 11 in 2024-06-27T21:38:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:26 smithi190 ceph-mon[32247]: 1.1e deep-scrub starts 2024-06-27T21:38:27.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:27.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: 6.a scrub starts 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: 6.0 scrub starts 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: 6.0 scrub ok 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: 6.a scrub ok 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/36430320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: osdmap e686: 12 total, 12 up, 11 in 2024-06-27T21:38:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:26 smithi192 ceph-mon[32956]: 1.1e deep-scrub starts 2024-06-27T21:38:27.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:27.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:27.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:27.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:27.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:27.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:27.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:27.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:27.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:27.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:27.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:27.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:28.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:28.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:28.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: 1.1e deep-scrub ok 2024-06-27T21:38:28.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: pgmap v1716: 67 pgs: 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 1 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 57 active+clean; 634 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 4/30402 objects degraded (0.013%); 2953/30402 objects misplaced (9.713%) 2024-06-27T21:38:28.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: osdmap e687: 12 total, 12 up, 11 in 2024-06-27T21:38:28.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:38:28.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:38:28.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:38:28.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:27 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:38:28.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:28.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:28.184 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7, 8] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:38:28.184 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:38:28.184 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 3, 4, 5, 6, 9, 11, 10, 2, 7, 8] 2024-06-27T21:38:28.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 7 2024-06-27T21:38:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: 1.1e deep-scrub ok 2024-06-27T21:38:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: pgmap v1716: 67 pgs: 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 1 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 57 active+clean; 634 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 4/30402 objects degraded (0.013%); 2953/30402 objects misplaced (9.713%) 2024-06-27T21:38:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: osdmap e687: 12 total, 12 up, 11 in 2024-06-27T21:38:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:38:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:38:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:38:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:27 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:38:28.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: 1.1e deep-scrub ok 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: pgmap v1716: 67 pgs: 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+remapped, 1 remapped+peering, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 57 active+clean; 634 MiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 4/30402 objects degraded (0.013%); 2953/30402 objects misplaced (9.713%) 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: osdmap e687: 12 total, 12 up, 11 in 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:38:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:27 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:38:28.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:28.376 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:38:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:38:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:38:28.377 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:38:28.377 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:38:28.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:28.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:28.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:28.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:28.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:28.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:28.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:28.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:28.868 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:28.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:28.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:29.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:29.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:29.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:29.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:29.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:38:29.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:28 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:38:29.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:28 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:38:29.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:28 smithi151 ceph-mon[28965]: osdmap e688: 12 total, 12 up, 11 in 2024-06-27T21:38:29.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:28 smithi151 ceph-mon[28965]: pgmap v1719: 67 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 600 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 2833/28800 objects misplaced (9.837%) 2024-06-27T21:38:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:28 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:38:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:28 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:38:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:28 smithi190 ceph-mon[32247]: osdmap e688: 12 total, 12 up, 11 in 2024-06-27T21:38:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:28 smithi190 ceph-mon[32247]: pgmap v1719: 67 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 600 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 2833/28800 objects misplaced (9.837%) 2024-06-27T21:38:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:28 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:38:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:28 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:38:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:28 smithi192 ceph-mon[32956]: osdmap e688: 12 total, 12 up, 11 in 2024-06-27T21:38:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:28 smithi192 ceph-mon[32956]: pgmap v1719: 67 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 600 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 2833/28800 objects misplaced (9.837%) 2024-06-27T21:38:29.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:29.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:29.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:29.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:29.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:29.572 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:29.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:29.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:29.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:29.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:29.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:29.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:29.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:30.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:30.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:38:30.081 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:30.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:30.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 6.16 scrub starts 2024-06-27T21:38:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 6.16 scrub ok 2024-06-27T21:38:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 6.c scrub starts 2024-06-27T21:38:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 6.c scrub ok 2024-06-27T21:38:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/30402 objects degraded (0.013%), 1 pg degraded) 2024-06-27T21:38:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:38:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:38:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:38:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:38:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:29 smithi190 ceph-mon[32247]: osdmap e689: 12 total, 12 up, 11 in 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 6.16 scrub starts 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 6.16 scrub ok 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 6.c scrub starts 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 6.c scrub ok 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/30402 objects degraded (0.013%), 1 pg degraded) 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:38:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:38:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:38:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:38:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:29 smithi192 ceph-mon[32956]: osdmap e689: 12 total, 12 up, 11 in 2024-06-27T21:38:30.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:30.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:30.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:30.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 6.16 scrub starts 2024-06-27T21:38:30.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 6.16 scrub ok 2024-06-27T21:38:30.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 6.c scrub starts 2024-06-27T21:38:30.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 6.c scrub ok 2024-06-27T21:38:30.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/30402 objects degraded (0.013%), 1 pg degraded) 2024-06-27T21:38:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:38:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:38:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:38:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:38:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:29 smithi151 ceph-mon[28965]: osdmap e689: 12 total, 12 up, 11 in 2024-06-27T21:38:30.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:30.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:30.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:30.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:30.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:30.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:30.944 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:30.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:30.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:30.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:31.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:31.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: pgmap v1721: 67 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 527 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 2701/25254 objects misplaced (10.695%) 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:31.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:31.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:38:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:38:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:38:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:38:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: pgmap v1721: 67 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 527 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 2701/25254 objects misplaced (10.695%) 2024-06-27T21:38:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:38:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:38:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:38:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:38:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: pgmap v1721: 67 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+clean+laggy, 1 active+clean+scrubbing+laggy, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 58 active+clean; 527 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 2701/25254 objects misplaced (10.695%) 2024-06-27T21:38:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:31.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:31.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:31.369 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:31.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:31.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:31.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:31.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:31.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:31.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:31.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:31.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:31.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:31.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:32.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:32.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:32.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:32.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: 6.11 scrub starts 2024-06-27T21:38:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: 6.11 scrub ok 2024-06-27T21:38:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: osdmap e690: 12 total, 12 up, 11 in 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: 6.12 deep-scrub starts 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:38:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:31 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: 6.11 scrub starts 2024-06-27T21:38:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: 6.11 scrub ok 2024-06-27T21:38:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: osdmap e690: 12 total, 12 up, 11 in 2024-06-27T21:38:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: 6.12 deep-scrub starts 2024-06-27T21:38:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:38:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:38:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:31 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:32.403 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:32.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: 6.11 scrub starts 2024-06-27T21:38:32.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: 6.11 scrub ok 2024-06-27T21:38:32.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:32.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: osdmap e690: 12 total, 12 up, 11 in 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: 6.12 deep-scrub starts 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:38:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:31 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:38:32.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:32.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:32.580 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:32.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:32.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:32.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:32.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:32.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:32.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:32.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:33.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:33.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:33.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:32 smithi151 ceph-mon[28965]: 6.10 scrub starts 2024-06-27T21:38:33.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:32 smithi151 ceph-mon[28965]: 6.10 scrub ok 2024-06-27T21:38:33.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:32 smithi151 ceph-mon[28965]: 6.12 deep-scrub ok 2024-06-27T21:38:33.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:32 smithi151 ceph-mon[28965]: pgmap v1723: 67 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 61 active+clean; 421 MiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 2317/20187 objects misplaced (11.478%) 2024-06-27T21:38:33.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:33.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:32 smithi190 ceph-mon[32247]: 6.10 scrub starts 2024-06-27T21:38:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:32 smithi190 ceph-mon[32247]: 6.10 scrub ok 2024-06-27T21:38:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:32 smithi190 ceph-mon[32247]: 6.12 deep-scrub ok 2024-06-27T21:38:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:32 smithi190 ceph-mon[32247]: pgmap v1723: 67 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 61 active+clean; 421 MiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 2317/20187 objects misplaced (11.478%) 2024-06-27T21:38:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:32 smithi192 ceph-mon[32956]: 6.10 scrub starts 2024-06-27T21:38:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:32 smithi192 ceph-mon[32956]: 6.10 scrub ok 2024-06-27T21:38:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:32 smithi192 ceph-mon[32956]: 6.12 deep-scrub ok 2024-06-27T21:38:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:32 smithi192 ceph-mon[32956]: pgmap v1723: 67 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 61 active+clean; 421 MiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 2317/20187 objects misplaced (11.478%) 2024-06-27T21:38:33.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:33.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:33.424 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:38:33.424 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:38:33.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:33.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:38:33.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:38:33.454 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:33.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:38:33.476 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:38:33.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:38:33.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:38:33.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:38:33.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:38:33.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:38:33.552 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:38:33.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:33.578 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:38:33.579 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:38:33.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:33.612 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:38:33.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:38:33.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:38:33.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:38:33.677 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:38:33.677 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:38:33.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:33.703 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:38:33.703 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:38:33.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:33.739 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:38:33.739 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:38:33.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:33.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:33.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:33.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:34.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:34.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:34.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:34.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:33 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:38:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:33 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:38:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1898909203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:33 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:38:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:33 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:38:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1898909203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:34.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:34.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:34.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:33 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:38:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:33 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:38:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1898909203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:38:34.474 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:34.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:34.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:34.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:34.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:34.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:34.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:34.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:34.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:34.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:35.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:35.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:35.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:35.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:35.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:34 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:38:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:34 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:38:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:34 smithi190 ceph-mon[32247]: pgmap v1724: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 61 active+clean; 315 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 1450/15084 objects misplaced (9.613%) 2024-06-27T21:38:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:34 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:38:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:34 smithi190 ceph-mon[32247]: osdmap e691: 12 total, 12 up, 11 in 2024-06-27T21:38:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:34 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:38:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:34 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:38:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:34 smithi192 ceph-mon[32956]: pgmap v1724: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 61 active+clean; 315 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 1450/15084 objects misplaced (9.613%) 2024-06-27T21:38:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:34 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:38:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:34 smithi192 ceph-mon[32956]: osdmap e691: 12 total, 12 up, 11 in 2024-06-27T21:38:35.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:35.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:35.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:34 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:38:35.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:35 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:38:35.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:35 smithi151 ceph-mon[28965]: pgmap v1724: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 61 active+clean; 315 MiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 1450/15084 objects misplaced (9.613%) 2024-06-27T21:38:35.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:35 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:38:35.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:35 smithi151 ceph-mon[28965]: osdmap e691: 12 total, 12 up, 11 in 2024-06-27T21:38:35.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:35.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:35.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:35.625 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:35.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:35.748 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:35.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:35.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:35.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:35.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:35.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:36.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:36.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:36.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:36.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: 6.18 scrub starts 2024-06-27T21:38:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: 6.18 scrub ok 2024-06-27T21:38:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:36 smithi190 ceph-mon[32247]: osdmap e692: 12 total, 12 up, 11 in 2024-06-27T21:38:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: 6.18 scrub starts 2024-06-27T21:38:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: 6.18 scrub ok 2024-06-27T21:38:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:36 smithi192 ceph-mon[32956]: osdmap e692: 12 total, 12 up, 11 in 2024-06-27T21:38:36.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:36.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:36.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:36.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:36.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: 6.18 scrub starts 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: 6.18 scrub ok 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:36.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:36 smithi151 ceph-mon[28965]: osdmap e692: 12 total, 12 up, 11 in 2024-06-27T21:38:36.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:36.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:36.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:36.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:36.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:36.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:36.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:36.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:36.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:36.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:36.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:37.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:37.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:37.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:37.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:37 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:38:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:37 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:38:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:37 smithi190 ceph-mon[32247]: pgmap v1727: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 62 active+clean; 204 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 994/9759 objects misplaced (10.185%) 2024-06-27T21:38:37.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:37 smithi190 ceph-mon[32247]: 6.4 scrub starts 2024-06-27T21:38:37.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:37 smithi190 ceph-mon[32247]: 6.4 scrub ok 2024-06-27T21:38:37.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:37 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:38:37.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:37 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:38:37.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:37 smithi192 ceph-mon[32956]: pgmap v1727: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 62 active+clean; 204 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 994/9759 objects misplaced (10.185%) 2024-06-27T21:38:37.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:37 smithi192 ceph-mon[32956]: 6.4 scrub starts 2024-06-27T21:38:37.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:37 smithi192 ceph-mon[32956]: 6.4 scrub ok 2024-06-27T21:38:37.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:37.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:37.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:37 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:38:37.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:37 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:38:37.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:37 smithi151 ceph-mon[28965]: pgmap v1727: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 62 active+clean; 204 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 994/9759 objects misplaced (10.185%) 2024-06-27T21:38:37.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:37 smithi151 ceph-mon[28965]: 6.4 scrub starts 2024-06-27T21:38:37.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:37 smithi151 ceph-mon[28965]: 6.4 scrub ok 2024-06-27T21:38:37.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:37.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:37.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:37.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:37.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:37.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:37.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:37.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:37.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:37.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:37.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:38.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:38.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:38 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:38:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:38 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:38:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:38 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:38:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:38 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:38:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:38 smithi151 ceph-mon[28965]: 6.13 scrub starts 2024-06-27T21:38:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:38 smithi151 ceph-mon[28965]: 6.13 scrub ok 2024-06-27T21:38:38.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:38.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:38 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:38:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:38 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:38:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:38 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:38:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:38 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:38:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:38 smithi190 ceph-mon[32247]: 6.13 scrub starts 2024-06-27T21:38:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:38 smithi190 ceph-mon[32247]: 6.13 scrub ok 2024-06-27T21:38:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:38 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:38:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:38 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:38:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:38 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:38:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:38 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:38:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:38 smithi192 ceph-mon[32956]: 6.13 scrub starts 2024-06-27T21:38:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:38 smithi192 ceph-mon[32956]: 6.13 scrub ok 2024-06-27T21:38:38.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:38.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:38.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:38.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:38.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:38.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:38.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:38.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:38.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:38.858 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:38.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:38.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:38.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:39.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:39.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:39.135 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:38:39.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:39.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:39 smithi190 ceph-mon[32247]: 6.8 scrub starts 2024-06-27T21:38:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:39 smithi190 ceph-mon[32247]: 6.8 scrub ok 2024-06-27T21:38:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:39 smithi190 ceph-mon[32247]: pgmap v1728: 67 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 63 active+clean; 158 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 607/7542 objects misplaced (8.048%) 2024-06-27T21:38:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4007696044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:38:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:39 smithi192 ceph-mon[32956]: 6.8 scrub starts 2024-06-27T21:38:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:39 smithi192 ceph-mon[32956]: 6.8 scrub ok 2024-06-27T21:38:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:39 smithi192 ceph-mon[32956]: pgmap v1728: 67 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 63 active+clean; 158 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 607/7542 objects misplaced (8.048%) 2024-06-27T21:38:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4007696044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:38:39.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:39.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:39.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:39 smithi151 ceph-mon[28965]: 6.8 scrub starts 2024-06-27T21:38:39.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:39 smithi151 ceph-mon[28965]: 6.8 scrub ok 2024-06-27T21:38:39.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:39 smithi151 ceph-mon[28965]: pgmap v1728: 67 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 63 active+clean; 158 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 607/7542 objects misplaced (8.048%) 2024-06-27T21:38:39.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4007696044' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:38:39.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:39.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:39.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:39.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:39.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:39.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:39.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:39.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:39.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:39.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:39.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:40.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:40.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:40.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:40.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:38:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: 6.1f scrub ok 2024-06-27T21:38:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:38:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-06-27T21:38:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4007696044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:38:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: osdmap e693: 12 total, 12 up, 11 in 2024-06-27T21:38:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:38:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: 6.1f scrub ok 2024-06-27T21:38:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:38:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-06-27T21:38:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4007696044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:38:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: osdmap e693: 12 total, 12 up, 11 in 2024-06-27T21:38:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:40.328 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:40.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:40.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:38:40.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: 6.1f scrub ok 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4007696044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: osdmap e693: 12 total, 12 up, 11 in 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:40.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:40.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:40.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:40.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:40.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:40.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:40.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:40.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:40.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:40.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:40.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:40.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:41.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:41.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:41.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:41.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:41 smithi192 ceph-mon[32956]: osdmap e694: 12 total, 12 up, 11 in 2024-06-27T21:38:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:41 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:38:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:41 smithi192 ceph-mon[32956]: pgmap v1731: 67 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 63 active+clean; 130 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 544 op/s; 345/6219 objects misplaced (5.548%); 1.7 MiB/s, 76 objects/s recovering 2024-06-27T21:38:41.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:41.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:41 smithi151 ceph-mon[28965]: osdmap e694: 12 total, 12 up, 11 in 2024-06-27T21:38:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:41 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:38:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:41 smithi151 ceph-mon[28965]: pgmap v1731: 67 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 63 active+clean; 130 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 544 op/s; 345/6219 objects misplaced (5.548%); 1.7 MiB/s, 76 objects/s recovering 2024-06-27T21:38:41.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:41 smithi190 ceph-mon[32247]: osdmap e694: 12 total, 12 up, 11 in 2024-06-27T21:38:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:41 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:38:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:41 smithi190 ceph-mon[32247]: pgmap v1731: 67 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 63 active+clean; 130 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 544 op/s; 345/6219 objects misplaced (5.548%); 1.7 MiB/s, 76 objects/s recovering 2024-06-27T21:38:41.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:41.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:41.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:41.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:41.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:41.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:41.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:41.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:41.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:42.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:42.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:42.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:42.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:42.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:42.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:42.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:42.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:42.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:42.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:42.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:42.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:42.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:42.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:42.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:42.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:42.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:43.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:43.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:43.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:43.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:43.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:43.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:43.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:43.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:43 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:43.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:43 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:43.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:43 smithi151 ceph-mon[28965]: osdmap e695: 12 total, 12 up, 11 in 2024-06-27T21:38:43.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:43 smithi151 ceph-mon[28965]: pgmap v1733: 67 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 63 active+clean; 127 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 203 op/s; 333/6090 objects misplaced (5.468%); 1.6 MiB/s, 68 objects/s recovering 2024-06-27T21:38:43.450 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:43 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:43 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:43 smithi190 ceph-mon[32247]: osdmap e695: 12 total, 12 up, 11 in 2024-06-27T21:38:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:43 smithi190 ceph-mon[32247]: pgmap v1733: 67 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 63 active+clean; 127 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 203 op/s; 333/6090 objects misplaced (5.468%); 1.6 MiB/s, 68 objects/s recovering 2024-06-27T21:38:43.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:43 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:43.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:43 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:43.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:43 smithi192 ceph-mon[32956]: osdmap e695: 12 total, 12 up, 11 in 2024-06-27T21:38:43.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:43 smithi192 ceph-mon[32956]: pgmap v1733: 67 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 63 active+clean; 127 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 203 op/s; 333/6090 objects misplaced (5.468%); 1.6 MiB/s, 68 objects/s recovering 2024-06-27T21:38:43.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:43.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:43.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:43.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:43.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:43.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:43.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:43.955 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:43.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:44.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:44.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:44.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:44.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:44.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:44.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:44.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:44.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:44.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:44.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:44.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:44.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:44.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:44.836 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:44.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:44.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:44.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:45.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:45.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:45.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:45.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:45.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:45.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:45.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:45.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:45 smithi190 ceph-mon[32247]: pgmap v1734: 67 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 80 objects/s recovering 2024-06-27T21:38:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:45 smithi190 ceph-mon[32247]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:45 smithi192 ceph-mon[32956]: pgmap v1734: 67 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 80 objects/s recovering 2024-06-27T21:38:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:45 smithi192 ceph-mon[32956]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:45.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:45.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:45.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:45.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:45 smithi151 ceph-mon[28965]: pgmap v1734: 67 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 80 objects/s recovering 2024-06-27T21:38:45.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:45 smithi151 ceph-mon[28965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:45.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:45.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:45.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:45.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:45.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:45.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:45.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:46.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.083 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:46.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:46.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:46.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:46.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:46.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:46.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:46.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:46.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:46 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:38:46.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:46 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:38:46.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:46.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:46.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:46 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:38:46.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:46.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:46.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:46.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:46.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:46.947 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:46.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:47.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:47.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:47.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:47.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:47.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:47.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:47.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:47.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:47 smithi151 ceph-mon[28965]: pgmap v1735: 67 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 59 objects/s recovering 2024-06-27T21:38:47.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:47 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:47.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:47 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:47.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:47 smithi190 ceph-mon[32247]: pgmap v1735: 67 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 59 objects/s recovering 2024-06-27T21:38:47.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:47 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:47.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:47 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:47.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:47.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:47 smithi192 ceph-mon[32956]: pgmap v1735: 67 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 59 objects/s recovering 2024-06-27T21:38:47.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:47 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:47.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:47 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:47.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:47.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:47.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:47.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:47.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:47.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:47.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:47.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:48.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:48.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:48.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:48.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:48.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:38:48.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:48.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:48.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:48.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:48.488 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:48 smithi151 ceph-mon[28965]: osdmap e696: 12 total, 12 up, 11 in 2024-06-27T21:38:48.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:48 smithi192 ceph-mon[32956]: osdmap e696: 12 total, 12 up, 11 in 2024-06-27T21:38:48.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:48.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:48.615 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:38:48.615 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:38:48.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:38:48.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:48.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:48.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:48 smithi190 ceph-mon[32247]: osdmap e696: 12 total, 12 up, 11 in 2024-06-27T21:38:48.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:48.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:48 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:38:48.597+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e696 definitely_dead 0 2024-06-27T21:38:48.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:48.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:48.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:49.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:49.113 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:49.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:49.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:49.292 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.7. 2024-06-27T21:38:49.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:49.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:49.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:49.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:49.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: pgmap v1737: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 93 objects/s recovering 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3392090579' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:38:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:49 smithi192 ceph-mon[32956]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: pgmap v1737: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 93 objects/s recovering 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3392090579' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:38:49.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:49 smithi151 ceph-mon[28965]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-27T21:38:49.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:49.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:49.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:49.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: pgmap v1737: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 64 active+clean; 126 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 93 objects/s recovering 2024-06-27T21:38:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:38:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:38:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3392090579' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:38:49.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-06-27T21:38:49.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:49 smithi190 ceph-mon[32247]: Client client.admin marked osd.7 out, while it was still marked up 2024-06-27T21:38:49.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:49.853 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:49.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:49.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:49.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:50.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:50.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:50.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:50.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:50.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:50.353 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:50.353 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:50.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:50.357 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:50.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-27T21:38:50.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:50 smithi151 ceph-mon[28965]: osdmap e697: 12 total, 12 up, 10 in 2024-06-27T21:38:50.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:50 smithi151 ceph-mon[28965]: osdmap e698: 12 total, 12 up, 10 in 2024-06-27T21:38:50.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:50 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:38:50.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:50.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:50.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-27T21:38:50.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:50 smithi192 ceph-mon[32956]: osdmap e697: 12 total, 12 up, 10 in 2024-06-27T21:38:50.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:50 smithi192 ceph-mon[32956]: osdmap e698: 12 total, 12 up, 10 in 2024-06-27T21:38:50.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:50 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:38:50.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:50.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:50.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:50.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:50.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-06-27T21:38:50.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:50 smithi190 ceph-mon[32247]: osdmap e697: 12 total, 12 up, 10 in 2024-06-27T21:38:50.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:50 smithi190 ceph-mon[32247]: osdmap e698: 12 total, 12 up, 10 in 2024-06-27T21:38:50.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:50 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:38:50.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:50.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:50.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:50.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:50.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:51.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:51.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:51.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:51.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:51.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:51.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:51.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:51.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:51.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:51 smithi192 ceph-mon[32956]: pgmap v1740: 67 pgs: 4 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 60 active+clean; 126 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 54 objects/s recovering 2024-06-27T21:38:51.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:51 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:38:51.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:51 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:38:51.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:51 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:38:51.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:51 smithi192 ceph-mon[32956]: osdmap e699: 12 total, 12 up, 10 in 2024-06-27T21:38:51.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:51 smithi192 ceph-mon[32956]: 6.b starting backfill to osd.10 from (0'0,0'0] MAX to 692'12171 2024-06-27T21:38:51.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:51.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:51 smithi151 ceph-mon[28965]: pgmap v1740: 67 pgs: 4 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 60 active+clean; 126 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 54 objects/s recovering 2024-06-27T21:38:51.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:51 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:38:51.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:51 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:38:51.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:51 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:38:51.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:51 smithi151 ceph-mon[28965]: osdmap e699: 12 total, 12 up, 10 in 2024-06-27T21:38:51.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:51 smithi151 ceph-mon[28965]: 6.b starting backfill to osd.10 from (0'0,0'0] MAX to 692'12171 2024-06-27T21:38:51.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:51 smithi190 ceph-mon[32247]: pgmap v1740: 67 pgs: 4 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 60 active+clean; 126 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 333/6021 objects misplaced (5.531%); 0 B/s, 54 objects/s recovering 2024-06-27T21:38:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:51 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:38:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:51 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:38:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:51 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:38:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:51 smithi190 ceph-mon[32247]: osdmap e699: 12 total, 12 up, 10 in 2024-06-27T21:38:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:51 smithi190 ceph-mon[32247]: 6.b starting backfill to osd.10 from (0'0,0'0] MAX to 692'12171 2024-06-27T21:38:51.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:51.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:51.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:51.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:51.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:52.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:52.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:52.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:52.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:52.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:52.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:52.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:52.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:52 smithi192 ceph-mon[32956]: 6.14 starting backfill to osd.5 from (0'0,0'0] MAX to 692'11876 2024-06-27T21:38:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:52 smithi192 ceph-mon[32956]: 6.c starting backfill to osd.4 from (0'0,0'0] MAX to 692'12832 2024-06-27T21:38:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:52 smithi192 ceph-mon[32956]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 692'12580 2024-06-27T21:38:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:52 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:38:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:52 smithi192 ceph-mon[32956]: osdmap e700: 12 total, 12 up, 10 in 2024-06-27T21:38:52.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:52.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:52.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:52 smithi151 ceph-mon[28965]: 6.14 starting backfill to osd.5 from (0'0,0'0] MAX to 692'11876 2024-06-27T21:38:52.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:52 smithi151 ceph-mon[28965]: 6.c starting backfill to osd.4 from (0'0,0'0] MAX to 692'12832 2024-06-27T21:38:52.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:52 smithi151 ceph-mon[28965]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 692'12580 2024-06-27T21:38:52.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:52 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:38:52.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:52 smithi151 ceph-mon[28965]: osdmap e700: 12 total, 12 up, 10 in 2024-06-27T21:38:52.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:52.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:52 smithi190 ceph-mon[32247]: 6.14 starting backfill to osd.5 from (0'0,0'0] MAX to 692'11876 2024-06-27T21:38:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:52 smithi190 ceph-mon[32247]: 6.c starting backfill to osd.4 from (0'0,0'0] MAX to 692'12832 2024-06-27T21:38:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:52 smithi190 ceph-mon[32247]: 6.e starting backfill to osd.11 from (0'0,0'0] MAX to 692'12580 2024-06-27T21:38:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:52 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:38:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:52 smithi190 ceph-mon[32247]: osdmap e700: 12 total, 12 up, 10 in 2024-06-27T21:38:52.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:52.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:52.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:52.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:52.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:53.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:53.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:53.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:53.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:53.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:53.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:53.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:53.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:53.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:53.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:53.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:53.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:53.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:53.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:53.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:54.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:54.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:54.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:54.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:53 smithi151 ceph-mon[28965]: pgmap v1743: 67 pgs: 7 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 57 active+clean; 126 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 333/6021 objects misplaced (5.531%) 2024-06-27T21:38:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:53 smithi151 ceph-mon[28965]: osdmap e701: 12 total, 12 up, 10 in 2024-06-27T21:38:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:53 smithi190 ceph-mon[32247]: pgmap v1743: 67 pgs: 7 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 57 active+clean; 126 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 333/6021 objects misplaced (5.531%) 2024-06-27T21:38:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:53 smithi190 ceph-mon[32247]: osdmap e701: 12 total, 12 up, 10 in 2024-06-27T21:38:54.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:53 smithi192 ceph-mon[32956]: pgmap v1743: 67 pgs: 7 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 57 active+clean; 126 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 333/6021 objects misplaced (5.531%) 2024-06-27T21:38:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:53 smithi192 ceph-mon[32956]: osdmap e701: 12 total, 12 up, 10 in 2024-06-27T21:38:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:54.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:54.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:54.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:54.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:54.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:54.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:54.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:54.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:54.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:38:54.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:38:54.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:54.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:54.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:55.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:55.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:55.172 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 8] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:38:55.172 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:38:55.173 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-06-27T21:38:55.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-06-27T21:38:55.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:55.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:55.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:55.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:55.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:55.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:55.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:55.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:55.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:55.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:55.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:55.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:55.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:55.830 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 107591 objects 2024-06-27T21:38:55.830 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :475.274 2024-06-27T21:38:55.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:38:55.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:55.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:38:56.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:55 smithi192 ceph-mon[32956]: pgmap v1745: 67 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 remapped+peering, 4 active+remapped+backfill_toofull, 53 active+clean; 113 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 44 op/s; 6/5427 objects degraded (0.111%); 428/5427 objects misplaced (7.886%); 2.6 MiB/s, 262 objects/s recovering 2024-06-27T21:38:56.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:55 smithi192 ceph-mon[32956]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:56.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:55 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6/5427 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:56.058 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:56.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:56.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:55 smithi151 ceph-mon[28965]: pgmap v1745: 67 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 remapped+peering, 4 active+remapped+backfill_toofull, 53 active+clean; 113 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 44 op/s; 6/5427 objects degraded (0.111%); 428/5427 objects misplaced (7.886%); 2.6 MiB/s, 262 objects/s recovering 2024-06-27T21:38:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:55 smithi151 ceph-mon[28965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:55 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6/5427 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:56.210 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:55 smithi190 ceph-mon[32247]: pgmap v1745: 67 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 7 remapped+peering, 4 active+remapped+backfill_toofull, 53 active+clean; 113 MiB data, 5.3 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 44 op/s; 6/5427 objects degraded (0.111%); 428/5427 objects misplaced (7.886%); 2.6 MiB/s, 262 objects/s recovering 2024-06-27T21:38:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:55 smithi190 ceph-mon[32247]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:38:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:55 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6/5427 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:38:56.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:56.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:56.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:56.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:56.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:56.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:56.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:56.710 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:56.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:56.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:56.908 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-06-27T21:38:56.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-06-27T21:38:56.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:38:56.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:56.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:38:57.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:56 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:38:57.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:56 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:38:57.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:56 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:38:57.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:56 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:38:57.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:56 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:38:57.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:56 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:38:57.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:56 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:38:57.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:56 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:38:57.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:57.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:57.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:57.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:57.255 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:38:57.255 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:38:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:56 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:38:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:56 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:38:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:56 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:38:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:56 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:38:57.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:38:57.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:38:57.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:57.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:57.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:57.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:57.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:57.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:57.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:57.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:57.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:57.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:57 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:38:57.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:57 smithi151 ceph-mon[28965]: pgmap v1746: 67 pgs: 7 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 53 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 7/2505 objects degraded (0.279%); 293/2505 objects misplaced (11.697%); 5.9 MiB/s, 545 objects/s recovering 2024-06-27T21:38:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:57.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:57.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:58.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:57 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:38:58.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:57 smithi192 ceph-mon[32956]: pgmap v1746: 67 pgs: 7 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 53 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 7/2505 objects degraded (0.279%); 293/2505 objects misplaced (11.697%); 5.9 MiB/s, 545 objects/s recovering 2024-06-27T21:38:58.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:58.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:58.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:58.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:57 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:38:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:57 smithi190 ceph-mon[32247]: pgmap v1746: 67 pgs: 7 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 53 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 7/2505 objects degraded (0.279%); 293/2505 objects misplaced (11.697%); 5.9 MiB/s, 545 objects/s recovering 2024-06-27T21:38:58.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:58.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:58.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:38:58.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:38:58.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:58.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:58.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:58.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:58.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:38:58.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:38:58.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:58.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:58.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:59.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:38:59.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:38:59.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:38:59.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:38:59.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:38:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:38:59.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:38:59.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:38:59.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:59.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:59.511 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:59.551 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:38:59.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:38:59.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:38:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:38:59.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:59.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:38:59.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:38:59.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:38:59.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:38:59.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:38:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:38:59.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:00.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:00.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:38:59 smithi151 ceph-mon[28965]: pgmap v1747: 67 pgs: 1 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 14 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 245 op/s; 1/639 objects degraded (0.156%); 54/639 objects misplaced (8.451%); 4.7 MiB/s, 515 objects/s recovering 2024-06-27T21:39:00.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:00.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:38:59 smithi190 ceph-mon[32247]: pgmap v1747: 67 pgs: 1 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 14 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 245 op/s; 1/639 objects degraded (0.156%); 54/639 objects misplaced (8.451%); 4.7 MiB/s, 515 objects/s recovering 2024-06-27T21:39:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:38:59 smithi192 ceph-mon[32956]: pgmap v1747: 67 pgs: 1 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 14 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 245 op/s; 1/639 objects degraded (0.156%); 54/639 objects misplaced (8.451%); 4.7 MiB/s, 515 objects/s recovering 2024-06-27T21:39:00.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:00.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:00.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:00.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:00.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:00.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:00.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:00.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:00.827 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:00.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:00.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:00 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:00.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:00.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:00.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:01.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:01.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:01.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:01.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:01.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:00 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:00 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:01.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:01.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:01.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:01.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:01.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:01.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:01.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:01.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:01.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:01.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:01.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:01.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:01.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:02.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:02.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:01 smithi151 ceph-mon[28965]: pgmap v1748: 67 pgs: 3 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 51 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 241 op/s; 1/6 objects degraded (16.667%); 4.2 MiB/s, 747 objects/s recovering 2024-06-27T21:39:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:01 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:01 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:39:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:01 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:02.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:02.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:01 smithi190 ceph-mon[32247]: pgmap v1748: 67 pgs: 3 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 51 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 241 op/s; 1/6 objects degraded (16.667%); 4.2 MiB/s, 747 objects/s recovering 2024-06-27T21:39:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:01 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:01 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:39:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:01 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:01 smithi192 ceph-mon[32956]: pgmap v1748: 67 pgs: 3 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_toofull, 51 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 241 op/s; 1/6 objects degraded (16.667%); 4.2 MiB/s, 747 objects/s recovering 2024-06-27T21:39:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:01 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:01 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:39:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:01 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:02.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:02.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:02.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:02.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:02.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:02.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:02.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:02.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:02.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:02.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:02.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:02.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:02 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:39:02.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:02 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:39:02.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:02.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:02.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:03.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:03.106 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.107 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:03.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:03.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:02 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:39:03.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:02 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:39:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:02 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:39:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:02 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:39:03.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:03.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:03.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:03.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:03.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:03.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:03.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:03.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:03.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:03.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:03.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:03.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:03 smithi151 ceph-mon[28965]: pgmap v1749: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 1/6 objects degraded (16.667%); 3.5 MiB/s, 702 objects/s recovering 2024-06-27T21:39:03.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:03 smithi151 ceph-mon[28965]: osdmap e702: 12 total, 12 up, 10 in 2024-06-27T21:39:03.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:04.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:04.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:04.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:03 smithi190 ceph-mon[32247]: pgmap v1749: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 1/6 objects degraded (16.667%); 3.5 MiB/s, 702 objects/s recovering 2024-06-27T21:39:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:03 smithi190 ceph-mon[32247]: osdmap e702: 12 total, 12 up, 10 in 2024-06-27T21:39:04.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:04.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:03 smithi192 ceph-mon[32956]: pgmap v1749: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 203 op/s; 1/6 objects degraded (16.667%); 3.5 MiB/s, 702 objects/s recovering 2024-06-27T21:39:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:03 smithi192 ceph-mon[32956]: osdmap e702: 12 total, 12 up, 10 in 2024-06-27T21:39:04.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:04.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:04.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:04.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:04.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:04.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:04.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:04.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:04.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:04.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:04.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:05.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:05.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:05.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:05.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:04 smithi151 ceph-mon[28965]: osdmap e703: 12 total, 12 up, 10 in 2024-06-27T21:39:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:04 smithi151 ceph-mon[28965]: osdmap e704: 12 total, 12 up, 10 in 2024-06-27T21:39:05.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:04 smithi190 ceph-mon[32247]: osdmap e703: 12 total, 12 up, 10 in 2024-06-27T21:39:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:04 smithi190 ceph-mon[32247]: osdmap e704: 12 total, 12 up, 10 in 2024-06-27T21:39:05.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:04 smithi192 ceph-mon[32956]: osdmap e703: 12 total, 12 up, 10 in 2024-06-27T21:39:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:04 smithi192 ceph-mon[32956]: osdmap e704: 12 total, 12 up, 10 in 2024-06-27T21:39:05.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:05.401 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:05.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:05.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:05.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:05.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:05.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:05.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:05.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:05.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:05.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:06.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:06.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:06.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:06.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:05 smithi151 ceph-mon[28965]: pgmap v1752: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 1/6 objects degraded (16.667%); 0 B/s, 599 objects/s recovering 2024-06-27T21:39:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:05 smithi151 ceph-mon[28965]: osdmap e705: 12 total, 12 up, 10 in 2024-06-27T21:39:06.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:05 smithi190 ceph-mon[32247]: pgmap v1752: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 1/6 objects degraded (16.667%); 0 B/s, 599 objects/s recovering 2024-06-27T21:39:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:05 smithi190 ceph-mon[32247]: osdmap e705: 12 total, 12 up, 10 in 2024-06-27T21:39:06.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:05 smithi192 ceph-mon[32956]: pgmap v1752: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+clean+scrubbing, 8 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 49 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 1/6 objects degraded (16.667%); 0 B/s, 599 objects/s recovering 2024-06-27T21:39:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:05 smithi192 ceph-mon[32956]: osdmap e705: 12 total, 12 up, 10 in 2024-06-27T21:39:06.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:06.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:06.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:06.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:06.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:06.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:06.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:06.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:06.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:06.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:07.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:07.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:07.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:07.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:06 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:06 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:07.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:06 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:06 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:07.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:06 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:06 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:07.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:07.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:07.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:07.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:07.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:07.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:07.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:07.787 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:07.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:07.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:07.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:08.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:08.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:08.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:08.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:08.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:07 smithi151 ceph-mon[28965]: pgmap v1755: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.31k objects/s recovering 2024-06-27T21:39:08.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:07 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:08.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:07 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:39:08.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:07 smithi190 ceph-mon[32247]: pgmap v1755: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.31k objects/s recovering 2024-06-27T21:39:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:07 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:07 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:39:08.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:08.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:07 smithi192 ceph-mon[32956]: pgmap v1755: 67 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.31k objects/s recovering 2024-06-27T21:39:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:07 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:07 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:39:08.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:08.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:08.500 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:08.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:08.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:08.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:08.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:08.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:39:08.755 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:39:08.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:08.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:08.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:39:08.786 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:39:08.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:39:08.812 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:39:08.837 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:39:08.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:39:08.861 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:39:08.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:39:08.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:08.889 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:39:08.890 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:39:08.890 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:08 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:39:08.890 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2162063384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:08.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:08.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:08.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:08.915 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:39:08.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:39:08.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:39:08.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:39:08.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:39:08.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:39:08.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:39:08.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:39:09.009 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:39:09.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:39:09.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:09.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:39:09.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:39:09.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:09.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:09.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:09.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:08 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:39:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2162063384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:09.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:08 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:39:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2162063384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:09.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:09.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:09.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:09.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:09.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:09.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:09.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:09.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:09.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:09.905 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:09.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:10.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:10.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:10.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:10.157 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:09 smithi151 ceph-mon[28965]: pgmap v1756: 67 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 peering, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 906 objects/s recovering 2024-06-27T21:39:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:10.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:10.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:09 smithi190 ceph-mon[32247]: pgmap v1756: 67 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 peering, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 906 objects/s recovering 2024-06-27T21:39:10.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:09 smithi192 ceph-mon[32956]: pgmap v1756: 67 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 peering, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 906 objects/s recovering 2024-06-27T21:39:10.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:10.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:10.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:10.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:10.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:10.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:10.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:10.783 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:10.838 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:39:10.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:10.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:10.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:10.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1880473735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:39:11.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:11.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:11.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:11.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:11.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1880473735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:39:11.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1880473735' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:39:11.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:11.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:11.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:11.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:11.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:11.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:11.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:11.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:11.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:11.914 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:11.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:12.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:12.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:12.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:11 smithi151 ceph-mon[28965]: pgmap v1757: 67 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.05k objects/s recovering 2024-06-27T21:39:12.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:11 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:12.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:11 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:12.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1880473735' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:39:12.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:11 smithi151 ceph-mon[28965]: osdmap e706: 12 total, 12 up, 10 in 2024-06-27T21:39:12.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:11 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:12.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:12.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:12.226 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:39:12.226 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:39:12.226 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:39:12.226 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:39:12.226 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:39:12.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:11 smithi190 ceph-mon[32247]: pgmap v1757: 67 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.05k objects/s recovering 2024-06-27T21:39:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:11 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:11 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1880473735' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:39:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:11 smithi190 ceph-mon[32247]: osdmap e706: 12 total, 12 up, 10 in 2024-06-27T21:39:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:11 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:12.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:11 smithi192 ceph-mon[32956]: pgmap v1757: 67 pgs: 3 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 4 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.05k objects/s recovering 2024-06-27T21:39:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:11 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:11 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1880473735' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:39:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:11 smithi192 ceph-mon[32956]: osdmap e706: 12 total, 12 up, 10 in 2024-06-27T21:39:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:11 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:12.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:12.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:12.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:12.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:12.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:12.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:12.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:12.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:12.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:12.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:12.908 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:12 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:12.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:12.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:39:13.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:13.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:13.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:13.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:13.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:12 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:13.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:12 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:13.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:13.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:13.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:13.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:13.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:39:13.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:13.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:13.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:13.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:13.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:13.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:13.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:13.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:13 smithi151 ceph-mon[28965]: pgmap v1759: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 52 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.15k objects/s recovering 2024-06-27T21:39:13.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:13 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:39:13.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:13 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:13.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:13 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:14.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:14.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:14.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:14.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:14.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:13 smithi190 ceph-mon[32247]: pgmap v1759: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 52 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.15k objects/s recovering 2024-06-27T21:39:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:13 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:39:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:13 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:13 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:14.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:13 smithi192 ceph-mon[32956]: pgmap v1759: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 52 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.15k objects/s recovering 2024-06-27T21:39:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:13 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:39:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:13 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:13 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:14.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:14.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:14.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:14.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:14.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:14.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:14.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:14.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:14.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:14.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:14.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:14.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:14 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:39:14.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:14 smithi151 ceph-mon[28965]: osdmap e707: 12 total, 12 up, 10 in 2024-06-27T21:39:14.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:15.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:15.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:15.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:15.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:14 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:39:15.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:14 smithi190 ceph-mon[32247]: osdmap e707: 12 total, 12 up, 10 in 2024-06-27T21:39:15.269 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:14 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:39:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:14 smithi192 ceph-mon[32956]: osdmap e707: 12 total, 12 up, 10 in 2024-06-27T21:39:15.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:15.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:15.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:15.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:15.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:15.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:15.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:15.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:15.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:15.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:15.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:15.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:16.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:16.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:16.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:16.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:15 smithi151 ceph-mon[28965]: pgmap v1761: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 636 objects/s recovering 2024-06-27T21:39:16.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:16.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:16.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:15 smithi190 ceph-mon[32247]: pgmap v1761: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 636 objects/s recovering 2024-06-27T21:39:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:15 smithi192 ceph-mon[32956]: pgmap v1761: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 636 objects/s recovering 2024-06-27T21:39:16.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:16.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:16.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:16.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:16.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:16.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:16.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:16.733 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:16.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:16.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:16.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:16.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:16.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:17.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:17.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:17.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:17.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:16 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:17.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:16 smithi151 ceph-mon[28965]: osdmap e708: 12 total, 12 up, 10 in 2024-06-27T21:39:17.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:16 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:17.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:17.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:16 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:16 smithi190 ceph-mon[32247]: osdmap e708: 12 total, 12 up, 10 in 2024-06-27T21:39:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:16 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:16 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:16 smithi192 ceph-mon[32956]: osdmap e708: 12 total, 12 up, 10 in 2024-06-27T21:39:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:16 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:17.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:17.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:17.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:17.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:17.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:17.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:17.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:17.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:17.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:17.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:17.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:17.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:17 smithi151 ceph-mon[28965]: pgmap v1763: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 56 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.14k objects/s recovering 2024-06-27T21:39:17.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:17 smithi151 ceph-mon[28965]: osdmap e709: 12 total, 12 up, 10 in 2024-06-27T21:39:17.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:17 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:17.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:18.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:18.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:18.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:18.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:18.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:17 smithi190 ceph-mon[32247]: pgmap v1763: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 56 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.14k objects/s recovering 2024-06-27T21:39:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:17 smithi190 ceph-mon[32247]: osdmap e709: 12 total, 12 up, 10 in 2024-06-27T21:39:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:17 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:17 smithi192 ceph-mon[32956]: pgmap v1763: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 56 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 1.14k objects/s recovering 2024-06-27T21:39:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:17 smithi192 ceph-mon[32956]: osdmap e709: 12 total, 12 up, 10 in 2024-06-27T21:39:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:17 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:18.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:18.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:18.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:18.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:18.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:18.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:18.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:18.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:18.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:18.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:18.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:18.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:18.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:19.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:19.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:19.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:18 smithi151 ceph-mon[28965]: osdmap e710: 12 total, 12 up, 10 in 2024-06-27T21:39:19.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:19.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:19.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:18 smithi190 ceph-mon[32247]: osdmap e710: 12 total, 12 up, 10 in 2024-06-27T21:39:19.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:18 smithi192 ceph-mon[32956]: osdmap e710: 12 total, 12 up, 10 in 2024-06-27T21:39:19.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:19.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:19.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:19.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:19.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:19.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:19.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:19.755 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:19.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:19.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:19.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:19.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:20.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:20.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:20.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:39:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.132 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:20.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:20.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:19 smithi151 ceph-mon[28965]: pgmap v1766: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 56 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 960 objects/s recovering 2024-06-27T21:39:20.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:19 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:39:20.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:19 smithi151 ceph-mon[28965]: osdmap e711: 12 total, 12 up, 10 in 2024-06-27T21:39:20.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:19 smithi151 ceph-mon[28965]: 6.1f scrub starts 2024-06-27T21:39:20.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:19 smithi151 ceph-mon[28965]: osdmap e712: 12 total, 12 up, 10 in 2024-06-27T21:39:20.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:20.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:19 smithi190 ceph-mon[32247]: pgmap v1766: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 56 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 960 objects/s recovering 2024-06-27T21:39:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:19 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:39:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:19 smithi190 ceph-mon[32247]: osdmap e711: 12 total, 12 up, 10 in 2024-06-27T21:39:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:19 smithi190 ceph-mon[32247]: 6.1f scrub starts 2024-06-27T21:39:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:19 smithi190 ceph-mon[32247]: osdmap e712: 12 total, 12 up, 10 in 2024-06-27T21:39:20.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:19 smithi192 ceph-mon[32956]: pgmap v1766: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 56 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 960 objects/s recovering 2024-06-27T21:39:20.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:19 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:39:20.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:19 smithi192 ceph-mon[32956]: osdmap e711: 12 total, 12 up, 10 in 2024-06-27T21:39:20.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:19 smithi192 ceph-mon[32956]: 6.1f scrub starts 2024-06-27T21:39:20.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:19 smithi192 ceph-mon[32956]: osdmap e712: 12 total, 12 up, 10 in 2024-06-27T21:39:20.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:20.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:20.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:20.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:20.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:20.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:20.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:20.751 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:20.788 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_4' removed 2024-06-27T21:39:20.853 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:20.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:20.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:20.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:21.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:21.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:21.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:21.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4067981022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:39:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4067981022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:39:21.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:20 smithi151 ceph-mon[28965]: osdmap e713: 12 total, 12 up, 10 in 2024-06-27T21:39:21.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4067981022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:39:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4067981022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:39:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:20 smithi190 ceph-mon[32247]: osdmap e713: 12 total, 12 up, 10 in 2024-06-27T21:39:21.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4067981022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:39:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4067981022' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:39:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:20 smithi192 ceph-mon[32956]: osdmap e713: 12 total, 12 up, 10 in 2024-06-27T21:39:21.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:21.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:21.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:21.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:21.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:21.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:21.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:21.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:21.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:21.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:21.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:22.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:22.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:22.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:22.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:21 smithi151 ceph-mon[28965]: pgmap v1769: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 58 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 381 objects/s recovering 2024-06-27T21:39:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:21 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:21 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:21 smithi190 ceph-mon[32247]: pgmap v1769: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 58 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 381 objects/s recovering 2024-06-27T21:39:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:21 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:21 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:22.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:22.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:21 smithi192 ceph-mon[32956]: pgmap v1769: 67 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 58 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 1/6 objects degraded (16.667%); 0 B/s, 381 objects/s recovering 2024-06-27T21:39:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:21 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:22.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:21 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:22.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:22.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:22.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:22.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:22.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:22.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-06-27T21:39:22.681 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:22.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:22.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:22.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:22.935 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:22.936 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:22.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:22.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:22.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:22 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:22.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:22 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:22.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:22 smithi151 ceph-mon[28965]: osdmap e714: 12 total, 12 up, 10 in 2024-06-27T21:39:23.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:23.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:23.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:23.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:22 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:22 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:23.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:22 smithi190 ceph-mon[32247]: osdmap e714: 12 total, 12 up, 10 in 2024-06-27T21:39:23.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:22 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:22 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:23.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:22 smithi192 ceph-mon[32956]: osdmap e714: 12 total, 12 up, 10 in 2024-06-27T21:39:23.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:23.404 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:39:23.405 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:39:23.405 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-06-27T21:39:23.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-06-27T21:39:23.406 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:23.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:23.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:23.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:23.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:23.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:23.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:23.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:23.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:23.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:23.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:23.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:23.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:23.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:23 smithi151 ceph-mon[28965]: pgmap v1772: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:23.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:23 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:23.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:23 smithi151 ceph-mon[28965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-06-27T21:39:23.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:23 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded) 2024-06-27T21:39:24.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:24.102 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:24.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:24.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:23 smithi190 ceph-mon[32247]: pgmap v1772: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:23 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:23 smithi190 ceph-mon[32247]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-06-27T21:39:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:23 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded) 2024-06-27T21:39:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:23 smithi192 ceph-mon[32956]: pgmap v1772: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:23 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:23 smithi192 ceph-mon[32956]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-06-27T21:39:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:23 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded) 2024-06-27T21:39:24.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:24.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:24.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:24.489 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:24.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:24.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:24.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:24.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:24.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:24.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:24.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:24.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:24.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:25.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:25.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:25.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:24 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:39:25.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:24 smithi151 ceph-mon[28965]: osdmap e715: 12 total, 12 up, 10 in 2024-06-27T21:39:25.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:25.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:24 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:39:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:24 smithi190 ceph-mon[32247]: osdmap e715: 12 total, 12 up, 10 in 2024-06-27T21:39:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:24 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:39:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:24 smithi192 ceph-mon[32956]: osdmap e715: 12 total, 12 up, 10 in 2024-06-27T21:39:25.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:25.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:25.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:25.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:25.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:25.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:25.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:25.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:25.854 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:25.881 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:25.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:25.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:26.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:26.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:26.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:26.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:25 smithi151 ceph-mon[28965]: pgmap v1773: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:26.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:25 smithi190 ceph-mon[32247]: pgmap v1773: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:26.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:26.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:26.289 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:25 smithi192 ceph-mon[32956]: pgmap v1773: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:26.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:26.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:26.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:26.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:26.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:26.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:26.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:26.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:26.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:26.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:26.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:27.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:27.028 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:27.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:27.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:27.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:26 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:27.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:26 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:27.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:26 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:27.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:27.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:27.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:27.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:27.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:27.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:27.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:27.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:27.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:27.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:27.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:27.915 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:39:28.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:28.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:28.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:28.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:27 smithi151 ceph-mon[28965]: pgmap v1775: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:27 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4229995352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:39:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:27 smithi190 ceph-mon[32247]: pgmap v1775: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:27 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4229995352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:39:28.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:28.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:27 smithi192 ceph-mon[32956]: pgmap v1775: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:27 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4229995352' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:39:28.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:28.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:28.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:28.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:28.631 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:28.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:28.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:28.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:28.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:28.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:28.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:28.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:29.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:28 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:29.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:28 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:29.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4229995352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:39:29.023 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:28 smithi151 ceph-mon[28965]: osdmap e716: 12 total, 12 up, 10 in 2024-06-27T21:39:29.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:29.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:29.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:29.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:29.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:28 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:29.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:28 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:29.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4229995352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:39:29.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:28 smithi192 ceph-mon[32956]: osdmap e716: 12 total, 12 up, 10 in 2024-06-27T21:39:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:28 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:28 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4229995352' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:39:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:28 smithi190 ceph-mon[32247]: osdmap e716: 12 total, 12 up, 10 in 2024-06-27T21:39:29.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:29.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:29.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:29.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:29.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:29.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:29.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:29.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:29.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:29.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:29.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:29.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:29.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:30.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:30.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:30.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:29 smithi151 ceph-mon[28965]: pgmap v1777: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:30.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:29 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:39:30.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:29 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:39:30.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:29 smithi151 ceph-mon[28965]: osdmap e717: 12 total, 12 up, 10 in 2024-06-27T21:39:30.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:30.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:29 smithi190 ceph-mon[32247]: pgmap v1777: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:29 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:39:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:29 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:39:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:29 smithi190 ceph-mon[32247]: osdmap e717: 12 total, 12 up, 10 in 2024-06-27T21:39:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:29 smithi192 ceph-mon[32956]: pgmap v1777: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:29 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:39:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:29 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:39:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:29 smithi192 ceph-mon[32956]: osdmap e717: 12 total, 12 up, 10 in 2024-06-27T21:39:30.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:30.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:30.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:30.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:30.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:30.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:30.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:30.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:30.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:30.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:30.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:30.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:31.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:31.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:39:31.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1525169825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:31.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:31.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:31.222 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:39:31.222 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:39:31.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:39:31.222 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:39:31.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.222 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:51275 6.2 6:4afb8263:::benchmark_data_smithi192_117994_object51274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:30:07.374220+0000", 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 563.84481695299996, 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46063098099999999, 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 51275 2024-06-27T21:39:31.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374220+0000", 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374220+0000", 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374221+0000", 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1200000000000001e-07 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374260+0000", 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8996999999999999e-05 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374271+0000", 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0910000000000001e-05 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374276+0000", 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7629999999999997e-06 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374325+0000", 2024-06-27T21:39:31.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8838000000000002e-05 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374463+0000", 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013785699999999999 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,11", 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.374567+0000", 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103649 2024-06-27T21:39:31.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.375755+0000", 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001188124 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.376074+0000", 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031923100000000001 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834774+0000", 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45869947599999999 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834818+0000", 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4359e-05 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834851+0000", 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2864999999999998e-05 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.229 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:51275 6.2 e486/467 ondisk, result = 0)", 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:30:07.376129+0000", 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 563.84290809100003, 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45872734100000001, 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.376129+0000", 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.376129+0000", 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.376129+0000", 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9599999999999999e-07 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.376132+0000", 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9390000000000002e-06 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834704+0000", 2024-06-27T21:39:31.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45857200300000001 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834711+0000", 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9720000000000003e-06 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834762+0000", 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0837000000000001e-05 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834769+0000", 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1210000000000001e-06 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:30:07.834856+0000", 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6972999999999996e-05 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:156111 6.f 6:fcae7150:::benchmark_data_smithi192_117994_object48517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:13.813504+0000", 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 377.40553270300001, 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.64207833999999997, 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156111 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813504+0000", 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813504+0000", 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813505+0000", 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813510+0000", 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813519+0000", 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4349999999999997e-06 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813524+0000", 2024-06-27T21:39:31.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5910000000000003e-06 2024-06-27T21:39:31.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813568+0000", 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3044000000000003e-05 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813663+0000", 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5563000000000006e-05 2024-06-27T21:39:31.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813747+0000", 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3908999999999999e-05 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.817639+0000", 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003892202 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.818732+0000", 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010923339999999999 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:14.455507+0000", 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63677589499999998 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:14.455549+0000", 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1356000000000002e-05 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:14.455582+0000", 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3569999999999999e-05 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:156223 6.e 6:74cd8b3d:::benchmark_data_smithi192_117994_object48629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:15.194773+0000", 2024-06-27T21:39:31.242 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 376.024264147, 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43812209600000002, 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156223 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194773+0000", 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194773+0000", 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194775+0000", 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-06-27T21:39:31.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194787+0000", 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1593000000000001e-05 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194827+0000", 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0667999999999998e-05 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194836+0000", 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1619999999999997e-06 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194893+0000", 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6388000000000003e-05 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.195005+0000", 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011181 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.195093+0000", 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7928000000000002e-05 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.198760+0000", 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036673819999999998 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.199244+0000", 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048398899999999999 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.632813+0000", 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43356867900000001 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.632855+0000", 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2410000000000002e-05 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.632895+0000", 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9650000000000002e-05 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.a e541,516 6:59eebc9d:::benchmark_data_smithi192_117994_object18140:head,511'7778)", 2024-06-27T21:39:31.249 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:19.275215+0000", 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 371.94382165299999, 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52697154400000001, 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275215+0000", 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275215+0000", 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275216+0000", 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0200000000000001e-07 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275220+0000", 2024-06-27T21:39:31.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8260000000000003e-06 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802078+0000", 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52685791000000004 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802083+0000", 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3730000000000001e-06 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802134+0000", 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1372000000000002e-05 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802141+0000", 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3999999999999997e-06 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802187+0000", 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5961000000000003e-05 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.13 551/550 [PushOp(6:cdfd5352:::benchmark_data_smithi192_117994_object50909:head, version: 486'3256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:cdfd5352:::benchmark_data_smithi192_117994_object50909:head@486'3256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 361.42525694199998, 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46384465499999999, 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:39:31.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7100000000000001e-07 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793880+0000", 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9273999999999999e-05 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256375+0000", 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.462494984 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256389+0000", 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3978999999999999e-05 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256461+0000", 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2417999999999997e-05 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256471+0000", 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0219e-05 2024-06-27T21:39:31.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.257624+0000", 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011531099999999999 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.13 e563,550 6:ca83749f:::benchmark_data_smithi192_117994_object37811:head,519'9236)", 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.718706113, 2024-06-27T21:39:31.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50509707500000001, 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:39:31.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000005e-07 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500337+0000", 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5650000000000001e-06 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005291+0000", 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50495398700000005 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005298+0000", 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.764e-06 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005368+0000", 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0366999999999999e-05 2024-06-27T21:39:31.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005375+0000", 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7259999999999996e-06 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005428+0000", 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2917000000000002e-05 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:168135 6.f 6:f725a463:::benchmark_data_smithi192_117994_object60541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:14.389388+0000", 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 256.82964896599998, 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62669164200000005, 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.263 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168135 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389388+0000", 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389388+0000", 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389389+0000", 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3799999999999999e-06 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389394+0000", 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2229999999999998e-06 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389407+0000", 2024-06-27T21:39:31.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3093e-05 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389416+0000", 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9150000000000002e-06 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389474+0000", 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7908000000000002e-05 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389634+0000", 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015965 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389793+0000", 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158658 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.393717+0000", 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039243730000000001 2024-06-27T21:39:31.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944686+0000", 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55096871300000005 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015949+0000", 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.071262961999999999 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.016021+0000", 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2797000000000001e-05 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.016079+0000", 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7970000000000002e-05 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.269 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:168135 6.f e587/467 ondisk, result = 0)", 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 256.824714094, 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55037249099999996, 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6899999999999997e-07 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394327+0000", 2024-06-27T21:39:31.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.614e-06 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944602+0000", 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55027552199999996 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944611+0000", 2024-06-27T21:39:31.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8000000000000004e-06 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944667+0000", 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5957e-05 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944678+0000", 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0702e-05 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944695+0000", 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7326999999999999e-05 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:168135 6.f e587/467 ondisk, result = 0)", 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:14.394376+0000", 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 256.82466055200001, 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62170963899999998, 2024-06-27T21:39:31.274 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394376+0000", 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394376+0000", 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394377+0000", 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.63e-07 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394380+0000", 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9660000000000001e-06 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015875+0000", 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62149521900000004 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015878+0000", 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6540000000000001e-06 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015929+0000", 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0705999999999999e-05 2024-06-27T21:39:31.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015940+0000", 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1209999999999999e-05 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.016086+0000", 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014542099999999999 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.278 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.5 594/548 [PushOp(6:a5ffe05c:::benchmark_data_smithi192_117994_object99841:head, version: 501'6175, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:a5ffe05c:::benchmark_data_smithi192_117994_object99841:head@501'6175, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:30.449103+0000", 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 240.76993367599999, 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61745724199999996, 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449103+0000", 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449103+0000", 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449104+0000", 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.42e-06 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449179+0000", 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4813000000000004e-05 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064721+0000", 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61554164700000003 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064753+0000", 2024-06-27T21:39:31.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2100000000000001e-05 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064899+0000", 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014559999999999999 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064915+0000", 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5981000000000002e-05 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.066560+0000", 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001645681 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170693 6.f 6:ff66131a:::benchmark_data_smithi192_117994_object63099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e595)", 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 236.512053178, 2024-06-27T21:39:31.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.480746165, 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170693 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:39:31.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4799999999999997e-07 2024-06-27T21:39:31.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706988+0000", 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5769999999999998e-06 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706997+0000", 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0219999999999993e-06 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707002+0000", 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9180000000000002e-06 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707047+0000", 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707144+0000", 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6757999999999998e-05 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707233+0000", 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8937000000000006e-05 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.711312+0000", 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040799169999999997 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.711381+0000", 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8541000000000007e-05 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:35.187612+0000", 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.476230663 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:35.187677+0000", 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5320999999999993e-05 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:35.187730+0000", 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2748999999999999e-05 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:172014 6.5 e597/548 6:a50abf01:::benchmark_data_smithi192_117994_object64420:head v 597'10229, mlcod=502'6233)", 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:45.302797+0000", 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 225.91623942199999, 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54581944999999998, 2024-06-27T21:39:31.290 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302797+0000", 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302797+0000", 2024-06-27T21:39:31.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302798+0000", 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2499999999999995e-07 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302802+0000", 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5260000000000004e-06 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844258+0000", 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54145553800000001 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844269+0000", 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1063999999999999e-05 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844347+0000", 2024-06-27T21:39:31.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7813999999999997e-05 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844381+0000", 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4703000000000003e-05 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.848563+0000", 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041811110000000004 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.848617+0000", 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4169000000000003e-05 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172333 6.f 6:f63ad1db:::benchmark_data_smithi192_117994_object64739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.615351+0000", 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 222.603685587, 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45670154899999998, 2024-06-27T21:39:31.295 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172333 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615351+0000", 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615351+0000", 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615352+0000", 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.125e-06 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615357+0000", 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6079999999999998e-06 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615367+0000", 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0329999999999999e-05 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:39:31.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:39:31.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:39:31.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1525169825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:39:31.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1525169825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615373+0000", 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9549999999999997e-06 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615416+0000", 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2525999999999999e-05 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615549+0000", 2024-06-27T21:39:31.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013355800000000001 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615699+0000", 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149903 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.619475+0000", 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003776223 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.619560+0000", 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4375000000000004e-05 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.071948+0000", 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45238802500000003 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.072009+0000", 2024-06-27T21:39:31.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0980999999999998e-05 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.072053+0000", 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3940000000000003e-05 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.20 657/654 [PushReplyOp(6:070deb76:::benchmark_data_smithi192_117994_object88876:head)])", 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:39.358439+0000", 2024-06-27T21:39:31.304 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 111.860597591, 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32801349899999999, 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358439+0000", 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358439+0000", 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358440+0000", 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7000000000000005e-07 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358443+0000", 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5240000000000001e-06 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.685723+0000", 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32728001000000001 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.685731+0000", 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5299999999999999e-06 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.685776+0000", 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5467999999999999e-05 2024-06-27T21:39:31.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.686453+0000", 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00067639700000000002 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:197630 6.14 6:2a19bfa2:::benchmark_data_smithi192_117994_object90036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:00.420281+0000", 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.798756084000004, 2024-06-27T21:39:31.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27922517600000002, 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197630 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420281+0000", 2024-06-27T21:39:31.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420281+0000", 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420282+0000", 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.13e-06 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420286+0000", 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5000000000000001e-06 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420297+0000", 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0769e-05 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420303+0000", 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.908e-06 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420349+0000", 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6419999999999999e-05 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420439+0000", 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9881000000000002e-05 2024-06-27T21:39:31.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420764+0000", 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000324498 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.422295+0000", 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015308019999999999 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.423275+0000", 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00098013799999999997 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699419+0000", 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27614467999999998 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699463+0000", 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3758999999999999e-05 2024-06-27T21:39:31.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699506+0000", 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2691000000000002e-05 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:197630 6.14 e668/663 ondisk, result = 0)", 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:00.426445+0000", 2024-06-27T21:39:31.315 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 90.792591375000001, 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.273065212, 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.426445+0000", 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.426445+0000", 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.426447+0000", 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.533e-06 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.426455+0000", 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4260000000000003e-06 2024-06-27T21:39:31.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699355+0000", 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27289948400000003 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699360+0000", 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1340000000000003e-06 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699408+0000", 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7985000000000002e-05 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699414+0000", 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8050000000000003e-06 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699511+0000", 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6844999999999999e-05 2024-06-27T21:39:31.319 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:202834 6.1e e673/654)", 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:07.812600+0000", 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 83.406436419000002, 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63657862700000001, 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:39:31.320 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812600+0000", 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812600+0000", 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812601+0000", 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812603+0000", 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6259999999999999e-06 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449019+0000", 2024-06-27T21:39:31.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63641561099999999 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449038+0000", 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8797e-05 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449166+0000", 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012861400000000001 2024-06-27T21:39:31.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449179+0000", 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2578000000000001e-05 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:203122 6.20 6:0683f0fd:::benchmark_data_smithi192_117994_object95528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:08.156724+0000", 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 83.062312954999996, 2024-06-27T21:39:31.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35273166299999997, 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 203122 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156724+0000", 2024-06-27T21:39:31.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156724+0000", 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156725+0000", 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1209999999999999e-06 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156729+0000", 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.228e-06 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156742+0000", 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.279e-05 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156751+0000", 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6969999999999999e-06 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156813+0000", 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2285000000000004e-05 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156957+0000", 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143893 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.157252+0000", 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029488700000000002 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.158755+0000", 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015033379999999999 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.159055+0000", 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029955500000000001 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.509388+0000", 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.350333216 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.509426+0000", 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7979999999999999e-05 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.509455+0000", 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9672999999999999e-05 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:203132 6.e 6:76f73f22:::benchmark_data_smithi192_117994_object95538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:08.184754+0000", 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 83.034282634999997, 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31415277899999999, 2024-06-27T21:39:31.331 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 203132 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184754+0000", 2024-06-27T21:39:31.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184754+0000", 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184755+0000", 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6299999999999993e-07 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184760+0000", 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4730000000000002e-06 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184770+0000", 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0234999999999999e-05 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184775+0000", 2024-06-27T21:39:31.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6400000000000002e-06 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184832+0000", 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6573000000000002e-05 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184928+0000", 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5589999999999998e-05 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.185337+0000", 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040961399999999999 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.187150+0000", 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018128510000000001 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.194613+0000", 2024-06-27T21:39:31.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074630190000000004 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.498805+0000", 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30419222899999998 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.498853+0000", 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7305000000000002e-05 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:39:31.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.498907+0000", 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4286999999999997e-05 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:39:31.338 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:39:31.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:31.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:31.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:31.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:31.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:31.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:31.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:31.727 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:39:31.727 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:39:31.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:31.748 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:39:31.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:39:31.769 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:39:31.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:39:31.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:39:31.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:39:31.813 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:39:31.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:39:31.835 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:39:31.835 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:39:31.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:31.856 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:39:31.857 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:39:31.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:31.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:31.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:31.976 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:39:31.976 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:39:31.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:39:31.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:31 smithi151 ceph-mon[28965]: pgmap v1779: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:31.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:31 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:39:31.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:31 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:39:31.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:31 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:39:31.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:31 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:39:31.998 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:39:31.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:39:32.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:32.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:39:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:39:32.043 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:39:32.044 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:39:32.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:39:32.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:39:32.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:32.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:32.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:31 smithi190 ceph-mon[32247]: pgmap v1779: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:31 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:39:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:31 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:39:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:31 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:39:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:31 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:39:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:31 smithi192 ceph-mon[32956]: pgmap v1779: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:31 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:39:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:31 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:39:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:31 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:39:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:31 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:39:32.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:32.374 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:32.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:32.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:32.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:32.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:32.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:32.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:32.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:32.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:32.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:32.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:33.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:39:33.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:39:33.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:39:33.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:39:33.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:39:33.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:39:33.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:32 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:39:33.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:33.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:39:33.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:32 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:39:33.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:39:33.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:33.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:39:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:39:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:39:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:39:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:39:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:39:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:32 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:39:33.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:33.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:33.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:33.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:33.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:33.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:33.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:33.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:33.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:33.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:33.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:34.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:34.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:34.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:34.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:34.161 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: pgmap v1780: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:34.161 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:39:34.161 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2914124064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:39:34.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:33 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:39:34.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-06-27T21:39:34.210 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:34.210 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:34.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: pgmap v1780: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:39:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:39:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2914124064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-27T21:39:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:39:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:39:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:39:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:33 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:39:34.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: pgmap v1780: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:39:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2914124064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:39:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:33 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:39:34.327 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_5' created 2024-06-27T21:39:34.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:34.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:34.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:34.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:34.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:34.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:34.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:34.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:34.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:34.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:35.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:39:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:39:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:39:35.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:39:35.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2914124064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-27T21:39:35.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: osdmap e718: 12 total, 12 up, 10 in 2024-06-27T21:39:35.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:34 smithi190 ceph-mon[32247]: osdmap e719: 12 total, 12 up, 10 in 2024-06-27T21:39:35.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:35.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:35.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:39:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:39:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:39:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:39:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2914124064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-27T21:39:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: osdmap e718: 12 total, 12 up, 10 in 2024-06-27T21:39:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:34 smithi151 ceph-mon[28965]: osdmap e719: 12 total, 12 up, 10 in 2024-06-27T21:39:35.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-06-27T21:39:35.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:35.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:39:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:39:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:39:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:39:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2914124064' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-06-27T21:39:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: osdmap e718: 12 total, 12 up, 10 in 2024-06-27T21:39:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:34 smithi192 ceph-mon[32956]: osdmap e719: 12 total, 12 up, 10 in 2024-06-27T21:39:35.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:35.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:35.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.527 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:35.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:35.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:35.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:35.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:35.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:35.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:35.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:35.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:36.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:36.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:36.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:36.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:36.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:35 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:39:36.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:35 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:39:36.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:35 smithi151 ceph-mon[28965]: pgmap v1781: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:36.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:35 smithi151 ceph-mon[28965]: osdmap e720: 12 total, 12 up, 10 in 2024-06-27T21:39:36.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:35 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:39:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:35 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:39:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:35 smithi190 ceph-mon[32247]: pgmap v1781: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:35 smithi190 ceph-mon[32247]: osdmap e720: 12 total, 12 up, 10 in 2024-06-27T21:39:36.273 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:36.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:35 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:39:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:35 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:39:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:35 smithi192 ceph-mon[32956]: pgmap v1781: 31 pgs: 31 active+clean; 577 KiB data, 5.0 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:35 smithi192 ceph-mon[32956]: osdmap e720: 12 total, 12 up, 10 in 2024-06-27T21:39:36.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:36.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:36.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:36.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:36.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:36.666 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:36.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:36.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:36.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:36.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:36.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:36.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:37.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:37.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:37.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:37.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:37.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:36 smithi190 ceph-mon[32247]: pgmap v1785: 47 pgs: 1 creating+activating, 2 creating+peering, 4 unknown, 40 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:36 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:39:37.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:36 smithi192 ceph-mon[32956]: pgmap v1785: 47 pgs: 1 creating+activating, 2 creating+peering, 4 unknown, 40 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:36 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:39:37.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:37.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:36 smithi151 ceph-mon[28965]: pgmap v1785: 47 pgs: 1 creating+activating, 2 creating+peering, 4 unknown, 40 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:37.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:36 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:39:37.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:37.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:37.546 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:37.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:37.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:37.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:37.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:37.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:37.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:37.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:38.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:38.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:38.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:38.164 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:38.172 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:38.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:39:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:37 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:39:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:37 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:39:38.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:37 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:39:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:37 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:39:38.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:38.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:38.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:37 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:39:38.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:37 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:39:38.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:38.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:38.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:38.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:38.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:38.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:38.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:38.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:38.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:38.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:39.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:39.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:39.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:39.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:38 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:39:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:38 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:39:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:38 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:39:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:38 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:39:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:38 smithi190 ceph-mon[32247]: pgmap v1786: 47 pgs: 1 creating+activating, 2 creating+peering, 2 unknown, 42 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:39.285 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:39.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:38 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:39:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:38 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:39:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:38 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:39:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:38 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:39:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:38 smithi192 ceph-mon[32956]: pgmap v1786: 47 pgs: 1 creating+activating, 2 creating+peering, 2 unknown, 42 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:39.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:39.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:38 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:39:39.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:38 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:39:39.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:38 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:39:39.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:38 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:39:39.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:38 smithi151 ceph-mon[28965]: pgmap v1786: 47 pgs: 1 creating+activating, 2 creating+peering, 2 unknown, 42 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:39.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:39.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:39.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:39.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:39.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:39.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:39.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:39.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:39.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:39.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:39.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:39.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:40.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:40.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:40.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:40.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:40.292 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:39:40.292 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:39:40.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:40.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:40.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:40.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:40.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:40.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:40.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:40.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:40.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:40.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:40.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:40.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:40 smithi151 ceph-mon[28965]: osdmap e721: 12 total, 12 up, 10 in 2024-06-27T21:39:40.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:40.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:41.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:40 smithi192 ceph-mon[32956]: osdmap e721: 12 total, 12 up, 10 in 2024-06-27T21:39:41.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:41.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:39:41.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:41.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:41.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:40 smithi190 ceph-mon[32247]: osdmap e721: 12 total, 12 up, 10 in 2024-06-27T21:39:41.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:41.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:41.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:41.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:41.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:41.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:41.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:41.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:41.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:41.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:41.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:41.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:41.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:42.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:41 smithi151 ceph-mon[28965]: pgmap v1788: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:42.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:42.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:42.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:41 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:39:42.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:41 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:39:42.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:42.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:42.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:42.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:41 smithi190 ceph-mon[32247]: pgmap v1788: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:41 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:39:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:41 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:39:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:41 smithi192 ceph-mon[32956]: pgmap v1788: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:41 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:39:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:41 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:39:42.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:42.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:42.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:42.459 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:42.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:42.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.587 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:42.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:42.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:42.811 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:39:42.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:42.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:42.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:42 smithi151 ceph-mon[28965]: osdmap e722: 12 total, 12 up, 10 in 2024-06-27T21:39:42.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1971605554' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:39:42.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:42.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:42.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:42.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:42.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:43.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:43.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:43.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:43.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:42 smithi190 ceph-mon[32247]: osdmap e722: 12 total, 12 up, 10 in 2024-06-27T21:39:43.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1971605554' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:39:43.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:43.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:42 smithi192 ceph-mon[32956]: osdmap e722: 12 total, 12 up, 10 in 2024-06-27T21:39:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1971605554' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:39:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:43.314 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:43.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:43.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:43.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:43.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:43.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:43.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:43.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:43.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:43.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:43.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:43.974 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:43.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:44.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: pgmap v1790: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:44.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:39:44.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:39:44.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:39:44.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1971605554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:39:44.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: osdmap e723: 12 total, 12 up, 10 in 2024-06-27T21:39:44.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:39:44.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:43 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:39:44.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:44.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:44.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:44.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: pgmap v1790: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:39:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:39:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:39:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1971605554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:39:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: osdmap e723: 12 total, 12 up, 10 in 2024-06-27T21:39:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:39:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:43 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:39:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: pgmap v1790: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1971605554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: osdmap e723: 12 total, 12 up, 10 in 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:39:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:43 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:39:44.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:44.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:44.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:44.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:44.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:44.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:44.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:44.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:44.788 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-06-27T21:39:44.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:44.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:44.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:44.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:44.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:45.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:45.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:45.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: osdmap e724: 12 total, 12 up, 10 in 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1805151625,config=2353167246,health=2095200028,kv=3661340479}) 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3249155906}) 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195297825}) 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2030402540}) 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942527502}) 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1175234200}) 2024-06-27T21:39:45.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2274791215}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401956763}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3996391476}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=450241051}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2673117186}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871638701}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3522937007}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1213543075}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523085675}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=16109540}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466864984}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925185255}) 2024-06-27T21:39:45.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078814924}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320948124}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1832831610}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523267717}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405714987}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=967970001}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=571356781}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349071982}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=855761480}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=555858979}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168265397}) 2024-06-27T21:39:45.054 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027239739}) 2024-06-27T21:39:45.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3122143378}) 2024-06-27T21:39:45.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=341360269}) 2024-06-27T21:39:45.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207801183}) 2024-06-27T21:39:45.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2459626780}) 2024-06-27T21:39:45.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277051312}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1259541996}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975892697}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095549956}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3565136999}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1633556252}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2552037399}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2301480247}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3194561803}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564577811}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2629353359}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429708977}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3452000053}) 2024-06-27T21:39:45.056 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=976935863}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776793539}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=156887537}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004391141}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3918023766}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=707341606}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589539638}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3056572892}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996144269}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605913326}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339260140}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2997244417}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2231763606}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173181324}) 2024-06-27T21:39:45.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091279710}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=204375543}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999015565}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716802452}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561414620}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=86,mdsmap=3,mgr=11} crc {logm=304405197,mdsmap=2339142170,mgr=1458972961}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=80} crc {mgr=2182222232,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=1367315846,mon_config_key=2528972433}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1436394348}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=635237681}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=311744889,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=207983073,osdmap=141961322}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=144945941}) 2024-06-27T21:39:45.058 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=37} crc {osdmap=2193049873}) 2024-06-27T21:39:45.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1828723199' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:39:45.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1828723199' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:39:45.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:44 smithi151 ceph-mon[28965]: osdmap e725: 12 total, 12 up, 10 in 2024-06-27T21:39:45.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:45.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:45.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:45.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:39:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:39:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: osdmap e724: 12 total, 12 up, 10 in 2024-06-27T21:39:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1805151625,config=2353167246,health=2095200028,kv=3661340479}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3249155906}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195297825}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2030402540}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942527502}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1175234200}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2274791215}) 2024-06-27T21:39:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401956763}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3996391476}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=450241051}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2673117186}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871638701}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3522937007}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1213543075}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523085675}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=16109540}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466864984}) 2024-06-27T21:39:45.268 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925185255}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078814924}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320948124}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1832831610}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523267717}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405714987}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=967970001}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=571356781}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349071982}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=855761480}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=555858979}) 2024-06-27T21:39:45.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168265397}) 2024-06-27T21:39:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027239739}) 2024-06-27T21:39:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3122143378}) 2024-06-27T21:39:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=341360269}) 2024-06-27T21:39:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207801183}) 2024-06-27T21:39:45.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2459626780}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277051312}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1259541996}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975892697}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095549956}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3565136999}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1633556252}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2552037399}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2301480247}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3194561803}) 2024-06-27T21:39:45.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564577811}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2629353359}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429708977}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3452000053}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=976935863}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776793539}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=156887537}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004391141}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3918023766}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=707341606}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589539638}) 2024-06-27T21:39:45.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3056572892}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996144269}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605913326}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339260140}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2997244417}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2231763606}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173181324}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091279710}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=204375543}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999015565}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716802452}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561414620}) 2024-06-27T21:39:45.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=86,mdsmap=3,mgr=11} crc {logm=304405197,mdsmap=2339142170,mgr=1458972961}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=80} crc {mgr=2182222232,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=1367315846,mon_config_key=2528972433}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1436394348}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=635237681}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=311744889,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=207983073,osdmap=141961322}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=144945941}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=37} crc {osdmap=2193049873}) 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1828723199' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1828723199' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:39:45.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:44 smithi190 ceph-mon[32247]: osdmap e725: 12 total, 12 up, 10 in 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: osdmap e724: 12 total, 12 up, 10 in 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:39:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1805151625,config=2353167246,health=2095200028,kv=3661340479}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=2370569707,logm=3249155906}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=195297825}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2030402540}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942527502}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1175234200}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2274791215}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401956763}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3996391476}) 2024-06-27T21:39:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=450241051}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2673117186}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871638701}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3522937007}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1213543075}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523085675}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=16109540}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=466864984}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1925185255}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078814924}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320948124}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1832831610}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2523267717}) 2024-06-27T21:39:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2405714987}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=967970001}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=571356781}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3349071982}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=855761480}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=555858979}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168265397}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027239739}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3122143378}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=341360269}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=207801183}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2459626780}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277051312}) 2024-06-27T21:39:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1259541996}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2975892697}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095549956}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3565136999}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1633556252}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2552037399}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2301480247}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3194561803}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564577811}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2629353359}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429708977}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3452000053}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=976935863}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2776793539}) 2024-06-27T21:39:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=156887537}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004391141}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3918023766}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=707341606}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589539638}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3056572892}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=996144269}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605913326}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339260140}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2997244417}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2231763606}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173181324}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3091279710}) 2024-06-27T21:39:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=204375543}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999015565}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716802452}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561414620}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=86,mdsmap=3,mgr=11} crc {logm=304405197,mdsmap=2339142170,mgr=1458972961}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=80} crc {mgr=2182222232,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=1367315846,mon_config_key=2528972433}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1436394348}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=635237681}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=11,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=71} crc {mon_config_key=311744889,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=207983073,osdmap=141961322}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=144945941}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=37} crc {osdmap=2193049873}) 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1828723199' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:39:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1828723199' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:39:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:44 smithi192 ceph-mon[32956]: osdmap e725: 12 total, 12 up, 10 in 2024-06-27T21:39:45.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:45.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:45.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:39:45.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:39:45.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:39:45.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:39:45.439 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:39:45.440 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:39:45.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:45.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:39:45.461 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:39:45.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:45.484 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:39:45.484 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:39:45.507 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:39:45.507 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:39:45.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:39:45.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:39:45.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:39:45.552 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:39:45.573 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:39:45.574 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:39:45.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:45.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:39:45.596 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:39:45.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:45.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:39:45.618 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:39:45.640 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:39:45.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:39:45.643 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:45.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:45.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:45.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:45.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:39:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:45.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:45.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:45 smithi151 ceph-mon[28965]: pgmap v1793: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:45.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/846597887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:45.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:45.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:45.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:46.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:46.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:46.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:46.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:45 smithi190 ceph-mon[32247]: pgmap v1793: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/846597887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:45 smithi192 ceph-mon[32956]: pgmap v1793: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/846597887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:46.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:46.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:46.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:46.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:46.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:46.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:46.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:46.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:46.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:46.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:46.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:46.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:46.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:47.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:46 smithi151 ceph-mon[28965]: 1.1e deep-scrub starts 2024-06-27T21:39:47.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:46 smithi151 ceph-mon[28965]: 1.1e deep-scrub ok 2024-06-27T21:39:47.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:47.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:47.206 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-06-27T21:39:47.209 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:39:47.210 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:39:47.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:47.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:47.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:46 smithi190 ceph-mon[32247]: 1.1e deep-scrub starts 2024-06-27T21:39:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:46 smithi190 ceph-mon[32247]: 1.1e deep-scrub ok 2024-06-27T21:39:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:46 smithi192 ceph-mon[32956]: 1.1e deep-scrub starts 2024-06-27T21:39:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:46 smithi192 ceph-mon[32956]: 1.1e deep-scrub ok 2024-06-27T21:39:47.337 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:39:47.337 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:39:47.337 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_150129 2024-06-27T21:39:47.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:39:47.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:39:47.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:47.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:47.481 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:47.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:47.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:47.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:47.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:47.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:47.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:47.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:47.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:47 smithi151 ceph-mon[28965]: pgmap v1795: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:47 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:39:47.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:47 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:39:47.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:48.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:48.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:48.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:48.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:47 smithi190 ceph-mon[32247]: pgmap v1795: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:47 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:39:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:47 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:39:48.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:47 smithi192 ceph-mon[32956]: pgmap v1795: 47 pgs: 47 active+clean; 577 KiB data, 4.9 GiB used, 889 GiB / 894 GiB avail 2024-06-27T21:39:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:47 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:39:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:47 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:39:48.337 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 2781 2765 172.803 172.812 0.00681513 0.00572195 2024-06-27T21:39:48.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:48.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:48.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:48.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:48.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:48.652 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:48.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:48.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-06-27T21:39:48.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:48.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:48.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:48.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:48.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:48 smithi151 ceph-mon[28965]: osdmap e726: 12 total, 12 up, 10 in 2024-06-27T21:39:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:48.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:49.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:49.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:49.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:39:49.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:49.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:48 smithi190 ceph-mon[32247]: osdmap e726: 12 total, 12 up, 10 in 2024-06-27T21:39:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:48 smithi192 ceph-mon[32956]: osdmap e726: 12 total, 12 up, 10 in 2024-06-27T21:39:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:49.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:49.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:49.337 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 4813 4797 149.896 127 0.00336464 0.00664474 2024-06-27T21:39:49.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:49.450 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:49.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:49.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:49.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:49.693 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:49.708 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:49.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:49.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:49.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:49.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:49.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:50.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:50.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:50.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:49 smithi151 ceph-mon[28965]: pgmap v1797: 47 pgs: 47 active+clean; 5.9 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 1019 KiB/s wr, 15 op/s 2024-06-27T21:39:50.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:49 smithi151 ceph-mon[28965]: osdmap e727: 12 total, 12 up, 10 in 2024-06-27T21:39:50.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:50.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:50.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:50.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:49 smithi190 ceph-mon[32247]: pgmap v1797: 47 pgs: 47 active+clean; 5.9 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 1019 KiB/s wr, 15 op/s 2024-06-27T21:39:50.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:49 smithi190 ceph-mon[32247]: osdmap e727: 12 total, 12 up, 10 in 2024-06-27T21:39:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:49 smithi192 ceph-mon[32956]: pgmap v1797: 47 pgs: 47 active+clean; 5.9 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 1019 KiB/s wr, 15 op/s 2024-06-27T21:39:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:49 smithi192 ceph-mon[32956]: osdmap e727: 12 total, 12 up, 10 in 2024-06-27T21:39:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:50.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:50.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 16 6483 6467 134.719 104.375 0.0571395 0.00720255 2024-06-27T21:39:50.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:50.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:50.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:50.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:50.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:50.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:50.712 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:50.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:50.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:50.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:50.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:50.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:51.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:51.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:51.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:50 smithi151 ceph-mon[28965]: osdmap e728: 12 total, 12 up, 10 in 2024-06-27T21:39:51.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:51.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:50 smithi190 ceph-mon[32247]: osdmap e728: 12 total, 12 up, 10 in 2024-06-27T21:39:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:50 smithi192 ceph-mon[32956]: osdmap e728: 12 total, 12 up, 10 in 2024-06-27T21:39:51.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:51.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:51.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 7236 7220 112.803 47.0625 0.00148739 0.00795084 2024-06-27T21:39:51.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:51.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:51.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:51.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:51.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:51.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:51.767 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:51.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:51.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:51.867 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:39:51.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:51.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:51.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:52.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:52.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: pgmap v1800: 47 pgs: 47 active+clean; 134 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 25 MiB/s wr, 394 op/s 2024-06-27T21:39:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:39:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:39:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:52.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:52.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: osdmap e729: 12 total, 12 up, 10 in 2024-06-27T21:39:52.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/85349425' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:39:52.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:39:52.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:52.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: pgmap v1800: 47 pgs: 47 active+clean; 134 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 25 MiB/s wr, 394 op/s 2024-06-27T21:39:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:39:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:39:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: osdmap e729: 12 total, 12 up, 10 in 2024-06-27T21:39:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/85349425' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:39:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:39:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: pgmap v1800: 47 pgs: 47 active+clean; 134 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 25 MiB/s wr, 394 op/s 2024-06-27T21:39:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:39:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:39:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: osdmap e729: 12 total, 12 up, 10 in 2024-06-27T21:39:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/85349425' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:39:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:39:52.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:52.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:52.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 7276 7260 90.7423 2.5 0.124606 0.00808655 2024-06-27T21:39:52.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:52.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:52.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:52.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:52.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:52.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:52.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:52.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:52.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:52.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:52.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:53.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:53.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:53.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:53.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:52 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:39:53.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:52 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:39:53.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:52 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:53.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:52 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:39:53.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:52 smithi151 ceph-mon[28965]: osdmap e730: 12 total, 12 up, 10 in 2024-06-27T21:39:53.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:52 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:39:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:52 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:39:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:52 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:52 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:39:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:52 smithi190 ceph-mon[32247]: osdmap e730: 12 total, 12 up, 10 in 2024-06-27T21:39:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:52 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:39:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:52 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:39:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:52 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:39:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:52 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:39:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:52 smithi192 ceph-mon[32956]: osdmap e730: 12 total, 12 up, 10 in 2024-06-27T21:39:53.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:53.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:53.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 7276 7260 75.618 0 - 0.00808655 2024-06-27T21:39:53.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:53.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:53.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:53.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:53.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:53.715 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:53.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:53.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:53.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:53 smithi151 ceph-mon[28965]: pgmap v1803: 47 pgs: 47 active+clean; 320 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:39:53.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:39:53.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:53.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:39:54.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:54.111 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:54.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:54.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:53 smithi190 ceph-mon[32247]: pgmap v1803: 47 pgs: 47 active+clean; 320 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:39:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:53 smithi192 ceph-mon[32956]: pgmap v1803: 47 pgs: 47 active+clean; 320 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-06-27T21:39:54.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:54.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 8264 8248 73.6355 30.875 0.00146726 0.0133116 2024-06-27T21:39:54.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:54.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:54.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:54.561 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:39:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:39:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:39:54.561 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:39:54.561 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:39:54.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:54.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:54.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:54.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:54.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:54.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:54.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:54.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:55.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:55.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:55.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:55.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:54 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:55.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:54 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:55.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:54 smithi151 ceph-mon[28965]: osdmap e731: 12 total, 12 up, 10 in 2024-06-27T21:39:55.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:55.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:54 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:54 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:54 smithi190 ceph-mon[32247]: osdmap e731: 12 total, 12 up, 10 in 2024-06-27T21:39:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:54 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:54 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:54 smithi192 ceph-mon[32956]: osdmap e731: 12 total, 12 up, 10 in 2024-06-27T21:39:55.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 10189 10173 79.4689 120.312 0.00559957 0.0125666 2024-06-27T21:39:55.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:55.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:55.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:55.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:55.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:55.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:55.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:39:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:39:55.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:39:55.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:55.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:55.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:55.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:56.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:56.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:56.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:56.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:55 smithi151 ceph-mon[28965]: pgmap v1805: 47 pgs: 47 active+clean; 405 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:39:56.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:55 smithi190 ceph-mon[32247]: pgmap v1805: 47 pgs: 47 active+clean; 405 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:39:56.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:56.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:55 smithi192 ceph-mon[32956]: pgmap v1805: 47 pgs: 47 active+clean; 405 MiB data, 6.0 GiB used, 888 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:39:56.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:39:56.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 11275 11259 78.1801 67.875 0.00729663 0.0127801 2024-06-27T21:39:56.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:56.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:56.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:56.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:56.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:56.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:56.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-06-27T21:39:56.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:56.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:56.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:56.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:56.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:57.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:57.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:57.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:57.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:57.186 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:57.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:57.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:57.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 15 12945 12930 80.805 104.438 0.00637088 0.0123649 2024-06-27T21:39:57.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:39:57.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:39:57.485 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:39:57.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:57.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:57.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:57.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:57.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:57.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:57.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:57.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:57.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:58.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:58.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:58.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:58.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:57 smithi151 ceph-mon[28965]: pgmap v1806: 47 pgs: 47 active+clean; 574 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:39:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:57 smithi190 ceph-mon[32247]: pgmap v1806: 47 pgs: 47 active+clean; 574 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:39:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:57 smithi192 ceph-mon[32956]: pgmap v1806: 47 pgs: 47 active+clean; 574 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:39:58.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:39:58.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:39:58.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 14565 14549 82.6574 101.188 0.0113224 0.0120869 2024-06-27T21:39:58.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:58.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:58.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:39:58.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:39:58.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:58.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:58.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:58.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:58.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:39:58.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:39:58.915 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:39:58.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:39:58.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:39:58.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:39:58.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:39:58.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:58.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:39:58.961 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:39:58.962 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:39:58.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:58 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:39:58.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:58 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:39:58.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2732842059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:58.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:39:58.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:39:59.011 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:39:59.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:39:59.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:39:59.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:39:59.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:39:59.056 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:39:59.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:39:59.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:39:59.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:39:59.082 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:39:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:39:59.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:39:59.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:39:59.132 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:39:59.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:39:59.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:39:59.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:39:59.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:39:59.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:58 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:39:59.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:58 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:39:59.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2732842059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:59.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:39:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:58 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:39:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:58 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:39:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2732842059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:39:59.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:39:59.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:39:59.338 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 15512 15496 80.7009 59.1875 0.0584905 0.0122708 2024-06-27T21:39:59.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:59.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:59.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:39:59.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:59.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:39:59.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:59.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:39:59.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:39:59.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:39:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:39:59.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:39:59.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:39:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:39:59.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:00.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:00.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:00.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:00.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:59 smithi151 ceph-mon[28965]: pgmap v1807: 47 pgs: 47 active+clean; 618 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 40 MiB/s wr, 647 op/s 2024-06-27T21:40:00.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:59 smithi151 ceph-mon[28965]: osdmap e732: 12 total, 12 up, 10 in 2024-06-27T21:40:00.175 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:39:59 smithi151 ceph-mon[28965]: osdmap e733: 12 total, 12 up, 10 in 2024-06-27T21:40:00.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:00.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:59 smithi190 ceph-mon[32247]: pgmap v1807: 47 pgs: 47 active+clean; 618 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 40 MiB/s wr, 647 op/s 2024-06-27T21:40:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:59 smithi190 ceph-mon[32247]: osdmap e732: 12 total, 12 up, 10 in 2024-06-27T21:40:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:39:59 smithi190 ceph-mon[32247]: osdmap e733: 12 total, 12 up, 10 in 2024-06-27T21:40:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:59 smithi192 ceph-mon[32956]: pgmap v1807: 47 pgs: 47 active+clean; 618 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 40 MiB/s wr, 647 op/s 2024-06-27T21:40:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:59 smithi192 ceph-mon[32956]: osdmap e732: 12 total, 12 up, 10 in 2024-06-27T21:40:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:39:59 smithi192 ceph-mon[32956]: osdmap e733: 12 total, 12 up, 10 in 2024-06-27T21:40:00.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:00.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:00.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 17248 17232 82.8385 108.5 0.0029242 0.0120619 2024-06-27T21:40:00.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:00.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:00.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:00.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:00.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:00.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:00.811 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:00.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:00.917 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:40:00.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:00.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:00.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:01.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:00 smithi151 ceph-mon[28965]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-06-27T21:40:01.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:00 smithi151 ceph-mon[28965]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-06-27T21:40:01.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/218250869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:01.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:01.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:01.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:01.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:01.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:00 smithi190 ceph-mon[32247]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-06-27T21:40:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:00 smithi190 ceph-mon[32247]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-06-27T21:40:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/218250869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:00 smithi192 ceph-mon[32956]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-06-27T21:40:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:00 smithi192 ceph-mon[32956]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-06-27T21:40:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/218250869' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:01.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:01.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:01.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 18455 18439 82.3094 75.4375 0.00143874 0.0120743 2024-06-27T21:40:01.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:01.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:01.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:01.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:01.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:01.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:01.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:01.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:01.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:01.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:01.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:02.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:01 smithi151 ceph-mon[28965]: pgmap v1810: 47 pgs: 47 active+clean; 820 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:40:02.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:01 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:02.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/218250869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:02.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:01 smithi151 ceph-mon[28965]: osdmap e734: 12 total, 12 up, 10 in 2024-06-27T21:40:02.050 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:40:02.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:40:02.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:02.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:02.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:02.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:01 smithi190 ceph-mon[32247]: pgmap v1810: 47 pgs: 47 active+clean; 820 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:40:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:01 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/218250869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:02.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:01 smithi190 ceph-mon[32247]: osdmap e734: 12 total, 12 up, 10 in 2024-06-27T21:40:02.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:40:02.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:40:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:01 smithi192 ceph-mon[32956]: pgmap v1810: 47 pgs: 47 active+clean; 820 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:40:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:01 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/218250869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:01 smithi192 ceph-mon[32956]: osdmap e734: 12 total, 12 up, 10 in 2024-06-27T21:40:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:40:02.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:40:02.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:02.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 20198 20182 84.0841 108.938 0.0254881 0.0118746 2024-06-27T21:40:02.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:02.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:02.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:02.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:02.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:02.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:02.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:02.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:02.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:02.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:02.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:03.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:03.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:03.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:02 smithi151 ceph-mon[28965]: osdmap e735: 12 total, 12 up, 10 in 2024-06-27T21:40:03.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:03.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:03.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:03.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:02 smithi190 ceph-mon[32247]: osdmap e735: 12 total, 12 up, 10 in 2024-06-27T21:40:03.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:02 smithi192 ceph-mon[32956]: osdmap e735: 12 total, 12 up, 10 in 2024-06-27T21:40:03.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 21524 21508 84.0081 82.875 0.00191838 0.0118837 2024-06-27T21:40:03.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:03.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:03.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:03.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:03.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:03.642 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:03.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:03.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:03.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:03.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:03.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:03.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:03.993 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:04.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:04.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:03 smithi151 ceph-mon[28965]: pgmap v1813: 47 pgs: 47 active+clean; 1008 MiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-06-27T21:40:04.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:04.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:03 smithi190 ceph-mon[32247]: pgmap v1813: 47 pgs: 47 active+clean; 1008 MiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-06-27T21:40:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:03 smithi192 ceph-mon[32956]: pgmap v1813: 47 pgs: 47 active+clean; 1008 MiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-06-27T21:40:04.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 22994 22978 84.4705 91.875 0.00365499 0.0118324 2024-06-27T21:40:04.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:04.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:04.510 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:04.536 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:04.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:40:04.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:04.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:04.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:04.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:04.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:04.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:04.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:04.949 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:04.950 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:05.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:05.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:05.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:05.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:05.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:05.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:05.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 16 24735 24719 85.8223 108.812 0.0102444 0.0116436 2024-06-27T21:40:05.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:05.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:05.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:05.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:05.677 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:05.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:40:05.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:05.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:05.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:05.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:05.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:05.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:05.982 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:06.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:05 smithi190 ceph-mon[32247]: pgmap v1814: 47 pgs: 47 active+clean; 1.1 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:40:06.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-06-27T21:40:06.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:06.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:06.109 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:05 smithi151 ceph-mon[28965]: pgmap v1814: 47 pgs: 47 active+clean; 1.1 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:40:06.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:06.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:05 smithi192 ceph-mon[32956]: pgmap v1814: 47 pgs: 47 active+clean; 1.1 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:40:06.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:40:06.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:06.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 26392 26376 86.7555 103.562 0.00822808 0.0115186 2024-06-27T21:40:06.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:06.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:06.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:06.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:06.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:06.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:06.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:06.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:06.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:06.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:06.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:06.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:07.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:07.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:07.218 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:07.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:07.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:40:07.338658+0000 min lat: 0.00110269 max lat: 3.09321 avg lat: 0.0116394 2024-06-27T21:40:07.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:40:07.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 27493 27477 85.8581 68.8125 0.0133782 0.0116394 2024-06-27T21:40:07.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:07.368 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:07.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:07.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:07.425 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:07.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:07.502 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:07.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:07.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:07.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:07.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:07.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:07.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:07.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:07.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:08.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:08.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:08.136 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:08.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:07 smithi151 ceph-mon[28965]: pgmap v1815: 47 pgs: 47 active+clean; 1.4 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-06-27T21:40:08.210 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:07 smithi190 ceph-mon[32247]: pgmap v1815: 47 pgs: 47 active+clean; 1.4 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-06-27T21:40:08.216 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:08.216 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:08.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:08.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:07 smithi192 ceph-mon[32956]: pgmap v1815: 47 pgs: 47 active+clean; 1.4 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-06-27T21:40:08.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 16 28786 28770 85.6172 80.8125 0.0454858 0.011642 2024-06-27T21:40:08.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:08.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:08.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:08.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:08.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:08.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:08.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:08.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:08.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:08.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:08.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:08.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:40:08.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:09.022 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:09.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:09.149 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:09.208 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:09.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:09.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:09.339 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 30074 30058 85.3843 80.5 0.0152739 0.0117051 2024-06-27T21:40:09.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:09.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:09.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:09.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:09.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:09.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:09.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:09.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:09.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:09.906 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:09.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:10.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:10.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:10.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:10.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:09 smithi151 ceph-mon[28965]: pgmap v1816: 47 pgs: 47 active+clean; 1.5 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:40:10.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:09 smithi190 ceph-mon[32247]: pgmap v1816: 47 pgs: 47 active+clean; 1.5 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:40:10.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:09 smithi192 ceph-mon[32956]: pgmap v1816: 47 pgs: 47 active+clean; 1.5 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-06-27T21:40:10.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 31647 31631 85.9461 98.3125 0.0106871 0.0116217 2024-06-27T21:40:10.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:10.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:10.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:10.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:10.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:10.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:10.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:10.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:10.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:10.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:10.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:11.010 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:11.011 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:11.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:11.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:11.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:11.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:11.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:11.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:11.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 16 33178 33162 86.3517 95.6875 0.00376493 0.0115731 2024-06-27T21:40:11.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:11.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:11.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:11.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:11.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:40:11.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:11.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:11.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:11.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:11.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:11.924 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:11.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:11.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:11.978 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:40:12.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:12.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:12.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:12.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:12.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:11 smithi151 ceph-mon[28965]: pgmap v1817: 47 pgs: 47 active+clean; 1.7 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-27T21:40:12.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2929659264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:12.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:11 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:11 smithi190 ceph-mon[32247]: pgmap v1817: 47 pgs: 47 active+clean; 1.7 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-27T21:40:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2929659264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:11 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:11 smithi192 ceph-mon[32956]: pgmap v1817: 47 pgs: 47 active+clean; 1.7 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-06-27T21:40:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2929659264' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:11 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:12.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:12.320 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:12.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 15 34574 34559 86.3898 87.3125 0.0114312 0.0115696 2024-06-27T21:40:12.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:12.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:12.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:12.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:12.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:12.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:12.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:12.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:12.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:12.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:12.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:12.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:13.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:12 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:13.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:12 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:40:13.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:12 smithi151 ceph-mon[28965]: osdmap e736: 12 total, 12 up, 10 in 2024-06-27T21:40:13.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:13.069 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:40:13.069 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:40:13.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:13.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:40:13.137 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:40:13.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:40:13.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:40:13.170 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:40:13.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:13.201 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:40:13.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:40:13.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:13.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:40:13.238 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:40:13.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:12 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:13.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:12 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:40:13.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:12 smithi190 ceph-mon[32247]: osdmap e736: 12 total, 12 up, 10 in 2024-06-27T21:40:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:12 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:12 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:40:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:12 smithi192 ceph-mon[32956]: osdmap e736: 12 total, 12 up, 10 in 2024-06-27T21:40:13.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:13.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:13.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 36129 36113 86.8024 97.125 0.00172252 0.0115057 2024-06-27T21:40:13.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:13.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:13.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:13.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:13.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:13.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:13.802 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e540/526 cost 1049576)", 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:15.926414+0000", 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 417.87305170100001, 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.705371831, 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.803 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.926414+0000", 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.926414+0000", 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.926415+0000", 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.091e-06 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.926429+0000", 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4679e-05 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.631343+0000", 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.70491378000000005 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.631351+0000", 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7670000000000003e-06 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.631405+0000", 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4219999999999999e-05 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:16.631785+0000", 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038029399999999999 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:157545 6.7 6:e2be8f79:::benchmark_data_smithi192_117994_object49951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:30.925605+0000", 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 402.87386005600001, 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36750161599999998, 2024-06-27T21:40:13.807 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 157545 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925605+0000", 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925605+0000", 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925607+0000", 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6419999999999999e-06 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925614+0000", 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8299999999999998e-06 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925631+0000", 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7252e-05 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925639+0000", 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7770000000000002e-06 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925697+0000", 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8241e-05 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925786+0000", 2024-06-27T21:40:13.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9189000000000003e-05 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.925889+0000", 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103324 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.929430+0000", 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035407049999999999 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:30.929638+0000", 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000207958 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:31.292993+0000", 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36335486500000003 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:31.293054+0000", 2024-06-27T21:40:13.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0804000000000003e-05 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:31.293107+0000", 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3029000000000003e-05 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.4 600/599 [PushReplyOp(6:20a2dd3e:::benchmark_data_smithi192_117994_object81280:head)])", 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:48.349293+0000", 2024-06-27T21:40:13.814 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 265.45017234300002, 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38083065300000002, 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.349293+0000", 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.349293+0000", 2024-06-27T21:40:13.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.349293+0000", 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.16e-07 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.349296+0000", 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3550000000000001e-06 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.729867+0000", 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38057088500000003 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.729878+0000", 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0169999999999999e-05 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.729949+0000", 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1852000000000003e-05 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:48.730124+0000", 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017417499999999999 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.4 601/599 [PushReplyOp(6:23ac47e1:::benchmark_data_smithi192_117994_object93762:head)])", 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:50.096058+0000", 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 263.70340709700002, 2024-06-27T21:40:13.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.59354186900000006, 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.096058+0000", 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.096058+0000", 2024-06-27T21:40:13.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.096058+0000", 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9299999999999999e-07 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.096061+0000", 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0570000000000001e-06 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.689371+0000", 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.59330912199999997 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.689378+0000", 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8380000000000002e-06 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.689424+0000", 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6072999999999997e-05 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.689600+0000", 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017558600000000001 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.4 603/599 [PushReplyOp(6:244e6395:::benchmark_data_smithi192_117994_object66494:head)])", 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:52.040078+0000", 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 261.75938737199999, 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.437050049, 2024-06-27T21:40:13.822 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.040078+0000", 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.040078+0000", 2024-06-27T21:40:13.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.040078+0000", 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.08e-07 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.040081+0000", 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9780000000000001e-06 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.476867+0000", 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43678608899999999 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.476877+0000", 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0274e-05 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.476956+0000", 2024-06-27T21:40:13.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8354000000000002e-05 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.477128+0000", 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000172246 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.4 603/599 [PushReplyOp(6:24db7079:::benchmark_data_smithi192_117994_object69688:head)])", 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:52.659445+0000", 2024-06-27T21:40:13.826 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 261.14002001, 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.422912909, 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.659445+0000", 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.659445+0000", 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.659445+0000", 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4600000000000001e-07 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.659447+0000", 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0040000000000002e-06 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.082069+0000", 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42262169100000002 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.082079+0000", 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5310000000000007e-06 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.082179+0000", 2024-06-27T21:40:13.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100887 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.082358+0000", 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017865000000000001 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.4 604/599 [PushReplyOp(6:25a7518d:::benchmark_data_smithi192_117994_object82691:head)])", 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:54.292436+0000", 2024-06-27T21:40:13.830 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 259.507029719, 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.421146933, 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.292436+0000", 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.292436+0000", 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.292436+0000", 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1500000000000001e-07 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.292439+0000", 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2399999999999999e-06 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.712780+0000", 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42034083999999999 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.712789+0000", 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1509999999999993e-06 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.713237+0000", 2024-06-27T21:40:13.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044848000000000002 2024-06-27T21:40:13.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.713582+0000", 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034500699999999999 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:176664 6.4 6:20a3f22e:::benchmark_data_smithi192_117994_object69070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:19.737488+0000", 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 234.06197738200001, 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41672330400000002, 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.836 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 176664 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737488+0000", 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737488+0000", 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737489+0000", 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.505e-06 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737495+0000", 2024-06-27T21:40:13.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8289999999999996e-06 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737510+0000", 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4579000000000001e-05 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737518+0000", 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0360000000000006e-06 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737570+0000", 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1854000000000002e-05 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.737574+0000", 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2930000000000002e-06 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.738601+0000", 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001026924 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.738705+0000", 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010419799999999999 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 9,11", 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.739185+0000", 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048039900000000001 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.741482+0000", 2024-06-27T21:40:13.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022965049999999999 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:19.742303+0000", 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00082070099999999996 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:20.154149+0000", 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41184637299999999 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:20.154182+0000", 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3439999999999998e-05 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:20.154211+0000", 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8668000000000001e-05 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.843 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:177152 6.4 6:228c94ea:::benchmark_data_smithi192_117994_object69558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e624)", 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:24.040886+0000", 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 229.75857881900001, 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78470593499999997, 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 177152 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040886+0000", 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040886+0000", 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040887+0000", 2024-06-27T21:40:13.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5899999999999995e-07 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040891+0000", 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.313e-06 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040903+0000", 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1206000000000001e-05 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040908+0000", 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3399999999999997e-06 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.040962+0000", 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4410000000000003e-05 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.041085+0000", 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012290900000000001 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 9,11", 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.041403+0000", 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031718300000000002 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.043358+0000", 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019553869999999998 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.043437+0000", 2024-06-27T21:40:13.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9177000000000004e-05 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.825524+0000", 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78208670199999997 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.825560+0000", 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5873e-05 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:24.825592+0000", 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2675999999999997e-05 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:181719 6.7 6:e45a6468:::benchmark_data_smithi192_117994_object74125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e641)", 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:12.325633+0000", 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 181.47383266400001, 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43445835799999999, 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.850 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 181719 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325633+0000", 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325633+0000", 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325634+0000", 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6300000000000001e-06 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325641+0000", 2024-06-27T21:40:13.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5980000000000002e-06 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325657+0000", 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6487999999999999e-05 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325665+0000", 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3209999999999999e-06 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325728+0000", 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3218000000000004e-05 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.325866+0000", 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013852499999999999 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.326320+0000", 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045387599999999998 2024-06-27T21:40:13.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.328811+0000", 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024909699999999999 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.329169+0000", 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035750000000000002 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.760016+0000", 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43084755800000002 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.760058+0000", 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2080999999999999e-05 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:12.760091+0000", 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2592999999999998e-05 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.856 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:203120 6.7 6:e778c393:::benchmark_data_smithi192_117994_object95526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:08.152461+0000", 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 125.64700467900001, 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41691995100000001, 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:40:13.857 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 203120 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152461+0000", 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152461+0000", 2024-06-27T21:40:13.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152462+0000", 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6810000000000001e-06 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152469+0000", 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.153e-06 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152488+0000", 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8715e-05 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152498+0000", 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7319999999999993e-06 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152564+0000", 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6050999999999995e-05 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.152661+0000", 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6668000000000003e-05 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,4", 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.153172+0000", 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000511237 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.156103+0000", 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029316350000000001 2024-06-27T21:40:13.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.345835+0000", 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.189731971 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.569280+0000", 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22344459799999999 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.569336+0000", 2024-06-27T21:40:13.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5807000000000003e-05 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.569381+0000", 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4703000000000002e-05 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e705,698 6:a6704fcb:::benchmark_data_smithi192_117994_object48223:head,540'9656)", 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:08.659548+0000", 2024-06-27T21:40:13.863 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 65.139917451000002, 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52699047600000004, 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659548+0000", 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659548+0000", 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659549+0000", 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.53e-06 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659575+0000", 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5902e-05 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186447+0000", 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52687193300000001 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186452+0000", 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2399999999999998e-06 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186495+0000", 2024-06-27T21:40:13.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.227e-05 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186501+0000", 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8279999999999997e-06 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186538+0000", 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6773000000000002e-05 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.16 e706,698 6:6ed1fc8a:::benchmark_data_smithi192_117994_object66041:head,607'10323)", 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 60.819947732000003, 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43442441300000001, 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0299999999999999e-07 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979522+0000", 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0090000000000001e-06 2024-06-27T21:40:13.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413778+0000", 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.434255693 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413784+0000", 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2709999999999998e-06 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413870+0000", 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5923999999999999e-05 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413876+0000", 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2659999999999998e-06 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413942+0000", 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5647000000000004e-05 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.16 e707,698 6:69a24610:::benchmark_data_smithi192_117994_object70316:head,625'10450)", 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:14.103552+0000", 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 59.695913103000002, 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48983110000000002, 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103552+0000", 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103552+0000", 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103553+0000", 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.82e-07 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103556+0000", 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5899999999999999e-06 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593246+0000", 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.489689924 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593256+0000", 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7499999999999998e-06 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593315+0000", 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8947999999999999e-05 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593322+0000", 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8360000000000003e-06 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593383+0000", 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999995e-05 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a2547159:::benchmark_data_smithi192_117994_object90208:head,669'11030)", 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:16.952936+0000", 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.846529183999998, 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48239369700000001, 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.876 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952936+0000", 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952936+0000", 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952937+0000", 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2600000000000002e-07 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952943+0000", 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2670000000000003e-06 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435072+0000", 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48212887100000001 2024-06-27T21:40:13.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435085+0000", 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3072e-05 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435239+0000", 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000154067 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435253+0000", 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4606e-05 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435330+0000", 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6287999999999999e-05 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.880 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a6ffdd34:::benchmark_data_smithi192_117994_object90211:head,669'11031)", 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.844168650999997, 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48275176600000003, 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4999999999999998e-07 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955299+0000", 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3879999999999998e-06 2024-06-27T21:40:13.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.437969+0000", 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.482669815 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.437973+0000", 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1570000000000002e-06 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.438010+0000", 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6628000000000002e-05 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.438015+0000", 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1630000000000004e-06 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.438048+0000", 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3265000000000001e-05 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.884 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a6561f31:::benchmark_data_smithi192_117994_object91018:head,670'11058)", 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:17.493688+0000", 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.305776786000003, 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36878778600000001, 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493688+0000", 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493688+0000", 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493689+0000", 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0699999999999997e-07 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493696+0000", 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7719999999999997e-06 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862204+0000", 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.368508593 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862221+0000", 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6082999999999998e-05 2024-06-27T21:40:13.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862353+0000", 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000132053 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862372+0000", 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.929e-05 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862476+0000", 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104388 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a673efaf:::benchmark_data_smithi192_117994_object91022:head,670'11059)", 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:17.495674+0000", 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.303791212999997, 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36941172900000002, 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.889 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495674+0000", 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495674+0000", 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495675+0000", 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999996e-07 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495681+0000", 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3550000000000001e-06 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865007+0000", 2024-06-27T21:40:13.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36932614899999999 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865012+0000", 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3469999999999998e-06 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865046+0000", 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3961999999999998e-05 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865051+0000", 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.702e-06 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865086+0000", 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4610000000000002e-05 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e710,698 6:a11fa313:::benchmark_data_smithi192_117994_object92325:head,673'11094)", 2024-06-27T21:40:13.893 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 55.781901341000001, 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37082322099999998, 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0500000000000004e-07 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017567+0000", 2024-06-27T21:40:13.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9809999999999999e-06 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388209+0000", 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37064207999999998 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388219+0000", 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9699999999999994e-06 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388309+0000", 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0002999999999999e-05 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388317+0000", 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2790000000000001e-06 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388387+0000", 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0402999999999997e-05 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.897 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e710,698 6:a7bd4761:::benchmark_data_smithi192_117994_object92357:head,673'11095)", 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 55.779683007999999, 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.372040078, 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:13.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:40:13.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6e-07 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019785+0000", 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.419e-06 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391708+0000", 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.371923112 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391713+0000", 2024-06-27T21:40:13.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2490000000000001e-06 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391781+0000", 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7336999999999996e-05 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391786+0000", 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0769999999999998e-06 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391822+0000", 2024-06-27T21:40:13.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6624e-05 2024-06-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:13.902 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:40:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:13.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:13.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:14.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:14.031 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:14.073 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:40:14.074 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:40:14.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:14.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:14.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:13 smithi151 ceph-mon[28965]: pgmap v1819: 47 pgs: 47 active+clean; 1.9 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-06-27T21:40:14.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:13 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:40:14.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:13 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:40:14.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/284886674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:14.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:13 smithi190 ceph-mon[32247]: pgmap v1819: 47 pgs: 47 active+clean; 1.9 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-06-27T21:40:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:13 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:40:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:13 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:40:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/284886674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:14.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:13 smithi192 ceph-mon[32956]: pgmap v1819: 47 pgs: 47 active+clean; 1.9 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-06-27T21:40:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:13 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:40:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:13 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:40:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/284886674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:40:14.297 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:40:14.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:40:14.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:40:14.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 16 37447 37431 86.6382 82.375 0.00208321 0.0115245 2024-06-27T21:40:14.354 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:40:14.355 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:40:14.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:40:14.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:40:14.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:14.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:40:14.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:40:14.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:14.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:14.536 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:14.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:14.661 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:14.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:40:14.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:14.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:14.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:14.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:14.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:14.970 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:15.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:15.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:15.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:15.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:15.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:15.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:15.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 38712 38696 86.3674 79.0625 0.0157996 0.0115704 2024-06-27T21:40:15.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:15.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:15.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:15.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:15.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:15.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:15.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:15.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.827 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:15.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:15.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:15.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:16.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:16.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:16.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:40:16.151 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:15 smithi190 ceph-mon[32247]: pgmap v1820: 47 pgs: 47 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-27T21:40:16.151 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:15 smithi190 ceph-mon[32247]: osdmap e737: 12 total, 12 up, 10 in 2024-06-27T21:40:16.151 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:15 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:40:16.151 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:15 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:40:16.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:15 smithi151 ceph-mon[28965]: pgmap v1820: 47 pgs: 47 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-27T21:40:16.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:15 smithi151 ceph-mon[28965]: osdmap e737: 12 total, 12 up, 10 in 2024-06-27T21:40:16.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:15 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:40:16.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:15 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:40:16.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:16.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:15 smithi192 ceph-mon[32956]: pgmap v1820: 47 pgs: 47 active+clean; 2.0 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-06-27T21:40:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:15 smithi192 ceph-mon[32956]: osdmap e737: 12 total, 12 up, 10 in 2024-06-27T21:40:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:15 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:40:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:15 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:40:16.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:16.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 40019 40003 86.2058 81.6875 0.0314077 0.0115941 2024-06-27T21:40:16.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:16.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:16.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:16.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:16.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:16.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:16.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:16.724 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:40:16.724 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:40:16.725 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:40:16.725 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:40:16.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:16.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:16.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:16.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:16.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:16.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:17.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:16 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:40:17.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:16 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:40:17.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:17.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:16 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:40:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:16 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:40:17.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:17.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:16 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:40:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:16 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:40:17.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 41521 41505 86.4612 93.875 0.0162668 0.011556 2024-06-27T21:40:17.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:40:17.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:17.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:17.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:17.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:17.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:17.631 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:17.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:17.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:17.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:17.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:17.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:17.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:17.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:18.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:18.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:17 smithi190 ceph-mon[32247]: pgmap v1822: 47 pgs: 47 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-06-27T21:40:18.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:18.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:18.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:18.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:17 smithi151 ceph-mon[28965]: pgmap v1822: 47 pgs: 47 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-06-27T21:40:18.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:18.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:17 smithi192 ceph-mon[32956]: pgmap v1822: 47 pgs: 47 active+clean; 2.3 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-06-27T21:40:18.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 15 42707 42692 86.0651 74.1875 0.0130193 0.0116141 2024-06-27T21:40:18.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:18.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:18.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:18.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:18.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:18.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:18.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:18.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:18.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:18.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:18.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:19.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:19.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-06-27T21:40:19.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:19.138 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:19.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:19.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:19.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:19.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:19.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:19.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 15 44137 44122 86.1683 89.375 0.00429891 0.011599 2024-06-27T21:40:19.394 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:40:19.394 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:40:19.394 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:40:19.395 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:40:19.395 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:40:19.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:19.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:19.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:19.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:19.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:19.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:19.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:19.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:19 smithi151 ceph-mon[28965]: pgmap v1823: 47 pgs: 47 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:40:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:19 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:40:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:19 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:40:19.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:19.960 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:19.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:19.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:20.048 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:19 smithi190 ceph-mon[32247]: pgmap v1823: 47 pgs: 47 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:40:20.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:19 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:40:20.049 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:19 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:40:20.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:40:20.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:20.109 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:19 smithi192 ceph-mon[32956]: pgmap v1823: 47 pgs: 47 active+clean; 2.4 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-06-27T21:40:20.109 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:19 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:40:20.110 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:19 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:40:20.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:20.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:20.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:20.332 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:20.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 45401 45385 85.949 78.9375 0.0128832 0.0116257 2024-06-27T21:40:20.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:20.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:20.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:20.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:20.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:20.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:20.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:20.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:20.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:20.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:20.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:21.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:40:21.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:21.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:21.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:21.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:21.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:21.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 46807 46791 86.0054 87.875 0.0248528 0.0116216 2024-06-27T21:40:21.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:21.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:21.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:21.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:21.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:21.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:21.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:21.817 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:40:21.850 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:21.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:21 smithi151 ceph-mon[28965]: pgmap v1824: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:40:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/144518099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:21.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:21 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:21.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:21.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:21.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:21.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:22.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:22.105 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:21 smithi190 ceph-mon[32247]: pgmap v1824: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:40:22.105 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/144518099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:22.105 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:21 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:22.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:103777 6.3 e503/467 6:c849b376:::benchmark_data_smithi192_117994_object103776:head v 503'6566, mlcod=503'6563)", 2024-06-27T21:40:22.123 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:30:55.525358+0000", 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 566.59555573199998, 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30888744200000001, 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.525358+0000", 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.525358+0000", 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.525358+0000", 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1800000000000005e-07 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.525381+0000", 2024-06-27T21:40:22.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2433e-05 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.831375+0000", 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30599368700000001 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.831381+0000", 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8689999999999998e-06 2024-06-27T21:40:22.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.831452+0000", 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0623999999999993e-05 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.831468+0000", 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5469999999999999e-05 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.834215+0000", 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027477059999999999 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:30:55.834245+0000", 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9935000000000001e-05 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:116642 6.c 6:31da7807:::benchmark_data_smithi192_117994_object9048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:31:10.448964+0000", 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 551.67194983299999, 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27721074699999998, 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 116642 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.448964+0000", 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.448964+0000", 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.448965+0000", 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.034e-06 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.448970+0000", 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9429999999999999e-06 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.448982+0000", 2024-06-27T21:40:22.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2581e-05 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.448989+0000", 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7240000000000003e-06 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.449039+0000", 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9694000000000002e-05 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.449163+0000", 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124373 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 5,8", 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.449254+0000", 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0705999999999995e-05 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:40:22.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.451960+0000", 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027062990000000001 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.457112+0000", 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051521709999999997 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.726114+0000", 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26900169600000001 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.726145+0000", 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1606999999999997e-05 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:31:10.726174+0000", 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8918999999999999e-05 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:161667 6.a e562/545 6:59b12ea2:::benchmark_data_smithi192_117994_object54073:head v 562'10079, mlcod=562'10077)", 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:08.117515+0000", 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 374.00339819800001, 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33725649000000002, 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117515+0000", 2024-06-27T21:40:22.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117515+0000", 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117516+0000", 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.116e-06 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117524+0000", 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1570000000000003e-06 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449716+0000", 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33219267099999999 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449723+0000", 2024-06-27T21:40:22.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8909999999999998e-06 2024-06-27T21:40:22.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449783+0000", 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9747999999999998e-05 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449809+0000", 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6126e-05 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.454734+0000", 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049245469999999996 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.454772+0000", 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8234e-05 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:172585 6.a e601/545 6:508e154a:::benchmark_data_smithi192_117994_object64991:head v 601'10775, mlcod=601'10773)", 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:50.005917+0000", 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 272.11499620699999, 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51760722800000003, 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.142 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005917+0000", 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005917+0000", 2024-06-27T21:40:22.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005918+0000", 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0419999999999999e-06 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005925+0000", 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1470000000000004e-06 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519722+0000", 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51379602700000004 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519729+0000", 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3320000000000003e-06 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519777+0000", 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7945999999999998e-05 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519792+0000", 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5098e-05 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.523488+0000", 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036964519999999998 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.523525+0000", 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6183999999999998e-05 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_backfill(progress 6.b e 603/603 lb 6:d3c288e1:::benchmark_data_smithi192_117994_object105693:head)", 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 269.320919608, 2024-06-27T21:40:22.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28496729500000001, 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0100000000000001e-07 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799998+0000", 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.844e-06 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084805+0000", 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28480739300000002 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084813+0000", 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8970000000000008e-06 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084905+0000", 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1991999999999996e-05 2024-06-27T21:40:22.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084917+0000", 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2104e-05 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084961+0000", 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3863999999999999e-05 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e629/526 cost 1049576)", 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:37.697950+0000", 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 224.42296321699999, 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34383320699999997, 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697950+0000", 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697950+0000", 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697951+0000", 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8400000000000003e-07 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697962+0000", 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0906e-05 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041190+0000", 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34322828100000002 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041215+0000", 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4195000000000001e-05 2024-06-27T21:40:22.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041284+0000", 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9418999999999994e-05 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041783+0000", 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049952199999999997 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.155 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e629/526 cost 1049576)", 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:38.041150+0000", 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 224.079763117, 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37134862200000002, 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041150+0000", 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041150+0000", 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041152+0000", 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.311e-06 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041165+0000", 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3446e-05 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.411855+0000", 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37068976999999997 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.411861+0000", 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5769999999999999e-06 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.411926+0000", 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4894999999999994e-05 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.412499+0000", 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057262299999999997 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:179811 6.b e629/612 6:d8080e2b:::benchmark_data_smithi192_117994_object72217:head v 629'10993, mlcod=629'10989)", 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:42.327863+0000", 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 219.79305019099999, 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42010233600000002, 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327863+0000", 2024-06-27T21:40:22.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327863+0000", 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327866+0000", 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4439999999999998e-06 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327871+0000", 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4099999999999999e-06 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745267+0000", 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41739600700000001 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745276+0000", 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7309999999999999e-06 2024-06-27T21:40:22.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745336+0000", 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9886000000000002e-05 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745352+0000", 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5999999999999999e-05 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.747915+0000", 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00256347 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.747966+0000", 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0387999999999999e-05 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:179812 6.b e629/612 6:ddc36af4:::benchmark_data_smithi192_117994_object72218:head v 629'10994, mlcod=629'10990)", 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:42.332970+0000", 2024-06-27T21:40:22.164 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 219.78794338899999, 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43448742499999998, 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332970+0000", 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332970+0000", 2024-06-27T21:40:22.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332971+0000", 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332977+0000", 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1310000000000003e-06 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761778+0000", 2024-06-27T21:40:22.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42880081399999997 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761788+0000", 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0161e-05 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761853+0000", 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5106999999999999e-05 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761868+0000", 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5330999999999999e-05 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.767414+0000", 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055453519999999999 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.767458+0000", 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3687999999999999e-05 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.168 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e660/651 cost 1049576)", 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 153.68910186700001, 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38737757699999997, 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5499999999999997e-07 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431826+0000", 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3038e-05 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.818693+0000", 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38686758300000001 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.818699+0000", 2024-06-27T21:40:22.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5430000000000001e-06 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.818749+0000", 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0827e-05 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.819189+0000", 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043973100000000001 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e669/651 cost 1049576)", 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:01.097374+0000", 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 141.02353925200001, 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35812077399999998, 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097374+0000", 2024-06-27T21:40:22.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097374+0000", 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097375+0000", 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8400000000000003e-07 2024-06-27T21:40:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097384+0000", 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9050000000000003e-06 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.454745+0000", 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35736057199999999 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.454757+0000", 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2823000000000001e-05 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.454898+0000", 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000140276 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.455495+0000", 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000597414 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.176 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:203244 6.1a e674/654 6:5cb63d6a:::benchmark_data_smithi192_117994_object95650:head v 674'12085, mlcod=674'12082)", 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:08.709463+0000", 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 133.41145037000001, 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26052489099999998, 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709463+0000", 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709463+0000", 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709464+0000", 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0899999999999999e-06 2024-06-27T21:40:22.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709472+0000", 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5329999999999997e-06 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968046+0000", 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25857432699999999 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968054+0000", 2024-06-27T21:40:22.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9500000000000001e-06 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968160+0000", 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105979 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968187+0000", 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6894e-05 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.969941+0000", 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0017538370000000001 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.969988+0000", 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7280999999999997e-05 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.7 684/683 [PushOp(6:e38f712a:::benchmark_data_smithi192_117994_object97546:head, version: 499'5997, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e38f712a:::benchmark_data_smithi192_117994_object97546:head@499'5997, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:23.417763+0000", 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 118.703150702, 2024-06-27T21:40:22.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33887941300000002, 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417763+0000", 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.182 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417763+0000", 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417764+0000", 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.483e-06 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417825+0000", 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0652000000000001e-05 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754466+0000", 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33664066300000001 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754480+0000", 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4008e-05 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754586+0000", 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010654600000000001 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754596+0000", 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0299000000000001e-05 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.756642+0000", 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002045762 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.185 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.7 685/683 [PushOp(6:e2087deb:::benchmark_data_smithi192_117994_object102906:head, version: 502'6332, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e2087deb:::benchmark_data_smithi192_117994_object102906:head@502'6332, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:25.269923+0000", 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 116.850990357, 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24171029199999999, 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269923+0000", 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269923+0000", 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269924+0000", 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269951+0000", 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7229e-05 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509327+0000", 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.239376003 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509337+0000", 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8949999999999993e-06 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509404+0000", 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6612999999999994e-05 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509413+0000", 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.511633+0000", 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022206729999999998 2024-06-27T21:40:22.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:207989 6.a e688/651 6:57913744:::benchmark_data_smithi192_117994_object100395:head v 688'12240, mlcod=688'12237)", 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:29.064520+0000", 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 113.05639362300001, 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055321000000002, 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064520+0000", 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064520+0000", 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064521+0000", 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064525+0000", 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6609999999999999e-06 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362612+0000", 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29808666 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362625+0000", 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2726e-05 2024-06-27T21:40:22.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362724+0000", 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9574000000000006e-05 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362768+0000", 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3479000000000002e-05 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.365040+0000", 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002272086 2024-06-27T21:40:22.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.365073+0000", 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3293000000000001e-05 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e705,698 6:17d4eb95:::benchmark_data_smithi192_117994_object66222:head,607'10400)", 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 72.891812724000005, 2024-06-27T21:40:22.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28567531600000001, 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:40:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0299999999999999e-07 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229107+0000", 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.806e-06 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514622+0000", 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28551521499999999 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514630+0000", 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9049999999999997e-06 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514724+0000", 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3922e-05 2024-06-27T21:40:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514731+0000", 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4740000000000002e-06 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514776+0000", 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5491000000000002e-05 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.199 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e706,698 6:10363e48:::benchmark_data_smithi192_117994_object81348:head,659'10912)", 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:11.552360+0000", 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 70.568553363999996, 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73961317900000001, 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552360+0000", 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552360+0000", 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552361+0000", 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6400000000000002e-07 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552365+0000", 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4750000000000004e-06 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291750+0000", 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73938458600000001 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291773+0000", 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2864999999999999e-05 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291907+0000", 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013467000000000001 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291915+0000", 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291973+0000", 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8165999999999998e-05 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e706,698 6:12477890:::benchmark_data_smithi192_117994_object82990:head,660'10965)", 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:12.440774+0000", 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 69.680139440999994, 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78605415199999995, 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.204 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440774+0000", 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440774+0000", 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440781+0000", 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5200000000000003e-06 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440790+0000", 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7459999999999995e-06 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226686+0000", 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78589590899999995 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226693+0000", 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9639999999999997e-06 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226763+0000", 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9511000000000006e-05 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226770+0000", 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3470000000000001e-06 2024-06-27T21:40:22.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226828+0000", 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8155000000000001e-05 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e706,698 6:11d68537:::benchmark_data_smithi192_117994_object83145:head,660'10975)", 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 68.865962824999997, 2024-06-27T21:40:22.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52791518199999998, 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7199999999999999e-07 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254957+0000", 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8869999999999997e-06 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782724+0000", 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52776675900000003 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782731+0000", 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4919999999999996e-06 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782792+0000", 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0925999999999999e-05 2024-06-27T21:40:22.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782802+0000", 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7699999999999996e-06 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782866+0000", 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3775999999999994e-05 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.212 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e707,698 6:10e09e4c:::benchmark_data_smithi192_117994_object98629:head,686'11458)", 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:14.832210+0000", 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.288703429999998, 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.85005609900000001, 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832210+0000", 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:40:22.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832210+0000", 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832211+0000", 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832214+0000", 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6610000000000002e-06 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682103+0000", 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84988828800000005 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682119+0000", 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6116999999999998e-05 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682203+0000", 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4399000000000003e-05 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682212+0000", 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1689999999999998e-06 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682266+0000", 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3881000000000001e-05 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:40:22.216 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:40:22.217 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:40:22.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:22.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:22.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:21 smithi192 ceph-mon[32956]: pgmap v1824: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-06-27T21:40:22.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/144518099' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:22.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:21 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:22.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 15 47998 47983 85.6765 74.5 0.00444376 0.0116664 2024-06-27T21:40:22.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:22.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:22.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:22.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:22.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:22.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:22.753 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:22.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:22.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:22.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:22.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:40:22.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:23.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:23.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:23.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:22 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:40:23.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:22 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:40:23.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:22 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:23.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:22 smithi151 ceph-mon[28965]: osdmap e738: 12 total, 12 up, 10 in 2024-06-27T21:40:23.225 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:23.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:22 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:40:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:22 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:40:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:22 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:22 smithi190 ceph-mon[32247]: osdmap e738: 12 total, 12 up, 10 in 2024-06-27T21:40:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:23.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:22 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:40:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:22 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:40:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:22 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:22 smithi192 ceph-mon[32956]: osdmap e738: 12 total, 12 up, 10 in 2024-06-27T21:40:23.340 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 49414 49398 85.7531 88.4375 0.018606 0.0116553 2024-06-27T21:40:23.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:23.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:23.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:23.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:23.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:23.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:23.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:23.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:23.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:23.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:23.934 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:24.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:24.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:23 smithi190 ceph-mon[32247]: pgmap v1826: 47 pgs: 47 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-27T21:40:24.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:24.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:24.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:24.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:23 smithi151 ceph-mon[28965]: pgmap v1826: 47 pgs: 47 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-27T21:40:24.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:24.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:23 smithi192 ceph-mon[32956]: pgmap v1826: 47 pgs: 47 active+clean; 2.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-06-27T21:40:24.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 50470 50454 85.2191 66 0.0222623 0.0117284 2024-06-27T21:40:24.393 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:24.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:24.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:24.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:24.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:24.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:24.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:24.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:24.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:24.923 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:24 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:40:24.923 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:24 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:40:24.923 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:24 smithi190 ceph-mon[32247]: osdmap e739: 12 total, 12 up, 10 in 2024-06-27T21:40:24.932 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:40:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:40:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:40:24.933 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:40:24.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:24.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:25.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:25.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:25.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:24 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:40:25.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:24 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:40:25.171 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:24 smithi151 ceph-mon[28965]: osdmap e739: 12 total, 12 up, 10 in 2024-06-27T21:40:25.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:25.218 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:24 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:40:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:24 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:40:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:24 smithi192 ceph-mon[32956]: osdmap e739: 12 total, 12 up, 10 in 2024-06-27T21:40:25.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:40:25.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:25.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 52046 52030 85.5684 98.5 0.00315457 0.0116805 2024-06-27T21:40:25.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:25.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:25.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:25.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:25.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:40:25.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:25.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:25.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:25.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:25.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:25.892 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:25.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:25.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-06-27T21:40:25.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:25.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:26.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:26.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:25 smithi151 ceph-mon[28965]: pgmap v1827: 47 pgs: 47 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-27T21:40:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:26.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:26.147 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:25 smithi190 ceph-mon[32247]: pgmap v1827: 47 pgs: 47 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-27T21:40:26.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:26.242 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:25 smithi192 ceph-mon[32956]: pgmap v1827: 47 pgs: 47 active+clean; 2.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-06-27T21:40:26.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 53538 53522 85.7651 93.25 0.00827993 0.0116533 2024-06-27T21:40:26.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:26.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:26.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:26.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:26.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:26.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:26.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:26.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:26.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:26.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:26.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:26.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:27.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:27.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:27.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:27.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:26 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:40:27.153 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:26 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:40:27.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:26 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:40:27.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:26 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:40:27.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:27.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:26 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:40:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:26 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:40:27.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:40:27.340304+0000 min lat: 0.00110269 max lat: 3.09321 avg lat: 0.0117403 2024-06-27T21:40:27.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:40:27.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 54506 54490 85.1334 60.5 0.00969624 0.0117403 2024-06-27T21:40:27.355 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:27.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:27.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:27.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:27.468 INFO:teuthology.orchestra.run.smithi151.stderr:2024-06-27T21:40:27.465+0000 7f83f77fe640 0 --2- 172.21.15.151:0/3915897647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f83f8103530 0x7f83f819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-27T21:40:27.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:27.495 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:40:27.496 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:40:27.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:27.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:40:27.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:40:27.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:40:27.547 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:40:27.571 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:40:27.572 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:40:27.600 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:40:27.601 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:40:27.601 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:40:27.601 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:40:27.601 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:40:27.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:40:27.607 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:40:27.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:27.640 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:40:27.640 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:40:27.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:27.676 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:40:27.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:40:27.705 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:40:27.705 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:40:27.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:40:27.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:40:27.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:27.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:27.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:40:27.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:40:27.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:40:27.809 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:40:27.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:40:27.839 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:40:27.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:27.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:27.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:28.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:28.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:28.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:28.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: pgmap v1829: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-27T21:40:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:40:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:40:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: osdmap e740: 12 total, 12 up, 10 in 2024-06-27T21:40:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3915897647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:40:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:40:28.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: pgmap v1829: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-27T21:40:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:40:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:40:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: osdmap e740: 12 total, 12 up, 10 in 2024-06-27T21:40:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3915897647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:40:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:40:28.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: pgmap v1829: 47 pgs: 47 active+clean; 3.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-27T21:40:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:40:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:40:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: osdmap e740: 12 total, 12 up, 10 in 2024-06-27T21:40:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3915897647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:40:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:40:28.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 16 55882 55866 85.1543 86 0.00507438 0.0117376 2024-06-27T21:40:28.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:40:28.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:28.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:28.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:28.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:28.669 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:40:28.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:28.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:28.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:28.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:28.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:28.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:28.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:28.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:29.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:29.084 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:28 smithi190 ceph-mon[32247]: osdmap e741: 12 total, 12 up, 10 in 2024-06-27T21:40:29.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:29.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:28 smithi151 ceph-mon[28965]: osdmap e741: 12 total, 12 up, 10 in 2024-06-27T21:40:29.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:28 smithi192 ceph-mon[32956]: osdmap e741: 12 total, 12 up, 10 in 2024-06-27T21:40:29.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:29.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:29.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:29.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 57154 57138 85.0196 79.5 0.0040068 0.011755 2024-06-27T21:40:29.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:29.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:29.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:29.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:29.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:29.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:29.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:29.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:29.865 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:29.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:29.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:29.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:30.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:30.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:30.191 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:29 smithi190 ceph-mon[32247]: pgmap v1832: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:40:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:29 smithi151 ceph-mon[28965]: pgmap v1832: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:40:30.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:30.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:29 smithi192 ceph-mon[32956]: pgmap v1832: 47 pgs: 47 active+clean; 3.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-06-27T21:40:30.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 58237 58221 84.6164 67.6875 0.00493503 0.0118067 2024-06-27T21:40:30.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:30.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:30.439 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:30.439 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:30.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:30.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:30.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:40:30.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:30.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:30.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:30.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:30.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:30.878 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:30.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:30.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:31.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:31.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:40:31.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:31.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:31.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:31.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:31.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 15 59409 59394 84.3593 73.3125 0.0261689 0.0118441 2024-06-27T21:40:31.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:31.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:31.409 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:31.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:31.508 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:31.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:31.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:31.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:31.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:31.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:31.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:31.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:31.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:32.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:32.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:32.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:31 smithi151 ceph-mon[28965]: pgmap v1833: 47 pgs: 47 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:40:32.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:31 smithi190 ceph-mon[32247]: pgmap v1833: 47 pgs: 47 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:40:32.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:31 smithi192 ceph-mon[32956]: pgmap v1833: 47 pgs: 47 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-06-27T21:40:32.341 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 60362 60346 83.8068 59.5 0.0026878 0.0119089 2024-06-27T21:40:32.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:32.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:32.402 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:32.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:32.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:32.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:32.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:32.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:32.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:32.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:32.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:32.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:33.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:33.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:33.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:33.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:33.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:33.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:33.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 61586 61570 83.6455 76.5 0.00412124 0.0119488 2024-06-27T21:40:33.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:33.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:33.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:33.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:33.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:33.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:33.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:33.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:33.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:33.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:33.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:34.026 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:34.027 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:34.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:34.064 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:34.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:33 smithi151 ceph-mon[28965]: pgmap v1834: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:40:34.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1136624617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:40:34.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:33 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:40:34.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:33 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:40:34.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-06-27T21:40:34.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:34.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:34.253 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:40:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:33 smithi190 ceph-mon[32247]: pgmap v1834: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:40:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1136624617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:40:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:33 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:40:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:33 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:40:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:33 smithi192 ceph-mon[32956]: pgmap v1834: 47 pgs: 47 active+clean; 3.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:40:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1136624617' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:40:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:33 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:40:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:33 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:40:34.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:34.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:34.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 62614 62598 83.2327 64.25 0.00762153 0.0120075 2024-06-27T21:40:34.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:34.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:34.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:34.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:34.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:34.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:34.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:34.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:40:34.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:34.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:34.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:35.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:35.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:35.128 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:34 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:35.128 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:34 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:40:35.128 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:34 smithi192 ceph-mon[32956]: osdmap e742: 12 total, 12 up, 10 in 2024-06-27T21:40:35.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:34 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:34 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:40:35.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:34 smithi151 ceph-mon[28965]: osdmap e742: 12 total, 12 up, 10 in 2024-06-27T21:40:35.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:35.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:34 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:34 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:40:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:34 smithi190 ceph-mon[32247]: osdmap e742: 12 total, 12 up, 10 in 2024-06-27T21:40:35.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 63739 63723 82.9634 70.3125 0.0133302 0.0120482 2024-06-27T21:40:35.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:35.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:35.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:35.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:35.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:35.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:35.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:35.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:35.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:35.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:35.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:36.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:36.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:36.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:36.132 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:35 smithi192 ceph-mon[32956]: pgmap v1836: 47 pgs: 47 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:40:36.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:36.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:35 smithi151 ceph-mon[28965]: pgmap v1836: 47 pgs: 47 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:40:36.239 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:36.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:35 smithi190 ceph-mon[32247]: pgmap v1836: 47 pgs: 47 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-06-27T21:40:36.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:36.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 16 64746 64730 82.5546 62.9375 0.00277261 0.0121062 2024-06-27T21:40:36.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:36.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:36.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:36.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:36.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:36.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:36.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:36.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:36.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:36.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:36.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:40:36.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:36.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:36.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:37.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:37.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:37.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:37.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:40:37.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:40:37.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:37.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:40:37.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:40:37.272 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:40:37.272 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:40:37.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:40:37.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:40:37.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:37.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:40:37.332 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:40:37.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 65921 65905 82.3721 73.4375 0.0134943 0.0121345 2024-06-27T21:40:37.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:37.364 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:40:37.364 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:40:37.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:40:37.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:40:37.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:40:37.428 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:40:37.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:37.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:40:37.459 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:40:37.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:37.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:40:37.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:40:37.520 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:40:37.521 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:40:37.556 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:40:37.556 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:40:37.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:37.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:37.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:37.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:37.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:37.861 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:37.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:37.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:37.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:37 smithi192 ceph-mon[32956]: pgmap v1837: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-27T21:40:38.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2898303433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:38.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:38.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:37 smithi151 ceph-mon[28965]: pgmap v1837: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-27T21:40:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2898303433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:38.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:38.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:37 smithi190 ceph-mon[32247]: pgmap v1837: 47 pgs: 47 active+clean; 3.8 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-06-27T21:40:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2898303433' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:38.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 67118 67102 82.2237 74.8125 0.00317831 0.0121563 2024-06-27T21:40:38.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:38.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:38.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:38.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:38.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:38.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:38.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:38.764 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:38.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:38.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:38.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:39.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:39.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:39.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:39.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:39.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:39.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:39.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 68160 68144 81.8948 65.125 0.0268346 0.0122025 2024-06-27T21:40:39.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:39.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:39.478 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:39.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:39.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:39.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:39.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:39.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:39.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:39.866 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:40:39.867 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:40:39.867 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:40:39.867 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:40:39.867 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:40:39.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:39.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:39.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:40.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:40.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:40.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:40.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:40.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:39 smithi151 ceph-mon[28965]: pgmap v1838: 47 pgs: 47 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:40:40.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:39 smithi190 ceph-mon[32247]: pgmap v1838: 47 pgs: 47 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:40:40.272 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:39 smithi192 ceph-mon[32956]: pgmap v1838: 47 pgs: 47 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:40:40.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:40.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 68873 68857 81.1904 44.5625 0.0117723 0.0123107 2024-06-27T21:40:40.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:40.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:40.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:40.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:40.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:40.648 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:40:40.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:40.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:40.800 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.801 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:40.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:40.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:40.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:41.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:41.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:41.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:41.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:41.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:41.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:41.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 69783 69767 80.74 56.875 0.00483887 0.0123767 2024-06-27T21:40:41.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:41.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:41.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:40:41.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:41.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:41.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:41.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:41.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:41.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:41.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:41.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:41 smithi151 ceph-mon[28965]: pgmap v1839: 47 pgs: 47 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:40:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:41.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:41.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:42.009 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:42.019 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:42.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:42.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:42.146 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:41 smithi192 ceph-mon[32956]: pgmap v1839: 47 pgs: 47 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:40:42.200 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:42.230 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:41 smithi190 ceph-mon[32247]: pgmap v1839: 47 pgs: 47 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-06-27T21:40:42.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:42.343 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 70835 70819 80.4674 65.75 0.0486282 0.0124142 2024-06-27T21:40:42.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.358 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:42.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:42.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:42.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:42.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:42.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:42.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:42.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:42.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:42.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:42.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:42.986 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:40:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:43.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:43.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:40:43.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:40:43.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3185278419' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:43.084 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:42 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:43.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:43.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:43.222 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.223 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.223 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:40:43.223 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:40:43.223 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.223 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3185278419' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:43.223 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:42 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:43.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:43.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3185278419' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:42 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:40:43.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 72016 72000 80.3482 73.8125 0.00865289 0.0124406 2024-06-27T21:40:43.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:43.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:43.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:43.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:43.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:43.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:43.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:43.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:43.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:43.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:43.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:43.915 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:40:43.915 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:40:43.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.9 634/633 [PushOp(6:974523ba:::benchmark_data_smithi192_117994_object74787:head, version: 492'4691, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:974523ba:::benchmark_data_smithi192_117994_object74787:head@492'4691, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:53.414547+0000", 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 230.497082132, 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.38933674899999998, 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.916 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.414547+0000", 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.414547+0000", 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.414548+0000", 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.414574+0000", 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.599e-05 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.802007+0000", 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.38743359900000002 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.802019+0000", 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1491999999999999e-05 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.802090+0000", 2024-06-27T21:40:43.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0599000000000006e-05 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.802099+0000", 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0310000000000004e-06 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:53.803884+0000", 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017854139999999999 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.920 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.9 635/633 [PushOp(6:900f3549:::benchmark_data_smithi192_117994_object86299:head, version: 496'5411, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:900f3549:::benchmark_data_smithi192_117994_object86299:head@496'5411, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:56.542302+0000", 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 227.369327782, 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.72485049899999998, 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:56.542302+0000", 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:56.542302+0000", 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:56.542302+0000", 2024-06-27T21:40:43.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0800000000000005e-07 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:56.542384+0000", 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2094000000000004e-05 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:57.265319+0000", 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.72293521299999997 2024-06-27T21:40:43.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:57.265334+0000", 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5130000000000001e-05 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:57.265418+0000", 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3738000000000004e-05 2024-06-27T21:40:43.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:57.265428+0000", 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5680000000000005e-06 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:57.267152+0000", 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017242480000000001 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.9 636/633 [PushOp(6:90812c7e:::benchmark_data_smithi192_117994_object89664:head, version: 496'5609, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:90812c7e:::benchmark_data_smithi192_117994_object89664:head@496'5609, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:58.164562+0000", 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 225.747066932, 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.43049343499999998, 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.164562+0000", 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.164562+0000", 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.164564+0000", 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5239999999999999e-06 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.164626+0000", 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1579000000000001e-05 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.593042+0000", 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42841623000000001 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.593070+0000", 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8384000000000001e-05 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.593300+0000", 2024-06-27T21:40:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000230206 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.593342+0000", 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2014999999999997e-05 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.595056+0000", 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001713497 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.1 636/633 [PushOp(6:83a6a273:::benchmark_data_smithi192_117994_object87245:head, version: 496'5275, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:83a6a273:::benchmark_data_smithi192_117994_object87245:head@496'5275, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:58.349639+0000", 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 225.561990033, 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57284844700000004, 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.349639+0000", 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.349639+0000", 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.349640+0000", 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1799999999999995e-07 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.349664+0000", 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4167000000000001e-05 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.921230+0000", 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.57156567899999999 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.921241+0000", 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0795000000000001e-05 2024-06-27T21:40:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.921295+0000", 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4681e-05 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.921304+0000", 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5320000000000006e-06 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:58.922488+0000", 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011839750000000001 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.9 637/633 [PushOp(6:918a1ac9:::benchmark_data_smithi192_117994_object96765:head, version: 498'6069, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:918a1ac9:::benchmark_data_smithi192_117994_object96765:head@498'6069, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:00.081576+0000", 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 223.83005339799999, 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64310218799999996, 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.081576+0000", 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.081576+0000", 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.081577+0000", 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5030000000000001e-06 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.081637+0000", 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9695999999999999e-05 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.723034+0000", 2024-06-27T21:40:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.641397094 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.723044+0000", 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9669999999999996e-06 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.723097+0000", 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3096000000000001e-05 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.723104+0000", 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7989999999999996e-06 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:00.724678+0000", 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015740330000000001 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.3 638/633 [PushOp(6:c1d95904:::benchmark_data_smithi192_117994_object97826:head, version: 499'6193, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:c1d95904:::benchmark_data_smithi192_117994_object97826:head@499'6193, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:01.657339+0000", 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 222.254290578, 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35508967499999999, 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:01.657339+0000", 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:01.657339+0000", 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:01.657344+0000", 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5770000000000001e-06 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:01.657403+0000", 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8482e-05 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:02.010602+0000", 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35319931700000001 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:02.010617+0000", 2024-06-27T21:40:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.447e-05 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:02.010684+0000", 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7027999999999996e-05 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:02.010693+0000", 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9400000000000008e-06 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:02.012428+0000", 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001735861 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPull(6.c e657/651 cost 1049576)", 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:39.017045+0000", 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 184.89458457399999, 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.172596819, 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.017045+0000", 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.017045+0000", 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.017045+0000", 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.017054+0000", 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9069999999999996e-06 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.017068+0000", 2024-06-27T21:40:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3772e-05 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.017073+0000", 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3890000000000004e-06 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:40.189226+0000", 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.172152332 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:40.189642+0000", 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041580799999999999 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPull(6.c e657/651 cost 1049576)", 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:39.782026+0000", 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 184.129603575, 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51727964199999998, 2024-06-27T21:40:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.782026+0000", 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.782026+0000", 2024-06-27T21:40:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.782026+0000", 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.23e-07 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.782038+0000", 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1926000000000001e-05 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.782069+0000", 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0623000000000001e-05 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:39.782076+0000", 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7700000000000004e-06 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:40.298966+0000", 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.51689035900000002 2024-06-27T21:40:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:40.299305+0000", 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000339241 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPull(6.c e657/651 cost 1049576)", 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:41.922139+0000", 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 181.98949057799999, 2024-06-27T21:40:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.62616057599999997, 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:41.922139+0000", 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:41.922139+0000", 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:41.922139+0000", 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:41.922151+0000", 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1752e-05 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:41.922176+0000", 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4899999999999999e-05 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:41.922182+0000", 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8220000000000003e-06 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.547137+0000", 2024-06-27T21:40:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.62495509500000002 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.548299+0000", 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011623880000000001 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPull(6.c e657/651 cost 1049576)", 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:42.156788+0000", 2024-06-27T21:40:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 181.75484164299999, 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40843573999999999, 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.156788+0000", 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.156788+0000", 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.156788+0000", 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2899999999999999e-07 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.156796+0000", 2024-06-27T21:40:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7020000000000001e-06 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.161796+0000", 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050003950000000004 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.161803+0000", 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8290000000000002e-06 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.564096+0000", 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40229330499999999 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:42.565223+0000", 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011270799999999999 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPull(6.c e659/651 cost 1049576)", 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:37:46.788071+0000", 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 177.12355847500001, 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.423507466, 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:46.788071+0000", 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:46.788071+0000", 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:46.788072+0000", 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1600000000000001e-07 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:46.788175+0000", 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010358500000000001 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:46.788205+0000", 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0212000000000001e-05 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:46.788212+0000", 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4110000000000001e-06 2024-06-27T21:40:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:47.211045+0000", 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42283314399999999 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:37:47.211578+0000", 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00053339800000000003 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:197879 6.23 6:c457d065:::benchmark_data_smithi192_117994_object90285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e669)", 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:38:01.030672+0000", 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 162.88095736400001, 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39232403399999999, 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 197879 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030672+0000", 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030672+0000", 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030673+0000", 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3829999999999999e-06 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030679+0000", 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.925e-06 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030693+0000", 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3322999999999999e-05 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030700+0000", 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.272e-06 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030743+0000", 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3328000000000003e-05 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.030832+0000", 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9105999999999997e-05 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,9", 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.031093+0000", 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000260704 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.032262+0000", 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011689809999999999 2024-06-27T21:40:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.032296+0000", 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4232e-05 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.422884+0000", 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39058791199999998 2024-06-27T21:40:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.422940+0000", 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5832999999999999e-05 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:01.422996+0000", 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6035000000000001e-05 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:203112 6.23 6:c5331a6a:::benchmark_data_smithi192_117994_object95518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:38:08.134104+0000", 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 155.777525637, 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.486375744, 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:40:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:40:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 203112 2024-06-27T21:40:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134104+0000", 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134104+0000", 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134105+0000", 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.624e-06 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134111+0000", 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.925e-06 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134195+0000", 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3844000000000006e-05 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134203+0000", 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0970000000000006e-06 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134259+0000", 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5469e-05 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134382+0000", 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012361800000000001 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3,9", 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.134687+0000", 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030465400000000001 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.137289+0000", 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026019709999999998 2024-06-27T21:40:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.137442+0000", 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015299500000000001 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.620418+0000", 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.48297621400000001 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.620453+0000", 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5135999999999997e-05 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:08.620479+0000", 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6197e-05 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.c e684,651 6:338dd4c0:::benchmark_data_smithi192_117994_object62774:head,594'10948)", 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:38:22.942394+0000", 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 140.96923553900001, 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54797872700000005, 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:22.942394+0000", 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:22.942394+0000", 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:22.942394+0000", 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2699999999999999e-07 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:22.942399+0000", 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2740000000000001e-06 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:23.490212+0000", 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54781321500000002 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:23.490227+0000", 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4766000000000001e-05 2024-06-27T21:40:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:23.490323+0000", 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6084999999999999e-05 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:23.490330+0000", 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.182e-06 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:38:23.490373+0000", 2024-06-27T21:40:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2577999999999998e-05 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.6 e705,698 6:65f8acd3:::benchmark_data_smithi192_117994_object67347:head,613'10359)", 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:08.554932+0000", 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 95.356697570999998, 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.975829685, 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.554932+0000", 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.554932+0000", 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.554932+0000", 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5600000000000005e-07 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.554937+0000", 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4209999999999997e-06 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.530596+0000", 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97565877199999995 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.530605+0000", 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8600000000000005e-06 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.530680+0000", 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4031999999999999e-05 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.530686+0000", 2024-06-27T21:40:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4649999999999999e-06 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.530761+0000", 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5479e-05 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.6 e706,698 6:66a49e2c:::benchmark_data_smithi192_117994_object73793:head,641'10549)", 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:10.879819+0000", 2024-06-27T21:40:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 93.031810774999997, 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41206710699999999, 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:10.879819+0000", 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:10.879819+0000", 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:10.879819+0000", 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:10.879824+0000", 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0499999999999999e-06 2024-06-27T21:40:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.291718+0000", 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.41189330800000001 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.291725+0000", 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2599999999999999e-06 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.291790+0000", 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5575999999999997e-05 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.291802+0000", 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1955e-05 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.291886+0000", 2024-06-27T21:40:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3347e-05 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.6 e706,698 6:63896bfc:::benchmark_data_smithi192_117994_object74269:head,642'10564)", 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:11.346772+0000", 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 92.564857783999997, 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35296932399999997, 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.346772+0000", 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.346772+0000", 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.346772+0000", 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3499999999999995e-07 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.346777+0000", 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9710000000000003e-06 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.699600+0000", 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35282243699999999 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.699607+0000", 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6310000000000003e-06 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.699667+0000", 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9784999999999998e-05 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.699674+0000", 2024-06-27T21:40:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1969999999999999e-06 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.699741+0000", 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6568000000000004e-05 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.6 e706,698 6:63f58ae4:::benchmark_data_smithi192_117994_object74285:head,642'10565)", 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:11.350128+0000", 2024-06-27T21:40:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 92.561500860999999, 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35319356400000002, 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.350128+0000", 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.350128+0000", 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.350129+0000", 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6700000000000003e-07 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.350134+0000", 2024-06-27T21:40:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.42e-06 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.703205+0000", 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.35307190700000002 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.703211+0000", 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0000000000000002e-06 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.703262+0000", 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0185999999999997e-05 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.703271+0000", 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5279999999999992e-06 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:11.703322+0000", 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0856000000000002e-05 2024-06-27T21:40:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.6 e707,698 6:6043dd2f:::benchmark_data_smithi192_117994_object96460:head,675'11246)", 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:14.030103+0000", 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 89.881526027000007, 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37871898500000001, 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:40:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.030103+0000", 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.030103+0000", 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.030104+0000", 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6000000000000004e-07 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.030107+0000", 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4429999999999999e-06 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.408576+0000", 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37846876800000001 2024-06-27T21:40:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.408592+0000", 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5733999999999999e-05 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.408711+0000", 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011958799999999999 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.408726+0000", 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4266e-05 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.408822+0000", 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6626000000000006e-05 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:47594 7.7 7:efdbed25:::benchmark_data_smithi192_150129_object47593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e738)", 2024-06-27T21:40:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:40:22.010685+0000", 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 21.900944216999999, 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14920191499999999, 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 47594 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010685+0000", 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010685+0000", 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010687+0000", 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6360000000000001e-06 2024-06-27T21:40:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010707+0000", 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0013999999999998e-05 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010719+0000", 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-05 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:40:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010725+0000", 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7490000000000004e-06 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010775+0000", 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9836000000000002e-05 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.010892+0000", 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011754799999999999 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,10", 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.011019+0000", 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012715599999999999 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.012738+0000", 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017183840000000001 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.015789+0000", 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030511100000000001 2024-06-27T21:40:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.159788+0000", 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.14399952599999999 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.159843+0000", 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4416999999999998e-05 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:40:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:22.159887+0000", 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4168999999999997e-05 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:40:44.013 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:40:44.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:44.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:44.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:44.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:44.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:43 smithi151 ceph-mon[28965]: pgmap v1840: 47 pgs: 47 active+clean; 4.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-27T21:40:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:43 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:44.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:43 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:44.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:43 smithi151 ceph-mon[28965]: osdmap e743: 12 total, 12 up, 10 in 2024-06-27T21:40:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:43 smithi190 ceph-mon[32247]: pgmap v1840: 47 pgs: 47 active+clean; 4.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-27T21:40:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:43 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:43 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:43 smithi190 ceph-mon[32247]: osdmap e743: 12 total, 12 up, 10 in 2024-06-27T21:40:44.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:43 smithi192 ceph-mon[32956]: pgmap v1840: 47 pgs: 47 active+clean; 4.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-27T21:40:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:43 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:43 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:40:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:43 smithi192 ceph-mon[32956]: osdmap e743: 12 total, 12 up, 10 in 2024-06-27T21:40:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:44.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:44.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 73266 73250 80.3091 78.125 0.0595571 0.0124463 2024-06-27T21:40:44.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:44.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:44.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:44.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:44.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:44.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:44.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:44.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:44.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:44.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:40:44.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:44.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:45.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:45.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:45.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:45.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:45.260 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:45.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:45.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:45.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 74018 74002 79.7347 47 0.0934953 0.0125358 2024-06-27T21:40:45.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:45.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:45.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:45.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:45.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:45.701 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:45.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:45.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:45.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:45.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:46.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:45 smithi192 ceph-mon[32956]: pgmap v1842: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:40:46.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:46.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:46.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:46.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:46.238 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:45 smithi151 ceph-mon[28965]: pgmap v1842: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:40:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:45 smithi190 ceph-mon[32247]: pgmap v1842: 47 pgs: 47 active+clean; 4.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-06-27T21:40:46.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:46.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:46.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 75029 75013 79.4541 63.1875 0.0108593 0.0125802 2024-06-27T21:40:46.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:46.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:46.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:46.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:46.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:40:46.662 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:40:46.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:46.694 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:40:46.695 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:40:46.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:46.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:40:46.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:40:46.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:40:46.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:40:46.784 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:40:46.784 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:40:46.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:40:46.817 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:40:46.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:46.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:46.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:40:46.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:40:46.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:40:46.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:40:46.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:40:46.896 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:40:46.920 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:40:46.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:40:46.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:46.960 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:40:46.960 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:40:46.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:46.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:46.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:40:46.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:40:47.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:47.094 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:47.150 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2210630284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:47.153 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:47.153 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:47.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:47.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2210630284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:47.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:47.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-06-27T21:40:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2210630284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:47.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:47.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:47.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:40:47.343522+0000 min lat: 0.00110269 max lat: 3.09321 avg lat: 0.0126445 2024-06-27T21:40:47.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:40:47.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 75896 75880 79.033 54.1875 0.0327995 0.0126445 2024-06-27T21:40:47.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:47.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:47.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:47.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:47.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:47.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:47.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:47.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:47.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:47.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:47.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:48.046 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:48.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:48.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:40:48.111 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:47 smithi192 ceph-mon[32956]: pgmap v1843: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:40:48.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:48.165 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:40:48.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:47 smithi151 ceph-mon[28965]: pgmap v1843: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:40:48.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:48.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:47 smithi190 ceph-mon[32247]: pgmap v1843: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:40:48.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:48.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 77109 77093 78.9801 75.8125 0.0202392 0.0126534 2024-06-27T21:40:48.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:48.464 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:48.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:48.518 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:48.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:48.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:48.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:48.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:48.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:48.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:48.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:48.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:49.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:49.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:49.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:49.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:48 smithi151 ceph-mon[28965]: pgmap v1844: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:40:49.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:49.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:49.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:48 smithi190 ceph-mon[32247]: pgmap v1844: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:40:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:48 smithi192 ceph-mon[32956]: pgmap v1844: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:40:49.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 78053 78037 78.6577 59 0.00569522 0.0127081 2024-06-27T21:40:49.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:49.384 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:49.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:49.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:49.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:49.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:49.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:49.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:49.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:49.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:49.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:49.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:49.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:40:50.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:50.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:50.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:50.179 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:50.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:50.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:50.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 79133 79117 78.4805 67.5 0.00718904 0.0127333 2024-06-27T21:40:50.371 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:40:50.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:50.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:50.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:50.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:50.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:50.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:50.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:50.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:50.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:50.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:50.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:51.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:51.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:51.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:51.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:51.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:51.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:51.344 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 80068 80052 78.1672 58.4375 0.00367213 0.0127878 2024-06-27T21:40:51.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:51.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:51.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:51 smithi192 ceph-mon[32956]: pgmap v1845: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:40:51.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:51.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:51.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:51 smithi151 ceph-mon[28965]: pgmap v1845: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:40:51.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:51.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:51 smithi190 ceph-mon[32247]: pgmap v1845: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-06-27T21:40:51.826 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:51.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:51.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:51.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:52.074 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:52.074 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:52.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:52.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:52.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:52.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:52.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 81130 81114 77.9857 66.375 0.00836114 0.0128174 2024-06-27T21:40:52.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:52.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:52.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:52.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:52.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:52.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:52.735 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:40:52.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:52.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:52.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:52.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:52.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:53.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:53.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:53.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:53.146 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:53.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:53.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:53.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:53.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 82145 82129 77.7652 63.4375 0.00416701 0.0128543 2024-06-27T21:40:53.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:53.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:53.519 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:53.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:53 smithi192 ceph-mon[32956]: pgmap v1846: 47 pgs: 47 active+clean; 4.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:40:53.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:53.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:53.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:53 smithi151 ceph-mon[28965]: pgmap v1846: 47 pgs: 47 active+clean; 4.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:40:53.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:53.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:53.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:53 smithi190 ceph-mon[32247]: pgmap v1846: 47 pgs: 47 active+clean; 4.8 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:40:53.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:53.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:53.914 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:53.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:53.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:54.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:54.078 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:54.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:54.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:54.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:54.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 16 83338 83322 77.7173 74.5625 0.00634222 0.012862 2024-06-27T21:40:54.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:54.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:54.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:40:54.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:40:54.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:54.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:54.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:54.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:54.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:54.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:55.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:55.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:55.124 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:55.125 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:55.142 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:55.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:55.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:55.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 16 84523 84507 77.6635 74.0625 0.0184391 0.0128705 2024-06-27T21:40:55.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:55.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:55.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:55.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:55.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:55.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:55.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:55.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:55.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:40:55.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:55.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:55.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:55.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:55 smithi151 ceph-mon[28965]: pgmap v1847: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 60 MiB/s wr, 962 op/s 2024-06-27T21:40:55.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:40:55.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:40:56.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:56.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:55 smithi192 ceph-mon[32956]: pgmap v1847: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 60 MiB/s wr, 962 op/s 2024-06-27T21:40:56.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:40:56.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:40:56.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:56.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:56.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:56.226 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:55 smithi190 ceph-mon[32247]: pgmap v1847: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 60 MiB/s wr, 962 op/s 2024-06-27T21:40:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:40:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:40:56.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:56.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:56.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 85582 85566 77.4971 66.1875 0.022209 0.0128982 2024-06-27T21:40:56.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:56.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:56.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:56.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:56.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:56.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:56.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:56.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:56.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:56.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:56.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:56.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:57.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:56 smithi192 ceph-mon[32956]: osdmap e744: 12 total, 12 up, 10 in 2024-06-27T21:40:57.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:57.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:57.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:56 smithi151 ceph-mon[28965]: osdmap e744: 12 total, 12 up, 10 in 2024-06-27T21:40:57.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:57.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:56 smithi190 ceph-mon[32247]: osdmap e744: 12 total, 12 up, 10 in 2024-06-27T21:40:57.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:57.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 86691 86675 77.3801 69.3125 0.00694019 0.0129175 2024-06-27T21:40:57.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:57.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:57.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:57.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:57.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:40:57.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:40:57.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:57.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:57.831 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:40:57.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:57.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:57.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:57.970 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:57 smithi192 ceph-mon[32956]: pgmap v1849: 47 pgs: 47 active+clean; 5.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:40:57.970 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2568150499' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:57.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:58.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:58.022 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:40:58.022 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:40:58.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:40:58.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:40:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:57 smithi151 ceph-mon[28965]: pgmap v1849: 47 pgs: 47 active+clean; 5.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:40:58.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2568150499' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:58.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:57 smithi190 ceph-mon[32247]: pgmap v1849: 47 pgs: 47 active+clean; 5.1 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:40:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2568150499' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:40:58.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:40:58.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 87559 87543 77.0543 54.25 0.00363685 0.0129517 2024-06-27T21:40:58.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:58.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:58.495 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:40:58.526 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:40:58.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:58.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:58.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:40:58.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:40:58.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:40:58.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:40:58.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:40:58.826 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:40:58.826 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:40:58.856 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:40:58.856 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:40:58.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:40:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:40:58.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:58.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:58.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:40:58.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:40:58.937 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:40:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:58 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2568150499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:40:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:58 smithi151 ceph-mon[28965]: osdmap e745: 12 total, 12 up, 10 in 2024-06-27T21:40:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1149940337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:40:58.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:40:58.992 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:40:59.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:40:59.021 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:40:59.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:59.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:58 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:59.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2568150499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:40:59.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:58 smithi192 ceph-mon[32956]: osdmap e745: 12 total, 12 up, 10 in 2024-06-27T21:40:59.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1149940337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:40:59.056 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:40:59.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:40:59.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:40:59.102 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:40:59.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:40:59.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:40:59.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:40:59.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:40:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:58 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:40:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2568150499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:40:59.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:58 smithi190 ceph-mon[32247]: osdmap e745: 12 total, 12 up, 10 in 2024-06-27T21:40:59.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1149940337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:40:59.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:40:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:59.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:40:59.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 88596 88580 76.8842 64.8125 0.00715139 0.0130016 2024-06-27T21:40:59.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:40:59.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:40:59.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:40:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:40:59.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:40:59.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:40:59.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:40:59.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:40:59.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:40:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:40:59.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:40:59.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:40:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:40:59.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:00.046 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:00.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:00.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:00.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: pgmap v1851: 47 pgs: 47 active+clean; 5.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:41:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: osdmap e746: 12 total, 12 up, 10 in 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:41:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:40:59 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:41:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: pgmap v1851: 47 pgs: 47 active+clean; 5.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: osdmap e746: 12 total, 12 up, 10 in 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:41:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:40:59 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:41:00.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: pgmap v1851: 47 pgs: 47 active+clean; 5.2 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: osdmap e746: 12 total, 12 up, 10 in 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:41:00.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:40:59 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:41:00.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:00.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:00.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 89519 89503 76.6211 57.6875 0.00642636 0.0130427 2024-06-27T21:41:00.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:00.461 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:00.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:00.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:00.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:00.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:00.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:41:00.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:00.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:00.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:41:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:41:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 7.3 scrub starts 2024-06-27T21:41:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:41:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:41:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:41:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:00 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:41:00.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:00.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:00.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:01.066 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:41:01.066 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:41:01.066 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:41:01.066 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:41:01.066 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:41:01.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:01.113 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:01.129 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:01.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:01.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:41:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:41:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 7.3 scrub starts 2024-06-27T21:41:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:41:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:41:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:41:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:00 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:41:01.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:41:01.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:41:01.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 7.3 scrub starts 2024-06-27T21:41:01.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:41:01.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:41:01.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:41:01.289 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:00 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:41:01.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 90507 90491 76.4201 61.75 0.014848 0.0130805 2024-06-27T21:41:01.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:01.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:01.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:01.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:01.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:01.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:01.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:01.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:01.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:41:01.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:41:01.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:01.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:01.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:01.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:02.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:02.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:02.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:02.184 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:41:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:01 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:41:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:01 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:41:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:01 smithi151 ceph-mon[28965]: pgmap v1853: 47 pgs: 47 active+clean; 5.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:41:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:01 smithi151 ceph-mon[28965]: 7.3 scrub ok 2024-06-27T21:41:02.241 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 8] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:41:02.241 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:41:02.241 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-06-27T21:41:02.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-06-27T21:41:02.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:01 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:41:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:01 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:41:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:01 smithi190 ceph-mon[32247]: pgmap v1853: 47 pgs: 47 active+clean; 5.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:41:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:01 smithi190 ceph-mon[32247]: 7.3 scrub ok 2024-06-27T21:41:02.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:01 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:41:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:01 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:41:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:01 smithi192 ceph-mon[32956]: pgmap v1853: 47 pgs: 47 active+clean; 5.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:41:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:01 smithi192 ceph-mon[32956]: 7.3 scrub ok 2024-06-27T21:41:02.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 91441 91425 76.1794 58.375 0.0209445 0.0131217 2024-06-27T21:41:02.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:02.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:02.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:02.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:02.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:02.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:02.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:02.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:02.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:02.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:02.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:02.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:02 smithi151 ceph-mon[28965]: 7.e scrub starts 2024-06-27T21:41:03.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:03.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:02 smithi192 ceph-mon[32956]: 7.e scrub starts 2024-06-27T21:41:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:03.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:03.136 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:03.180 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:03.206 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:03.207 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:03.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:02 smithi190 ceph-mon[32247]: 7.e scrub starts 2024-06-27T21:41:03.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:03.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:03.345 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 92455 92439 76.0109 63.375 0.00476003 0.0131498 2024-06-27T21:41:03.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:03.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:03.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:03.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:03.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:03.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:03.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:03.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:03.873 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:41:03.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:03.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:03.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:04.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: pgmap v1854: 47 pgs: 47 active+clean; 5.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 50 MiB/s wr, 803 op/s 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2596790030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:04.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:04.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:04.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: pgmap v1854: 47 pgs: 47 active+clean; 5.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 50 MiB/s wr, 803 op/s 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:04.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2596790030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:04.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:04.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: pgmap v1854: 47 pgs: 47 active+clean; 5.4 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 50 MiB/s wr, 803 op/s 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2596790030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:04.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:04.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:04.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:04.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 93404 93388 75.794 59.3125 0.010079 0.0131868 2024-06-27T21:41:04.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:04.372 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:04.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:04.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:04.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:04.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:04.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:04.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:04.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:04.884 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: 7.d scrub starts 2024-06-27T21:41:04.884 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:41:04.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:41:04.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:04.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: osdmap e747: 12 total, 12 up, 10 in 2024-06-27T21:41:04.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:41:04.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:41:04.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:04 smithi151 ceph-mon[28965]: 7.e scrub ok 2024-06-27T21:41:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:04.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:04.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:04.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:05.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.0 530/513 [PushOp(6:02d98c5a:::benchmark_data_smithi192_117994_object60336:head, version: 488'3783, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:02d98c5a:::benchmark_data_smithi192_117994_object60336:head@488'3783, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:30.490554+0000", 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 514.30280572300001, 2024-06-27T21:41:05.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36376302100000002, 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.490554+0000", 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.490554+0000", 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.490555+0000", 2024-06-27T21:41:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1090000000000001e-06 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.490597+0000", 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1860999999999998e-05 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.851039+0000", 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36044171600000002 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.851057+0000", 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8110000000000001e-05 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.851164+0000", 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010707799999999999 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.851180+0000", 2024-06-27T21:41:05.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5773999999999998e-05 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:30.854317+0000", 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031373730000000002 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:153642 6.1 e533/526 6:883c986a:::benchmark_data_smithi192_117994_object46048:head v 533'9286, mlcod=492'4566)", 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:32:48.606983+0000", 2024-06-27T21:41:05.079 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 496.18637624199999, 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.49619919800000001, 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606983+0000", 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606983+0000", 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606985+0000", 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.345e-06 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:48.606993+0000", 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0180000000000001e-06 2024-06-27T21:41:05.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.096358+0000", 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.48936513300000001 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.096364+0000", 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2920000000000001e-06 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.096410+0000", 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5627000000000002e-05 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.096430+0000", 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9643000000000001e-05 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.103134+0000", 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0067042200000000003 2024-06-27T21:41:05.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:32:49.103183+0000", 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8919999999999999e-05 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.d e547,516 6:b71bdd3d:::benchmark_data_smithi192_117994_object35202:head,519'8913)", 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:25.142210+0000", 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 459.65114928600002, 2024-06-27T21:41:05.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54641082500000004, 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.142210+0000", 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.142210+0000", 2024-06-27T21:41:05.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.142211+0000", 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5899999999999996e-07 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.142215+0000", 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6540000000000001e-06 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.687863+0000", 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.54564788399999997 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.687872+0000", 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7950000000000002e-06 2024-06-27T21:41:05.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.688295+0000", 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00042268299999999998 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:25.688621+0000", 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032624999999999999 2024-06-27T21:41:05.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:157608 6.d 6:b3b508ef:::benchmark_data_smithi192_117994_object50014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e551)", 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:31.768288+0000", 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 453.02507148299998, 2024-06-27T21:41:05.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.396680649, 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 157608 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768288+0000", 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768288+0000", 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768290+0000", 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5960000000000001e-06 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768296+0000", 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5400000000000001e-06 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768312+0000", 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5124e-05 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768319+0000", 2024-06-27T21:41:05.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2810000000000003e-06 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768382+0000", 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2757999999999998e-05 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768514+0000", 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013199899999999999 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.768642+0000", 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012810499999999999 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.772775+0000", 2024-06-27T21:41:05.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041336350000000001 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.772900+0000", 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012471199999999999 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164898+0000", 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39199756600000002 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164939+0000", 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1106000000000003e-05 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164969+0000", 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0227e-05 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.096 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.26020.0:157608 6.d e551/550 ondisk, result = 0)", 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:33:31.773542+0000", 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 453.01981822099998, 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39144087700000002, 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.773542+0000", 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.773542+0000", 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.773542+0000", 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7000000000000004e-07 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:31.773547+0000", 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5159999999999996e-06 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164801+0000", 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.391254516 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164810+0000", 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9539999999999993e-06 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164888+0000", 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7584999999999997e-05 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164894+0000", 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9279999999999998e-06 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:33:32.164982+0000", 2024-06-27T21:41:05.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8708000000000005e-05 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.17 563/550 [PushOp(6:eeb1df39:::benchmark_data_smithi192_117994_object67297:head, version: 490'4163, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:eeb1df39:::benchmark_data_smithi192_117994_object67297:head@490'4163, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:34:12.505492+0000", 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 412.28786783100003, 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50981167800000005, 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:41:05.101 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.505492+0000", 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.505492+0000", 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.505494+0000", 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6130000000000001e-06 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:12.505561+0000", 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7461999999999999e-05 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:13.013789+0000", 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50822814699999996 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:13.013805+0000", 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6121999999999999e-05 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:13.013878+0000", 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3158999999999995e-05 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:13.013888+0000", 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3519999999999996e-06 2024-06-27T21:41:05.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:34:13.015304+0000", 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0014158230000000001 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:169144 6.d 6:b5439742:::benchmark_data_smithi192_117994_object61550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e591)", 2024-06-27T21:41:05.105 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:22.049490+0000", 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 342.743869292, 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.67063501299999995, 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:41:05.106 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 169144 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049490+0000", 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049490+0000", 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049492+0000", 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1710000000000001e-06 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049497+0000", 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2299999999999999e-06 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049509+0000", 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1946e-05 2024-06-27T21:41:05.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049514+0000", 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6069999999999998e-06 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049562+0000", 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7932000000000001e-05 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049654+0000", 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1959000000000005e-05 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1", 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.049730+0000", 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5972999999999999e-05 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.053249+0000", 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035187500000000002 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:41:05.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.053347+0000", 2024-06-27T21:41:05.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7959000000000002e-05 2024-06-27T21:41:05.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.720058+0000", 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66671148700000005 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.720097+0000", 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8121000000000003e-05 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:22.720125+0000", 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8878e-05 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:172065 6.d 6.740498d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:46.363635+0000", 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 318.429724701, 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.48398128200000001, 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.115 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172065 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363635+0000", 2024-06-27T21:41:05.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363635+0000", 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363637+0000", 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.765e-06 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363643+0000", 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.959e-06 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363657+0000", 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4280999999999999e-05 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363679+0000", 2024-06-27T21:41:05.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2374e-05 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363736+0000", 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6759999999999999e-05 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.363744+0000", 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6299999999999998e-06 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.847590+0000", 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.48384619000000001 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:46.847616+0000", 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6322999999999999e-05 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.120 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(6.d 600/599 [PushReplyOp(6:b039814c:::benchmark_data_smithi192_117994_object94020:head)])", 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:48.531055+0000", 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 316.262304771, 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55648814499999999, 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.531055+0000", 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.531055+0000", 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.531055+0000", 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8900000000000001e-07 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.531059+0000", 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3629999999999998e-06 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.087211+0000", 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.55615270500000002 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.087219+0000", 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4560000000000003e-06 2024-06-27T21:41:05.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.087295+0000", 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6637000000000005e-05 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.087543+0000", 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024779499999999998 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:172329 6.19 6:9ee6969d:::benchmark_data_smithi192_117994_object64735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:48.581583+0000", 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 316.211777052, 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.483802917, 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 172329 2024-06-27T21:41:05.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581583+0000", 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581583+0000", 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581584+0000", 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.48e-06 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581590+0000", 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4929999999999998e-06 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581601+0000", 2024-06-27T21:41:05.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.167e-05 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581607+0000", 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.728e-06 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581657+0000", 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9647000000000001e-05 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.581751+0000", 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4022000000000002e-05 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,1,2", 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.582076+0000", 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032508100000000001 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.583272+0000", 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011958979999999999 2024-06-27T21:41:05.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.584649+0000", 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013776369999999999 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.615910+0000", 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031261030000000002 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.065257+0000", 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.44934686600000001 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.065298+0000", 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0289e-05 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.065386+0000", 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8076000000000002e-05 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.131 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:172333 6.f e600/467 6:f63ad1db:::benchmark_data_smithi192_117994_object64739:head v 600'10881, mlcod=600'10879)", 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:48.615797+0000", 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 316.17756254699998, 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45584349400000002, 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.132 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.615797+0000", 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.615797+0000", 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.615798+0000", 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2550000000000001e-06 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:48.615806+0000", 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7689999999999996e-06 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.068014+0000", 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.45220749199999999 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.068020+0000", 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.068059+0000", 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.068076+0000", 2024-06-27T21:41:05.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6215e-05 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.071610+0000", 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035340409999999999 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:49.071641+0000", 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1034999999999997e-05 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(6.d 601/599 [PushReplyOp(6:b182a85b:::benchmark_data_smithi192_117994_object89983:head)])", 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:35:50.042124+0000", 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 314.75123528, 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50613118700000004, 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.042124+0000", 2024-06-27T21:41:05.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.042124+0000", 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.042125+0000", 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.29e-07 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.042127+0000", 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3599999999999999e-06 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.548023+0000", 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50589565999999997 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.548030+0000", 2024-06-27T21:41:05.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6369999999999999e-06 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.548091+0000", 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0924000000000001e-05 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:35:50.548256+0000", 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000164377 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.140 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26020.0:178771 6.d 6:b7e9fa9c:::benchmark_data_smithi192_117994_object71177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e628)", 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:36.462678+0000", 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 268.33068124300002, 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.40331631200000001, 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26020", 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 178771 2024-06-27T21:41:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462678+0000", 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462678+0000", 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462680+0000", 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.184e-06 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462684+0000", 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2919999999999997e-06 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462695+0000", 2024-06-27T21:41:05.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0773e-05 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462701+0000", 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1430000000000003e-06 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462747+0000", 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6031e-05 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462833+0000", 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6458000000000005e-05 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0,11", 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.462909+0000", 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5896999999999995e-05 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.466625+0000", 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037158069999999998 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.467537+0000", 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00091201300000000004 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.865920+0000", 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.39838245999999999 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.865954+0000", 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4582999999999997e-05 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:36.865995+0000", 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0670999999999997e-05 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(6.15 635/550 [PushOp(6:ad5fb3ee:::benchmark_data_smithi192_117994_object81126:head, version: 495'5051, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:ad5fb3ee:::benchmark_data_smithi192_117994_object81126:head@495'5051, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:41:05.147 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:54.799716+0000", 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 249.993643537, 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.76879277700000004, 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:54.799716+0000", 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:54.799716+0000", 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:54.799717+0000", 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6000000000000003e-07 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:54.799782+0000", 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5388000000000005e-05 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:55.566945+0000", 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.76716300699999995 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:55.566958+0000", 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2731000000000001e-05 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:55.567055+0000", 2024-06-27T21:41:05.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7322000000000001e-05 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:55.567065+0000", 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4749999999999999e-06 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:55.568509+0000", 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001444194 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.151 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.18 e705,698 6:1bce29ce:::benchmark_data_smithi192_117994_object50854:head,553'9933)", 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:08.933374+0000", 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 115.859985639, 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.420359013, 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.933374+0000", 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.933374+0000", 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.933375+0000", 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6599999999999996e-07 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.933378+0000", 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.489e-06 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.353597+0000", 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42021838 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.353604+0000", 2024-06-27T21:41:05.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3830000000000003e-06 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.353661+0000", 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7423999999999998e-05 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.353669+0000", 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5499999999999997e-06 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.353733+0000", 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4220999999999994e-05 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.15 e705,698 6:afa8eefe:::benchmark_data_smithi192_117994_object49611:head,551'9740)", 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:08.937585+0000", 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 115.855775053, 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42109352500000002, 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.156 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.937585+0000", 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.937585+0000", 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.937586+0000", 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5099999999999999e-06 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:08.937596+0000", 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0026e-05 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.358170+0000", 2024-06-27T21:41:05.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.42057401700000002 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.358178+0000", 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2309999999999996e-06 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.358243+0000", 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4086999999999998e-05 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:09.358678+0000", 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000435654 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.15 e707,698 6:ac8daefc:::benchmark_data_smithi192_117994_object70792:head,625'10362)", 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:14.954982+0000", 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.838377628, 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50928232799999995, 2024-06-27T21:41:05.160 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.954982+0000", 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.954982+0000", 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.954983+0000", 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6799999999999999e-07 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.954989+0000", 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.224e-06 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.463821+0000", 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50883216499999995 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.463829+0000", 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6529999999999994e-06 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.463894+0000", 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5195000000000002e-05 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.464264+0000", 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037022299999999998 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDelete(6.18 e707,698 6:1b4dfaa5:::benchmark_data_smithi192_117994_object74093:head,641'10646)", 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:14.958179+0000", 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.835180944, 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50993370000000005, 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.164 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.958179+0000", 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.958179+0000", 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.958179+0000", 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.958183+0000", 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.41e-06 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.467969+0000", 2024-06-27T21:41:05.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.50978577800000002 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.467999+0000", 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0209000000000002e-05 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.468045+0000", 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5939999999999997e-05 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.468052+0000", 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.939e-06 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.468112+0000", 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0856999999999997e-05 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.15 e708,698 6:af9d87a3:::benchmark_data_smithi192_117994_object75449:head,643'10493)", 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:16.230422+0000", 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 108.562937342, 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64107011300000005, 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.230422+0000", 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.230422+0000", 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.230423+0000", 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0200000000000002e-07 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.230427+0000", 2024-06-27T21:41:05.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.732e-06 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.871197+0000", 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64077015999999998 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.871211+0000", 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4555999999999999e-05 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.871304+0000", 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2357999999999999e-05 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:16.871493+0000", 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000188705 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.172 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:54120 7.2 7:4683f384:::benchmark_data_smithi192_150129_object54119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e739)", 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:40:26.842229+0000", 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.951130327999998, 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.239527828, 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 54120 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:41:05.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842229+0000", 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842229+0000", 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842231+0000", 2024-06-27T21:41:05.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.356e-06 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842253+0000", 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.26e-05 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842267+0000", 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.331e-05 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842275+0000", 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4819999999999994e-06 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842329+0000", 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3667e-05 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842438+0000", 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010891999999999999 2024-06-27T21:41:05.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,6", 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.842555+0000", 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117007 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.844854+0000", 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022994420000000001 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:26.846303+0000", 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001448845 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:27.081519+0000", 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.23521610400000001 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:27.081710+0000", 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000191323 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:41:05.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:40:27.081757+0000", 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6771999999999999e-05 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:41:05.179 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:41:05.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:05.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: 7.d scrub starts 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: osdmap e747: 12 total, 12 up, 10 in 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:41:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:41:05.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:04 smithi190 ceph-mon[32247]: 7.e scrub ok 2024-06-27T21:41:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: 7.d scrub starts 2024-06-27T21:41:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:41:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:41:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: osdmap e747: 12 total, 12 up, 10 in 2024-06-27T21:41:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:41:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:41:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:04 smithi192 ceph-mon[32956]: 7.e scrub ok 2024-06-27T21:41:05.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:05.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:05.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 94293 94277 75.5346 55.5625 0.0164836 0.0132334 2024-06-27T21:41:05.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:05.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:05.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:05.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:05.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:05.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:05.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:05.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:05.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: pgmap v1856: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 878 op/s 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 7.a scrub starts 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 1.0 deep-scrub starts 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 1.0 deep-scrub ok 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 7.f scrub starts 2024-06-27T21:41:05.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:05 smithi151 ceph-mon[28965]: 7.8 scrub starts 2024-06-27T21:41:05.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:05.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:06.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:41:06.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:06.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:06.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:06.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:41:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: pgmap v1856: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 878 op/s 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 7.a scrub starts 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 1.0 deep-scrub starts 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 1.0 deep-scrub ok 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 7.f scrub starts 2024-06-27T21:41:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:05 smithi190 ceph-mon[32247]: 7.8 scrub starts 2024-06-27T21:41:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: pgmap v1856: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 55 MiB/s wr, 878 op/s 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 7.a scrub starts 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 1.0 deep-scrub starts 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 1.0 deep-scrub ok 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 7.f scrub starts 2024-06-27T21:41:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:05 smithi192 ceph-mon[32956]: 7.8 scrub starts 2024-06-27T21:41:06.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:06.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 95127 95111 75.2382 52.125 0.0263626 0.0132853 2024-06-27T21:41:06.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:06.455 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:41:06.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:06.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:06.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:06.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:06.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:41:06.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:06.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:06.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:06.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:06.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:06.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:06 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:41:06.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:06 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:41:06.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:06 smithi151 ceph-mon[28965]: 7.5 scrub starts 2024-06-27T21:41:06.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:06 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:41:06.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:06 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:41:06.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:07.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:07.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:07.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:07.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:06 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:41:07.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:06 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:41:07.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:06 smithi192 ceph-mon[32956]: 7.5 scrub starts 2024-06-27T21:41:07.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:06 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:41:07.192 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:06 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:41:07.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:07.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:06 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:41:07.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:06 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:41:07.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:06 smithi190 ceph-mon[32247]: 7.5 scrub starts 2024-06-27T21:41:07.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:06 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:41:07.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:06 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:41:07.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:07.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:41:07.345247+0000 min lat: 0.00110269 max lat: 3.09321 avg lat: 0.0133496 2024-06-27T21:41:07.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:41:07.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 95853 95837 74.8649 45.375 0.0133442 0.0133496 2024-06-27T21:41:07.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:07.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:07.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:07.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:07.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:07.672 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:07.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:07.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:07.888 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: pgmap v1857: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 5.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:41:07.888 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.d scrub ok 2024-06-27T21:41:07.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:07.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:07.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:08.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:08.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: pgmap v1857: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 5.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.d scrub ok 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.f scrub ok 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.a scrub ok 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.0 scrub starts 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.c scrub starts 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/633224569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.7 deep-scrub starts 2024-06-27T21:41:08.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:07 smithi151 ceph-mon[28965]: 7.8 scrub ok 2024-06-27T21:41:08.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:08.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:41:08.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:41:08.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-06-27T21:41:08.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:41:08.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:41:08.128 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:41:08.128 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:41:08.142 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.f scrub ok 2024-06-27T21:41:08.142 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.a scrub ok 2024-06-27T21:41:08.142 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.0 scrub starts 2024-06-27T21:41:08.142 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.c scrub starts 2024-06-27T21:41:08.143 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:41:08.143 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:41:08.143 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/633224569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:08.143 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.7 deep-scrub starts 2024-06-27T21:41:08.143 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:07 smithi192 ceph-mon[32956]: 7.8 scrub ok 2024-06-27T21:41:08.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:08.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:41:08.160 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:41:08.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:08.188 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:41:08.188 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:41:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:08.218 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:41:08.218 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:41:08.248 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:41:08.248 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:41:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: pgmap v1857: 47 pgs: 4 active+clean+scrubbing, 43 active+clean; 5.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-06-27T21:41:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.d scrub ok 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.f scrub ok 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.a scrub ok 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.0 scrub starts 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.c scrub starts 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/633224569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.7 deep-scrub starts 2024-06-27T21:41:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:07 smithi190 ceph-mon[32247]: 7.8 scrub ok 2024-06-27T21:41:08.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:41:08.278 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:41:08.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:08.310 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:41:08.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:41:08.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:08.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 96467 96451 74.4143 38.375 0.0242933 0.0134328 2024-06-27T21:41:08.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:41:08.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:41:08.376 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:41:08.377 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:41:08.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:41:08.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:41:08.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:08.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:08.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:08.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:08.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:08.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:08.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:08.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:08.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:08.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:08.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:09.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:09.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:09.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:09.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:09.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:08 smithi151 ceph-mon[28965]: 7.5 scrub ok 2024-06-27T21:41:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:08 smithi151 ceph-mon[28965]: 7.1 scrub starts 2024-06-27T21:41:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:08 smithi151 ceph-mon[28965]: 7.c scrub ok 2024-06-27T21:41:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:08 smithi151 ceph-mon[28965]: 7.0 scrub ok 2024-06-27T21:41:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:09.202 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:08 smithi192 ceph-mon[32956]: 7.5 scrub ok 2024-06-27T21:41:09.202 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:08 smithi192 ceph-mon[32956]: 7.1 scrub starts 2024-06-27T21:41:09.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:08 smithi192 ceph-mon[32956]: 7.c scrub ok 2024-06-27T21:41:09.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:08 smithi192 ceph-mon[32956]: 7.0 scrub ok 2024-06-27T21:41:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:08 smithi190 ceph-mon[32247]: 7.5 scrub ok 2024-06-27T21:41:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:08 smithi190 ceph-mon[32247]: 7.1 scrub starts 2024-06-27T21:41:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:08 smithi190 ceph-mon[32247]: 7.c scrub ok 2024-06-27T21:41:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:08 smithi190 ceph-mon[32247]: 7.0 scrub ok 2024-06-27T21:41:09.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:09.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:09.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 97304 97288 74.1447 52.3125 0.00443895 0.0134814 2024-06-27T21:41:09.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:09.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:09.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:09.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:09.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:09.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:09.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:09.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:09.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:09.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:09.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:10.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:10.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:10.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:10.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:10 smithi192 ceph-mon[32956]: pgmap v1858: 47 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 42 active+clean; 5.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 61 MiB/s wr, 971 op/s 2024-06-27T21:41:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:10 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:41:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:10 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:41:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:10 smithi192 ceph-mon[32956]: 7.1 scrub ok 2024-06-27T21:41:10.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:10.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 15 98088 98073 73.8425 49.0625 0.00624447 0.0135344 2024-06-27T21:41:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:10.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:10 smithi151 ceph-mon[28965]: pgmap v1858: 47 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 42 active+clean; 5.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 61 MiB/s wr, 971 op/s 2024-06-27T21:41:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:10 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:41:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:10 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:41:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:10 smithi151 ceph-mon[28965]: 7.1 scrub ok 2024-06-27T21:41:10.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.479 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:10.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:10 smithi190 ceph-mon[32247]: pgmap v1858: 47 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 42 active+clean; 5.8 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 61 MiB/s wr, 971 op/s 2024-06-27T21:41:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:10 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:41:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:10 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:41:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:10 smithi190 ceph-mon[32247]: 7.1 scrub ok 2024-06-27T21:41:10.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:10.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:10.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:10.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:10.726 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:41:10.726 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:41:10.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:10.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:10.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:10.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:10.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:10.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:11.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:11.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:11.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:11.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: 1.2 deep-scrub starts 2024-06-27T21:41:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: 1.2 deep-scrub ok 2024-06-27T21:41:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:41:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:41:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: pgmap v1859: 47 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 42 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 57 MiB/s wr, 911 op/s 2024-06-27T21:41:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:11 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:11.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 15 98854 98839 73.5333 47.875 0.00450017 0.0135924 2024-06-27T21:41:11.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: 1.2 deep-scrub starts 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: 1.2 deep-scrub ok 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: pgmap v1859: 47 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 42 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 57 MiB/s wr, 911 op/s 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:11.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:11 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:11.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:11.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:41:11.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:11.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: 1.2 deep-scrub starts 2024-06-27T21:41:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: 1.2 deep-scrub ok 2024-06-27T21:41:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:41:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:41:11.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: pgmap v1859: 47 pgs: 1 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 42 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 57 MiB/s wr, 911 op/s 2024-06-27T21:41:11.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:11.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:11 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:11.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:11.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:11.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:11.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:11.888 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:41:11.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:11.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:11.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:12.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:12.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:41:12.059 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:12.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:12.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:12.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:12 smithi192 ceph-mon[32956]: osdmap e748: 12 total, 12 up, 10 in 2024-06-27T21:41:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:12 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:12 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:12.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:12.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 99671 99655 73.2682 51 0.0074953 0.0136423 2024-06-27T21:41:12.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:12.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:12.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:12 smithi151 ceph-mon[28965]: osdmap e748: 12 total, 12 up, 10 in 2024-06-27T21:41:12.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:12 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:12.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:12 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:12.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:12 smithi190 ceph-mon[32247]: osdmap e748: 12 total, 12 up, 10 in 2024-06-27T21:41:12.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:12 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:12.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:12 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:12.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:12.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:12.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:12.703 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:12.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:12.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:12.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:12.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:12.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:13.046 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:41:13.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:13.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:13.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:13.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:13 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:13.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:13 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:13.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:13 smithi151 ceph-mon[28965]: osdmap e749: 12 total, 12 up, 10 in 2024-06-27T21:41:13.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:13 smithi151 ceph-mon[28965]: pgmap v1862: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 59 MiB/s wr, 937 op/s 2024-06-27T21:41:13.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2391699782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:41:13.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:13.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:13.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:13.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 15 100399 100384 72.946 45.5625 0.0415465 0.0136995 2024-06-27T21:41:13.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:13 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:13.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:13 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:13.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:13 smithi192 ceph-mon[32956]: osdmap e749: 12 total, 12 up, 10 in 2024-06-27T21:41:13.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:13 smithi192 ceph-mon[32956]: pgmap v1862: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 59 MiB/s wr, 937 op/s 2024-06-27T21:41:13.409 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2391699782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:41:13.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:13.466 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:13 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:13.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:13 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:13.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:13 smithi190 ceph-mon[32247]: osdmap e749: 12 total, 12 up, 10 in 2024-06-27T21:41:13.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:13 smithi190 ceph-mon[32247]: pgmap v1862: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 59 MiB/s wr, 937 op/s 2024-06-27T21:41:13.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2391699782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:41:13.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:13.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:13.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:13.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:13.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:13.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:13.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:13.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:13.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:14.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:14.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:14.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:14.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:14.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:14.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 101024 101008 72.5558 39 0.00670846 0.0137753 2024-06-27T21:41:14.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:14.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:14.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:14 smithi151 ceph-mon[28965]: 7.b scrub starts 2024-06-27T21:41:14.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:14 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:14.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2391699782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:41:14.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:14 smithi151 ceph-mon[28965]: osdmap e750: 12 total, 12 up, 10 in 2024-06-27T21:41:14.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:14 smithi190 ceph-mon[32247]: 7.b scrub starts 2024-06-27T21:41:14.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:14 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:14.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2391699782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:41:14.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:14 smithi190 ceph-mon[32247]: osdmap e750: 12 total, 12 up, 10 in 2024-06-27T21:41:14.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:14 smithi192 ceph-mon[32956]: 7.b scrub starts 2024-06-27T21:41:14.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:14 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:14.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2391699782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:41:14.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:14 smithi192 ceph-mon[32956]: osdmap e750: 12 total, 12 up, 10 in 2024-06-27T21:41:14.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:14.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:14.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:14.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-06-27T21:41:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:14.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:14.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:14.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:14.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:14.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:15.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:15.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:15.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:15.184 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:15.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:15.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:15.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:15 smithi151 ceph-mon[28965]: pgmap v1864: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 45 MiB/s wr, 717 op/s 2024-06-27T21:41:15.346 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 101849 101833 72.3172 51.5625 0.00745611 0.0138211 2024-06-27T21:41:15.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:15.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:15.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:15 smithi190 ceph-mon[32247]: pgmap v1864: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 45 MiB/s wr, 717 op/s 2024-06-27T21:41:15.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:15 smithi192 ceph-mon[32956]: pgmap v1864: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 45 MiB/s wr, 717 op/s 2024-06-27T21:41:15.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:15.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:15.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:15.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:15.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:15.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:15.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:15.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:16.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:16.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:16.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:16.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:16 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:16.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:16 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:16.162 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:16.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:16.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:16.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 15 102607 102592 72.0376 47.4375 0.00566107 0.0138758 2024-06-27T21:41:16.393 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:16 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:16.393 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:16 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:16.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:16.423 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:16.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:16 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:16.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:16 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:16.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:16.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:16.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:16.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:16.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:16.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:16.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:16.941 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:16.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:17.034 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:41:17.035 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:41:17.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:17.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:17.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:17.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:17 smithi151 ceph-mon[28965]: osdmap e751: 12 total, 12 up, 10 in 2024-06-27T21:41:17.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:17 smithi151 ceph-mon[28965]: pgmap v1866: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 46 MiB/s wr, 734 op/s 2024-06-27T21:41:17.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:41:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:17 smithi192 ceph-mon[32956]: osdmap e751: 12 total, 12 up, 10 in 2024-06-27T21:41:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:17 smithi192 ceph-mon[32956]: pgmap v1866: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 46 MiB/s wr, 734 op/s 2024-06-27T21:41:17.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:17.339 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:17.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 6 103302 103296 71.726 44 0.0252989 0.0139365 2024-06-27T21:41:17.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:17.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:17.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:17 smithi190 ceph-mon[32247]: osdmap e751: 12 total, 12 up, 10 in 2024-06-27T21:41:17.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:17 smithi190 ceph-mon[32247]: pgmap v1866: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 46 MiB/s wr, 734 op/s 2024-06-27T21:41:17.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:17.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:17.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:17.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:17.708 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:17.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:41:17.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:17.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:17.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:17.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:17.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:18.041 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:41:18.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:18.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:18.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:18.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:18.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:18.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.8151 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 103302 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 71.0936 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 24.6655 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 172.812 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 0 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1137 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 394.648 2024-06-27T21:41:18.347 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 2765 2024-06-27T21:41:18.348 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 0 2024-06-27T21:41:18.348 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0139582 2024-06-27T21:41:18.348 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0393034 2024-06-27T21:41:18.348 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 3.09321 2024-06-27T21:41:18.348 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00110269 2024-06-27T21:41:18.353 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:41:18.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:18.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:18.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:18.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:18.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:18.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:18.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:18.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:18.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:18.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:18.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:19.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:19.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:19.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:19.234 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:19.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:19.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:19.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:19.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:19.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:19.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:19 smithi192 ceph-mon[32956]: pgmap v1867: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 43 MiB/s wr, 683 op/s 2024-06-27T21:41:19.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:19.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:19.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:19 smithi151 ceph-mon[28965]: pgmap v1867: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 43 MiB/s wr, 683 op/s 2024-06-27T21:41:19.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:19.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:19.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:19 smithi190 ceph-mon[32247]: pgmap v1867: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 43 MiB/s wr, 683 op/s 2024-06-27T21:41:19.833 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:41:19.833 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:41:19.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:19.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:19.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:19.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:20.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:20.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:20.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:20.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:20.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:20.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:20.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:20.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:20.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:41:20.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:20.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:20.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:20.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:20.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:20.817 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:41:20.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:20.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:20.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:20.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:20 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:20.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:20.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:20 smithi151 ceph-mon[28965]: osdmap e752: 12 total, 12 up, 10 in 2024-06-27T21:41:21.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:21.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:21.085 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:20 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:21.085 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:21.085 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:20 smithi190 ceph-mon[32247]: osdmap e752: 12 total, 12 up, 10 in 2024-06-27T21:41:21.157 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:21.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:21.272 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:21.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:20 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:21.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:20 smithi192 ceph-mon[32956]: osdmap e752: 12 total, 12 up, 10 in 2024-06-27T21:41:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:21.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:21.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:21.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:21.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:21.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:21.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:21.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:21.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:21.823 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:21.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:21.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:21.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:22.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:41:22.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:41:22.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:41:22.041 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:41:22.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:22.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:41:22.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:41:22.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:21 smithi151 ceph-mon[28965]: pgmap v1868: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 41 MiB/s wr, 696 op/s 2024-06-27T21:41:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:22.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:22.100 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:41:22.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:41:22.129 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:41:22.129 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:41:22.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:41:22.154 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:41:22.181 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:41:22.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:22.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:41:22.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:41:22.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:22.241 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:21 smithi190 ceph-mon[32247]: pgmap v1868: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 41 MiB/s wr, 696 op/s 2024-06-27T21:41:22.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:41:22.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:41:22.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:41:22.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:41:22.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:41:22.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:41:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:21 smithi192 ceph-mon[32956]: pgmap v1868: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 41 MiB/s wr, 696 op/s 2024-06-27T21:41:22.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:22.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:41:22.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:41:22.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:22.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:22.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:22.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:22.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:22.651 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:106574 7.b e752/718 7:dc075adb:::benchmark_data_smithi192_150129_object3269:head v 752'6774, mlcod=752'6772)", 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:41:22.648086+0000", 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0018043720000000001, 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018235860000000001, 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648086+0000", 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648086+0000", 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648086+0000", 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1699999999999998e-07 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648091+0000", 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2599999999999996e-06 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648115+0000", 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3323999999999999e-05 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648120+0000", 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0529999999999997e-06 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648171+0000", 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1369000000000002e-05 2024-06-27T21:41:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648193+0000", 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1841e-05 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:106575 7.1 e752/718 7:8281284a:::benchmark_data_smithi192_150129_object3270:head v 752'6583, mlcod=752'6581)", 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:41:22.648395+0000", 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0014946029999999999, 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015460020000000001, 2024-06-27T21:41:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648395+0000", 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648395+0000", 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648396+0000", 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.39e-07 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648399+0000", 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6950000000000002e-06 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648418+0000", 2024-06-27T21:41:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8746999999999999e-05 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648421+0000", 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2310000000000001e-06 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648465+0000", 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4121e-05 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:22.648488+0000", 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2646999999999998e-05 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 2 2024-06-27T21:41:22.660 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:41:22.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:22.730 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:22.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:22.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:22.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:22.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:22.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:23.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1881079510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:23.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:23.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1881079510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:23.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:23.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:23.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1881079510' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:23.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:23.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:23.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:41:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:23.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:23.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:23.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:23.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:23.645 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:41:23.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:23.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:23.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:23.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:24.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:24.059 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:23 smithi190 ceph-mon[32247]: pgmap v1870: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 127 B/s rd, 37 MiB/s wr, 740 op/s 2024-06-27T21:41:24.063 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:23 smithi151 ceph-mon[28965]: pgmap v1870: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 127 B/s rd, 37 MiB/s wr, 740 op/s 2024-06-27T21:41:24.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:24.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:24.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:24.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:23 smithi192 ceph-mon[32956]: pgmap v1870: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 127 B/s rd, 37 MiB/s wr, 740 op/s 2024-06-27T21:41:24.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:24.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:24.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:24.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:24.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:24.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:24.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:24.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:24.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:24.898 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:24.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:25.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:25.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.025 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:25.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:25.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:25.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:25.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:25.279 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:25.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:25.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:25.468 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:41:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:41:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:41:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:41:25.469 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:41:25.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:25.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:25.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:25.662 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:25.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:25.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:25.837 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:41:25.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:41:25.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:25.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:25.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:26.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:26.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:26.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:25 smithi190 ceph-mon[32247]: pgmap v1871: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 125 B/s rd, 15 MiB/s wr, 478 op/s 2024-06-27T21:41:26.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2094073186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:26.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:41:26.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:26.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:25 smithi151 ceph-mon[28965]: pgmap v1871: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 125 B/s rd, 15 MiB/s wr, 478 op/s 2024-06-27T21:41:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2094073186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:26.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:25 smithi192 ceph-mon[32956]: pgmap v1871: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 125 B/s rd, 15 MiB/s wr, 478 op/s 2024-06-27T21:41:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2094073186' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:26.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:26.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:26.413 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:41:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:26.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:26.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:26.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:26.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-06-27T21:41:26.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:26.676 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:26.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:26.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:26.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:26.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:26.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:26 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:27.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2094073186' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:27.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:26 smithi151 ceph-mon[28965]: osdmap e753: 12 total, 12 up, 10 in 2024-06-27T21:41:27.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:27.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:27.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:27.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:27.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:27.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:27.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:26 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2094073186' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:26 smithi190 ceph-mon[32247]: osdmap e753: 12 total, 12 up, 10 in 2024-06-27T21:41:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:27.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:26 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2094073186' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:26 smithi192 ceph-mon[32956]: osdmap e753: 12 total, 12 up, 10 in 2024-06-27T21:41:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:27.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:27.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:27.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:27.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:27.646 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:27.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:27.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:27.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:27.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:27.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:27.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:27.969 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:27.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:27 smithi190 ceph-mon[32247]: pgmap v1873: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 12 MiB/s wr, 799 op/s 2024-06-27T21:41:27.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:27 smithi190 ceph-mon[32247]: osdmap e754: 12 total, 12 up, 10 in 2024-06-27T21:41:28.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:28.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:28.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:28.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:27 smithi151 ceph-mon[28965]: pgmap v1873: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 12 MiB/s wr, 799 op/s 2024-06-27T21:41:28.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:27 smithi151 ceph-mon[28965]: osdmap e754: 12 total, 12 up, 10 in 2024-06-27T21:41:28.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:28.193 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:28.221 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:41:28.221 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:41:28.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.6 532/515 [PushReplyOp(6:6854d079:::benchmark_data_smithi192_117994_object75012:head)])", 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 524.84622360699996, 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.60063333399999996, 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371635+0000", 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3799999999999998e-07 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.371638+0000", 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.326e-06 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.969696+0000", 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.59805774700000003 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.969702+0000", 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7169999999999996e-06 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.969760+0000", 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8294999999999997e-05 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:43.972268+0000", 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002507811 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.6 533/515 [PushReplyOp(6:6292732d:::benchmark_data_smithi192_117994_object82946:head)])", 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:32:51.047303+0000", 2024-06-27T21:41:28.226 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 517.17055485100002, 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42797745300000001, 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047303+0000", 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047303+0000", 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047304+0000", 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.047309+0000", 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7899999999999999e-06 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.473245+0000", 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42593658000000001 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.473256+0000", 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0553999999999999e-05 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.473309+0000", 2024-06-27T21:41:28.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3248999999999998e-05 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:32:51.475281+0000", 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001971549 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.230 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e540,515 6:6fcfe693:::benchmark_data_smithi192_117994_object14778:head,510'7668)", 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:14.560235+0000", 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 493.65762299900001, 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.59406511399999995, 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560235+0000", 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560235+0000", 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560236+0000", 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6100000000000001e-07 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:14.560240+0000", 2024-06-27T21:41:28.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0929999999999996e-06 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.153309+0000", 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.593068924 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.153316+0000", 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.678e-06 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.153975+0000", 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065804500000000001 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.154300+0000", 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032581299999999999 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:156223 6.e e540/467 6:74cd8b3d:::benchmark_data_smithi192_117994_object48629:head v 540'9858, mlcod=540'9856)", 2024-06-27T21:41:28.234 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:15.195227+0000", 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 493.02263086200003, 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43734508599999999, 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195227+0000", 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195227+0000", 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195228+0000", 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0419999999999999e-06 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.195237+0000", 2024-06-27T21:41:28.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6179999999999995e-06 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628577+0000", 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43333962999999998 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628584+0000", 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4240000000000004e-06 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628646+0000", 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2119999999999995e-05 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.628671+0000", 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5228e-05 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.632531+0000", 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038593740000000001 2024-06-27T21:41:28.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:15.632572+0000", 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1650000000000003e-05 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e540,515 6:6157eff0:::benchmark_data_smithi192_117994_object18148:head,511'7888)", 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:17.212981+0000", 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 491.00487747300002, 2024-06-27T21:41:28.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49524637900000001, 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212981+0000", 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212981+0000", 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212982+0000", 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8000000000000005e-07 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.212987+0000", 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1019999999999996e-06 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.707161+0000", 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49417447399999997 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.707170+0000", 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2019999999999993e-06 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.707758+0000", 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058814200000000005 2024-06-27T21:41:28.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:17.708227+0000", 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000468679 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e540,515 6:6fc38716:::benchmark_data_smithi192_117994_object18210:head,511'7892)", 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:18.026250+0000", 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 490.19160776699999, 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37025059999999999, 2024-06-27T21:41:28.243 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026250+0000", 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026250+0000", 2024-06-27T21:41:28.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026251+0000", 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7400000000000002e-07 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.026258+0000", 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6479999999999997e-06 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396215+0000", 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36995685299999997 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396223+0000", 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1159999999999999e-06 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396271+0000", 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8344999999999999e-05 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:18.396501+0000", 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000229764 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.16 551/550 [PushReplyOp(6:6955813a:::benchmark_data_smithi192_117994_object59286:head)])", 2024-06-27T21:41:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:33.187949+0000", 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 475.02990898299998, 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.447962637, 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187949+0000", 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187949+0000", 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187951+0000", 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3960000000000001e-06 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.187956+0000", 2024-06-27T21:41:28.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3580000000000002e-06 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.633549+0000", 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44559270699999998 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.633554+0000", 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2039999999999996e-06 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.633613+0000", 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8850000000000001e-05 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:33.635912+0000", 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022991219999999998 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.251 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.16 551/550 [PushReplyOp(6:6ba9cbd0:::benchmark_data_smithi192_117994_object60030:head)])", 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:33:34.122133+0000", 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 474.09572567499998, 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58028879200000005, 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122133+0000", 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122133+0000", 2024-06-27T21:41:28.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122134+0000", 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1289999999999999e-06 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.122138+0000", 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6829999999999999e-06 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.700499+0000", 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.57836112900000003 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.700512+0000", 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2306999999999999e-05 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.700595+0000", 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3567999999999997e-05 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:33:34.702421+0000", 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001825976 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "replica_scrub(pg: 6.16,from:0'0,to:560'9928,epoch:562/561,start:6:6d94347c:::benchmark_data_smithi192_117994_object81448:0,end:6:6e05da67:::benchmark_data_smithi192_117994_object101194:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:06.073868+0000", 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 442.14399007200001, 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44751875699999999, 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073868+0000", 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073868+0000", 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073869+0000", 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7599999999999996e-07 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.073875+0000", 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1979999999999998e-06 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521287+0000", 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44741202499999999 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521298+0000", 2024-06-27T21:41:28.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0920999999999999e-05 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521349+0000", 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1128999999999998e-05 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521358+0000", 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5820000000000002e-06 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.521387+0000", 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9125999999999999e-05 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:161547 6.16 e562/561 6:6fc2e18a:::benchmark_data_smithi192_117994_object53953:head v 562'9945, mlcod=562'9943)", 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:06.982216+0000", 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 441.23564180699998, 2024-06-27T21:41:28.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58225666300000001, 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982216+0000", 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982216+0000", 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982217+0000", 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1999999999999998e-07 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.982225+0000", 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7689999999999996e-06 2024-06-27T21:41:28.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562719+0000", 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.58049409299999999 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562728+0000", 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6270000000000006e-06 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562780+0000", 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1952e-05 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.562797+0000", 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7552000000000001e-05 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.564443+0000", 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001645229 2024-06-27T21:41:28.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:07.564473+0000", 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0521000000000001e-05 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.15 636/550 [PushOp(6:adc7d4ea:::benchmark_data_smithi192_117994_object88705:head, version: 496'5501, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:adc7d4ea:::benchmark_data_smithi192_117994_object88705:head@496'5501, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 269.14345429799999, 2024-06-27T21:41:28.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.545111978, 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074406+0000", 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7090000000000001e-06 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074452+0000", 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5967999999999998e-05 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617875+0000", 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54342337500000004 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617887+0000", 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.236e-05 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617955+0000", 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7934000000000004e-05 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617964+0000", 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8130000000000002e-06 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.619516+0000", 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015518190000000001 2024-06-27T21:41:28.270 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.21 657/654 [PushOp(6:86e4cd7e:::benchmark_data_smithi192_117994_object88829:head, version: 496'5366, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:86e4cd7e:::benchmark_data_smithi192_117994_object88829:head@496'5366, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:38.954212+0000", 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 229.26364573399999, 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44072743800000003, 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:28.271 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954212+0000", 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954212+0000", 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954213+0000", 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4300000000000002e-07 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954267+0000", 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3646000000000001e-05 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393751+0000", 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43948427699999998 2024-06-27T21:41:28.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393764+0000", 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2349e-05 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393850+0000", 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6161000000000004e-05 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393859+0000", 2024-06-27T21:41:28.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8990000000000007e-06 2024-06-27T21:41:28.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.394940+0000", 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010813629999999999 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.21 658/654 [PushOp(6:86f4a616:::benchmark_data_smithi192_117994_object106008:head, version: 503'6416, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:86f4a616:::benchmark_data_smithi192_117994_object106008:head@503'6416, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:43.012631+0000", 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 225.20522750000001, 2024-06-27T21:41:28.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52881486700000002, 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012631+0000", 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012631+0000", 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012632+0000", 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8700000000000005e-07 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012685+0000", 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3356000000000002e-05 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540285+0000", 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52760042299999999 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540302+0000", 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6521e-05 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540375+0000", 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2912999999999998e-05 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540383+0000", 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9230000000000002e-06 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.541446+0000", 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010629439999999999 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.21 e659,654 6:8749a36d:::benchmark_data_smithi192_117994_object5123:head,507'6835)", 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:45.334891+0000", 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 222.882967235, 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43374579200000002, 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334891+0000", 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334891+0000", 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334892+0000", 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334896+0000", 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7709999999999997e-06 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768507+0000", 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43361023900000001 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768515+0000", 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8449999999999992e-06 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768576+0000", 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0411000000000001e-05 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768586+0000", 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9499999999999996e-06 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768637+0000", 2024-06-27T21:41:28.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.092e-05 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:208043 6.6 6:60c693c3:::benchmark_data_smithi192_117994_object100449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:29.151456+0000", 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 179.06640177200001, 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35514869399999999, 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.288 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 208043 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151456+0000", 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151456+0000", 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151458+0000", 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1570000000000001e-06 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151462+0000", 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7330000000000003e-06 2024-06-27T21:41:28.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151475+0000", 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.314e-05 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151484+0000", 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4069999999999993e-06 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151536+0000", 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2608000000000002e-05 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151615+0000", 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8022000000000006e-05 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151904+0000", 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028929699999999999 2024-06-27T21:41:28.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.153958+0000", 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020541629999999999 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.154599+0000", 2024-06-27T21:41:28.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064101700000000002 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.506497+0000", 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35189832500000001 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.506554+0000", 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6774000000000002e-05 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.506605+0000", 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1051000000000003e-05 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e705,698 6:601ec4c4:::benchmark_data_smithi192_117994_object69452:head,624'10423)", 2024-06-27T21:41:28.295 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:09.763980+0000", 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 138.45387804200001, 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54553509600000005, 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763980+0000", 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763980+0000", 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763981+0000", 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763987+0000", 2024-06-27T21:41:28.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2149999999999998e-06 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309162+0000", 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.545174661 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309171+0000", 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1020000000000003e-06 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309223+0000", 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1712999999999998e-05 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309515+0000", 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029271000000000001 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.299 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e706,698 6:643d305c:::benchmark_data_smithi192_117994_object73633:head,641'10548)", 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 137.339181373, 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41167358999999998, 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6599999999999996e-07 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878681+0000", 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1990000000000002e-06 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290033+0000", 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41135234700000001 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290043+0000", 2024-06-27T21:41:28.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6709999999999994e-06 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290092+0000", 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9014000000000002e-05 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290350+0000", 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000258793 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.303 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e707,698 6:616e8dab:::benchmark_data_smithi192_117994_object96457:head,675'11245)", 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 134.188363463, 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37847040300000001, 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:41:28.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9200000000000001e-07 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029507+0000", 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1748e-05 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407628+0000", 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37812072299999999 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407644+0000", 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6121999999999999e-05 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407766+0000", 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000122533 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407965+0000", 2024-06-27T21:41:28.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019868500000000001 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "replica_scrub(pg: 7.3,from:0'0,to:744'5258,epoch:746/718,start:7:c4a6ae38:::benchmark_data_smithi192_150129_object82486:0,end:7:c4efb0f7:::benchmark_data_smithi192_150129_object62197:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:40:59.786510+0000", 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.431348056000001, 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15067423999999999, 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:41:28.308 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.786510+0000", 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.786510+0000", 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.786511+0000", 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1900000000000002e-07 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.786515+0000", 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1860000000000002e-06 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.937091+0000", 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.150575766 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.937098+0000", 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6350000000000006e-06 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.937148+0000", 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9418e-05 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.937157+0000", 2024-06-27T21:41:28.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5710000000000004e-06 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:40:59.937184+0000", 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7044999999999999e-05 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:93933 7.3 e747/718 7:c2e2da8e:::benchmark_data_smithi192_150129_object93932:head v 747'5703, mlcod=747'5700)", 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:41:04.930580+0000", 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 23.287278615999998, 2024-06-27T21:41:28.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.120276232, 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:04.930580+0000", 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:41:28.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:04.930580+0000", 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:04.930580+0000", 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9299999999999997e-07 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:04.930602+0000", 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2101000000000001e-05 2024-06-27T21:41:28.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:05.048710+0000", 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.118107156 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:05.048716+0000", 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1840000000000004e-06 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.315 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:05.048782+0000", 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6438000000000003e-05 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:05.048797+0000", 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4796e-05 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:05.050826+0000", 2024-06-27T21:41:28.316 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020288630000000001 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:05.050856+0000", 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0001e-05 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:41:28.317 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:41:28.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:28.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:27 smithi192 ceph-mon[32956]: pgmap v1873: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 127 B/s rd, 12 MiB/s wr, 799 op/s 2024-06-27T21:41:28.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:27 smithi192 ceph-mon[32956]: osdmap e754: 12 total, 12 up, 10 in 2024-06-27T21:41:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:28.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:28.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:28.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:28.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:28.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:28.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:28.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:28.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:28.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:28.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:28.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:29.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:41:29.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:29.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:29.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:41:29.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:29.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:29.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:29.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:29.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:29.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:29.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:29.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:29.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:29.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:29.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:29.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:29.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:30.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:30.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:30.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:29 smithi151 ceph-mon[28965]: pgmap v1875: 47 pgs: 47 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 128 B/s rd, 4.3 MiB/s wr, 704 op/s 2024-06-27T21:41:30.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:30.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:30.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:29 smithi190 ceph-mon[32247]: pgmap v1875: 47 pgs: 47 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 128 B/s rd, 4.3 MiB/s wr, 704 op/s 2024-06-27T21:41:30.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:29 smithi192 ceph-mon[32956]: pgmap v1875: 47 pgs: 47 active+clean; 6.0 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 128 B/s rd, 4.3 MiB/s wr, 704 op/s 2024-06-27T21:41:30.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:30.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:30.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:30.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:30.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:30.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:30.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:30.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:30.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:30.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:30.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:31.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:31.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:31.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:31.154 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:31.154 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:31.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:30 smithi151 ceph-mon[28965]: osdmap e755: 12 total, 12 up, 10 in 2024-06-27T21:41:31.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:30 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:31.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:30 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:31.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:30 smithi190 ceph-mon[32247]: osdmap e755: 12 total, 12 up, 10 in 2024-06-27T21:41:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:30 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:30 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:31.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:30 smithi192 ceph-mon[32956]: osdmap e755: 12 total, 12 up, 10 in 2024-06-27T21:41:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:30 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:30 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:31.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:31.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:31.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:31.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:31.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:31.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:31.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:31.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:31.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:31.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:31.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:31.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:31 smithi151 ceph-mon[28965]: pgmap v1877: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-06-27T21:41:31.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:31 smithi151 ceph-mon[28965]: osdmap e756: 12 total, 12 up, 10 in 2024-06-27T21:41:32.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:32.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:32.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:32.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:31 smithi190 ceph-mon[32247]: pgmap v1877: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-06-27T21:41:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:31 smithi190 ceph-mon[32247]: osdmap e756: 12 total, 12 up, 10 in 2024-06-27T21:41:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:31 smithi192 ceph-mon[32956]: pgmap v1877: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-06-27T21:41:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:31 smithi192 ceph-mon[32956]: osdmap e756: 12 total, 12 up, 10 in 2024-06-27T21:41:32.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:32.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:32.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:32.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:32.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:32.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:32.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:41:32.697 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:41:32.713 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:32.726 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:41:32.726 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:41:32.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:32.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:41:32.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:41:32.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:41:32.787 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:41:32.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:41:32.817 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:41:32.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:32.847 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:41:32.848 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:41:32.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:32.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:32.875 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:41:32.875 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:41:32.900 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:41:32.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:41:32.931 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:41:32.931 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:41:32.933 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:32.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:41:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:41:32.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:41:32.996 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:41:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:33.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:33.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:41:33.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:41:33.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1414033709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:33.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:33.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:33.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:33.183 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1414033709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:33.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:33.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1414033709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:33.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:33.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:33.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:33.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:33.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:33.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:33.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:33.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:33.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:33.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:33.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:33.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:34.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:34.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:34.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:34.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:33 smithi190 ceph-mon[32247]: pgmap v1879: 47 pgs: 47 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 766 op/s 2024-06-27T21:41:34.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:33 smithi151 ceph-mon[28965]: pgmap v1879: 47 pgs: 47 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 766 op/s 2024-06-27T21:41:34.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:34.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:33 smithi192 ceph-mon[32956]: pgmap v1879: 47 pgs: 47 active+clean; 5.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 766 op/s 2024-06-27T21:41:34.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:34.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:34.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:34.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:34.576 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:34.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-06-27T21:41:34.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:34.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:34.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:34.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:34.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:34.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:34.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:35.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:35.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:35.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:35.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:35.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:35.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:35.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:35.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:35.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:35.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:35.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:35.703 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:35.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:35.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:35.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:35.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:41:35.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:35.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:36.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:36.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:36.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:35 smithi151 ceph-mon[28965]: pgmap v1880: 47 pgs: 47 active+clean; 5.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 762 op/s 2024-06-27T21:41:36.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:36.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:35 smithi190 ceph-mon[32247]: pgmap v1880: 47 pgs: 47 active+clean; 5.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 762 op/s 2024-06-27T21:41:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:35 smithi192 ceph-mon[32956]: pgmap v1880: 47 pgs: 47 active+clean; 5.7 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 762 op/s 2024-06-27T21:41:36.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:36.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:36.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:36.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:36.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:36.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:36.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:36.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:36.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:36.825 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:36.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:36.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:36.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:36.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:37.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:37.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:37.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:37.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:37.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:37.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:37.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:37.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:37.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:37.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:37.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:37.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:37.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:37.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:37.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:37 smithi151 ceph-mon[28965]: pgmap v1881: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-27T21:41:37.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:37.984 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:37.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:38.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:38.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:38.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:38.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:38.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:37 smithi190 ceph-mon[32247]: pgmap v1881: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-27T21:41:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:37 smithi192 ceph-mon[32956]: pgmap v1881: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-06-27T21:41:38.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:38.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:38.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:38.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:38.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:38.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:38.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:38.780 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:41:38.780 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:41:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:38.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:38.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:38.909 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:41:38.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:38.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:39.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:39.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:39.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:39.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:39.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2858763777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:41:39.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:38 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:41:39.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2858763777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:41:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:38 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:41:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2858763777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:41:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:38 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:41:39.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:39.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:39.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:39.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:39.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:39.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:39.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:39.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:39.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:39.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:39.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:39.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:39.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:39.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:39.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:40.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:40.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:40.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:39 smithi151 ceph-mon[28965]: pgmap v1882: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 958 op/s 2024-06-27T21:41:40.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:39 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:40.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:39 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:41:40.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:39 smithi151 ceph-mon[28965]: osdmap e757: 12 total, 12 up, 10 in 2024-06-27T21:41:40.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:39 smithi151 ceph-mon[28965]: 7.2 scrub starts 2024-06-27T21:41:40.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:40.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:39 smithi190 ceph-mon[32247]: pgmap v1882: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 958 op/s 2024-06-27T21:41:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:39 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:39 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:41:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:39 smithi190 ceph-mon[32247]: osdmap e757: 12 total, 12 up, 10 in 2024-06-27T21:41:40.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:39 smithi190 ceph-mon[32247]: 7.2 scrub starts 2024-06-27T21:41:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:39 smithi192 ceph-mon[32956]: pgmap v1882: 47 pgs: 47 active+clean; 5.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 958 op/s 2024-06-27T21:41:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:39 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:41:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:39 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:41:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:39 smithi192 ceph-mon[32956]: osdmap e757: 12 total, 12 up, 10 in 2024-06-27T21:41:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:39 smithi192 ceph-mon[32956]: 7.2 scrub starts 2024-06-27T21:41:40.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:40.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:40.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:40.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:40.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:40.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:40.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:40.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:40.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:40.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:40.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:40.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:41.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:41.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:41.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:40 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:41:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:40 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:41:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:40 smithi151 ceph-mon[28965]: 7.2 scrub ok 2024-06-27T21:41:41.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.266 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:41.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:40 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:41:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:40 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:41:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:40 smithi190 ceph-mon[32247]: 7.2 scrub ok 2024-06-27T21:41:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:40 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:41:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:40 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:41:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:40 smithi192 ceph-mon[32956]: 7.2 scrub ok 2024-06-27T21:41:41.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:41.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:41.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:41.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:41.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:41.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:41.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:41.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:41.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:41.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:41:41.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:41.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:41.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:41:42.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:42.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:42.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:42.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:41 smithi151 ceph-mon[28965]: pgmap v1884: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 763 op/s 2024-06-27T21:41:42.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:42.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:42.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:41 smithi190 ceph-mon[32247]: pgmap v1884: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 763 op/s 2024-06-27T21:41:42.284 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:41 smithi192 ceph-mon[32956]: pgmap v1884: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 763 op/s 2024-06-27T21:41:42.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:42.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:42.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:42.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:42.659 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:42.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:42.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:42.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:42.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:42.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:42.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:42.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:43.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:43.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:43.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:43.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:43.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:42 smithi151 ceph-mon[28965]: 7.b scrub starts 2024-06-27T21:41:43.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:42 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:41:43.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:42 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:41:43.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:42 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:41:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:42 smithi190 ceph-mon[32247]: 7.b scrub starts 2024-06-27T21:41:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:42 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:41:43.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:42 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:41:43.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:42 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:41:43.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:43.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:42 smithi192 ceph-mon[32956]: 7.b scrub starts 2024-06-27T21:41:43.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:42 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:41:43.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:42 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:41:43.288 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:42 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:41:43.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:43.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:43.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:43.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:43.548 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:43.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:43.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:43.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:43.803 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:43.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:43.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-06-27T21:41:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:43.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:43.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:44.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:44.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:44.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:44.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:43 smithi151 ceph-mon[28965]: pgmap v1885: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 906 op/s 2024-06-27T21:41:44.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:43 smithi151 ceph-mon[28965]: 7.7 scrub starts 2024-06-27T21:41:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.219 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:44.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:43 smithi190 ceph-mon[32247]: pgmap v1885: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 906 op/s 2024-06-27T21:41:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:43 smithi190 ceph-mon[32247]: 7.7 scrub starts 2024-06-27T21:41:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:43 smithi192 ceph-mon[32956]: pgmap v1885: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 906 op/s 2024-06-27T21:41:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:43 smithi192 ceph-mon[32956]: 7.7 scrub starts 2024-06-27T21:41:44.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:44.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:44.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:44.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:44.534 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:41:44.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:41:44.564 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:41:44.564 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:41:44.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:44.589 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:44.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:44.598 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:41:44.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:41:44.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:44.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:41:44.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:41:44.643 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:44.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:41:44.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:41:44.704 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:41:44.705 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:41:44.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:44.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:44.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:41:44.734 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:41:44.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:41:44.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:41:44.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:41:44.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:41:44.815 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:41:44.815 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:41:44.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:44.846 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:41:44.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:41:44.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:44.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:41:44.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:41:44.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:44.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:44.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:45.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:45.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:45.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:45.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:45.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:45.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: 7.b scrub ok 2024-06-27T21:41:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: 7.7 scrub ok 2024-06-27T21:41:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:41:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1578494520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:41:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:41:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:45 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: 7.b scrub ok 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: 7.7 scrub ok 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1578494520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:41:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:41:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:41:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:41:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:41:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:41:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:45 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: 7.b scrub ok 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: 7.7 scrub ok 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1578494520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:41:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:45 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:41:45.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:45.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:45.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:45.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:45.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:45.906 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:45.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:46.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:46.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:46.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:46.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:46.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:46.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:46.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:46 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:41:46.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:46.432 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:46 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:41:46.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:46.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:46 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:41:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:46.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:46.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:46.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:46.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:46.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:46.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:46.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:46.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:47.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:47.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:47.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:47.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:47.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:47 smithi151 ceph-mon[28965]: pgmap v1887: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 864 op/s 2024-06-27T21:41:47.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:47.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:47.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:47.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:47.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:47 smithi190 ceph-mon[32247]: pgmap v1887: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 864 op/s 2024-06-27T21:41:47.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:47 smithi192 ceph-mon[32956]: pgmap v1887: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 864 op/s 2024-06-27T21:41:47.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:47.598 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:47.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:47.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:47.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:47.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:47.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:47.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:47.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:48.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:48.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:48.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:48.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:48.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:48.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:48.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:48.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:48.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:48.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:48.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:48.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:48.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:48.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:48.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:48.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:48.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:49.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:49.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:49.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:49.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:49.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:49.301 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:41:49.369 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:49.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:49.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:49.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:49 smithi192 ceph-mon[32956]: pgmap v1888: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 866 op/s 2024-06-27T21:41:49.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2637660617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:49.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:49.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:49.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:49 smithi151 ceph-mon[28965]: pgmap v1888: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 866 op/s 2024-06-27T21:41:49.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2637660617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:49.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:49 smithi190 ceph-mon[32247]: pgmap v1888: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 866 op/s 2024-06-27T21:41:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2637660617' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:41:49.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:49.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:49.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:49.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:50.010 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:50.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:50.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:50.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:50.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:50.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:50.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:50.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:50.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:41:50.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:41:50.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:41:50.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:41:50.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2637660617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:50.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: osdmap e758: 12 total, 12 up, 10 in 2024-06-27T21:41:50.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:50 smithi192 ceph-mon[32956]: 7.4 scrub starts 2024-06-27T21:41:50.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:50.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2637660617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: osdmap e758: 12 total, 12 up, 10 in 2024-06-27T21:41:50.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:50 smithi151 ceph-mon[28965]: 7.4 scrub starts 2024-06-27T21:41:50.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2637660617' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: osdmap e758: 12 total, 12 up, 10 in 2024-06-27T21:41:50.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:50 smithi190 ceph-mon[32247]: 7.4 scrub starts 2024-06-27T21:41:50.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:50.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:50.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:50.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:50.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:50.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:51.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:51.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:51.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:51.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:41:51.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:51.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:51.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:51.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:51 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:41:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:51 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:41:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:51 smithi151 ceph-mon[28965]: pgmap v1890: 47 pgs: 47 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 858 op/s 2024-06-27T21:41:51.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:51.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:51.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:51.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:51.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:51.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:51 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:41:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:51 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:41:51.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:51 smithi190 ceph-mon[32247]: pgmap v1890: 47 pgs: 47 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 858 op/s 2024-06-27T21:41:51.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:51 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:41:51.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:51 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:41:51.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:51 smithi192 ceph-mon[32956]: pgmap v1890: 47 pgs: 47 active+clean; 4.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 858 op/s 2024-06-27T21:41:51.822 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:41:51.822 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:41:51.822 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:41:51.822 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:41:51.822 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:41:51.837 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:51.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:51.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:51.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:51.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:51.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:52.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:52.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:52.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:52.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:52.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:52.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:52.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:52.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:52.607 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:41:52.607 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:41:52.607 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:41:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:41:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 7.4 scrub ok 2024-06-27T21:41:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:41:52.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:52 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:41:52.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:52.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:52.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:41:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:41:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:41:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:41:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 7.4 scrub ok 2024-06-27T21:41:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:41:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:52 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:41:52.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:52.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:41:52.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:41:52.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:41:52.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:41:52.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 7.4 scrub ok 2024-06-27T21:41:52.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:41:52.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:52 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:41:52.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:52.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:52.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:53.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:53.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:53.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:53.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:53.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:53.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:53.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:53.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:53.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:53.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:53.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:53.564 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:53 smithi151 ceph-mon[28965]: pgmap v1891: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 862 op/s 2024-06-27T21:41:53.564 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:53 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:41:53.564 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:53 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:41:53.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:41:53.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:53.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:53.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:53.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:53 smithi190 ceph-mon[32247]: pgmap v1891: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 862 op/s 2024-06-27T21:41:53.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:53 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:41:53.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:53 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:41:53.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:53.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:53 smithi192 ceph-mon[32956]: pgmap v1891: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 862 op/s 2024-06-27T21:41:53.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:53 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:41:53.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:53 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:41:53.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:53.929 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:53.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:54.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:54.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:54.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:54.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:54.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:54.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:54.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:54.440 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:54.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:41:54.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:54.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:54.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:54.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:54.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:41:54.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:41:54.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:54.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:54.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:55.019 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:41:55.057 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:55.089 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:55.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:55.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:55.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:55.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:55.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:55.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:55.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:55 smithi151 ceph-mon[28965]: pgmap v1892: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:41:55.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:55 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:41:55.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:55 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:41:55.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:55 smithi151 ceph-mon[28965]: 7.9 deep-scrub starts 2024-06-27T21:41:55.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:55.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:55.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:55.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:55.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:55 smithi190 ceph-mon[32247]: pgmap v1892: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:41:55.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:55 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:41:55.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:55 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:41:55.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:55 smithi190 ceph-mon[32247]: 7.9 deep-scrub starts 2024-06-27T21:41:55.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:55 smithi192 ceph-mon[32956]: pgmap v1892: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.6 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:41:55.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:55 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:41:55.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:55 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:41:55.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:55 smithi192 ceph-mon[32956]: 7.9 deep-scrub starts 2024-06-27T21:41:55.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:55.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:55.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:55.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:55.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:56.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:56.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:56.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:56.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:41:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:41:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:41:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:41:56.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:41:56.248 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:41:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:41:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:41:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:41:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:41:56.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:41:56.250 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:41:56.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:41:56.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:41:56.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:41:56.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:41:56.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:41:56.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:56.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:56.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:56.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:56.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:56.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:56.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:56.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:56 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:41:56.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:56 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:41:56.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/28233862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:56.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:56 smithi151 ceph-mon[28965]: osdmap e759: 12 total, 12 up, 10 in 2024-06-27T21:41:56.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:56 smithi151 ceph-mon[28965]: 7.6 scrub starts 2024-06-27T21:41:56.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:56.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:56.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:56 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:41:56.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:56 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:41:56.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/28233862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:56.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:56 smithi190 ceph-mon[32247]: osdmap e759: 12 total, 12 up, 10 in 2024-06-27T21:41:56.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:56 smithi190 ceph-mon[32247]: 7.6 scrub starts 2024-06-27T21:41:56.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:56 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:41:56.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:56 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:41:56.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/28233862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:41:56.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:56 smithi192 ceph-mon[32956]: osdmap e759: 12 total, 12 up, 10 in 2024-06-27T21:41:56.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:56 smithi192 ceph-mon[32956]: 7.6 scrub starts 2024-06-27T21:41:56.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:56.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:41:56.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:56.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:56.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:57.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:57.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:57.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:57.248 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:57.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:41:57.372 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:41:57.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:57.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:57.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:57.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:57.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:57.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:57.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:57.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:41:57.881 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:57.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:41:57.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:41:58.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:41:58.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:58.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:57 smithi151 ceph-mon[28965]: pgmap v1894: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:41:58.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:57 smithi151 ceph-mon[28965]: 1.b deep-scrub starts 2024-06-27T21:41:58.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:57 smithi151 ceph-mon[28965]: 1.b deep-scrub ok 2024-06-27T21:41:58.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:58.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:57 smithi190 ceph-mon[32247]: pgmap v1894: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:41:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:57 smithi190 ceph-mon[32247]: 1.b deep-scrub starts 2024-06-27T21:41:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:57 smithi190 ceph-mon[32247]: 1.b deep-scrub ok 2024-06-27T21:41:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:57 smithi192 ceph-mon[32956]: pgmap v1894: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:41:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:57 smithi192 ceph-mon[32956]: 1.b deep-scrub starts 2024-06-27T21:41:58.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:57 smithi192 ceph-mon[32956]: 1.b deep-scrub ok 2024-06-27T21:41:58.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:41:58.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:41:58.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:41:58.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:41:58.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:41:58.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:41:58.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:58.761 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:58.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:58.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:58.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:58.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:59.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:59.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:41:59.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:41:59.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:58 smithi151 ceph-mon[28965]: 7.6 scrub ok 2024-06-27T21:41:59.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:41:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:58 smithi190 ceph-mon[32247]: 7.6 scrub ok 2024-06-27T21:41:59.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:41:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:58 smithi192 ceph-mon[32956]: 7.6 scrub ok 2024-06-27T21:41:59.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:59.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:59.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:59.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:59.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:41:59.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:41:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:41:59.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:41:59.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:41:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:41:59.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:41:59.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:41:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:41:59.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:00.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:00.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:00.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:00.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:00.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:00.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:41:59 smithi192 ceph-mon[32956]: pgmap v1895: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 898 op/s 2024-06-27T21:42:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:41:59 smithi151 ceph-mon[28965]: pgmap v1895: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 898 op/s 2024-06-27T21:42:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:41:59 smithi190 ceph-mon[32247]: pgmap v1895: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 898 op/s 2024-06-27T21:42:00.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:00.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:00.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:00.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:00.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:00.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:00.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:00.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:00.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-06-27T21:42:00.712 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:00.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:00.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:00.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:00.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:42:00.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:42:00.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:42:00.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:42:00.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:42:00.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:42:00.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:00 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:42:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:00.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:00.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:01.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:01.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:01.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:01.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:01.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:01.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:42:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:42:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:42:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:42:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:42:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:42:01.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:00 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:42:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:42:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:42:01.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:42:01.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:42:01.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:42:01.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:42:01.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:00 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:42:01.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:01.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:01.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:01.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:01.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:01.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:01.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:01.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:01.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:01.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:01.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:01.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:01.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:02.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:02.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:02.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:01 smithi151 ceph-mon[28965]: pgmap v1896: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 805 op/s 2024-06-27T21:42:02.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:02.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:01 smithi190 ceph-mon[32247]: pgmap v1896: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 805 op/s 2024-06-27T21:42:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:01 smithi192 ceph-mon[32956]: pgmap v1896: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.3 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 805 op/s 2024-06-27T21:42:02.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:02.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:02.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:02.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:02.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:02.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:02.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:02.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:02.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:02.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:02.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:02.966 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:03.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:03.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:03.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:03.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:03.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:42:03.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:03.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:03.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:03.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:03.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:03.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:03.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:03.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:03.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:03.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:03.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:03.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:03.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:03.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:04.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:04.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:04.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:04.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:03 smithi151 ceph-mon[28965]: pgmap v1897: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 746 op/s 2024-06-27T21:42:04.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:04.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:04.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:04.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:03 smithi190 ceph-mon[32247]: pgmap v1897: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 746 op/s 2024-06-27T21:42:04.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:03 smithi192 ceph-mon[32956]: pgmap v1897: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.2 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 746 op/s 2024-06-27T21:42:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:04.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:04.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:04.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:04.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:04.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:04.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:04.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:04.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:04.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:04.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:04.868 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:04.894 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:42:04.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:04.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:04.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:05.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:05.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:05.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:05.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:04 smithi151 ceph-mon[28965]: osdmap e760: 12 total, 12 up, 10 in 2024-06-27T21:42:05.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3592268933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:05.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:05.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:04 smithi190 ceph-mon[32247]: osdmap e760: 12 total, 12 up, 10 in 2024-06-27T21:42:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3592268933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:04 smithi192 ceph-mon[32956]: osdmap e760: 12 total, 12 up, 10 in 2024-06-27T21:42:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3592268933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:05.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:05.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:05.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:05.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:05.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:05.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:05.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:05.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:05.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:05.892 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:05.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:05.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:06.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:06.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:05 smithi151 ceph-mon[28965]: pgmap v1899: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 484 op/s 2024-06-27T21:42:06.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:05 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:06.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3592268933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:42:06.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:05 smithi151 ceph-mon[28965]: osdmap e761: 12 total, 12 up, 10 in 2024-06-27T21:42:06.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:05 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:42:06.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:06.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:06.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:05 smithi190 ceph-mon[32247]: pgmap v1899: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 484 op/s 2024-06-27T21:42:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:05 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3592268933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:42:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:05 smithi190 ceph-mon[32247]: osdmap e761: 12 total, 12 up, 10 in 2024-06-27T21:42:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:05 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:42:06.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:05 smithi192 ceph-mon[32956]: pgmap v1899: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 484 op/s 2024-06-27T21:42:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:05 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3592268933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:42:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:05 smithi192 ceph-mon[32956]: osdmap e761: 12 total, 12 up, 10 in 2024-06-27T21:42:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:05 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:42:06.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:06.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:06.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:06.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:06.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:06.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:06.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:06.786 INFO:teuthology.orchestra.run.smithi151.stderr:2024-06-27T21:42:06.056+0000 7f8bf67fc640 0 --2- 172.21.15.151:0/2583363621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bf8102bb0 0x7f8bf8102ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-27T21:42:06.788 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:06.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:06.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:06.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:07.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:07.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:07.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:07.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:07.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.293 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:07.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:07.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:07.424 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:07.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:07.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:07.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-06-27T21:42:07.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:07.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:07.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:07.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:07.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:07.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:07.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:07.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:08.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:08.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:08.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:08.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:07 smithi151 ceph-mon[28965]: pgmap v1901: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.0 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 740 op/s 2024-06-27T21:42:08.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:07 smithi190 ceph-mon[32247]: pgmap v1901: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.0 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 740 op/s 2024-06-27T21:42:08.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:42:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:07 smithi192 ceph-mon[32956]: pgmap v1901: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.0 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 740 op/s 2024-06-27T21:42:08.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:08.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:08.385 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:42:08.385 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:42:08.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:08.487 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:08.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:08.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:08.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:08.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:08.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:08.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:08.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:08.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:09.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:09.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:09.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:09.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:09.276 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:09.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:09.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:42:09.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:09.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:09.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:09.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:09.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:09.713 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:09.734 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:42:09.734 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:42:09.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:09.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:42:09.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:42:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:09.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:09.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:42:09.809 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:42:09.837 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:42:09.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:42:09.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:42:09.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:42:09.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:09.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:42:09.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:42:09.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:42:09.929 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:42:09.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:09.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:09.956 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:42:09.956 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:42:09.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:42:09.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:42:10.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:42:10.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:42:10.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:10.041 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:42:10.041 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:42:10.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:10.067 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:42:10.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:42:10.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:10.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:09 smithi151 ceph-mon[28965]: pgmap v1902: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 610 op/s 2024-06-27T21:42:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3769593068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:09 smithi190 ceph-mon[32247]: pgmap v1902: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 610 op/s 2024-06-27T21:42:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3769593068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:10.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:09 smithi192 ceph-mon[32956]: pgmap v1902: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.0 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 610 op/s 2024-06-27T21:42:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3769593068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:10.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:10.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:10.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:10.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:10.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:10.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:10.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:10.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.812 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:10.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:10.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:10.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:10.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:11.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:11.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:11.137 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:11.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:11.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:11.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:11.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:11.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:11.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:11.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:11.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:11.612 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:11.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:11.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:11.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:11.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:11.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:11.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:11 smithi151 ceph-mon[28965]: pgmap v1903: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 614 op/s 2024-06-27T21:42:11.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:11.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:12.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:11 smithi192 ceph-mon[32956]: pgmap v1903: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 614 op/s 2024-06-27T21:42:12.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:12.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:12.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:12.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:11 smithi190 ceph-mon[32247]: pgmap v1903: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 614 op/s 2024-06-27T21:42:12.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:12.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:12.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:12.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:12.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:12.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:12.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:12.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:12.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:12.850 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:12.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:12.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:12.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:13.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:13.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:13.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:42:13.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:13.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:13.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:13.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:13.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:13.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:13.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:13.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:13.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:13.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:13.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:13.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:13.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:13.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:14.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:14.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:14.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:14.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:13 smithi151 ceph-mon[28965]: pgmap v1904: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 659 op/s 2024-06-27T21:42:14.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:13 smithi190 ceph-mon[32247]: pgmap v1904: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 659 op/s 2024-06-27T21:42:14.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:13 smithi192 ceph-mon[32956]: pgmap v1904: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 659 op/s 2024-06-27T21:42:14.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:14.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:14.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:14.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:14.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:14.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:14.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:14.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:14.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:14.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:14.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:14.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:15.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:15.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:15.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:15.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:15.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:15.290 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:15.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:15.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:15.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:15.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:15.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:15.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:15.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:15.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:15.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:15.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:15.923 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:42:15.945 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:15.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:16.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:16.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:16.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:16.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:15 smithi151 ceph-mon[28965]: pgmap v1905: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 626 op/s 2024-06-27T21:42:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3650325233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:42:16.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:16.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:15 smithi190 ceph-mon[32247]: pgmap v1905: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 626 op/s 2024-06-27T21:42:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3650325233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:42:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:15 smithi192 ceph-mon[32956]: pgmap v1905: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 626 op/s 2024-06-27T21:42:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3650325233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:42:16.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:16.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:16.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:16.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:16.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:16.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:16.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:16.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:16.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:16.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:16.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:16.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:16 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:16.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3650325233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:42:16.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:16 smithi151 ceph-mon[28965]: osdmap e762: 12 total, 12 up, 10 in 2024-06-27T21:42:16.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:16.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:17.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:17.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:17.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:17.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:16 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3650325233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:42:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:16 smithi190 ceph-mon[32247]: osdmap e762: 12 total, 12 up, 10 in 2024-06-27T21:42:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:16 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3650325233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:42:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:16 smithi192 ceph-mon[32956]: osdmap e762: 12 total, 12 up, 10 in 2024-06-27T21:42:17.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:17.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:17.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:17.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:17.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:17.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:17.633 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:42:17.633 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:42:17.634 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:42:17.634 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:42:17.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:17.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:17.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:17.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:17.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:17.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:17.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:18.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:18.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:18.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:17 smithi151 ceph-mon[28965]: pgmap v1907: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 637 op/s 2024-06-27T21:42:18.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:18.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:17 smithi190 ceph-mon[32247]: pgmap v1907: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 637 op/s 2024-06-27T21:42:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:17 smithi192 ceph-mon[32956]: pgmap v1907: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 637 op/s 2024-06-27T21:42:18.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:18.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:18.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:18.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:18.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:18.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:18.711 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:18.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:18.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:42:18.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:18.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:18.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:18.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-06-27T21:42:18.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:19.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:19.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:19.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:19.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:19.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:19.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:19.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:19.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:19.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:19.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:19.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:19.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:19.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:19.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:19.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:19.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:20.043 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:20.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:20.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:20.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:19 smithi151 ceph-mon[28965]: pgmap v1908: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 640 op/s 2024-06-27T21:42:20.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1958630931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:20.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:20.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:42:20.257 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:19 smithi190 ceph-mon[32247]: pgmap v1908: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 640 op/s 2024-06-27T21:42:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1958630931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:19 smithi192 ceph-mon[32956]: pgmap v1908: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.6 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 640 op/s 2024-06-27T21:42:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1958630931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:20.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:20.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:20.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:20.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:20.476 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:20.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:42:20.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:42:20.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:42:20.549 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:42:20.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:20.593 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:42:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:42:20.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:20.628 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:42:20.629 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:42:20.660 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:42:20.661 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:42:20.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:42:20.692 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:42:20.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:20.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:42:20.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:42:20.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:20.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:42:20.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:42:20.769 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:42:20.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:42:20.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:42:20.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:42:20.822 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:42:20.822 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:42:20.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:20.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:20.852 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:42:20.852 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:42:20.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:20.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:20.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:21.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:21.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:21.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:20 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:21.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:21.174 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:20 smithi151 ceph-mon[28965]: osdmap e763: 12 total, 12 up, 10 in 2024-06-27T21:42:21.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:21.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:20 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:21.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:21.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:20 smithi190 ceph-mon[32247]: osdmap e763: 12 total, 12 up, 10 in 2024-06-27T21:42:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:20 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:20 smithi192 ceph-mon[32956]: osdmap e763: 12 total, 12 up, 10 in 2024-06-27T21:42:21.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:21.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:21.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:21.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:21.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:21.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:21.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:21.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:21.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:21.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:21.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:21.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:22.012 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:22.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:22.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:21 smithi151 ceph-mon[28965]: pgmap v1910: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 642 op/s 2024-06-27T21:42:22.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:21 smithi190 ceph-mon[32247]: pgmap v1910: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 642 op/s 2024-06-27T21:42:22.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:21 smithi192 ceph-mon[32956]: pgmap v1910: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 3.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 642 op/s 2024-06-27T21:42:22.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:22.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:22.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:22.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:22.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:22.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:22.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:22.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:22.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:22.900 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:22.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:23.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:23.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:23.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:23.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:23.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:23.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:23.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:23.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:23.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:23.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:23.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:23.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:23.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:23.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:23.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:42:23.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:23.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:24.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:23 smithi151 ceph-mon[28965]: pgmap v1911: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:42:24.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:24.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:24.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:24.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:23 smithi190 ceph-mon[32247]: pgmap v1911: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:42:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:23 smithi192 ceph-mon[32956]: pgmap v1911: 47 pgs: 47 active+clean; 3.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:42:24.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:24.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:24.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:24.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:24.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:24.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:24.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:24.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:24.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:24.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:24.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:24.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:24.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:25.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:25.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:25.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-06-27T21:42:25.170 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:25.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:25.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:25.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:25.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:25.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:25.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:25.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:25.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:25.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:25.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:25.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:25.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:25.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:25.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:26.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:26.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:26.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:26.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:26.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:25 smithi151 ceph-mon[28965]: pgmap v1912: 47 pgs: 47 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 678 op/s 2024-06-27T21:42:26.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:25 smithi190 ceph-mon[32247]: pgmap v1912: 47 pgs: 47 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 678 op/s 2024-06-27T21:42:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:25 smithi192 ceph-mon[32956]: pgmap v1912: 47 pgs: 47 active+clean; 3.3 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 678 op/s 2024-06-27T21:42:26.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:26.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:26.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:26.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:26.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:26.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:26.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:26.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:26.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:26.876 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:26.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:26.965 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:42:26.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:26.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:27.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:27.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:27.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1653713764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:42:27.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:26 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:42:27.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:27.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1653713764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:42:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:26 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:42:27.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1653713764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:42:27.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:26 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:42:27.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:27.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:27.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:27.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:27.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:27.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:27.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:27.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:27.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:27.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:27.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:27.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:27.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:28.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:28.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:28.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: pgmap v1913: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 770 op/s 2024-06-27T21:42:28.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:28.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:28.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:28.167 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:28.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:28.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:42:28.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: osdmap e764: 12 total, 12 up, 10 in 2024-06-27T21:42:28.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:28.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:28.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:28.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: pgmap v1913: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 770 op/s 2024-06-27T21:42:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: osdmap e764: 12 total, 12 up, 10 in 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: pgmap v1913: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 770 op/s 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: osdmap e764: 12 total, 12 up, 10 in 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:28.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:28.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:28.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:28.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:28.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:28.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:28.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:28.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:28.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:28.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:28.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:28.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:29.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:29.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:29.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:29.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:28 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:42:29.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:28 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:42:29.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:28 smithi151 ceph-mon[28965]: osdmap e765: 12 total, 12 up, 10 in 2024-06-27T21:42:29.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:28 smithi151 ceph-mon[28965]: 7.e scrub starts 2024-06-27T21:42:29.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:29.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:28 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:42:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:28 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:42:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:28 smithi190 ceph-mon[32247]: osdmap e765: 12 total, 12 up, 10 in 2024-06-27T21:42:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:28 smithi190 ceph-mon[32247]: 7.e scrub starts 2024-06-27T21:42:29.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:28 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:42:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:28 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:42:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:28 smithi192 ceph-mon[32956]: osdmap e765: 12 total, 12 up, 10 in 2024-06-27T21:42:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:28 smithi192 ceph-mon[32956]: 7.e scrub starts 2024-06-27T21:42:29.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:29.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:29.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:29.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:29.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:29.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:29.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:29.792 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:29.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:29.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:29.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:30.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:30.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:30.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:30.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: pgmap v1916: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 774 op/s 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: 7.a scrub starts 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: 7.e scrub ok 2024-06-27T21:42:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:29 smithi190 ceph-mon[32247]: osdmap e766: 12 total, 12 up, 10 in 2024-06-27T21:42:30.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: pgmap v1916: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 774 op/s 2024-06-27T21:42:30.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: 7.a scrub starts 2024-06-27T21:42:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:42:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:42:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: 7.e scrub ok 2024-06-27T21:42:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:29 smithi192 ceph-mon[32956]: osdmap e766: 12 total, 12 up, 10 in 2024-06-27T21:42:30.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:30.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:30.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: pgmap v1916: 47 pgs: 47 active+clean; 3.1 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 774 op/s 2024-06-27T21:42:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: 7.a scrub starts 2024-06-27T21:42:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:42:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:42:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:30.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: 7.e scrub ok 2024-06-27T21:42:30.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:29 smithi151 ceph-mon[28965]: osdmap e766: 12 total, 12 up, 10 in 2024-06-27T21:42:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:30.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:30.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:30.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:30.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:30.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:30.854 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:30.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:30.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:30.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:31.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:31.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:30 smithi151 ceph-mon[28965]: 7.a scrub ok 2024-06-27T21:42:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:30 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:42:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:30 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:42:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:30 smithi151 ceph-mon[28965]: pgmap v1918: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:42:31.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:31.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:30 smithi190 ceph-mon[32247]: 7.a scrub ok 2024-06-27T21:42:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:30 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:42:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:30 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:42:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:30 smithi190 ceph-mon[32247]: pgmap v1918: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:42:31.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:30 smithi192 ceph-mon[32956]: 7.a scrub ok 2024-06-27T21:42:31.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:30 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:42:31.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:30 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:42:31.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:30 smithi192 ceph-mon[32956]: pgmap v1918: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:42:31.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:31.379 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:31.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:31.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:31.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:31.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:31.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:31.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:31.761 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:42:31.761 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:42:31.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:31.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:31.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:31.901 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:31.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:32.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:32.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 7.9 scrub starts 2024-06-27T21:42:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 7.0 scrub starts 2024-06-27T21:42:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 7.3 scrub starts 2024-06-27T21:42:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:42:32.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:42:32.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 7.9 scrub ok 2024-06-27T21:42:32.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: 7.0 scrub ok 2024-06-27T21:42:32.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:32.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:32.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:31 smithi190 ceph-mon[32247]: 7.9 scrub starts 2024-06-27T21:42:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:31 smithi190 ceph-mon[32247]: 7.0 scrub starts 2024-06-27T21:42:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: 7.3 scrub starts 2024-06-27T21:42:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:42:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:42:32.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: 7.9 scrub ok 2024-06-27T21:42:32.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: 7.0 scrub ok 2024-06-27T21:42:32.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:32.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:32.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:31 smithi192 ceph-mon[32956]: 7.9 scrub starts 2024-06-27T21:42:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:31 smithi192 ceph-mon[32956]: 7.0 scrub starts 2024-06-27T21:42:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: 7.3 scrub starts 2024-06-27T21:42:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:42:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:42:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: 7.9 scrub ok 2024-06-27T21:42:32.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: 7.0 scrub ok 2024-06-27T21:42:32.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:32.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:32.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.400 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:32.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:32.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:32.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:32.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:42:32.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:32.675 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:32.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:32.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:32.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:32.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:32.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:33.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:33.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:33.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:33.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:42:33.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: osdmap e767: 12 total, 12 up, 10 in 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 7.3 scrub ok 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 7.8 scrub starts 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: pgmap v1920: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 730 op/s 2024-06-27T21:42:33.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3452000209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:33.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:42:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: osdmap e767: 12 total, 12 up, 10 in 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 7.3 scrub ok 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 7.8 scrub starts 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:42:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:42:33.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: pgmap v1920: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 730 op/s 2024-06-27T21:42:33.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3452000209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:33.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:33.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:42:33.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:33.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:33.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:42:33.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:42:33.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:42:33.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:42:33.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:42:33.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:42:33.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:42:33.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:42:33.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: osdmap e767: 12 total, 12 up, 10 in 2024-06-27T21:42:33.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 7.3 scrub ok 2024-06-27T21:42:33.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 7.8 scrub starts 2024-06-27T21:42:33.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:42:33.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:42:33.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: pgmap v1920: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 2.9 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 730 op/s 2024-06-27T21:42:33.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3452000209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:33.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:42:33.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:42:33.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:33.563 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:42:33.563 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:42:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:33.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:33.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:42:33.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:42:33.634 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:42:33.634 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:42:33.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:42:33.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:42:33.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:33.694 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:42:33.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:42:33.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:42:33.718 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:42:33.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:42:33.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:42:33.772 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:42:33.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:42:33.808 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:42:33.809 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:42:33.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:33.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:33.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:42:33.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:42:33.875 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:42:33.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:42:33.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:33.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:33.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:34.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:34.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:34.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:34.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:34 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:42:34.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:34 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:42:34.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:34 smithi151 ceph-mon[28965]: 7.8 scrub ok 2024-06-27T21:42:34.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:34 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:42:34.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:34 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:42:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:34.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:34 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:42:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:34 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:42:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:34 smithi192 ceph-mon[32956]: 7.8 scrub ok 2024-06-27T21:42:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:34 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:42:34.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:34 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:42:34.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:34.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:34.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:34 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:42:34.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:34 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:42:34.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:34 smithi190 ceph-mon[32247]: 7.8 scrub ok 2024-06-27T21:42:34.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:34 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:42:34.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:34 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:42:34.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:34.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:34.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:34.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:34.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:34.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:34.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:34.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:34.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:34.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:34.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:35.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:35.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:35.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:35.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:35 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:42:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:35 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:42:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:35 smithi192 ceph-mon[32956]: pgmap v1921: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 733 op/s 2024-06-27T21:42:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:35 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:42:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:35 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:42:35.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:35.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:35.392 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:35 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:42:35.392 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:35 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:42:35.392 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:35 smithi151 ceph-mon[28965]: pgmap v1921: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 733 op/s 2024-06-27T21:42:35.392 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:35 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:42:35.393 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:35 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:42:35.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:35.514 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:35.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:35 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:42:35.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:35 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:42:35.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:35 smithi190 ceph-mon[32247]: pgmap v1921: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 733 op/s 2024-06-27T21:42:35.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:35 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:42:35.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:35 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:42:35.615 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:35.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:35.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:35.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:35.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:35.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:35.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:35.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:36.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:36.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:36.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:36.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:36.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:36.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:36.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:36.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:36.519 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:36.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:36.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:36.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:36.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:36.780 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:36.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:36.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:36.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:36.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:37.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:37.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:37.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:37.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:37.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:37.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:37.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:37.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.516 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:37.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:37.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:37.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:37 smithi151 ceph-mon[28965]: pgmap v1922: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 929 op/s 2024-06-27T21:42:37.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:37 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:42:37.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:37 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:42:37.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:37.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:37.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:37.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:37 smithi190 ceph-mon[32247]: pgmap v1922: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 929 op/s 2024-06-27T21:42:37.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:37 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:42:37.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:37 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:42:37.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:42:37.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:37 smithi192 ceph-mon[32956]: pgmap v1922: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 929 op/s 2024-06-27T21:42:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:37 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:42:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:37 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:42:37.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:37.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:37.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:37.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:38.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:38.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:38.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:38.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:38.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:38.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:38.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:38.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:38.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:38.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:38.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:38.649 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:38.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:38.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:38.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:38.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:38.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:39.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:39.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:39.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:39.152 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:39.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:39.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:39.382 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:39.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:39.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:39.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:39.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:39.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:39 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:42:39.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:39 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:42:39.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:39 smithi151 ceph-mon[28965]: pgmap v1923: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 790 op/s 2024-06-27T21:42:39.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:39.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:39.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:39.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:39.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:39.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:39 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:42:39.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:39 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:42:39.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:39 smithi190 ceph-mon[32247]: pgmap v1923: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 790 op/s 2024-06-27T21:42:39.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:39.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:39.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:39 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:42:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:39 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:42:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:39 smithi192 ceph-mon[32956]: pgmap v1923: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 790 op/s 2024-06-27T21:42:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:39.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:39.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:39.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:40.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:40.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:40.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:40.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:40.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:40.285 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:40.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:40.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:40.486 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:40.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:40.541 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:40.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:40.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:40.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:40 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:42:40.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:40 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:42:40.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:40 smithi151 ceph-mon[28965]: osdmap e768: 12 total, 12 up, 10 in 2024-06-27T21:42:40.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:40 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:42:40.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:40 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:42:40.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:40 smithi190 ceph-mon[32247]: osdmap e768: 12 total, 12 up, 10 in 2024-06-27T21:42:40.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:40.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:40 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:42:40.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:40 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:42:40.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:40 smithi192 ceph-mon[32956]: osdmap e768: 12 total, 12 up, 10 in 2024-06-27T21:42:40.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:40.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:40.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:40.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:41.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:41.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:41.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:41.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:41.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:41.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:41.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:41.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:41.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:41.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:41.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:41.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.680 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:41.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:41 smithi151 ceph-mon[28965]: pgmap v1925: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 618 op/s 2024-06-27T21:42:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:41 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:41.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:41 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:41.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:41 smithi190 ceph-mon[32247]: pgmap v1925: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 618 op/s 2024-06-27T21:42:41.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:41 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:41.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:41 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:41.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:41.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:41 smithi192 ceph-mon[32956]: pgmap v1925: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 618 op/s 2024-06-27T21:42:41.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:41 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:41.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:41 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:41.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:41.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:41.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:41.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:42.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:42.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:42.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:42.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:42.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:42.310 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:42.364 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:42:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:42:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:42:42.364 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157108 6.19 6.5fbcb8f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:26.762363+0000", 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 555.56739283800005, 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93288493800000005, 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157108 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762363+0000", 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762363+0000", 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762365+0000", 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8649999999999999e-06 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762373+0000", 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8150000000000006e-06 2024-06-27T21:42:42.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762389+0000", 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6154000000000002e-05 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762397+0000", 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9759999999999995e-06 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762451+0000", 2024-06-27T21:42:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4725000000000002e-05 2024-06-27T21:42:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.762458+0000", 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.404e-06 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695237+0000", 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.93277944899999998 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695248+0000", 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0550000000000001e-05 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:157114 6.19 6.78756779 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:26.886811+0000", 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 555.44294432599997, 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80846879699999996, 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.371 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157114 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886811+0000", 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886811+0000", 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886813+0000", 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.606e-06 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886820+0000", 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.973e-06 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886836+0000", 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6116999999999998e-05 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886844+0000", 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9689999999999994e-06 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886896+0000", 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1560000000000001e-05 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:26.886901+0000", 2024-06-27T21:42:42.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5979999999999996e-06 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695272+0000", 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80837113599999999 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:27.695280+0000", 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8380000000000002e-06 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.10 553/550 [PushOp(6:0ccb8fbe:::benchmark_data_smithi192_117994_object58114:head, version: 487'3656, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:0ccb8fbe:::benchmark_data_smithi192_117994_object58114:head@487'3656, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:35.987284+0000", 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 546.34247186799996, 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60961001800000003, 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987284+0000", 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987284+0000", 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987286+0000", 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7689999999999999e-06 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987325+0000", 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9020999999999998e-05 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.595603+0000", 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60827812000000003 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.595615+0000", 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1839e-05 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.595678+0000", 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3705000000000001e-05 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.595686+0000", 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0739999999999993e-06 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.596894+0000", 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012074900000000001 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:161955 6.14 6:2ff3d028:::benchmark_data_smithi192_117994_object54361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e563)", 2024-06-27T21:42:42.381 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:10.593360+0000", 2024-06-27T21:42:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 511.73639573899999, 2024-06-27T21:42:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70676338100000002, 2024-06-27T21:42:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.382 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 161955 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593360+0000", 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593360+0000", 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593361+0000", 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.266e-06 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593367+0000", 2024-06-27T21:42:42.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5790000000000003e-06 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593381+0000", 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3720000000000001e-05 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593387+0000", 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9360000000000002e-06 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593439+0000", 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1468999999999998e-05 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593538+0000", 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9215000000000005e-05 2024-06-27T21:42:42.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4,5,7,9", 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.593835+0000", 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000296817 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595342+0000", 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015069440000000001 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595434+0000", 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1782999999999999e-05 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595702+0000", 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026865299999999999 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.596631+0000", 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092821500000000003 2024-06-27T21:42:42.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.638522+0000", 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.041891607999999997 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.299899+0000", 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.66137690500000001 2024-06-27T21:42:42.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.299979+0000", 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9864999999999995e-05 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.300123+0000", 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144406 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:161955 6.14 e563/550 ondisk, result = 0)", 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:10.595633+0000", 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 511.73412268599998, 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70449732300000001, 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595633+0000", 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595633+0000", 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595634+0000", 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4499999999999997e-07 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:10.595637+0000", 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.45e-06 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.299819+0000", 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70418150099999999 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.299826+0000", 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0899999999999999e-06 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.299882+0000", 2024-06-27T21:42:42.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6079000000000003e-05 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.299892+0000", 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7310000000000005e-06 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.300130+0000", 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000238827 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.13 e563,550 6:cf658f50:::benchmark_data_smithi192_117994_object36168:head,519'9128)", 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:11.471704+0000", 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 510.85805179699997, 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.68774031099999999, 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.471704+0000", 2024-06-27T21:42:42.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.471704+0000", 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.471705+0000", 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5300000000000004e-07 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:11.471711+0000", 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7270000000000001e-06 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.159301+0000", 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.68759008899999996 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.159326+0000", 2024-06-27T21:42:42.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4587999999999999e-05 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.159380+0000", 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3622000000000003e-05 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.159386+0000", 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.241e-06 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.159444+0000", 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8390999999999997e-05 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.10 e563,550 6:0f8cb0ff:::benchmark_data_smithi192_117994_object6255:head,507'7123)", 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:13.830793+0000", 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 508.49896301899997, 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63728283100000005, 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.400 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.830793+0000", 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.830793+0000", 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.830793+0000", 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0900000000000001e-07 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.830798+0000", 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.673e-06 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.467909+0000", 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63711055599999999 2024-06-27T21:42:42.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.467917+0000", 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5399999999999996e-06 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.467984+0000", 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6947999999999997e-05 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.467994+0000", 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4080000000000004e-06 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:14.468076+0000", 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1996999999999994e-05 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.13 e564,550 6:c9b522fa:::benchmark_data_smithi192_117994_object48604:head,540'9922)", 2024-06-27T21:42:42.404 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:16.511342+0000", 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 505.81841389700003, 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53765198599999997, 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:16.511342+0000", 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:16.511342+0000", 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:16.511343+0000", 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5899999999999995e-07 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:16.511350+0000", 2024-06-27T21:42:42.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5290000000000003e-06 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.048853+0000", 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53750247600000001 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.048860+0000", 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9829999999999999e-06 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.048922+0000", 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2603999999999999e-05 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.048932+0000", 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.55e-06 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.048994+0000", 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2084999999999999e-05 2024-06-27T21:42:42.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:169144 6.d e591/550 6:b5439742:::benchmark_data_smithi192_117994_object61550:head v 591'10547, mlcod=591'10545)", 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:22.049850+0000", 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 440.27990534600002, 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67006246899999999, 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.049850+0000", 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.049850+0000", 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.049851+0000", 2024-06-27T21:42:42.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8899999999999999e-07 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.049859+0000", 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7819999999999994e-06 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.715695+0000", 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.66583604900000004 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.715702+0000", 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.08e-06 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.715757+0000", 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5340999999999998e-05 2024-06-27T21:42:42.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.715779+0000", 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1957999999999999e-05 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.719866+0000", 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040869799999999996 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:22.719913+0000", 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6589999999999999e-05 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170050 6.19 6:9f0fb5cd:::benchmark_data_smithi192_117994_object62456:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.786352+0000", 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.54340329899998, 2024-06-27T21:42:42.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69005087899999995, 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170050 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786352+0000", 2024-06-27T21:42:42.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786352+0000", 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786354+0000", 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0049999999999999e-06 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786360+0000", 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8340000000000004e-06 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786372+0000", 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1494999999999999e-05 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786382+0000", 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0533999999999999e-05 2024-06-27T21:42:42.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786740+0000", 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000357282 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786749+0000", 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2469999999999997e-06 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.476317+0000", 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.68956853200000001 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.476403+0000", 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5950000000000002e-05 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170056 6.19 6:9b36ec32:::benchmark_data_smithi192_117994_object62462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.786401+0000", 2024-06-27T21:42:42.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.543354529, 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69700956599999997, 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170056 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786401+0000", 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786401+0000", 2024-06-27T21:42:42.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786402+0000", 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9200000000000001e-07 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786405+0000", 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.427e-06 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786412+0000", 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.294e-06 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786420+0000", 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6850000000000001e-06 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786774+0000", 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035387800000000002 2024-06-27T21:42:42.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786780+0000", 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6729999999999997e-06 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.476835+0000", 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69005515299999998 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.476949+0000", 2024-06-27T21:42:42.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113708 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.477311+0000", 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000362784 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483237+0000", 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059258410000000003 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483280+0000", 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3183999999999999e-05 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483334+0000", 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3871e-05 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483386+0000", 2024-06-27T21:42:42.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1629999999999999e-05 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483411+0000", 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4845999999999999e-05 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170059 6.19 6:9f0eac08:::benchmark_data_smithi192_117994_object62465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.786428+0000", 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.54332763600001, 2024-06-27T21:42:42.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69937375099999999, 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170059 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786428+0000", 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786428+0000", 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786429+0000", 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4000000000000002e-07 2024-06-27T21:42:42.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786431+0000", 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4899999999999999e-06 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786437+0000", 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9869999999999996e-06 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786443+0000", 2024-06-27T21:42:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4930000000000003e-06 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786804+0000", 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00036031399999999998 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786809+0000", 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5600000000000001e-06 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.478760+0000", 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69195025499999996 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.478855+0000", 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5768999999999997e-05 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.479167+0000", 2024-06-27T21:42:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031157400000000001 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483242+0000", 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040747860000000004 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483483+0000", 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000241022 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.485726+0000", 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002243189 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.485760+0000", 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4479e-05 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.485802+0000", 2024-06-27T21:42:42.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1393000000000002e-05 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170071 6.19 6:995bd25b:::benchmark_data_smithi192_117994_object62477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.786450+0000", 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.54330563899998, 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70192359400000004, 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170071 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786450+0000", 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786450+0000", 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786450+0000", 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.58e-07 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786453+0000", 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2819999999999999e-06 2024-06-27T21:42:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786459+0000", 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7939999999999997e-06 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786464+0000", 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8400000000000002e-06 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786848+0000", 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038384999999999998 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786853+0000", 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-06 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.480741+0000", 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69388857100000001 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.480842+0000", 2024-06-27T21:42:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100967 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.481139+0000", 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029706300000000003 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483292+0000", 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002152242 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.485876+0000", 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025838559999999998 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.488268+0000", 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002392852 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.488329+0000", 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1033999999999999e-05 2024-06-27T21:42:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.488374+0000", 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4205000000000001e-05 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170085 6.19 6:9cc92bf8:::benchmark_data_smithi192_117994_object62491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.786470+0000", 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.54328556600001, 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.704081234, 2024-06-27T21:42:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170085 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786470+0000", 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786470+0000", 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786470+0000", 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5400000000000002e-07 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786473+0000", 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0810000000000001e-06 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786478+0000", 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8540000000000002e-06 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786484+0000", 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0159999999999997e-06 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786937+0000", 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045207800000000002 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786943+0000", 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3929999999999996e-06 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483520+0000", 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69657737900000005 2024-06-27T21:42:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483632+0000", 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111457 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.483898+0000", 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026669500000000001 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.485941+0000", 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020421549999999999 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490420+0000", 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044798640000000001 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490456+0000", 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5927999999999999e-05 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490490+0000", 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3176000000000002e-05 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490551+0000", 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1804000000000007e-05 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170098 6.19 6:9b14a3cc:::benchmark_data_smithi192_117994_object62504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.786490+0000", 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.54326576, 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70423395700000002, 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170098 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786490+0000", 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786490+0000", 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786490+0000", 2024-06-27T21:42:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4499999999999998e-07 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786492+0000", 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0439999999999998e-06 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786498+0000", 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5820000000000001e-06 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786503+0000", 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7199999999999997e-06 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786970+0000", 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046780600000000003 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.786976+0000", 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8390000000000003e-06 2024-06-27T21:42:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.485976+0000", 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69899995800000003 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.486093+0000", 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011628 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.486377+0000", 2024-06-27T21:42:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028496199999999999 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490596+0000", 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042183990000000003 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490605+0000", 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4599999999999992e-06 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490638+0000", 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2317999999999998e-05 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490687+0000", 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9662e-05 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.490724+0000", 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6582000000000003e-05 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170118 6.19 6:9d30ca97:::benchmark_data_smithi192_117994_object62524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e594)", 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:29.874790+0000", 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 432.45496539700002, 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61853313300000001, 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:42:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170118 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874790+0000", 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874790+0000", 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874792+0000", 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3379999999999999e-06 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874797+0000", 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1109999999999999e-06 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874809+0000", 2024-06-27T21:42:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2126999999999999e-05 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874815+0000", 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1489999999999999e-06 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874860+0000", 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5429999999999997e-05 2024-06-27T21:42:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:29.874869+0000", 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1259999999999998e-06 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.488431+0000", 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61356232600000005 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:42:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.488539+0000", 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010802300000000001 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.488839+0000", 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030019399999999999 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.492840+0000", 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040012770000000001 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.492906+0000", 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5939999999999995e-05 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.493225+0000", 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031898000000000001 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.493279+0000", 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3712999999999999e-05 2024-06-27T21:42:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.493323+0000", 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4399e-05 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.14 657/550 [PushReplyOp(6:2b5356dc:::benchmark_data_smithi192_117994_object107310:head)])", 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 303.22635671099999, 2024-06-27T21:42:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70161021400000001, 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:42:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.03e-07 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103401+0000", 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5680000000000001e-06 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:42:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.804560+0000", 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70115915399999995 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.804568+0000", 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.804629+0000", 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1503999999999999e-05 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:42:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.805009+0000", 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037987699999999998 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:42:42.470 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:42:42.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:42.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:42.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:42.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:42.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:42 smithi151 ceph-mon[28965]: 7.f scrub starts 2024-06-27T21:42:42.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:42 smithi151 ceph-mon[28965]: osdmap e769: 12 total, 12 up, 10 in 2024-06-27T21:42:42.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:42 smithi151 ceph-mon[28965]: 7.f scrub ok 2024-06-27T21:42:42.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:42.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:42.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:42 smithi190 ceph-mon[32247]: 7.f scrub starts 2024-06-27T21:42:42.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:42 smithi190 ceph-mon[32247]: osdmap e769: 12 total, 12 up, 10 in 2024-06-27T21:42:42.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:42 smithi190 ceph-mon[32247]: 7.f scrub ok 2024-06-27T21:42:42.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:42 smithi192 ceph-mon[32956]: 7.f scrub starts 2024-06-27T21:42:42.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:42 smithi192 ceph-mon[32956]: osdmap e769: 12 total, 12 up, 10 in 2024-06-27T21:42:42.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:42 smithi192 ceph-mon[32956]: 7.f scrub ok 2024-06-27T21:42:42.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:42.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:42.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:42.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:43.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:43.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:43.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:43.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:43.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:43.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:43.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:43.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:43.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:43.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:43 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:42:43.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:43 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:42:43.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:43 smithi151 ceph-mon[28965]: pgmap v1927: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 541 op/s 2024-06-27T21:42:43.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:43.634 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 11, 10, 2, 8] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:42:43.634 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:42:43.635 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 3, 4, 5, 6, 9, 11, 10, 2, 8] 2024-06-27T21:42:43.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 11 2024-06-27T21:42:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:43.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:43.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:43.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:43 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:42:43.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:43 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:42:43.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:43 smithi190 ceph-mon[32247]: pgmap v1927: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 541 op/s 2024-06-27T21:42:43.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:43.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:43 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:42:43.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:43 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:42:43.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:43 smithi192 ceph-mon[32956]: pgmap v1927: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 541 op/s 2024-06-27T21:42:43.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:43.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:43.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:44.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:44.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:44.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:44.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:44.271 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:44.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:44.332 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:42:44.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:44.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:44.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:42:44.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:44.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:44.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:44.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:44 smithi151 ceph-mon[28965]: 7.5 scrub starts 2024-06-27T21:42:44.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3040546098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:42:44.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:44 smithi151 ceph-mon[28965]: 7.5 scrub ok 2024-06-27T21:42:44.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:44.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:44 smithi190 ceph-mon[32247]: 7.5 scrub starts 2024-06-27T21:42:44.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3040546098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:42:44.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:44 smithi190 ceph-mon[32247]: 7.5 scrub ok 2024-06-27T21:42:44.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:44 smithi192 ceph-mon[32956]: 7.5 scrub starts 2024-06-27T21:42:44.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3040546098' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:42:44.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:44 smithi192 ceph-mon[32956]: 7.5 scrub ok 2024-06-27T21:42:44.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:44.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:44.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:44.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:44.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:42:44.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:42:45.030 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:42:45.031 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:42:45.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:42:45.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:45.090 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:42:45.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:42:45.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:42:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:42:45.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:45.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:45.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:45.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:42:45.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:42:45.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:45.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:45.398 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:42:45.399 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:42:45.401 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:45.420 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:45.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:42:45.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:42:45.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:45.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:42:45.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:42:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:45.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:45.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:42:45.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:42:45.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:45.613 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:42:45.613 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:42:45.613 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: 7.d scrub starts 2024-06-27T21:42:45.614 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: pgmap v1928: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 158 op/s 2024-06-27T21:42:45.614 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:42:45.614 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:42:45.614 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3040546098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:42:45.614 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: osdmap e770: 12 total, 12 up, 10 in 2024-06-27T21:42:45.614 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/23185602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:45.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:45.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:45.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:42:45.758 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:42:45.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:45.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:42:45.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:42:45.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: 7.d scrub starts 2024-06-27T21:42:45.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: pgmap v1928: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 158 op/s 2024-06-27T21:42:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:42:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:42:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3040546098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:42:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: osdmap e770: 12 total, 12 up, 10 in 2024-06-27T21:42:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/23185602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: 7.d scrub starts 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: pgmap v1928: 47 pgs: 47 active+clean; 2.6 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 158 op/s 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3040546098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: osdmap e770: 12 total, 12 up, 10 in 2024-06-27T21:42:45.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/23185602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:45.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:45.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:45.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:45.917 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:42:45.917 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:42:45.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:46.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:46.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:46.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:46.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:46.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:46.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:46.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:46.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:46.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:46.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:46.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:46 smithi151 ceph-mon[28965]: 7.d scrub ok 2024-06-27T21:42:46.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:46.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:46 smithi190 ceph-mon[32247]: 7.d scrub ok 2024-06-27T21:42:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:46.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:46.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:46 smithi192 ceph-mon[32956]: 7.d scrub ok 2024-06-27T21:42:46.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:46.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:46.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:47.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:47.029 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:47.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:47.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:47.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:47.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:47.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:42:47.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:47.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:47.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:47 smithi151 ceph-mon[28965]: 7.1 scrub starts 2024-06-27T21:42:47.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:47 smithi151 ceph-mon[28965]: 7.7 scrub starts 2024-06-27T21:42:47.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:47 smithi151 ceph-mon[28965]: pgmap v1930: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 235 op/s 2024-06-27T21:42:47.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:47 smithi151 ceph-mon[28965]: 7.1 scrub ok 2024-06-27T21:42:47.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:47 smithi151 ceph-mon[28965]: 7.c deep-scrub starts 2024-06-27T21:42:47.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:47.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:47.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:47.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:47.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:47 smithi190 ceph-mon[32247]: 7.1 scrub starts 2024-06-27T21:42:47.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:47 smithi190 ceph-mon[32247]: 7.7 scrub starts 2024-06-27T21:42:47.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:47 smithi190 ceph-mon[32247]: pgmap v1930: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 235 op/s 2024-06-27T21:42:47.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:47 smithi190 ceph-mon[32247]: 7.1 scrub ok 2024-06-27T21:42:47.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:47 smithi190 ceph-mon[32247]: 7.c deep-scrub starts 2024-06-27T21:42:47.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:47 smithi192 ceph-mon[32956]: 7.1 scrub starts 2024-06-27T21:42:47.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:47 smithi192 ceph-mon[32956]: 7.7 scrub starts 2024-06-27T21:42:47.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:47 smithi192 ceph-mon[32956]: pgmap v1930: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 235 op/s 2024-06-27T21:42:47.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:47 smithi192 ceph-mon[32956]: 7.1 scrub ok 2024-06-27T21:42:47.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:47 smithi192 ceph-mon[32956]: 7.c deep-scrub starts 2024-06-27T21:42:47.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:47.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:47.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:47.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:47.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:48.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:48.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:48.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:48.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:48.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:48.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:48.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:48 smithi151 ceph-mon[28965]: 7.7 scrub ok 2024-06-27T21:42:48.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:48.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:48.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:48.588 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:42:48.588 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:42:48.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:48.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.734 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:48.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:48.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:48 smithi190 ceph-mon[32247]: 7.7 scrub ok 2024-06-27T21:42:48.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:48 smithi192 ceph-mon[32956]: 7.7 scrub ok 2024-06-27T21:42:48.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:48.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:48.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:48.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:49.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:49.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:49.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:49.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:49.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:49.404 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:49 smithi151 ceph-mon[28965]: pgmap v1931: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 197 op/s 2024-06-27T21:42:49.404 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:49 smithi151 ceph-mon[28965]: 7.b deep-scrub starts 2024-06-27T21:42:49.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:49.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:49.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:49.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:49.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:49.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:49 smithi190 ceph-mon[32247]: pgmap v1931: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 197 op/s 2024-06-27T21:42:49.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:49 smithi190 ceph-mon[32247]: 7.b deep-scrub starts 2024-06-27T21:42:49.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:49 smithi192 ceph-mon[32956]: pgmap v1931: 47 pgs: 47 active+clean; 2.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 197 op/s 2024-06-27T21:42:49.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:49 smithi192 ceph-mon[32956]: 7.b deep-scrub starts 2024-06-27T21:42:49.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:49.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:49.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:49.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:49.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:49.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:50.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:42:50.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:50.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:50.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:50.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:50.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:50.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:50.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:50 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:42:50.381+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e770 definitely_dead 0 2024-06-27T21:42:50.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:50.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:50.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:50.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:50.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:50.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:50.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:50.873 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.11. 2024-06-27T21:42:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:50.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:50.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:50.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:50.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:51.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:51.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3842463424' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:42:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:42:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:50 smithi151 ceph-mon[28965]: Client client.admin marked osd.11 out, while it was still marked up 2024-06-27T21:42:51.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:51.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:42:51.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3842463424' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:42:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:42:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:50 smithi190 ceph-mon[32247]: Client client.admin marked osd.11 out, while it was still marked up 2024-06-27T21:42:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3842463424' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:42:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:42:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:50 smithi192 ceph-mon[32956]: Client client.admin marked osd.11 out, while it was still marked up 2024-06-27T21:42:51.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:51.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:51.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:51.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:51.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.619 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:51.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:51.720 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.743 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:51.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:51.844 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:51.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:51.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:51 smithi151 ceph-mon[28965]: pgmap v1932: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 2.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 311 op/s 2024-06-27T21:42:51.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-06-27T21:42:51.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:51 smithi151 ceph-mon[28965]: osdmap e771: 12 total, 12 up, 9 in 2024-06-27T21:42:51.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:51.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:51.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:52.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:52.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:52.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:52.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:52.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:51 smithi190 ceph-mon[32247]: pgmap v1932: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 2.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 311 op/s 2024-06-27T21:42:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-06-27T21:42:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:51 smithi190 ceph-mon[32247]: osdmap e771: 12 total, 12 up, 9 in 2024-06-27T21:42:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:51 smithi192 ceph-mon[32956]: pgmap v1932: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 2.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 311 op/s 2024-06-27T21:42:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-06-27T21:42:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:51 smithi192 ceph-mon[32956]: osdmap e771: 12 total, 12 up, 9 in 2024-06-27T21:42:52.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:52.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:52.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:52.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:52.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:52.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:52.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:52.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:52.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:52.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:52.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:52.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:53.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:53.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:53.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: osdmap e772: 12 total, 12 up, 9 in 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:42:53.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:52 smithi151 ceph-mon[28965]: osdmap e773: 12 total, 12 up, 9 in 2024-06-27T21:42:53.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:53.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: osdmap e772: 12 total, 12 up, 9 in 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:42:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:52 smithi190 ceph-mon[32247]: osdmap e773: 12 total, 12 up, 9 in 2024-06-27T21:42:53.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: osdmap e772: 12 total, 12 up, 9 in 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:42:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:52 smithi192 ceph-mon[32956]: osdmap e773: 12 total, 12 up, 9 in 2024-06-27T21:42:53.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:53.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:53.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:53.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:53.501 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:53.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:53.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:53.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:53.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:53.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:53.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:53.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:53.899 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:42:53.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:54.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:54.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:54.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: pgmap v1935: 47 pgs: 3 peering, 1 unknown, 2 active+clean+scrubbing+deep, 41 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 521 op/s 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 7.a starting backfill to osd.10 from (0'0,0'0] MAX to 770'10902 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 7.9 starting backfill to osd.10 from (0'0,0'0] MAX to 770'13454 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 7.5 starting backfill to osd.6 from (0'0,0'0] MAX to 770'10759 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 7.1 starting backfill to osd.5 from (0'0,0'0] MAX to 770'10465 2024-06-27T21:42:54.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 7.b starting backfill to osd.0 from (0'0,0'0] MAX to 770'10980 2024-06-27T21:42:54.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:42:54.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: 7.c starting backfill to osd.10 from (0'0,0'0] MAX to 770'11035 2024-06-27T21:42:54.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:54.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:54.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2375760299' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:54.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:53 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:54.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:54.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: pgmap v1935: 47 pgs: 3 peering, 1 unknown, 2 active+clean+scrubbing+deep, 41 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 521 op/s 2024-06-27T21:42:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:42:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:42:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 7.a starting backfill to osd.10 from (0'0,0'0] MAX to 770'10902 2024-06-27T21:42:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 7.9 starting backfill to osd.10 from (0'0,0'0] MAX to 770'13454 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 7.5 starting backfill to osd.6 from (0'0,0'0] MAX to 770'10759 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 7.1 starting backfill to osd.5 from (0'0,0'0] MAX to 770'10465 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 7.b starting backfill to osd.0 from (0'0,0'0] MAX to 770'10980 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: 7.c starting backfill to osd.10 from (0'0,0'0] MAX to 770'11035 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2375760299' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:53 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: pgmap v1935: 47 pgs: 3 peering, 1 unknown, 2 active+clean+scrubbing+deep, 41 active+clean; 2.3 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 521 op/s 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 7.a starting backfill to osd.10 from (0'0,0'0] MAX to 770'10902 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 7.9 starting backfill to osd.10 from (0'0,0'0] MAX to 770'13454 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 7.5 starting backfill to osd.6 from (0'0,0'0] MAX to 770'10759 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 7.1 starting backfill to osd.5 from (0'0,0'0] MAX to 770'10465 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 7.b starting backfill to osd.0 from (0'0,0'0] MAX to 770'10980 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:42:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: 7.c starting backfill to osd.10 from (0'0,0'0] MAX to 770'11035 2024-06-27T21:42:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2375760299' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:54.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:53 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:42:54.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:54.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:54.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:54.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:54.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:54.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:54.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:54.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:54.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:54.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:54.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:55.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:55.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:55.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: osdmap e774: 12 total, 12 up, 9 in 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:42:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:54 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:42:55.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:42:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: osdmap e774: 12 total, 12 up, 9 in 2024-06-27T21:42:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:42:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:42:55.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:42:55.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:54 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:42:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:42:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:42:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: osdmap e774: 12 total, 12 up, 9 in 2024-06-27T21:42:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:42:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:42:55.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:42:55.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:54 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:42:55.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:55.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:55.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:55.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:55.538 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:42:55.538 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:42:55.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:55.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:55.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.732 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:55.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:55.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:55.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:55.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:42:55.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:55.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:42:56.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:56.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:56.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:42:56.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:42:56.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:42:56.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:42:56.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:55 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:42:56.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:56.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:42:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:42:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:42:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:42:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:55 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:42:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:42:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:42:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:42:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:42:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:55 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:42:56.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:56.382 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:56.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:42:56.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:56.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:56.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:42:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:56.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:42:56.674 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8] out_osds: [1, 7, 11] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:42:56.675 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:42:56.675 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-06-27T21:42:56.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 11 2024-06-27T21:42:56.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:56.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:56.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:56.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:56.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:56.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:56 smithi151 ceph-mon[28965]: osdmap e775: 12 total, 12 up, 9 in 2024-06-27T21:42:57.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:57.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:57.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:42:57.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:57.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:56 smithi190 ceph-mon[32247]: osdmap e775: 12 total, 12 up, 9 in 2024-06-27T21:42:57.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:42:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:56 smithi192 ceph-mon[32956]: osdmap e775: 12 total, 12 up, 9 in 2024-06-27T21:42:57.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:42:57.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:57.448 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:57.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:57.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:57.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:57.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:57.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:57.825 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:57.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:42:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:57.962 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:57.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:42:58.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:58.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:58.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:58.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:57 smithi151 ceph-mon[28965]: pgmap v1940: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 427 op/s; 13422/108807 objects misplaced (12.336%); 3.8 MiB/s, 64 objects/s recovering 2024-06-27T21:42:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:57 smithi151 ceph-mon[28965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:42:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:57 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:42:58.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:57 smithi190 ceph-mon[32247]: pgmap v1940: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 427 op/s; 13422/108807 objects misplaced (12.336%); 3.8 MiB/s, 64 objects/s recovering 2024-06-27T21:42:58.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:57 smithi190 ceph-mon[32247]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:42:58.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:57 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:42:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:57 smithi192 ceph-mon[32956]: pgmap v1940: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 427 op/s; 13422/108807 objects misplaced (12.336%); 3.8 MiB/s, 64 objects/s recovering 2024-06-27T21:42:58.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:57 smithi192 ceph-mon[32956]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:42:58.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:57 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:42:58.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:42:58.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:42:58.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:42:58.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:42:58.414 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:42:58.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:42:58.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:42:58.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:42:58.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:58.473 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:42:58.474 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:42:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:58.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:58.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:42:58.514 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:42:58.542 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:42:58.543 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:42:58.579 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:42:58.580 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:42:58.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:58.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:58.609 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:58.610 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:42:58.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:42:58.619 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:42:58.651 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:42:58.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:42:58.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:42:58.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:42:58.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:42:58.726 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:42:58.727 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:42:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.738 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:58.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:42:58.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:42:58.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:42:58.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:42:58.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:42:58.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:58.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:58.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:42:59.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1230767542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:59.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:59.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:59.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:59.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1230767542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1230767542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:42:59.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:42:59.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:42:59.477 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:42:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:42:59.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:42:59.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:42:59.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:42:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:42:59.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:42:59.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:42:59.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:42:59.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:42:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:42:59.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:00.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:00.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:00.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:42:59 smithi151 ceph-mon[28965]: pgmap v1941: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 388 op/s; 13422/107451 objects misplaced (12.491%); 2.8 MiB/s, 47 objects/s recovering 2024-06-27T21:43:00.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:00.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:00.197 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:42:59 smithi192 ceph-mon[32956]: pgmap v1941: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 388 op/s; 13422/107451 objects misplaced (12.491%); 2.8 MiB/s, 47 objects/s recovering 2024-06-27T21:43:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:42:59 smithi190 ceph-mon[32247]: pgmap v1941: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.2 GiB data, 11 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 388 op/s; 13422/107451 objects misplaced (12.491%); 2.8 MiB/s, 47 objects/s recovering 2024-06-27T21:43:00.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:00.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:00.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:00.442 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:00.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:00.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:00.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:00.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:00.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:00.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:00.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:00.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:00.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:01.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:01.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:01.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:01.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:01.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:01.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:01.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:01.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:01.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:01.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:01.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:01.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:01.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:01.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:01.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:01.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:01.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:02.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:02.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:01 smithi151 ceph-mon[28965]: pgmap v1942: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 11210/103287 objects misplaced (10.853%); 16 MiB/s, 253 objects/s recovering 2024-06-27T21:43:02.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:02.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:01 smithi190 ceph-mon[32247]: pgmap v1942: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 11210/103287 objects misplaced (10.853%); 16 MiB/s, 253 objects/s recovering 2024-06-27T21:43:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:01 smithi192 ceph-mon[32956]: pgmap v1942: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.1 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 482 op/s; 11210/103287 objects misplaced (10.853%); 16 MiB/s, 253 objects/s recovering 2024-06-27T21:43:02.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:02.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:02.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:02.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:02.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:02.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:02.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:02.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:02.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:02.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:02.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:02.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:03.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:03.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:03.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:03.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:03.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:03.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:03.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:03.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:03.552 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:03.654 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:03.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:03.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:03.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:03.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:03.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:03.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:03.993 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:03 smithi151 ceph-mon[28965]: pgmap v1943: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 611 op/s; 10495/97566 objects misplaced (10.757%); 14 MiB/s, 221 objects/s recovering 2024-06-27T21:43:04.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:04.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:04.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:04.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:04.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:03 smithi190 ceph-mon[32247]: pgmap v1943: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 611 op/s; 10495/97566 objects misplaced (10.757%); 14 MiB/s, 221 objects/s recovering 2024-06-27T21:43:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:03 smithi192 ceph-mon[32956]: pgmap v1943: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 2.0 GiB data, 10 GiB used, 794 GiB / 805 GiB avail; 0 B/s wr, 611 op/s; 10495/97566 objects misplaced (10.757%); 14 MiB/s, 221 objects/s recovering 2024-06-27T21:43:04.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:04.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:04.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:04.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:04.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:04.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:04.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:04.722 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:04.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:04.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:04.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:04.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:04.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:05.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:05.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:05.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:05.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:05.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:05.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:05.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:05.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:05.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:05.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:05.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:05.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:05.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:05.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:05.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:06.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:06.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:06.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:06.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:43:06.140 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:06.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:05 smithi151 ceph-mon[28965]: pgmap v1944: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 626 op/s; 9786/94080 objects misplaced (10.402%); 15 MiB/s, 243 objects/s recovering 2024-06-27T21:43:06.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:05 smithi190 ceph-mon[32247]: pgmap v1944: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 626 op/s; 9786/94080 objects misplaced (10.402%); 15 MiB/s, 243 objects/s recovering 2024-06-27T21:43:06.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:05 smithi192 ceph-mon[32956]: pgmap v1944: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.9 GiB data, 10 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 626 op/s; 9786/94080 objects misplaced (10.402%); 15 MiB/s, 243 objects/s recovering 2024-06-27T21:43:06.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:06.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:06.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:06.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:06.643 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:06.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:06.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:06.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:06.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:06.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:06.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:07.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:07.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:07.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:07.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:07.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:07.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:07.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:07.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:07.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:07.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:07.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:07.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:07.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:07.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:07.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:07.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:07.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:08.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:08.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:08.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:07 smithi151 ceph-mon[28965]: pgmap v1945: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.8 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 796 op/s; 8019/86325 objects misplaced (9.289%); 18 MiB/s, 287 objects/s recovering 2024-06-27T21:43:08.092 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:08.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:08.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:07 smithi151 ceph-mon[28965]: osdmap e776: 12 total, 12 up, 9 in 2024-06-27T21:43:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:08.112 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:08.141 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:43:08.141 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:43:08.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:43:08.141 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:43:08.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:156111 6.f 6:fcae7150:::benchmark_data_smithi192_117994_object48517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:13.813504+0000", 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 594.32309805099999, 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.64207833999999997, 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.142 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156111 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813504+0000", 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813504+0000", 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813505+0000", 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813510+0000", 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2249999999999999e-06 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813519+0000", 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4349999999999997e-06 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813524+0000", 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5910000000000003e-06 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813568+0000", 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3044000000000003e-05 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813663+0000", 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5563000000000006e-05 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.813747+0000", 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3908999999999999e-05 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.817639+0000", 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003892202 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:13.818732+0000", 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010923339999999999 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:14.455507+0000", 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63677589499999998 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:14.455549+0000", 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1356000000000002e-05 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:14.455582+0000", 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3569999999999999e-05 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:156223 6.e 6:74cd8b3d:::benchmark_data_smithi192_117994_object48629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:15.194773+0000", 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 592.94182949499998, 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43812209600000002, 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.148 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 156223 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194773+0000", 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194773+0000", 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194775+0000", 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4370000000000001e-06 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194787+0000", 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1593000000000001e-05 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194827+0000", 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0667999999999998e-05 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194836+0000", 2024-06-27T21:43:08.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1619999999999997e-06 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.194893+0000", 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6388000000000003e-05 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.195005+0000", 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011181 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.195093+0000", 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7928000000000002e-05 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.198760+0000", 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036673819999999998 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.199244+0000", 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048398899999999999 2024-06-27T21:43:08.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.632813+0000", 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43356867900000001 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.632855+0000", 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2410000000000002e-05 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:15.632895+0000", 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9650000000000002e-05 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.a e541,516 6:59eebc9d:::benchmark_data_smithi192_117994_object18140:head,511'7778)", 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:19.275215+0000", 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 588.86138700100003, 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52697154400000001, 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.155 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275215+0000", 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275215+0000", 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275216+0000", 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0200000000000001e-07 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.275220+0000", 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8260000000000003e-06 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802078+0000", 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52685791000000004 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802083+0000", 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3730000000000001e-06 2024-06-27T21:43:08.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802134+0000", 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1372000000000002e-05 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802141+0000", 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3999999999999997e-06 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:19.802187+0000", 2024-06-27T21:43:08.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5961000000000003e-05 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.13 551/550 [PushOp(6:cdfd5352:::benchmark_data_smithi192_117994_object50909:head, version: 486'3256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:cdfd5352:::benchmark_data_smithi192_117994_object50909:head@486'3256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 578.34282228999996, 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46384465499999999, 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793780+0000", 2024-06-27T21:43:08.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7100000000000001e-07 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:29.793880+0000", 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9273999999999999e-05 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256375+0000", 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.462494984 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256389+0000", 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3978999999999999e-05 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256461+0000", 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2417999999999997e-05 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.256471+0000", 2024-06-27T21:43:08.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0219e-05 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:30.257624+0000", 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011531099999999999 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.13 e563,550 6:ca83749f:::benchmark_data_smithi192_117994_object37811:head,519'9236)", 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:43:08.164 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 535.63627146099998, 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50509707500000001, 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:43:08.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500331+0000", 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000005e-07 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:12.500337+0000", 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5650000000000001e-06 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005291+0000", 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50495398700000005 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005298+0000", 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.764e-06 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005368+0000", 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0366999999999999e-05 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005375+0000", 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7259999999999996e-06 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:13.005428+0000", 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2917000000000002e-05 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.168 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:168135 6.f 6:f725a463:::benchmark_data_smithi192_117994_object60541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:14.389388+0000", 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 473.74721431400002, 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62669164200000005, 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168135 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389388+0000", 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389388+0000", 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389389+0000", 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3799999999999999e-06 2024-06-27T21:43:08.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389394+0000", 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2229999999999998e-06 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389407+0000", 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3093e-05 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389416+0000", 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9150000000000002e-06 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389474+0000", 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7908000000000002e-05 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389634+0000", 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015965 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.389793+0000", 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158658 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.393717+0000", 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039243730000000001 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944686+0000", 2024-06-27T21:43:08.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55096871300000005 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015949+0000", 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.071262961999999999 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.016021+0000", 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2797000000000001e-05 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.016079+0000", 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7970000000000002e-05 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:168135 6.f e587/467 ondisk, result = 0)", 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 473.74227944199998, 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55037249099999996, 2024-06-27T21:43:08.175 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394323+0000", 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6899999999999997e-07 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394327+0000", 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.614e-06 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944602+0000", 2024-06-27T21:43:08.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55027552199999996 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944611+0000", 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8000000000000004e-06 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944667+0000", 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5957e-05 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944678+0000", 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0702e-05 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.944695+0000", 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7326999999999999e-05 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.179 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:168135 6.f e587/467 ondisk, result = 0)", 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:14.394376+0000", 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 473.74222589999999, 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62170963899999998, 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394376+0000", 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394376+0000", 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394377+0000", 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.63e-07 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:14.394380+0000", 2024-06-27T21:43:08.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9660000000000001e-06 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015875+0000", 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62149521900000004 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015878+0000", 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6540000000000001e-06 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015929+0000", 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0705999999999999e-05 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.015940+0000", 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1209999999999999e-05 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:15.016086+0000", 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014542099999999999 2024-06-27T21:43:08.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.5 594/548 [PushOp(6:a5ffe05c:::benchmark_data_smithi192_117994_object99841:head, version: 501'6175, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:a5ffe05c:::benchmark_data_smithi192_117994_object99841:head@501'6175, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:30.449103+0000", 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 457.68749902399998, 2024-06-27T21:43:08.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61745724199999996, 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449103+0000", 2024-06-27T21:43:08.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449103+0000", 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449104+0000", 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.42e-06 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:30.449179+0000", 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4813000000000004e-05 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064721+0000", 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61554164700000003 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064753+0000", 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2100000000000001e-05 2024-06-27T21:43:08.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064899+0000", 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014559999999999999 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.064915+0000", 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5981000000000002e-05 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:31.066560+0000", 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001645681 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:170693 6.f 6:ff66131a:::benchmark_data_smithi192_117994_object63099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e595)", 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 453.42961852600001, 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.480746165, 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.189 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170693 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706984+0000", 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4799999999999997e-07 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706988+0000", 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5769999999999998e-06 2024-06-27T21:43:08.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.706997+0000", 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0219999999999993e-06 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707002+0000", 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9180000000000002e-06 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707047+0000", 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4913999999999997e-05 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707144+0000", 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6757999999999998e-05 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.707233+0000", 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8937000000000006e-05 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.711312+0000", 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040799169999999997 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:34.711381+0000", 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8541000000000007e-05 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:35.187612+0000", 2024-06-27T21:43:08.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.476230663 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:35.187677+0000", 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5320999999999993e-05 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:35.187730+0000", 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2748999999999999e-05 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:172014 6.5 e597/548 6:a50abf01:::benchmark_data_smithi192_117994_object64420:head v 597'10229, mlcod=502'6233)", 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:45.302797+0000", 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 442.83380476999997, 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54581944999999998, 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302797+0000", 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302797+0000", 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302798+0000", 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2499999999999995e-07 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.302802+0000", 2024-06-27T21:43:08.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5260000000000004e-06 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844258+0000", 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54145553800000001 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844269+0000", 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1063999999999999e-05 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844347+0000", 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7813999999999997e-05 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.844381+0000", 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4703000000000003e-05 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.848563+0000", 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041811110000000004 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:45.848617+0000", 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4169000000000003e-05 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172333 6.f 6:f63ad1db:::benchmark_data_smithi192_117994_object64739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.615351+0000", 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 439.52125093500001, 2024-06-27T21:43:08.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45670154899999998, 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172333 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615351+0000", 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615351+0000", 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615352+0000", 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.125e-06 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615357+0000", 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6079999999999998e-06 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615367+0000", 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0329999999999999e-05 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615373+0000", 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9549999999999997e-06 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615416+0000", 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2525999999999999e-05 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615549+0000", 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013355800000000001 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.615699+0000", 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000149903 2024-06-27T21:43:08.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.619475+0000", 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003776223 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.619560+0000", 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4375000000000004e-05 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.071948+0000", 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45238802500000003 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.072009+0000", 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0980999999999998e-05 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.072053+0000", 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3940000000000003e-05 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.206 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.20 657/654 [PushReplyOp(6:070deb76:::benchmark_data_smithi192_117994_object88876:head)])", 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:39.358439+0000", 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 328.77816293900003, 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32801349899999999, 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358439+0000", 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358439+0000", 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358440+0000", 2024-06-27T21:43:08.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7000000000000005e-07 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.358443+0000", 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5240000000000001e-06 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.685723+0000", 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32728001000000001 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.685731+0000", 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5299999999999999e-06 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.685776+0000", 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5467999999999999e-05 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.686453+0000", 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00067639700000000002 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:197630 6.14 6:2a19bfa2:::benchmark_data_smithi192_117994_object90036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:00.420281+0000", 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 307.71632143199997, 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27922517600000002, 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.211 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197630 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420281+0000", 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420281+0000", 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420282+0000", 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.13e-06 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420286+0000", 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5000000000000001e-06 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420297+0000", 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0769e-05 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420303+0000", 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.908e-06 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420349+0000", 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6419999999999999e-05 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420439+0000", 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9881000000000002e-05 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-06-27T21:43:08.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.420764+0000", 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000324498 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.422295+0000", 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015308019999999999 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.423275+0000", 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00098013799999999997 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699419+0000", 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27614467999999998 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699463+0000", 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3758999999999999e-05 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.699506+0000", 2024-06-27T21:43:08.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2691000000000002e-05 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:202834 6.1e e673/654)", 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:07.812600+0000", 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 300.32400176700003, 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63657862700000001, 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:43:08.218 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812600+0000", 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812600+0000", 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812601+0000", 2024-06-27T21:43:08.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:07.812603+0000", 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6259999999999999e-06 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449019+0000", 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63641561099999999 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449038+0000", 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8797e-05 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449166+0000", 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012861400000000001 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.449179+0000", 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2578000000000001e-05 2024-06-27T21:43:08.221 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:203122 6.20 6:0683f0fd:::benchmark_data_smithi192_117994_object95528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:08.156724+0000", 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 299.97987830300002, 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35273166299999997, 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.222 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 203122 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156724+0000", 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156724+0000", 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156725+0000", 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1209999999999999e-06 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156729+0000", 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.228e-06 2024-06-27T21:43:08.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156742+0000", 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.279e-05 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156751+0000", 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6969999999999999e-06 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156813+0000", 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2285000000000004e-05 2024-06-27T21:43:08.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.156957+0000", 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143893 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,11", 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.157252+0000", 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029488700000000002 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.158755+0000", 2024-06-27T21:43:08.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015033379999999999 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.159055+0000", 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029955500000000001 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.509388+0000", 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.350333216 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.509426+0000", 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7979999999999999e-05 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.509455+0000", 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9672999999999999e-05 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:203132 6.e 6:76f73f22:::benchmark_data_smithi192_117994_object95538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:08.184754+0000", 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 299.95184798299999, 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31415277899999999, 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 203132 2024-06-27T21:43:08.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184754+0000", 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184754+0000", 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184755+0000", 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6299999999999993e-07 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184760+0000", 2024-06-27T21:43:08.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4730000000000002e-06 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184770+0000", 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0234999999999999e-05 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184775+0000", 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6400000000000002e-06 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184832+0000", 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6573000000000002e-05 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.184928+0000", 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5589999999999998e-05 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-06-27T21:43:08.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.185337+0000", 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040961399999999999 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.187150+0000", 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018128510000000001 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.194613+0000", 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074630190000000004 2024-06-27T21:43:08.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.498805+0000", 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30419222899999998 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.498853+0000", 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7305000000000002e-05 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.498907+0000", 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4286999999999997e-05 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:7197 7.8 7:1e052b92:::benchmark_data_smithi192_150129_object7196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:39:50.924156+0000", 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 197.21244618399999, 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30122880400000002, 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.237 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7197 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924156+0000", 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924156+0000", 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924158+0000", 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.649e-06 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924187+0000", 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9043e-05 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924200+0000", 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3145999999999999e-05 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924206+0000", 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1500000000000004e-06 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924254+0000", 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8173000000000001e-05 2024-06-27T21:43:08.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.924260+0000", 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0950000000000001e-06 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.925426+0000", 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001165393 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.925538+0000", 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011216499999999999 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.925647+0000", 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108879 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.926342+0000", 2024-06-27T21:43:08.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00069508899999999997 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:50.927225+0000", 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088353900000000003 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:51.225303+0000", 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29807735499999999 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:51.225350+0000", 2024-06-27T21:43:08.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7348000000000001e-05 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:39:51.225385+0000", 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4780000000000002e-05 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164277 7.8 7:1b41cc32:::benchmark_data_smithi192_150129_object60972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.485696+0000", 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.650905786999999, 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.247969036, 2024-06-27T21:43:08.244 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164277 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485696+0000", 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485696+0000", 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485697+0000", 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.92e-07 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485700+0000", 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4139999999999998e-06 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485711+0000", 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0723999999999999e-05 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485716+0000", 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7709999999999997e-06 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485772+0000", 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5730999999999999e-05 2024-06-27T21:43:08.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485859+0000", 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7387999999999997e-05 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.485951+0000", 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1490000000000007e-05 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.489950+0000", 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039997720000000004 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.489963+0000", 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2542e-05 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733583+0000", 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.243620423 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:08.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733631+0000", 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7771000000000001e-05 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733665+0000", 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4217999999999997e-05 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26656.0:164277 7.8 e767/718 ondisk, result = 0)", 2024-06-27T21:43:08.250 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.489274+0000", 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.647328351999999, 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.244396161, 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.489274+0000", 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.489274+0000", 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.489275+0000", 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.04e-06 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.489280+0000", 2024-06-27T21:43:08.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6949999999999997e-06 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733521+0000", 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.244240606 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733527+0000", 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0379999999999996e-06 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733569+0000", 2024-06-27T21:43:08.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1823999999999998e-05 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733577+0000", 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2360000000000004e-06 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.733670+0000", 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2721999999999998e-05 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.254 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.255 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:08.255 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:08.255 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:43:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:08.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:07 smithi190 ceph-mon[32247]: pgmap v1945: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.8 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 796 op/s; 8019/86325 objects misplaced (9.289%); 18 MiB/s, 287 objects/s recovering 2024-06-27T21:43:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:07 smithi190 ceph-mon[32247]: osdmap e776: 12 total, 12 up, 9 in 2024-06-27T21:43:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:07 smithi192 ceph-mon[32956]: pgmap v1945: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.8 GiB data, 9.8 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 796 op/s; 8019/86325 objects misplaced (9.289%); 18 MiB/s, 287 objects/s recovering 2024-06-27T21:43:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:08.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:08.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:07 smithi192 ceph-mon[32956]: osdmap e776: 12 total, 12 up, 9 in 2024-06-27T21:43:08.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:08.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:08.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:08.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:08.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:08.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:08.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:08.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:08.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:08.904 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:08.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:08.957 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:43:09.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:09.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:09.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:09.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1883958501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:09.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1883958501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1883958501' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:09.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:09.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:09.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:09.434 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:09.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:09.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:09.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:09.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:09.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:43:09.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:09.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:09.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:09.946 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:09.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:10.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:09 smithi151 ceph-mon[28965]: pgmap v1947: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 746 op/s; 8019/85056 objects misplaced (9.428%); 19 MiB/s, 298 objects/s recovering 2024-06-27T21:43:10.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:09 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:10.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1883958501' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:10.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:09 smithi151 ceph-mon[28965]: osdmap e777: 12 total, 12 up, 9 in 2024-06-27T21:43:10.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:09 smithi151 ceph-mon[28965]: osdmap e778: 12 total, 12 up, 9 in 2024-06-27T21:43:10.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:10.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:10.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:10.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:09 smithi190 ceph-mon[32247]: pgmap v1947: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 746 op/s; 8019/85056 objects misplaced (9.428%); 19 MiB/s, 298 objects/s recovering 2024-06-27T21:43:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:09 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1883958501' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:09 smithi190 ceph-mon[32247]: osdmap e777: 12 total, 12 up, 9 in 2024-06-27T21:43:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:09 smithi190 ceph-mon[32247]: osdmap e778: 12 total, 12 up, 9 in 2024-06-27T21:43:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:09 smithi192 ceph-mon[32956]: pgmap v1947: 47 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.7 GiB data, 9.7 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 746 op/s; 8019/85056 objects misplaced (9.428%); 19 MiB/s, 298 objects/s recovering 2024-06-27T21:43:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:09 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1883958501' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:09 smithi192 ceph-mon[32956]: osdmap e777: 12 total, 12 up, 9 in 2024-06-27T21:43:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:09 smithi192 ceph-mon[32956]: osdmap e778: 12 total, 12 up, 9 in 2024-06-27T21:43:10.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:10.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:10.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:10.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:10.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:10.578 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:10.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:10.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:10.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:10.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:10.858 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.11. 2024-06-27T21:43:10.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:10 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:43:10.604+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e778 definitely_dead 0 2024-06-27T21:43:10.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:10.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:10.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:11.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:11.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2345481555' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-06-27T21:43:11.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2345481555' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-06-27T21:43:11.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:10 smithi151 ceph-mon[28965]: osdmap e779: 12 total, 12 up, 10 in 2024-06-27T21:43:11.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2345481555' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-06-27T21:43:11.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2345481555' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-06-27T21:43:11.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:10 smithi190 ceph-mon[32247]: osdmap e779: 12 total, 12 up, 10 in 2024-06-27T21:43:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2345481555' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-06-27T21:43:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2345481555' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-06-27T21:43:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:10 smithi192 ceph-mon[32956]: osdmap e779: 12 total, 12 up, 10 in 2024-06-27T21:43:11.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:11.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:11.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:11.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:11.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:11.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:11.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:11.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:11.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:11.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:11.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:11.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:11.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:12.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:12.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:12.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:12.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:11 smithi190 ceph-mon[32247]: pgmap v1950: 47 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 725 op/s; 5887/81012 objects misplaced (7.267%); 29 MiB/s, 457 objects/s recovering 2024-06-27T21:43:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:11 smithi190 ceph-mon[32247]: osdmap e780: 12 total, 12 up, 10 in 2024-06-27T21:43:12.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:11 smithi192 ceph-mon[32956]: pgmap v1950: 47 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 725 op/s; 5887/81012 objects misplaced (7.267%); 29 MiB/s, 457 objects/s recovering 2024-06-27T21:43:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:11 smithi192 ceph-mon[32956]: osdmap e780: 12 total, 12 up, 10 in 2024-06-27T21:43:12.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:12.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:11 smithi151 ceph-mon[28965]: pgmap v1950: 47 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 41 active+clean; 1.6 GiB data, 9.5 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 725 op/s; 5887/81012 objects misplaced (7.267%); 29 MiB/s, 457 objects/s recovering 2024-06-27T21:43:12.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:11 smithi151 ceph-mon[28965]: osdmap e780: 12 total, 12 up, 10 in 2024-06-27T21:43:12.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:12.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:12.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:12.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:12.586 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-06-27T21:43:12.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:12.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:12.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:12.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:12.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:12.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:12.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:13.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:13.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:13.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:13.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:13.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:13.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:13.332 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:13.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:13.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:13.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:13.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:13.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:13.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:13.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:13.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:13.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:13.996 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:13.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:14.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:13 smithi151 ceph-mon[28965]: pgmap v1953: 47 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 40 active+clean; 1.6 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 719 op/s; 2182/76416 objects misplaced (2.855%); 30 MiB/s, 482 objects/s recovering 2024-06-27T21:43:14.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:13 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:43:14.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:13 smithi151 ceph-mon[28965]: osdmap e781: 12 total, 12 up, 10 in 2024-06-27T21:43:14.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:13 smithi151 ceph-mon[28965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:43:14.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:14.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:14.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:14.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:13 smithi190 ceph-mon[32247]: pgmap v1953: 47 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 40 active+clean; 1.6 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 719 op/s; 2182/76416 objects misplaced (2.855%); 30 MiB/s, 482 objects/s recovering 2024-06-27T21:43:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:13 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:43:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:13 smithi190 ceph-mon[32247]: osdmap e781: 12 total, 12 up, 10 in 2024-06-27T21:43:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:13 smithi190 ceph-mon[32247]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:43:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:13 smithi192 ceph-mon[32956]: pgmap v1953: 47 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 40 active+clean; 1.6 GiB data, 9.3 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 719 op/s; 2182/76416 objects misplaced (2.855%); 30 MiB/s, 482 objects/s recovering 2024-06-27T21:43:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:13 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:43:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:13 smithi192 ceph-mon[32956]: osdmap e781: 12 total, 12 up, 10 in 2024-06-27T21:43:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:13 smithi192 ceph-mon[32956]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:43:14.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:14.375 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:14.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:43:14.466 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:43:14.466 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:43:14.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:14.502 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:43:14.502 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:43:14.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:14.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:43:14.545 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:43:14.579 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:43:14.579 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:43:14.610 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:43:14.610 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:43:14.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:14.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:14.639 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:43:14.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:43:14.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:43:14.672 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:43:14.704 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:43:14.705 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:43:14.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:14.734 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:43:14.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:43:14.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:14.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:43:14.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:43:14.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:43:14.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:43:14.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:43:14.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:43:14.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:14.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:14.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:15.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:15.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:15.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:15.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/205764884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:15.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/205764884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:15.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:15.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/205764884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:15.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:15.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:15.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:15.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:15.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:15.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:15.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:15.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:15.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:15.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:15.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:16.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:16.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:16.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:16.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:16.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:15 smithi151 ceph-mon[28965]: pgmap v1955: 47 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 40 active+clean; 1.5 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 457 op/s; 2182/74970 objects misplaced (2.910%); 45 KiB/s, 0 objects/s recovering 2024-06-27T21:43:16.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:15 smithi151 ceph-mon[28965]: osdmap e782: 12 total, 12 up, 10 in 2024-06-27T21:43:16.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:43:16.190 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:43:16.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:15 smithi190 ceph-mon[32247]: pgmap v1955: 47 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 40 active+clean; 1.5 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 457 op/s; 2182/74970 objects misplaced (2.910%); 45 KiB/s, 0 objects/s recovering 2024-06-27T21:43:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:15 smithi190 ceph-mon[32247]: osdmap e782: 12 total, 12 up, 10 in 2024-06-27T21:43:16.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:15 smithi192 ceph-mon[32956]: pgmap v1955: 47 pgs: 2 remapped+peering, 1 peering, 1 active+clean+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 40 active+clean; 1.5 GiB data, 9.2 GiB used, 795 GiB / 805 GiB avail; 0 B/s wr, 457 op/s; 2182/74970 objects misplaced (2.910%); 45 KiB/s, 0 objects/s recovering 2024-06-27T21:43:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:15 smithi192 ceph-mon[32956]: osdmap e782: 12 total, 12 up, 10 in 2024-06-27T21:43:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:16.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:16.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:16.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:16.473 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:16.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:16.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:16.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:16.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:16.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:16.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:16.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:16.959 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:16.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:17.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:17.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:17.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:17.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:43:17.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:17.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:16 smithi151 ceph-mon[28965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:43:17.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:17.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:16 smithi190 ceph-mon[32247]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:43:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:16 smithi192 ceph-mon[32956]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:43:17.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:17.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:17.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:17.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:17.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:43:17.586 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:43:17.587 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-27T21:43:17.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:43:17.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:17.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:17.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:17.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:17.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:17.883 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:17.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:17.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:43:17.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:18.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:18.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:18.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:18.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:17 smithi151 ceph-mon[28965]: pgmap v1957: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:43:18.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:18.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:17 smithi190 ceph-mon[32247]: pgmap v1957: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:43:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:17 smithi192 ceph-mon[32956]: pgmap v1957: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 624 op/s; 4.4 MiB/s, 69 objects/s recovering 2024-06-27T21:43:18.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:18.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:18.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:18.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:18.611 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:18.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:18.651 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:18.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:18.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:18.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:18.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:18.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:18.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:19.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:19.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:19.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:19.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:19.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:19.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:19.384 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:19.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:19.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:19.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:19.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:19.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:19.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:19.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:19.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:19.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:19.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:20.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:20.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:19 smithi151 ceph-mon[28965]: pgmap v1958: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:43:20.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:19 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:43:20.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:20.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:20.247 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:19 smithi190 ceph-mon[32247]: pgmap v1958: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:43:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:19 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:43:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:20.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:19 smithi192 ceph-mon[32956]: pgmap v1958: 47 pgs: 47 active+clean; 1.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 3.7 MiB/s, 59 objects/s recovering 2024-06-27T21:43:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:19 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:43:20.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:20.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:20.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:20.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.542 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:20.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:20.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:20.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:20.690 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:43:20.690 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:43:20.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:20.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:20.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:20.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:20.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:20.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:20.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:21.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:21.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:21.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:21.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:21.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:21.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:21.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:21.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:21.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:21.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:21.600 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:21.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:21.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:21.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:21.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:21.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:21.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:21.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:22.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:22.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:21 smithi151 ceph-mon[28965]: pgmap v1959: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.3 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 549 op/s; 3.0 MiB/s, 47 objects/s recovering 2024-06-27T21:43:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:21 smithi151 ceph-mon[28965]: osdmap e783: 12 total, 12 up, 10 in 2024-06-27T21:43:22.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:22.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:21 smithi190 ceph-mon[32247]: pgmap v1959: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.3 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 549 op/s; 3.0 MiB/s, 47 objects/s recovering 2024-06-27T21:43:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:21 smithi190 ceph-mon[32247]: osdmap e783: 12 total, 12 up, 10 in 2024-06-27T21:43:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:21 smithi192 ceph-mon[32956]: pgmap v1959: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.3 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 549 op/s; 3.0 MiB/s, 47 objects/s recovering 2024-06-27T21:43:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:21 smithi192 ceph-mon[32956]: osdmap e783: 12 total, 12 up, 10 in 2024-06-27T21:43:22.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:22.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:22.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:22.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:22.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:22.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:22.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:22.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:22.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:22.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:22.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:22.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:23.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:23.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:43:23.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:23.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:23.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:23.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:23.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:23.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:23.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:23.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:23.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:23.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:23.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:23.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:23.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:23.922 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:23.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:23.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:23 smithi151 ceph-mon[28965]: pgmap v1961: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.2 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 692 op/s; 2.9 MiB/s, 47 objects/s recovering 2024-06-27T21:43:24.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:24.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:24.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:43:24.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:43:24.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.183 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:24.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:24.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:43:24.198 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:43:24.231 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:43:24.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:43:24.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:43:24.265 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:43:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:23 smithi190 ceph-mon[32247]: pgmap v1961: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.2 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 692 op/s; 2.9 MiB/s, 47 objects/s recovering 2024-06-27T21:43:24.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:24.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:23 smithi192 ceph-mon[32956]: pgmap v1961: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.2 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 692 op/s; 2.9 MiB/s, 47 objects/s recovering 2024-06-27T21:43:24.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:43:24.305 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:43:24.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:24.344 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:43:24.344 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:43:24.375 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:43:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:43:24.402 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:43:24.402 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:43:24.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:24.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:43:24.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:43:24.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:24.465 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:43:24.465 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:43:24.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:43:24.494 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:43:24.521 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:43:24.522 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:43:24.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:24.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:24.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:24.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:24.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:24.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:24.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:24.949 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:43:24.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:24.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:25.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:25.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:24 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3348050219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1513783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:43:25.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:24 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:43:25.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:25.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:24 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:43:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3348050219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1513783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:43:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:24 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:43:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:24 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:43:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3348050219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1513783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:43:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:24 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:43:25.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:25.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:25.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:25.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:25.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:25.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:25.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:25.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:25.912 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:25.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:26.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:26.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:25 smithi151 ceph-mon[28965]: pgmap v1962: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.1 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 665 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:43:26.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:25 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:26.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:25 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:43:26.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:25 smithi151 ceph-mon[28965]: osdmap e784: 12 total, 12 up, 10 in 2024-06-27T21:43:26.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:26.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:26.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:25 smithi190 ceph-mon[32247]: pgmap v1962: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.1 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 665 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:43:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:25 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:25 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:43:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:25 smithi190 ceph-mon[32247]: osdmap e784: 12 total, 12 up, 10 in 2024-06-27T21:43:26.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:25 smithi192 ceph-mon[32956]: pgmap v1962: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.1 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 665 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-06-27T21:43:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:25 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:25 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:43:26.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:25 smithi192 ceph-mon[32956]: osdmap e784: 12 total, 12 up, 10 in 2024-06-27T21:43:26.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:26.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:26.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:26.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:26.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:26.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:26.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:26.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:26.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:26.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:26.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:26.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:26.991 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:27.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:27.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:27.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:27.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:27.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:27.336 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:27.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:27.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:27.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:27.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:27.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:27.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:27.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:27.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:27.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:27.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:43:27.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:27.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:28.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:28.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:28.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:27 smithi151 ceph-mon[28965]: pgmap v1964: 47 pgs: 47 active+clean; 969 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 900 op/s 2024-06-27T21:43:28.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:27 smithi151 ceph-mon[28965]: 7.c scrub starts 2024-06-27T21:43:28.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:27 smithi151 ceph-mon[28965]: 7.c scrub ok 2024-06-27T21:43:28.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:27 smithi190 ceph-mon[32247]: pgmap v1964: 47 pgs: 47 active+clean; 969 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 900 op/s 2024-06-27T21:43:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:27 smithi190 ceph-mon[32247]: 7.c scrub starts 2024-06-27T21:43:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:27 smithi190 ceph-mon[32247]: 7.c scrub ok 2024-06-27T21:43:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:27 smithi192 ceph-mon[32956]: pgmap v1964: 47 pgs: 47 active+clean; 969 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 900 op/s 2024-06-27T21:43:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:27 smithi192 ceph-mon[32956]: 7.c scrub starts 2024-06-27T21:43:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:27 smithi192 ceph-mon[32956]: 7.c scrub ok 2024-06-27T21:43:28.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:28.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:28.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:28.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:28.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:28.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:28.658 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:43:28.659 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:43:28.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:28.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:28.842 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:28.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:28.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:28.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:29.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:29.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:43:29.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:29.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:29.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:29.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:29.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: osdmap e785: 12 total, 12 up, 10 in 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:43:29.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:28 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: osdmap e785: 12 total, 12 up, 10 in 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:43:29.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:28 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:43:29.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:29.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: osdmap e785: 12 total, 12 up, 10 in 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:43:29.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:43:29.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:29.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:29.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:43:29.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:28 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:43:29.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:29.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:29.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:29.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:29.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:29.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:29.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:29.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:29.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:29.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:30.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:30.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:30.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:30.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:29 smithi190 ceph-mon[32247]: pgmap v1966: 47 pgs: 47 active+clean; 939 MiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:43:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:29 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:43:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:29 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:43:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:29 smithi190 ceph-mon[32247]: from='client.26941 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:43:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:29 smithi190 ceph-mon[32247]: osdmap e786: 12 total, 12 up, 10 in 2024-06-27T21:43:30.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:29 smithi192 ceph-mon[32956]: pgmap v1966: 47 pgs: 47 active+clean; 939 MiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:43:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:29 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:43:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:29 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:43:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:29 smithi192 ceph-mon[32956]: from='client.26941 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:43:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:29 smithi192 ceph-mon[32956]: osdmap e786: 12 total, 12 up, 10 in 2024-06-27T21:43:30.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:30.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:29 smithi151 ceph-mon[28965]: pgmap v1966: 47 pgs: 47 active+clean; 939 MiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:43:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:29 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:43:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:29 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:43:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:29 smithi151 ceph-mon[28965]: from='client.26941 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:43:30.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:29 smithi151 ceph-mon[28965]: osdmap e786: 12 total, 12 up, 10 in 2024-06-27T21:43:30.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:30.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:30.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:30.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:30.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:30.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:30.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:30.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:30.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:30.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:30.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:30.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:31.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:31.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:31.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:30 smithi151 ceph-mon[28965]: 7.b scrub starts 2024-06-27T21:43:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:30 smithi151 ceph-mon[28965]: 7.b scrub ok 2024-06-27T21:43:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:30 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:43:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:30 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:43:31.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:30 smithi151 ceph-mon[28965]: 7.6 scrub starts 2024-06-27T21:43:31.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:30 smithi190 ceph-mon[32247]: 7.b scrub starts 2024-06-27T21:43:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:30 smithi190 ceph-mon[32247]: 7.b scrub ok 2024-06-27T21:43:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:30 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:43:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:30 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:43:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:30 smithi190 ceph-mon[32247]: 7.6 scrub starts 2024-06-27T21:43:31.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:30 smithi192 ceph-mon[32956]: 7.b scrub starts 2024-06-27T21:43:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:30 smithi192 ceph-mon[32956]: 7.b scrub ok 2024-06-27T21:43:31.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:30 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:43:31.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:30 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:43:31.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:30 smithi192 ceph-mon[32956]: 7.6 scrub starts 2024-06-27T21:43:31.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:31.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:31.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:31.530 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:31.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:31.655 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1966,"stamp":"2024-06-27T21:43:28.261438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":984613432,"num_objects":15018,"num_object_clones":0,"num_object_copies":45054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191590,"num_write_kb":6611329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5577,"num_bytes_recovered":364923008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163570,"ondisk_log_size":163570,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":9002296,"kb_used_data":3107480,"kb_used_omap":106,"kb_used_meta":5894613,"kb_avail":928449224,"statfs":{"total":959950356480,"available":950732005376,"internally_reserved":0,"allocated":3182059520,"data_stored":3179091353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109496,"internal_metadata":6036083784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-367394816,"num_objects":-5606,"num_object_clones":0,"num_object_copies":-16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.325876"},"pg_stats":[{"pgid":"7.d","version":"784'12270","reported_seq":12519,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:26.071205+0000","last_change":"2024-06-27T21:42:45.305102+0000","last_active":"2024-06-27T21:43:26.071205+0000","last_peered":"2024-06-27T21:43:26.071205+0000","last_clean":"2024-06-27T21:43:26.071205+0000","last_became_active":"2024-06-27T21:39:34.794590+0000","last_became_peered":"2024-06-27T21:39:34.794590+0000","last_unstale":"2024-06-27T21:43:26.071205+0000","last_undegraded":"2024-06-27T21:43:26.071205+0000","last_fullsized":"2024-06-27T21:43:26.071205+0000","mapping_epoch":718,"log_start":"736'2200","ondisk_log_start":"736'2200","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'10576","last_scrub_stamp":"2024-06-27T21:42:45.304931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:45.304931+0000","objects_scrubbed":2627,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:45.346998+0000","scrub_duration":1075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965464,"num_objects":916,"num_object_clones":0,"num_object_copies":2748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12270,"num_write_kb":421889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,9],"acting":[0,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":19,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.952726+0000","last_change":"2024-06-27T21:43:11.868366+0000","last_active":"2024-06-27T21:43:24.952726+0000","last_peered":"2024-06-27T21:43:24.952726+0000","last_clean":"2024-06-27T21:43:24.952726+0000","last_became_active":"2024-06-27T21:43:11.867925+0000","last_became_peered":"2024-06-27T21:43:11.867925+0000","last_unstale":"2024-06-27T21:43:24.952726+0000","last_undegraded":"2024-06-27T21:43:24.952726+0000","last_fullsized":"2024-06-27T21:43:24.952726+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.045022+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:56.646935+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.045022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:04.766571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"784'12359","reported_seq":12620,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:25.344387+0000","last_change":"2024-06-27T21:43:11.912672+0000","last_active":"2024-06-27T21:43:25.344387+0000","last_peered":"2024-06-27T21:43:25.344387+0000","last_clean":"2024-06-27T21:43:25.344387+0000","last_became_active":"2024-06-27T21:43:11.869383+0000","last_became_peered":"2024-06-27T21:43:11.869383+0000","last_unstale":"2024-06-27T21:43:25.344387+0000","last_undegraded":"2024-06-27T21:43:25.344387+0000","last_fullsized":"2024-06-27T21:43:25.344387+0000","mapping_epoch":779,"log_start":"736'2300","ondisk_log_start":"736'2300","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'10275","last_scrub_stamp":"2024-06-27T21:42:29.953958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:29.953958+0000","objects_scrubbed":3106,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:41.536294+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64290816,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12359,"num_write_kb":426880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":693,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.946554+0000","last_change":"2024-06-27T21:41:52.050291+0000","last_active":"2024-06-27T21:43:24.946554+0000","last_peered":"2024-06-27T21:43:24.946554+0000","last_clean":"2024-06-27T21:43:24.946554+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:43:24.946554+0000","last_undegraded":"2024-06-27T21:43:24.946554+0000","last_fullsized":"2024-06-27T21:43:24.946554+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:52.050178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:41:52.050178+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:25.001456+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"784'12323","reported_seq":16441,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.990831+0000","last_change":"2024-06-27T21:43:11.921887+0000","last_active":"2024-06-27T21:43:24.990831+0000","last_peered":"2024-06-27T21:43:24.990831+0000","last_clean":"2024-06-27T21:43:24.990831+0000","last_became_active":"2024-06-27T21:43:11.874242+0000","last_became_peered":"2024-06-27T21:43:11.874242+0000","last_unstale":"2024-06-27T21:43:24.990831+0000","last_undegraded":"2024-06-27T21:43:24.990831+0000","last_fullsized":"2024-06-27T21:43:24.990831+0000","mapping_epoch":779,"log_start":"736'2300","ondisk_log_start":"736'2300","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'7954","last_scrub_stamp":"2024-06-27T21:41:43.404111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:41:43.404111+0000","objects_scrubbed":286,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:25.442113+0000","scrub_duration":1859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67829760,"num_objects":1035,"num_object_clones":0,"num_object_copies":3105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12037,"num_write_kb":418304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":126681088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,4],"acting":[11,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":567,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-06-27T21:43:27.960694+0000","last_change":"2024-06-27T21:43:11.867849+0000","last_active":"2024-06-27T21:43:27.960694+0000","last_peered":"2024-06-27T21:43:27.960694+0000","last_clean":"2024-06-27T21:43:27.960694+0000","last_became_active":"2024-06-27T21:43:11.867436+0000","last_became_peered":"2024-06-27T21:43:11.867436+0000","last_unstale":"2024-06-27T21:43:27.960694+0000","last_undegraded":"2024-06-27T21:43:27.960694+0000","last_fullsized":"2024-06-27T21:43:27.960694+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:52.259029+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:41:52.259029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.8","version":"783'11808","reported_seq":12051,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:23.455982+0000","last_change":"2024-06-27T21:42:32.958982+0000","last_active":"2024-06-27T21:43:23.455982+0000","last_peered":"2024-06-27T21:43:23.455982+0000","last_clean":"2024-06-27T21:43:23.455982+0000","last_became_active":"2024-06-27T21:39:34.792732+0000","last_became_peered":"2024-06-27T21:39:34.792732+0000","last_unstale":"2024-06-27T21:43:23.455982+0000","last_undegraded":"2024-06-27T21:43:23.455982+0000","last_fullsized":"2024-06-27T21:43:23.455982+0000","mapping_epoch":718,"log_start":"735'1800","ondisk_log_start":"735'1800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'10082","last_scrub_stamp":"2024-06-27T21:42:32.958707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:32.958707+0000","objects_scrubbed":2777,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:35.094003+0000","scrub_duration":809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67502080,"num_objects":1030,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":410816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,10],"acting":[2,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":689,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.936408+0000","last_change":"2024-06-27T21:42:32.012085+0000","last_active":"2024-06-27T21:43:20.936408+0000","last_peered":"2024-06-27T21:43:20.936408+0000","last_clean":"2024-06-27T21:43:20.936408+0000","last_became_active":"2024-06-27T21:38:50.794172+0000","last_became_peered":"2024-06-27T21:38:50.794172+0000","last_unstale":"2024-06-27T21:43:20.936408+0000","last_undegraded":"2024-06-27T21:43:20.936408+0000","last_fullsized":"2024-06-27T21:43:20.936408+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:32.011989+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:42:32.011989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:53.095923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"784'14856","reported_seq":15168,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:26.096823+0000","last_change":"2024-06-27T21:43:12.905876+0000","last_active":"2024-06-27T21:43:26.096823+0000","last_peered":"2024-06-27T21:43:26.096823+0000","last_clean":"2024-06-27T21:43:26.096823+0000","last_became_active":"2024-06-27T21:43:12.866451+0000","last_became_peered":"2024-06-27T21:43:12.866451+0000","last_unstale":"2024-06-27T21:43:26.096823+0000","last_undegraded":"2024-06-27T21:43:26.096823+0000","last_fullsized":"2024-06-27T21:43:26.096823+0000","mapping_epoch":780,"log_start":"743'4800","ondisk_log_start":"743'4800","created":718,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'12890","last_scrub_stamp":"2024-06-27T21:42:31.140335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.140335+0000","objects_scrubbed":2884,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:55.984138+0000","scrub_duration":788,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1304,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.943335+0000","last_change":"2024-06-27T21:42:27.807611+0000","last_active":"2024-06-27T21:43:24.943335+0000","last_peered":"2024-06-27T21:43:24.943335+0000","last_clean":"2024-06-27T21:43:24.943335+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:43:24.943335+0000","last_undegraded":"2024-06-27T21:43:24.943335+0000","last_fullsized":"2024-06-27T21:43:24.943335+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:27.807540+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:42:27.807540+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:51.822239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"784'11977","reported_seq":12172,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:26.086547+0000","last_change":"2024-06-27T21:41:57.294236+0000","last_active":"2024-06-27T21:43:26.086547+0000","last_peered":"2024-06-27T21:43:26.086547+0000","last_clean":"2024-06-27T21:43:26.086547+0000","last_became_active":"2024-06-27T21:39:34.795560+0000","last_became_peered":"2024-06-27T21:39:34.795560+0000","last_unstale":"2024-06-27T21:43:26.086547+0000","last_undegraded":"2024-06-27T21:43:26.086547+0000","last_fullsized":"2024-06-27T21:43:26.086547+0000","mapping_epoch":718,"log_start":"735'1900","ondisk_log_start":"735'1900","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"759'8597","last_scrub_stamp":"2024-06-27T21:41:57.294090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:41:57.294090+0000","objects_scrubbed":4280,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:27.942360+0000","scrub_duration":1238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11977,"num_write_kb":411264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":28,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.952070+0000","last_change":"2024-06-27T21:43:24.952070+0000","last_active":"2024-06-27T21:43:24.952070+0000","last_peered":"2024-06-27T21:43:24.952070+0000","last_clean":"2024-06-27T21:43:24.952070+0000","last_became_active":"2024-06-27T21:43:11.867111+0000","last_became_peered":"2024-06-27T21:43:11.867111+0000","last_unstale":"2024-06-27T21:43:24.952070+0000","last_undegraded":"2024-06-27T21:43:24.952070+0000","last_fullsized":"2024-06-27T21:43:24.952070+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.616078+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.616078+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:28.546634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.5","version":"784'12127","reported_seq":12381,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:25.007114+0000","last_change":"2024-06-27T21:43:12.895534+0000","last_active":"2024-06-27T21:43:25.007114+0000","last_peered":"2024-06-27T21:43:25.007114+0000","last_clean":"2024-06-27T21:43:25.007114+0000","last_became_active":"2024-06-27T21:43:12.867649+0000","last_became_peered":"2024-06-27T21:43:12.867649+0000","last_unstale":"2024-06-27T21:43:25.007114+0000","last_undegraded":"2024-06-27T21:43:25.007114+0000","last_fullsized":"2024-06-27T21:43:25.007114+0000","mapping_epoch":780,"log_start":"735'2100","ondisk_log_start":"735'2100","created":718,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'10505","last_scrub_stamp":"2024-06-27T21:42:43.842122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:43.842122+0000","objects_scrubbed":2595,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:58.387402+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63635456,"num_objects":971,"num_object_clones":0,"num_object_copies":2913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":671,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.936365+0000","last_change":"2024-06-27T21:42:33.042246+0000","last_active":"2024-06-27T21:43:20.936365+0000","last_peered":"2024-06-27T21:43:20.936365+0000","last_clean":"2024-06-27T21:43:20.936365+0000","last_became_active":"2024-06-27T21:36:51.808187+0000","last_became_peered":"2024-06-27T21:36:51.808187+0000","last_unstale":"2024-06-27T21:43:20.936365+0000","last_undegraded":"2024-06-27T21:43:20.936365+0000","last_fullsized":"2024-06-27T21:43:20.936365+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:33.042137+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:42:33.042137+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:45.440599+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10],"acting":[4,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"784'12027","reported_seq":12226,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:26.104758+0000","last_change":"2024-06-27T21:41:51.482830+0000","last_active":"2024-06-27T21:43:26.104758+0000","last_peered":"2024-06-27T21:43:26.104758+0000","last_clean":"2024-06-27T21:43:26.104758+0000","last_became_active":"2024-06-27T21:39:34.791789+0000","last_became_peered":"2024-06-27T21:39:34.791789+0000","last_unstale":"2024-06-27T21:43:26.104758+0000","last_undegraded":"2024-06-27T21:43:26.104758+0000","last_fullsized":"2024-06-27T21:43:26.104758+0000","mapping_epoch":718,"log_start":"735'2000","ondisk_log_start":"735'2000","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"758'8222","last_scrub_stamp":"2024-06-27T21:41:51.482646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:41:51.482646+0000","objects_scrubbed":4742,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:27.942398+0000","scrub_duration":1381,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59047936,"num_objects":901,"num_object_clones":0,"num_object_copies":2703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":553,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.945973+0000","last_change":"2024-06-27T21:42:36.322385+0000","last_active":"2024-06-27T21:43:24.945973+0000","last_peered":"2024-06-27T21:43:24.945973+0000","last_clean":"2024-06-27T21:43:24.945973+0000","last_became_active":"2024-06-27T21:38:50.236296+0000","last_became_peered":"2024-06-27T21:38:50.236296+0000","last_unstale":"2024-06-27T21:43:24.945973+0000","last_undegraded":"2024-06-27T21:43:24.945973+0000","last_fullsized":"2024-06-27T21:43:24.945973+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:36.322266+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:09.861737+0000","last_clean_scrub_stamp":"2024-06-27T21:42:36.322266+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:06.261121+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"785'14553","reported_seq":14934,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-06-27T21:43:28.073746+0000","last_change":"2024-06-27T21:42:46.780469+0000","last_active":"2024-06-27T21:43:28.073746+0000","last_peered":"2024-06-27T21:43:28.073746+0000","last_clean":"2024-06-27T21:43:28.073746+0000","last_became_active":"2024-06-27T21:39:34.792965+0000","last_became_peered":"2024-06-27T21:39:34.792965+0000","last_unstale":"2024-06-27T21:43:28.073746+0000","last_undegraded":"2024-06-27T21:43:28.073746+0000","last_fullsized":"2024-06-27T21:43:28.073746+0000","mapping_epoch":718,"log_start":"743'4500","ondisk_log_start":"743'4500","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'12841","last_scrub_stamp":"2024-06-27T21:42:46.780247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:46.780247+0000","objects_scrubbed":2528,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:51.258041+0000","scrub_duration":796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":1094,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.941823+0000","last_change":"2024-06-27T21:42:31.936011+0000","last_active":"2024-06-27T21:43:20.941823+0000","last_peered":"2024-06-27T21:43:20.941823+0000","last_clean":"2024-06-27T21:43:20.941823+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:43:20.941823+0000","last_undegraded":"2024-06-27T21:43:20.941823+0000","last_fullsized":"2024-06-27T21:43:20.941823+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:31.935798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.935798+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:47.337667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"783'11826","reported_seq":15360,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.696426+0000","last_change":"2024-06-27T21:43:11.920558+0000","last_active":"2024-06-27T21:43:24.696426+0000","last_peered":"2024-06-27T21:43:24.696426+0000","last_clean":"2024-06-27T21:43:24.696426+0000","last_became_active":"2024-06-27T21:43:11.876666+0000","last_became_peered":"2024-06-27T21:43:11.876666+0000","last_unstale":"2024-06-27T21:43:24.696426+0000","last_undegraded":"2024-06-27T21:43:24.696426+0000","last_fullsized":"2024-06-27T21:43:24.696426+0000","mapping_epoch":779,"log_start":"735'1800","ondisk_log_start":"735'1800","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'10295","last_scrub_stamp":"2024-06-27T21:42:46.488398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:46.488398+0000","objects_scrubbed":2498,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:02.942738+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11826,"num_write_kb":409216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1640,"num_bytes_recovered":107479040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,11],"acting":[10,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1013,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.941013+0000","last_change":"2024-06-27T21:42:38.868502+0000","last_active":"2024-06-27T21:43:20.941013+0000","last_peered":"2024-06-27T21:43:20.941013+0000","last_clean":"2024-06-27T21:43:20.941013+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:43:20.941013+0000","last_undegraded":"2024-06-27T21:43:20.941013+0000","last_fullsized":"2024-06-27T21:43:20.941013+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:38.868396+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:42:38.868396+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:53.118741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"784'11871","reported_seq":12121,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:25.300383+0000","last_change":"2024-06-27T21:42:31.383661+0000","last_active":"2024-06-27T21:43:25.300383+0000","last_peered":"2024-06-27T21:43:25.300383+0000","last_clean":"2024-06-27T21:43:25.300383+0000","last_became_active":"2024-06-27T21:39:34.922315+0000","last_became_peered":"2024-06-27T21:39:34.922315+0000","last_unstale":"2024-06-27T21:43:25.300383+0000","last_undegraded":"2024-06-27T21:43:25.300383+0000","last_fullsized":"2024-06-27T21:43:25.300383+0000","mapping_epoch":718,"log_start":"735'1800","ondisk_log_start":"735'1800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'9880","last_scrub_stamp":"2024-06-27T21:42:31.383542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.383542+0000","objects_scrubbed":2980,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:53.801743+0000","scrub_duration":828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63111168,"num_objects":963,"num_object_clones":0,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11871,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1099,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.943064+0000","last_change":"2024-06-27T21:42:43.954829+0000","last_active":"2024-06-27T21:43:20.943064+0000","last_peered":"2024-06-27T21:43:20.943064+0000","last_clean":"2024-06-27T21:43:20.943064+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:43:20.943064+0000","last_undegraded":"2024-06-27T21:43:20.943064+0000","last_fullsized":"2024-06-27T21:43:20.943064+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:43.954718+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:42:43.954718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:06.528413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.3","version":"784'11757","reported_seq":11998,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:27.327817+0000","last_change":"2024-06-27T21:42:32.067992+0000","last_active":"2024-06-27T21:43:27.327817+0000","last_peered":"2024-06-27T21:43:27.327817+0000","last_clean":"2024-06-27T21:43:27.327817+0000","last_became_active":"2024-06-27T21:39:34.790415+0000","last_became_peered":"2024-06-27T21:39:34.790415+0000","last_unstale":"2024-06-27T21:43:27.327817+0000","last_undegraded":"2024-06-27T21:43:27.327817+0000","last_fullsized":"2024-06-27T21:43:27.327817+0000","mapping_epoch":718,"log_start":"735'1700","ondisk_log_start":"735'1700","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'9788","last_scrub_stamp":"2024-06-27T21:42:32.067694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:32.067694+0000","objects_scrubbed":2867,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:43.732140+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57475072,"num_objects":877,"num_object_clones":0,"num_object_copies":2631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":404288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1101,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.940403+0000","last_change":"2024-06-27T21:43:11.875271+0000","last_active":"2024-06-27T21:43:20.940403+0000","last_peered":"2024-06-27T21:43:20.940403+0000","last_clean":"2024-06-27T21:43:20.940403+0000","last_became_active":"2024-06-27T21:43:11.874833+0000","last_became_peered":"2024-06-27T21:43:11.874833+0000","last_unstale":"2024-06-27T21:43:20.940403+0000","last_undegraded":"2024-06-27T21:43:20.940403+0000","last_fullsized":"2024-06-27T21:43:20.940403+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:03.934365+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:41:03.934365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:23.100020+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.2","version":"783'11825","reported_seq":12024,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.709696+0000","last_change":"2024-06-27T21:41:40.442713+0000","last_active":"2024-06-27T21:43:24.709696+0000","last_peered":"2024-06-27T21:43:24.709696+0000","last_clean":"2024-06-27T21:43:24.709696+0000","last_became_active":"2024-06-27T21:39:34.922345+0000","last_became_peered":"2024-06-27T21:39:34.922345+0000","last_unstale":"2024-06-27T21:43:24.709696+0000","last_undegraded":"2024-06-27T21:43:24.709696+0000","last_fullsized":"2024-06-27T21:43:24.709696+0000","mapping_epoch":718,"log_start":"735'1800","ondisk_log_start":"735'1800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"757'7626","last_scrub_stamp":"2024-06-27T21:41:40.441402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:41:40.441402+0000","objects_scrubbed":5167,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:28.165422+0000","scrub_duration":1514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61407232,"num_objects":937,"num_object_clones":0,"num_object_copies":2811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11825,"num_write_kb":408384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":19,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.952256+0000","last_change":"2024-06-27T21:43:11.871587+0000","last_active":"2024-06-27T21:43:24.952256+0000","last_peered":"2024-06-27T21:43:24.952256+0000","last_clean":"2024-06-27T21:43:24.952256+0000","last_became_active":"2024-06-27T21:43:11.870685+0000","last_became_peered":"2024-06-27T21:43:11.870685+0000","last_unstale":"2024-06-27T21:43:24.952256+0000","last_undegraded":"2024-06-27T21:43:24.952256+0000","last_fullsized":"2024-06-27T21:43:24.952256+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:32.438808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:32.438808+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:55.843372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"784'12517","reported_seq":16764,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:25.041047+0000","last_change":"2024-06-27T21:43:11.920875+0000","last_active":"2024-06-27T21:43:25.041047+0000","last_peered":"2024-06-27T21:43:25.041047+0000","last_clean":"2024-06-27T21:43:25.041047+0000","last_became_active":"2024-06-27T21:43:11.873181+0000","last_became_peered":"2024-06-27T21:43:11.873181+0000","last_unstale":"2024-06-27T21:43:25.041047+0000","last_undegraded":"2024-06-27T21:43:25.041047+0000","last_fullsized":"2024-06-27T21:43:25.041047+0000","mapping_epoch":779,"log_start":"737'2500","ondisk_log_start":"737'2500","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'5998","last_scrub_stamp":"2024-06-27T21:41:07.995282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:41:07.995282+0000","objects_scrubbed":599,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:25.441766+0000","scrub_duration":1271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62652416,"num_objects":956,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":411968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":129957888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,2],"acting":[11,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":610,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.936360+0000","last_change":"2024-06-27T21:42:34.981060+0000","last_active":"2024-06-27T21:43:20.936360+0000","last_peered":"2024-06-27T21:43:20.936360+0000","last_clean":"2024-06-27T21:43:20.936360+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:43:20.936360+0000","last_undegraded":"2024-06-27T21:43:20.936360+0000","last_fullsized":"2024-06-27T21:43:20.936360+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:34.980956+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:42:34.980956+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:37.650685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"784'12000","reported_seq":12244,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:26.069679+0000","last_change":"2024-06-27T21:42:41.603867+0000","last_active":"2024-06-27T21:43:26.069679+0000","last_peered":"2024-06-27T21:43:26.069679+0000","last_clean":"2024-06-27T21:43:26.069679+0000","last_became_active":"2024-06-27T21:39:34.794604+0000","last_became_peered":"2024-06-27T21:39:34.794604+0000","last_unstale":"2024-06-27T21:43:26.069679+0000","last_undegraded":"2024-06-27T21:43:26.069679+0000","last_fullsized":"2024-06-27T21:43:26.069679+0000","mapping_epoch":718,"log_start":"735'1900","ondisk_log_start":"735'1900","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'10263","last_scrub_stamp":"2024-06-27T21:42:41.603682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:41.603682+0000","objects_scrubbed":2597,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:07.852539+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56360960,"num_objects":860,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":411520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":550,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.944246+0000","last_change":"2024-06-27T21:42:31.538654+0000","last_active":"2024-06-27T21:43:24.944246+0000","last_peered":"2024-06-27T21:43:24.944246+0000","last_clean":"2024-06-27T21:43:24.944246+0000","last_became_active":"2024-06-27T21:38:49.780122+0000","last_became_peered":"2024-06-27T21:38:49.780122+0000","last_unstale":"2024-06-27T21:43:24.944246+0000","last_undegraded":"2024-06-27T21:43:24.944246+0000","last_fullsized":"2024-06-27T21:43:24.944246+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:31.538537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.538537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:00.769936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"783'11722","reported_seq":11967,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:23.460461+0000","last_change":"2024-06-27T21:42:28.933056+0000","last_active":"2024-06-27T21:43:23.460461+0000","last_peered":"2024-06-27T21:43:23.460461+0000","last_clean":"2024-06-27T21:43:23.460461+0000","last_became_active":"2024-06-27T21:39:34.794749+0000","last_became_peered":"2024-06-27T21:39:34.794749+0000","last_unstale":"2024-06-27T21:43:23.460461+0000","last_undegraded":"2024-06-27T21:43:23.460461+0000","last_fullsized":"2024-06-27T21:43:23.460461+0000","mapping_epoch":718,"log_start":"735'1700","ondisk_log_start":"735'1700","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"765'9729","last_scrub_stamp":"2024-06-27T21:42:28.932962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:28.932962+0000","objects_scrubbed":3060,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:40.611965+0000","scrub_duration":784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68550656,"num_objects":1046,"num_object_clones":0,"num_object_copies":3138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11722,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":994,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.938463+0000","last_change":"2024-06-27T21:42:30.174565+0000","last_active":"2024-06-27T21:43:20.938463+0000","last_peered":"2024-06-27T21:43:20.938463+0000","last_clean":"2024-06-27T21:43:20.938463+0000","last_became_active":"2024-06-27T21:36:51.804872+0000","last_became_peered":"2024-06-27T21:36:51.804872+0000","last_unstale":"2024-06-27T21:43:20.938463+0000","last_undegraded":"2024-06-27T21:43:20.938463+0000","last_fullsized":"2024-06-27T21:43:20.938463+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:30.174449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:30.174449+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:51.800312+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,4],"acting":[2,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":232,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.940783+0000","last_change":"2024-06-27T21:42:37.883302+0000","last_active":"2024-06-27T21:43:20.940783+0000","last_peered":"2024-06-27T21:43:20.940783+0000","last_clean":"2024-06-27T21:43:20.940783+0000","last_became_active":"2024-06-27T21:38:49.781023+0000","last_became_peered":"2024-06-27T21:38:49.781023+0000","last_unstale":"2024-06-27T21:43:20.940783+0000","last_undegraded":"2024-06-27T21:43:20.940783+0000","last_fullsized":"2024-06-27T21:43:20.940783+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:37.883204+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:42:37.883204+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:05.961207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":699,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.946540+0000","last_change":"2024-06-27T21:42:31.337909+0000","last_active":"2024-06-27T21:43:24.946540+0000","last_peered":"2024-06-27T21:43:24.946540+0000","last_clean":"2024-06-27T21:43:24.946540+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:43:24.946540+0000","last_undegraded":"2024-06-27T21:43:24.946540+0000","last_fullsized":"2024-06-27T21:43:24.946540+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:31.337766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.337766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:48.121311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":727,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.943305+0000","last_change":"2024-06-27T21:43:11.867679+0000","last_active":"2024-06-27T21:43:24.943305+0000","last_peered":"2024-06-27T21:43:24.943305+0000","last_clean":"2024-06-27T21:43:24.943305+0000","last_became_active":"2024-06-27T21:43:11.867292+0000","last_became_peered":"2024-06-27T21:43:11.867292+0000","last_unstale":"2024-06-27T21:43:24.943305+0000","last_undegraded":"2024-06-27T21:43:24.943305+0000","last_fullsized":"2024-06-27T21:43:24.943305+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:28.788214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:28.788214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:55.177248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":19,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.952081+0000","last_change":"2024-06-27T21:43:11.921500+0000","last_active":"2024-06-27T21:43:24.952081+0000","last_peered":"2024-06-27T21:43:24.952081+0000","last_clean":"2024-06-27T21:43:24.952081+0000","last_became_active":"2024-06-27T21:43:11.874424+0000","last_became_peered":"2024-06-27T21:43:11.874424+0000","last_unstale":"2024-06-27T21:43:24.952081+0000","last_undegraded":"2024-06-27T21:43:24.952081+0000","last_fullsized":"2024-06-27T21:43:24.952081+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.286680+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.286680+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:20.820480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":691,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.938761+0000","last_change":"2024-06-27T21:42:31.123004+0000","last_active":"2024-06-27T21:43:20.938761+0000","last_peered":"2024-06-27T21:43:20.938761+0000","last_clean":"2024-06-27T21:43:20.938761+0000","last_became_active":"2024-06-27T21:38:49.776796+0000","last_became_peered":"2024-06-27T21:38:49.776796+0000","last_unstale":"2024-06-27T21:43:20.938761+0000","last_undegraded":"2024-06-27T21:43:20.938761+0000","last_fullsized":"2024-06-27T21:43:20.938761+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:31.122879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.122879+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:36.623890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":417,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-06-27T21:43:27.960587+0000","last_change":"2024-06-27T21:41:51.299366+0000","last_active":"2024-06-27T21:43:27.960587+0000","last_peered":"2024-06-27T21:43:27.960587+0000","last_clean":"2024-06-27T21:43:27.960587+0000","last_became_active":"2024-06-27T21:38:49.782627+0000","last_became_peered":"2024-06-27T21:38:49.782627+0000","last_unstale":"2024-06-27T21:43:27.960587+0000","last_undegraded":"2024-06-27T21:43:27.960587+0000","last_fullsized":"2024-06-27T21:43:27.960587+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:51.299252+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:41:51.299252+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":430,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.950085+0000","last_change":"2024-06-27T21:43:11.868065+0000","last_active":"2024-06-27T21:43:24.950085+0000","last_peered":"2024-06-27T21:43:24.950085+0000","last_clean":"2024-06-27T21:43:24.950085+0000","last_became_active":"2024-06-27T21:43:11.867640+0000","last_became_peered":"2024-06-27T21:43:11.867640+0000","last_unstale":"2024-06-27T21:43:24.950085+0000","last_undegraded":"2024-06-27T21:43:24.950085+0000","last_fullsized":"2024-06-27T21:43:24.950085+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:44.135940+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:41:44.135940+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:27.942436+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":358,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-06-27T21:43:27.961437+0000","last_change":"2024-06-27T21:41:54.257467+0000","last_active":"2024-06-27T21:43:27.961437+0000","last_peered":"2024-06-27T21:43:27.961437+0000","last_clean":"2024-06-27T21:43:27.961437+0000","last_became_active":"2024-06-27T21:36:51.808305+0000","last_became_peered":"2024-06-27T21:36:51.808305+0000","last_unstale":"2024-06-27T21:43:27.961437+0000","last_undegraded":"2024-06-27T21:43:27.961437+0000","last_fullsized":"2024-06-27T21:43:27.961437+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:54.257395+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:41:54.257395+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":1030,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.938630+0000","last_change":"2024-06-27T21:43:11.922640+0000","last_active":"2024-06-27T21:43:20.938630+0000","last_peered":"2024-06-27T21:43:20.938630+0000","last_clean":"2024-06-27T21:43:20.938630+0000","last_became_active":"2024-06-27T21:43:11.867791+0000","last_became_peered":"2024-06-27T21:43:11.867791+0000","last_unstale":"2024-06-27T21:43:20.938630+0000","last_undegraded":"2024-06-27T21:43:20.938630+0000","last_fullsized":"2024-06-27T21:43:20.938630+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:41.981772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:41:41.981772+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,10],"acting":[2,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":554,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.941739+0000","last_change":"2024-06-27T21:41:39.905343+0000","last_active":"2024-06-27T21:43:20.941739+0000","last_peered":"2024-06-27T21:43:20.941739+0000","last_clean":"2024-06-27T21:43:20.941739+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:43:20.941739+0000","last_undegraded":"2024-06-27T21:43:20.941739+0000","last_fullsized":"2024-06-27T21:43:20.941739+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:39.905223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:41:39.905223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:23.100060+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":630,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-06-27T21:43:20.936236+0000","last_change":"2024-06-27T21:43:11.868081+0000","last_active":"2024-06-27T21:43:20.936236+0000","last_peered":"2024-06-27T21:43:20.936236+0000","last_clean":"2024-06-27T21:43:20.936236+0000","last_became_active":"2024-06-27T21:43:11.867693+0000","last_became_peered":"2024-06-27T21:43:11.867693+0000","last_unstale":"2024-06-27T21:43:20.936236+0000","last_undegraded":"2024-06-27T21:43:20.936236+0000","last_fullsized":"2024-06-27T21:43:20.936236+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:34.044962+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:42:34.044962+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:53.315918+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,8],"acting":[4,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":341,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-06-27T21:43:27.960443+0000","last_change":"2024-06-27T21:41:44.352715+0000","last_active":"2024-06-27T21:43:27.960443+0000","last_peered":"2024-06-27T21:43:27.960443+0000","last_clean":"2024-06-27T21:43:27.960443+0000","last_became_active":"2024-06-27T21:36:51.803452+0000","last_became_peered":"2024-06-27T21:36:51.803452+0000","last_unstale":"2024-06-27T21:43:27.960443+0000","last_undegraded":"2024-06-27T21:43:27.960443+0000","last_fullsized":"2024-06-27T21:43:27.960443+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:44.352585+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:41:44.352585+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":359,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-06-27T21:43:27.960342+0000","last_change":"2024-06-27T21:43:11.872317+0000","last_active":"2024-06-27T21:43:27.960342+0000","last_peered":"2024-06-27T21:43:27.960342+0000","last_clean":"2024-06-27T21:43:27.960342+0000","last_became_active":"2024-06-27T21:43:11.871918+0000","last_became_peered":"2024-06-27T21:43:11.871918+0000","last_unstale":"2024-06-27T21:43:27.960342+0000","last_undegraded":"2024-06-27T21:43:27.960342+0000","last_fullsized":"2024-06-27T21:43:27.960342+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:32.177952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:32.177952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:43.261816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,11],"acting":[8,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":19,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.952996+0000","last_change":"2024-06-27T21:43:11.876604+0000","last_active":"2024-06-27T21:43:24.952996+0000","last_peered":"2024-06-27T21:43:24.952996+0000","last_clean":"2024-06-27T21:43:24.952996+0000","last_became_active":"2024-06-27T21:43:11.874600+0000","last_became_peered":"2024-06-27T21:43:11.874600+0000","last_unstale":"2024-06-27T21:43:24.952996+0000","last_undegraded":"2024-06-27T21:43:24.952996+0000","last_fullsized":"2024-06-27T21:43:24.952996+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.064655+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.064655+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:10.369032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,10],"acting":[11,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":19,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-06-27T21:43:24.952302+0000","last_change":"2024-06-27T21:43:11.874893+0000","last_active":"2024-06-27T21:43:24.952302+0000","last_peered":"2024-06-27T21:43:24.952302+0000","last_clean":"2024-06-27T21:43:24.952302+0000","last_became_active":"2024-06-27T21:43:11.874396+0000","last_became_peered":"2024-06-27T21:43:11.874396+0000","last_unstale":"2024-06-27T21:43:24.952302+0000","last_undegraded":"2024-06-27T21:43:24.952302+0000","last_fullsized":"2024-06-27T21:43:24.952302+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:41:48.657717+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:41:48.657717+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:25.441802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":984023064,"num_objects":15016,"num_object_clones":0,"num_object_copies":45048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15016,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191590,"num_write_kb":6611329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5573,"num_bytes_recovered":364118016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3166908416,"data_stored":3166896200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160718,"ondisk_log_size":160718,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253274,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139644,"kb_used_data":386548,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92605508,"statfs":{"total":95995035648,"available":94828040192,"internally_reserved":0,"allocated":395825152,"data_stored":395529328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514910,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175484,"kb_used_data":450164,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92569668,"statfs":{"total":95995035648,"available":94791340032,"internally_reserved":0,"allocated":460967936,"data_stored":460672112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":742717888},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743843,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1008184,"kb_used_data":377264,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92736968,"statfs":{"total":95995035648,"available":94962655232,"internally_reserved":0,"allocated":386318336,"data_stored":386021992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":646052288},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972776,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621824,"kb_used_data":161656,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93123328,"statfs":{"total":95995035648,"available":95358287872,"internally_reserved":0,"allocated":165535744,"data_stored":165238957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":227633267110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528753,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822404,"kb_used_data":181752,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92922748,"statfs":{"total":95995035648,"available":95152893952,"internally_reserved":0,"allocated":186114048,"data_stored":185817261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":656013766},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757687,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":428996,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93316156,"statfs":{"total":95995035648,"available":95555743744,"internally_reserved":0,"allocated":778240,"data_stored":481453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161392,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":836600,"kb_used_data":416044,"kb_used_omap":92,"kb_used_meta":420451,"kb_avail":92908552,"statfs":{"total":95995035648,"available":95138357248,"internally_reserved":0,"allocated":426029056,"data_stored":425736784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":430541922},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182847,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560644,"kb_used_data":114364,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93184508,"statfs":{"total":95995035648,"available":95420936192,"internally_reserved":0,"allocated":117108736,"data_stored":116807877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":456980935},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411779,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1180476,"kb_used_data":539828,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92564676,"statfs":{"total":95995035648,"available":94786228224,"internally_reserved":0,"allocated":552783872,"data_stored":552488048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738830,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228040,"kb_used_data":479100,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92517112,"statfs":{"total":95995035648,"available":94737522688,"internally_reserved":0,"allocated":490598400,"data_stored":490297541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485560320,"data_stored":485556248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":547618816,"data_stored":547618816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":113315840,"data_stored":113311768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":410648576,"data_stored":410648576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":66846720,"data_stored":66846720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":181469184,"data_stored":181469184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":155975680,"data_stored":155975680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":373755904,"data_stored":373751832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":448004096,"data_stored":448004096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":383713280,"data_stored":383713280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:43:31.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-06-27T21:43:31.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:31.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:31.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:31.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:31.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:31.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:32.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:32.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:32.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:32.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:31 smithi151 ceph-mon[28965]: 7.6 scrub ok 2024-06-27T21:43:32.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:31 smithi151 ceph-mon[28965]: pgmap v1968: 47 pgs: 47 active+clean; 822 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 890 op/s 2024-06-27T21:43:32.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:31 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:43:32.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:31 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:43:32.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:31 smithi190 ceph-mon[32247]: 7.6 scrub ok 2024-06-27T21:43:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:31 smithi190 ceph-mon[32247]: pgmap v1968: 47 pgs: 47 active+clean; 822 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 890 op/s 2024-06-27T21:43:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:31 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:43:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:31 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:43:32.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:31 smithi192 ceph-mon[32956]: 7.6 scrub ok 2024-06-27T21:43:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:31 smithi192 ceph-mon[32956]: pgmap v1968: 47 pgs: 47 active+clean; 822 MiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 890 op/s 2024-06-27T21:43:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:31 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:43:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:31 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:43:32.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.10 553/550 [PushOp(6:0ccb8fbe:::benchmark_data_smithi192_117994_object58114:head, version: 487'3656, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:0ccb8fbe:::benchmark_data_smithi192_117994_object58114:head@487'3656, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:33:35.987293+0000", 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.41332578100003, 2024-06-27T21:43:32.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.700879637, 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987293+0000", 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987293+0000", 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987294+0000", 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000005e-07 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:35.987341+0000", 2024-06-27T21:43:32.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7268000000000002e-05 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686231+0000", 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69889012800000005 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686248+0000", 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6549999999999999e-05 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686347+0000", 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9213e-05 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.686357+0000", 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8109999999999997e-06 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:33:36.688173+0000", 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018159179999999999 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.17 564/550 [PushReplyOp(6:ed0f1f09:::benchmark_data_smithi192_117994_object74567:head)])", 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 555.27814719800006, 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35817200500000002, 2024-06-27T21:43:32.416 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:43:32.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122472+0000", 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999996e-07 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.122479+0000", 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1570000000000003e-06 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.479318+0000", 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35683908399999997 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.479326+0000", 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9289999999999998e-06 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.479393+0000", 2024-06-27T21:43:32.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6268999999999999e-05 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:34:17.480644+0000", 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012509000000000001 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:169458 6.b 6:dedf206c:::benchmark_data_smithi192_117994_object61864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e591)", 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:24.371330+0000", 2024-06-27T21:43:32.420 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 488.02928922699999, 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38649861699999999, 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 169458 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371330+0000", 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371330+0000", 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371332+0000", 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2560000000000001e-06 2024-06-27T21:43:32.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371342+0000", 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0227000000000001e-05 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371372+0000", 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9578999999999999e-05 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371387+0000", 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5063e-05 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371499+0000", 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011266599999999999 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371621+0000", 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121459 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.371696+0000", 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4849000000000001e-05 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.375339+0000", 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003643271 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.375961+0000", 2024-06-27T21:43:32.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00062231000000000005 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.757766+0000", 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38180487400000002 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.757800+0000", 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3738999999999997e-05 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:24.757828+0000", 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8323999999999999e-05 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172061 6.b 6.70fd271b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:46.120027+0000", 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 466.28059164699999, 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72584151200000002, 2024-06-27T21:43:32.427 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172061 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120027+0000", 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120027+0000", 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120029+0000", 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4479999999999999e-06 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120035+0000", 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.883e-06 2024-06-27T21:43:32.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120051+0000", 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5925000000000001e-05 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120059+0000", 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.272e-06 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120126+0000", 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7019000000000003e-05 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.120135+0000", 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6290000000000005e-06 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845829+0000", 2024-06-27T21:43:32.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.72569375300000005 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845869+0000", 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9582999999999997e-05 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172064 6.b 6.ad4cb37b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-06-27T21:43:32.432 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:46.318009+0000", 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 466.08261021800001, 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52793522800000003, 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172064 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318009+0000", 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318009+0000", 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318010+0000", 2024-06-27T21:43:32.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7069999999999999e-06 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318018+0000", 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3560000000000004e-06 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318035+0000", 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7164e-05 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318043+0000", 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4270000000000008e-06 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318118+0000", 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4550999999999998e-05 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.318132+0000", 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3878e-05 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845915+0000", 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52778372299999998 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:46.845944+0000", 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8422e-05 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:172327 6.b 6:db698ede:::benchmark_data_smithi192_117994_object64733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e600)", 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.569266+0000", 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 463.83135257700002, 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.49585006199999998, 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172327 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569266+0000", 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569266+0000", 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569268+0000", 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5039999999999999e-06 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569275+0000", 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9079999999999998e-06 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569290+0000", 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.49e-05 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569297+0000", 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8900000000000007e-06 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569369+0000", 2024-06-27T21:43:32.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1151999999999999e-05 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569514+0000", 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014560699999999999 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,6,7", 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.569673+0000", 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158899 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.573853+0000", 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041802389999999997 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.574777+0000", 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00092400700000000002 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.682001+0000", 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.107223204 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065010+0000", 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38300970000000001 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065050+0000", 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9743999999999998e-05 2024-06-27T21:43:32.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065116+0000", 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6308000000000003e-05 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:172327 6.b e600/599 ondisk, result = 0)", 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:48.575752+0000", 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 463.82486690799999, 2024-06-27T21:43:32.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48936887200000001, 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575752+0000", 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575752+0000", 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575753+0000", 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:48.575760+0000", 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4580000000000004e-06 2024-06-27T21:43:32.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.064940+0000", 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48918015500000001 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.064947+0000", 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7320000000000001e-06 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.064997+0000", 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9684000000000001e-05 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065006+0000", 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3559999999999999e-06 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:49.065121+0000", 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114815 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.449 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.b 601/599 [PushReplyOp(6:d154f043:::benchmark_data_smithi192_117994_object79607:head)])", 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:35:50.309937+0000", 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 462.09068223000003, 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39866802699999998, 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.309937+0000", 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.309937+0000", 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.309937+0000", 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6500000000000001e-07 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.309939+0000", 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.525e-06 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.707501+0000", 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39756198300000001 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.707510+0000", 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9469999999999992e-06 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.708327+0000", 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081657699999999997 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:35:50.708605+0000", 2024-06-27T21:43:32.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027783000000000001 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:179811 6.b 6:d8080e2b:::benchmark_data_smithi192_117994_object72217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:36:42.327564+0000", 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 410.07305508299999, 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42117975800000002, 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.454 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179811 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327564+0000", 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327564+0000", 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327565+0000", 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9299999999999996e-07 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327569+0000", 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7230000000000004e-06 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:32.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327578+0000", 2024-06-27T21:43:32.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2940000000000006e-06 2024-06-27T21:43:32.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327582+0000", 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7400000000000004e-06 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327617+0000", 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4977000000000001e-05 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327711+0000", 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3398000000000003e-05 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.327779+0000", 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8625000000000001e-05 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.331120+0000", 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033409210000000002 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.336890+0000", 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057695919999999996 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.748658+0000", 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.411767736 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.748690+0000", 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2375000000000001e-05 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.748743+0000", 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3483999999999998e-05 2024-06-27T21:43:32.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:179812 6.b 6:ddc36af4:::benchmark_data_smithi192_117994_object72218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:36:42.332669+0000", 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 410.06795010000002, 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43508817100000002, 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 179812 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332669+0000", 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332669+0000", 2024-06-27T21:43:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332669+0000", 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8299999999999996e-07 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332673+0000", 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1370000000000002e-06 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332680+0000", 2024-06-27T21:43:32.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2819999999999999e-06 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332684+0000", 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9260000000000002e-06 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332718+0000", 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4681999999999998e-05 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332800+0000", 2024-06-27T21:43:32.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1848000000000006e-05 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.332876+0000", 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5693999999999997e-05 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.335872+0000", 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029958390000000001 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.507024+0000", 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17115172400000001 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.767637+0000", 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26061383500000002 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.767708+0000", 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0290000000000006e-05 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:36:42.767757+0000", 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9131000000000001e-05 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:181538 6.b 6:d4209804:::benchmark_data_smithi192_117994_object73944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e641)", 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:11.499785+0000", 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 380.90083389199998, 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40085342899999998, 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.468 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 181538 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499785+0000", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499785+0000", 2024-06-27T21:43:32.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499786+0000", 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2720000000000001e-06 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499791+0000", 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9579999999999998e-06 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499803+0000", 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1775e-05 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499809+0000", 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2179999999999996e-06 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499865+0000", 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5958999999999997e-05 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.499999+0000", 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134361 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.500120+0000", 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120928 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.503849+0000", 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037287829999999998 2024-06-27T21:43:32.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.505251+0000", 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014020059999999999 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900577+0000", 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.395326018 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900610+0000", 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3065999999999998e-05 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900638+0000", 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8085e-05 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26020.0:181538 6.b e641/612 ondisk, result = 0)", 2024-06-27T21:43:32.474 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:11.504144+0000", 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 380.89647514900003, 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39649921599999999, 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.504144+0000", 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.504144+0000", 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.504145+0000", 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.47e-07 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.504149+0000", 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3719999999999998e-06 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900529+0000", 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39637966200000002 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900534+0000", 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2129999999999999e-06 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900569+0000", 2024-06-27T21:43:32.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4711e-05 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900574+0000", 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3020000000000002e-06 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:11.900643+0000", 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9009000000000003e-05 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.478 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:181719 6.7 e641/547 6:e45a6468:::benchmark_data_smithi192_117994_object74125:head v 641'10456, mlcod=641'10453)", 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:12.326409+0000", 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 380.07420942700003, 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43340313699999999, 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.326409+0000", 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.326409+0000", 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.326411+0000", 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.362e-06 2024-06-27T21:43:32.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.326420+0000", 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1800000000000002e-06 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.756802+0000", 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43038198599999999 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.756812+0000", 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0597999999999999e-05 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.756889+0000", 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6818999999999998e-05 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.756916+0000", 2024-06-27T21:43:32.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7008e-05 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.759755+0000", 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028384619999999999 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:12.759812+0000", 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7722000000000001e-05 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26020.0:197551 6.1b 6:de283c46:::benchmark_data_smithi192_117994_object89957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:00.307338+0000", 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 332.093280924, 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39521258999999997, 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26020", 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 197551 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.484 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307338+0000", 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307338+0000", 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307339+0000", 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2330000000000001e-06 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307344+0000", 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6659999999999999e-06 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307354+0000", 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0205e-05 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307359+0000", 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9699999999999998e-06 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307418+0000", 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9265999999999998e-05 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.307566+0000", 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014735799999999999 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7,8,10,11", 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.308013+0000", 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044787700000000001 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.309792+0000", 2024-06-27T21:43:32.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017786340000000001 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.310262+0000", 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047006000000000001 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.310313+0000", 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0482000000000002e-05 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.310610+0000", 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029744800000000002 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.316798+0000", 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061876470000000001 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.702439+0000", 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38564165099999997 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.702474+0000", 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4421999999999997e-05 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:00.702550+0000", 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6670999999999998e-05 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.491 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26020.0:203120 6.7 e674/547 6:e778c393:::benchmark_data_smithi192_117994_object95526:head v 674'11087, mlcod=674'11084)", 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:38:08.153250+0000", 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 324.24736861600002, 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.415886798, 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.153250+0000", 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.153250+0000", 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.153251+0000", 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9999999999999995e-07 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.153259+0000", 2024-06-27T21:43:32.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5560000000000002e-06 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.566608+0000", 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.41334936 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.566615+0000", 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3300000000000001e-06 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.566703+0000", 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7206000000000005e-05 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.566725+0000", 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2308999999999999e-05 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.569105+0000", 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002379787 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:38:08.569137+0000", 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2249999999999998e-05 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:113174 7.3 7:c72682ec:::benchmark_data_smithi192_150129_object9869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:41:30.071658+0000", 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.328960654, 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.210938283, 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 113174 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.071658+0000", 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.071658+0000", 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.071659+0000", 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0500000000000004e-07 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.071661+0000", 2024-06-27T21:43:32.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5000000000000002e-06 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.071670+0000", 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6719999999999993e-06 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.071674+0000", 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8199999999999998e-06 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.074325+0000", 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026513539999999999 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.074411+0000", 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5978999999999998e-05 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.074476+0000", 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5327000000000007e-05 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.076813+0000", 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002336608 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.076954+0000", 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000141013 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282531+0000", 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20557736600000001 2024-06-27T21:43:32.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282566+0000", 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4742e-05 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282596+0000", 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0397000000000001e-05 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.505 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26656.0:113174 7.3 e755/718 ondisk, result = 0)", 2024-06-27T21:43:32.505 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:41:30.076122+0000", 2024-06-27T21:43:32.505 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.324496623, 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20647876000000001, 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.076122+0000", 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.076122+0000", 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.076123+0000", 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9499999999999998e-07 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.076125+0000", 2024-06-27T21:43:32.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6259999999999999e-06 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282476+0000", 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206350694 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282480+0000", 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3259999999999997e-06 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282521+0000", 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1050999999999997e-05 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282528+0000", 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6170000000000003e-06 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:41:30.282601+0000", 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3051000000000002e-05 2024-06-27T21:43:32.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:180349 7.3 7:c6ac4f18:::benchmark_data_smithi192_150129_object77044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e776)", 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:43:08.396850+0000", 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.003769009999999, 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.241185292, 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 180349 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396850+0000", 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396850+0000", 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396851+0000", 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4169999999999999e-06 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396858+0000", 2024-06-27T21:43:32.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4330000000000001e-06 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396875+0000", 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7024000000000002e-05 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396882+0000", 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9460000000000001e-06 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.396944+0000", 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2117000000000002e-05 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.397045+0000", 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101439 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-06-27T21:43:32.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.397113+0000", 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7982000000000001e-05 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.401431+0000", 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043176009999999999 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.401536+0000", 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105456 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.637942+0000", 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.236405583 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.637994+0000", 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2429000000000002e-05 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:08.638035+0000", 2024-06-27T21:43:32.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0865000000000003e-05 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:191382 7.3 7:caab44c9:::benchmark_data_smithi192_150129_object88077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e784)", 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:43:25.233287+0000", 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 7.1673317770000002, 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.095179248999999994, 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 191382 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233287+0000", 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233287+0000", 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233289+0000", 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.607e-06 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233294+0000", 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7610000000000004e-06 2024-06-27T21:43:32.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233309+0000", 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4735e-05 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233318+0000", 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9719999999999998e-06 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233383+0000", 2024-06-27T21:43:32.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4700000000000001e-05 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233496+0000", 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113001 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.233569+0000", 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3374999999999995e-05 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.238256+0000", 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046871949999999999 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.238274+0000", 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7748999999999999e-05 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.328397+0000", 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090123291999999994 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.328437+0000", 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9397999999999998e-05 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:25.328466+0000", 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9464000000000001e-05 2024-06-27T21:43:32.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26656.0:195337 7.d e786/718 7:b826c57a:::benchmark_data_smithi192_150129_object92032:head v 786'12472, mlcod=786'12469)", 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:43:30.115733+0000", 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 2.284885455, 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.194360056, 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:32.524 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.115733+0000", 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.115733+0000", 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.115734+0000", 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1319999999999999e-06 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.115743+0000", 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3459999999999994e-06 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.305435+0000", 2024-06-27T21:43:32.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18969244900000001 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.305442+0000", 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8240000000000002e-06 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.305510+0000", 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8102999999999995e-05 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.305541+0000", 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1127000000000002e-05 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.310066+0000", 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045243269999999999 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:30.310093+0000", 2024-06-27T21:43:32.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7747999999999999e-05 2024-06-27T21:43:32.529 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.529 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.529 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.529 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:32.529 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:32.529 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:43:32.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:32.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:32.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:32.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:32.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:32.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:32.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:32.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:32.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:33.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:33.063 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:33.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:33.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: 7.2 scrub starts 2024-06-27T21:43:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: 7.4 scrub starts 2024-06-27T21:43:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: 7.2 scrub ok 2024-06-27T21:43:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:43:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:43:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:33.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: 7.2 scrub starts 2024-06-27T21:43:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: 7.4 scrub starts 2024-06-27T21:43:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: 7.2 scrub ok 2024-06-27T21:43:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:43:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:43:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:33.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:33.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:33.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: 7.2 scrub starts 2024-06-27T21:43:33.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: 7.4 scrub starts 2024-06-27T21:43:33.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: 7.2 scrub ok 2024-06-27T21:43:33.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:43:33.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:43:33.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:33.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:33.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:33.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:33.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:33.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:33.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:33.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:33.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:33.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:33.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:33.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:34.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:34.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:34.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:34.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:33 smithi190 ceph-mon[32247]: 7.4 scrub ok 2024-06-27T21:43:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:33 smithi190 ceph-mon[32247]: pgmap v1969: 47 pgs: 47 active+clean; 714 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 901 op/s 2024-06-27T21:43:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:33 smithi190 ceph-mon[32247]: osdmap e787: 12 total, 12 up, 10 in 2024-06-27T21:43:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:33 smithi192 ceph-mon[32956]: 7.4 scrub ok 2024-06-27T21:43:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:33 smithi192 ceph-mon[32956]: pgmap v1969: 47 pgs: 47 active+clean; 714 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 901 op/s 2024-06-27T21:43:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:33 smithi192 ceph-mon[32956]: osdmap e787: 12 total, 12 up, 10 in 2024-06-27T21:43:34.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:34.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:34.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:33 smithi151 ceph-mon[28965]: 7.4 scrub ok 2024-06-27T21:43:34.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:33 smithi151 ceph-mon[28965]: pgmap v1969: 47 pgs: 47 active+clean; 714 MiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 901 op/s 2024-06-27T21:43:34.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:33 smithi151 ceph-mon[28965]: osdmap e787: 12 total, 12 up, 10 in 2024-06-27T21:43:34.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:34.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:34.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:34.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:34.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:34.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:34.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:34.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:34.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:34.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:35.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:35.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:43:35.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:35.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:35.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:35.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:35.286 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:35.287 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:35.354 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:35.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.408 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:35.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:35.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:35.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:35.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:35.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:35.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:35.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:35.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:35.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:35.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:35.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:35.983 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:43:36.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:35 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:43:36.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:35 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:43:36.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:35 smithi151 ceph-mon[28965]: pgmap v1971: 47 pgs: 47 active+clean; 651 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 768 op/s 2024-06-27T21:43:36.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:35 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:36.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:35 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:36.001 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/509406477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:36.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:36.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:36.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:36.182 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:35 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:43:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:35 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:43:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:35 smithi190 ceph-mon[32247]: pgmap v1971: 47 pgs: 47 active+clean; 651 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 768 op/s 2024-06-27T21:43:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:35 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:35 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/509406477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:36.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:35 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:43:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:35 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:43:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:35 smithi192 ceph-mon[32956]: pgmap v1971: 47 pgs: 47 active+clean; 651 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 768 op/s 2024-06-27T21:43:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:35 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:35 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/509406477' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:36.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:36.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:36.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:36.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:36.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:36.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:36.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:36.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:36.827 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:36.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:36.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:36.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:37.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:36 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:43:37.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:36 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:43:37.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:36 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:43:37.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:36 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:43:37.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/509406477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:37.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:36 smithi190 ceph-mon[32247]: osdmap e788: 12 total, 12 up, 10 in 2024-06-27T21:43:37.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:37.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:37.161 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:43:37.161 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:43:37.161 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:43:37.161 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:43:37.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:37.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:37.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:36 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:43:37.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:36 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:43:37.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:36 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:43:37.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:36 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:43:37.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/509406477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:37.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:36 smithi192 ceph-mon[32956]: osdmap e788: 12 total, 12 up, 10 in 2024-06-27T21:43:37.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:37.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:36 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:43:37.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:36 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:43:37.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:36 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:43:37.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:36 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:43:37.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/509406477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:37.333 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:36 smithi151 ceph-mon[28965]: osdmap e788: 12 total, 12 up, 10 in 2024-06-27T21:43:37.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:37.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:37.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:37.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:37.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:37.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:37.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:37.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:37.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:43:37.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:37.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:37.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:37.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:38.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:38.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:38.192 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:37 smithi151 ceph-mon[28965]: pgmap v1973: 47 pgs: 47 active+clean; 484 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 925 op/s 2024-06-27T21:43:38.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:37 smithi151 ceph-mon[28965]: 7.8 scrub starts 2024-06-27T21:43:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:37 smithi151 ceph-mon[28965]: 7.8 scrub ok 2024-06-27T21:43:38.226 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:38.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:37 smithi190 ceph-mon[32247]: pgmap v1973: 47 pgs: 47 active+clean; 484 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 925 op/s 2024-06-27T21:43:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:37 smithi190 ceph-mon[32247]: 7.8 scrub starts 2024-06-27T21:43:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:37 smithi190 ceph-mon[32247]: 7.8 scrub ok 2024-06-27T21:43:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:37 smithi192 ceph-mon[32956]: pgmap v1973: 47 pgs: 47 active+clean; 484 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 925 op/s 2024-06-27T21:43:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:37 smithi192 ceph-mon[32956]: 7.8 scrub starts 2024-06-27T21:43:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:37 smithi192 ceph-mon[32956]: 7.8 scrub ok 2024-06-27T21:43:38.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:38.392 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:38.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:38.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:38.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:38.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:38.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:38.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:38.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:38.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:38.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:39.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:39.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:39.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:39.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:39.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:38 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:43:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:38 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:43:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:38 smithi190 ceph-mon[32247]: 1.14 deep-scrub starts 2024-06-27T21:43:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:38 smithi190 ceph-mon[32247]: 1.14 deep-scrub ok 2024-06-27T21:43:39.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:38 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:43:39.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:38 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:43:39.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:38 smithi151 ceph-mon[28965]: 1.14 deep-scrub starts 2024-06-27T21:43:39.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:38 smithi151 ceph-mon[28965]: 1.14 deep-scrub ok 2024-06-27T21:43:39.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:38 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:43:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:38 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:43:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:38 smithi192 ceph-mon[32956]: 1.14 deep-scrub starts 2024-06-27T21:43:39.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:38 smithi192 ceph-mon[32956]: 1.14 deep-scrub ok 2024-06-27T21:43:39.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:39.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:39.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:39.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:39.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:39.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:39.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:39.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:43:39.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:39.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:39.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:39.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:39.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:40.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:40.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:40.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:40.179 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:43:40.179 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:43:40.179 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:43:40.179 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:43:40.179 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:43:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:40.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:40.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:39 smithi190 ceph-mon[32247]: pgmap v1974: 47 pgs: 47 active+clean; 457 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 730 op/s 2024-06-27T21:43:40.185 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:39 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:43:40.185 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:39 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:43:40.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:39 smithi192 ceph-mon[32956]: pgmap v1974: 47 pgs: 47 active+clean; 457 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 730 op/s 2024-06-27T21:43:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:39 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:43:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:39 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:43:40.313 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:40.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:39 smithi151 ceph-mon[28965]: pgmap v1974: 47 pgs: 47 active+clean; 457 MiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 730 op/s 2024-06-27T21:43:40.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:39 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:43:40.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:39 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:43:40.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:40.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:40.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:40.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:40.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:40.713 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:40.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:40.832 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:43:40.832 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:43:40.832 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:43:40.832 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:43:40.832 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:43:40.832 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:43:40.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:40.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:43:40.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:43:40.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:43:40.903 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:40.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:43:40.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:43:40.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:40.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:43:40.966 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:43:40.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:40.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:41.005 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:43:41.005 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:43:41.009 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pgp_num to 31 2024-06-27T21:43:41.044 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:43:41.044 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:43:41.076 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:41.085 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:43:41.085 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:43:41.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1166629290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:41.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2943725229' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:43:41.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:41.124 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:43:41.124 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:43:41.142 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:41.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:43:41.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:43:41.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:43:41.199 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:43:41.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:41.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1166629290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2943725229' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:43:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1166629290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2943725229' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:43:41.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:41.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:41.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:41.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:41.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:41.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:41.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:41.763 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:41.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:41.891 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:41.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:41.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:42.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:42.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:42.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:42.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:42 smithi190 ceph-mon[32247]: pgmap v1975: 47 pgs: 47 active+clean; 362 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 703 op/s 2024-06-27T21:43:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:42 smithi190 ceph-mon[32247]: 1.a deep-scrub starts 2024-06-27T21:43:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:42 smithi190 ceph-mon[32247]: 1.a deep-scrub ok 2024-06-27T21:43:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2943725229' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:43:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:42 smithi190 ceph-mon[32247]: osdmap e789: 12 total, 12 up, 10 in 2024-06-27T21:43:42.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:42.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:42 smithi151 ceph-mon[28965]: pgmap v1975: 47 pgs: 47 active+clean; 362 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 703 op/s 2024-06-27T21:43:42.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:42 smithi151 ceph-mon[28965]: 1.a deep-scrub starts 2024-06-27T21:43:42.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:42 smithi151 ceph-mon[28965]: 1.a deep-scrub ok 2024-06-27T21:43:42.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2943725229' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:43:42.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:42 smithi151 ceph-mon[28965]: osdmap e789: 12 total, 12 up, 10 in 2024-06-27T21:43:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:42 smithi192 ceph-mon[32956]: pgmap v1975: 47 pgs: 47 active+clean; 362 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 703 op/s 2024-06-27T21:43:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:42 smithi192 ceph-mon[32956]: 1.a deep-scrub starts 2024-06-27T21:43:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:42 smithi192 ceph-mon[32956]: 1.a deep-scrub ok 2024-06-27T21:43:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2943725229' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:43:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:42 smithi192 ceph-mon[32956]: osdmap e789: 12 total, 12 up, 10 in 2024-06-27T21:43:42.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:42.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:42.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:42.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:42.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:42.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:42.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:42.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:42.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:42.887 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:43:42.887 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:43:42.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:42.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:43.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:43.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:43.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:43.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: 7.a deep-scrub starts 2024-06-27T21:43:43.201 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:43 smithi190 ceph-mon[32247]: pgmap v1977: 47 pgs: 47 active+clean; 256 MiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 789 op/s 2024-06-27T21:43:43.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:43.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:43:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:43:43.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:43:43.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:43:43.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:43:43.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:43:43.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: 7.a deep-scrub starts 2024-06-27T21:43:43.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:43 smithi192 ceph-mon[32956]: pgmap v1977: 47 pgs: 47 active+clean; 256 MiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 789 op/s 2024-06-27T21:43:43.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:43.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: 7.a deep-scrub starts 2024-06-27T21:43:43.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:43 smithi151 ceph-mon[28965]: pgmap v1977: 47 pgs: 47 active+clean; 256 MiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 789 op/s 2024-06-27T21:43:43.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:43.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:43.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:43:43.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:43.671 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:43.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:43.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:43.884 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:43.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:43.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:43.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:44.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:44.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:44.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:44 smithi151 ceph-mon[28965]: 1.2 deep-scrub starts 2024-06-27T21:43:44.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:44 smithi151 ceph-mon[28965]: 1.2 deep-scrub ok 2024-06-27T21:43:44.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:44 smithi151 ceph-mon[28965]: 7.a deep-scrub ok 2024-06-27T21:43:44.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:44 smithi151 ceph-mon[28965]: 7.e scrub starts 2024-06-27T21:43:44.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:44 smithi151 ceph-mon[28965]: 7.e scrub ok 2024-06-27T21:43:44.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:44.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:44 smithi190 ceph-mon[32247]: 1.2 deep-scrub starts 2024-06-27T21:43:44.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:44 smithi190 ceph-mon[32247]: 1.2 deep-scrub ok 2024-06-27T21:43:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:44 smithi190 ceph-mon[32247]: 7.a deep-scrub ok 2024-06-27T21:43:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:44 smithi190 ceph-mon[32247]: 7.e scrub starts 2024-06-27T21:43:44.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:44 smithi190 ceph-mon[32247]: 7.e scrub ok 2024-06-27T21:43:44.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:44 smithi192 ceph-mon[32956]: 1.2 deep-scrub starts 2024-06-27T21:43:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:44 smithi192 ceph-mon[32956]: 1.2 deep-scrub ok 2024-06-27T21:43:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:44 smithi192 ceph-mon[32956]: 7.a deep-scrub ok 2024-06-27T21:43:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:44 smithi192 ceph-mon[32956]: 7.e scrub starts 2024-06-27T21:43:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:44 smithi192 ceph-mon[32956]: 7.e scrub ok 2024-06-27T21:43:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:44.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:44.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:44.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:44.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:44.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:44.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:44.700 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:44.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:44.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:44.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:44.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:44.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:44.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:45.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:45.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:45.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:45.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:45.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:45 smithi190 ceph-mon[32247]: 7.3 scrub starts 2024-06-27T21:43:45.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:45 smithi190 ceph-mon[32247]: 7.3 scrub ok 2024-06-27T21:43:45.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:45 smithi190 ceph-mon[32247]: pgmap v1978: 47 pgs: 47 active+clean; 198 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 680 op/s 2024-06-27T21:43:45.381 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:45 smithi192 ceph-mon[32956]: 7.3 scrub starts 2024-06-27T21:43:45.381 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:45 smithi192 ceph-mon[32956]: 7.3 scrub ok 2024-06-27T21:43:45.381 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:45 smithi192 ceph-mon[32956]: pgmap v1978: 47 pgs: 47 active+clean; 198 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 680 op/s 2024-06-27T21:43:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:45.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:45.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:45 smithi151 ceph-mon[28965]: 7.3 scrub starts 2024-06-27T21:43:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:45 smithi151 ceph-mon[28965]: 7.3 scrub ok 2024-06-27T21:43:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:45 smithi151 ceph-mon[28965]: pgmap v1978: 47 pgs: 47 active+clean; 198 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 680 op/s 2024-06-27T21:43:45.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:45.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:45.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:45.614 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:43:45.614 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:43:45.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:45.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:43:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:45.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:45.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:45.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:45.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:45.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:45.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:45.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:45.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:45.907 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:46.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:46.008 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 103302 objects 2024-06-27T21:43:46.009 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :147.655 2024-06-27T21:43:46.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:46.027 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:43:46.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:46.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:46.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:46.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:46.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2105496182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:43:46.275 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:46 smithi190 ceph-mon[32247]: 1.1 deep-scrub starts 2024-06-27T21:43:46.275 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:46 smithi190 ceph-mon[32247]: 1.1 deep-scrub ok 2024-06-27T21:43:46.275 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:46 smithi190 ceph-mon[32247]: 1.3 deep-scrub starts 2024-06-27T21:43:46.275 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:46 smithi190 ceph-mon[32247]: 1.3 deep-scrub ok 2024-06-27T21:43:46.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:43:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2105496182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:43:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:46 smithi192 ceph-mon[32956]: 1.1 deep-scrub starts 2024-06-27T21:43:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:46 smithi192 ceph-mon[32956]: 1.1 deep-scrub ok 2024-06-27T21:43:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:46 smithi192 ceph-mon[32956]: 1.3 deep-scrub starts 2024-06-27T21:43:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:46 smithi192 ceph-mon[32956]: 1.3 deep-scrub ok 2024-06-27T21:43:46.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:46.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:46.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2105496182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:43:46.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:46 smithi151 ceph-mon[28965]: 1.1 deep-scrub starts 2024-06-27T21:43:46.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:46 smithi151 ceph-mon[28965]: 1.1 deep-scrub ok 2024-06-27T21:43:46.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:46 smithi151 ceph-mon[28965]: 1.3 deep-scrub starts 2024-06-27T21:43:46.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:46 smithi151 ceph-mon[28965]: 1.3 deep-scrub ok 2024-06-27T21:43:46.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:46.521 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:46.563 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:46.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:46.643 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:46.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:46.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:46.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:46.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:46.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:46.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:47.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:47.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:47.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:47.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:47.264 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:47 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2105496182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:43:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:47 smithi192 ceph-mon[32956]: osdmap e790: 12 total, 12 up, 10 in 2024-06-27T21:43:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:47 smithi192 ceph-mon[32956]: pgmap v1980: 47 pgs: 47 active+clean; 32 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 848 op/s 2024-06-27T21:43:47.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:47 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:47.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2105496182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:43:47.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:47 smithi151 ceph-mon[28965]: osdmap e790: 12 total, 12 up, 10 in 2024-06-27T21:43:47.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:47 smithi151 ceph-mon[28965]: pgmap v1980: 47 pgs: 47 active+clean; 32 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 848 op/s 2024-06-27T21:43:47.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:47.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:47.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:47.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:47.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:47 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:47.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2105496182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:43:47.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:47 smithi190 ceph-mon[32247]: osdmap e790: 12 total, 12 up, 10 in 2024-06-27T21:43:47.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:47 smithi190 ceph-mon[32247]: pgmap v1980: 47 pgs: 47 active+clean; 32 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 848 op/s 2024-06-27T21:43:47.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:47.637 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:47.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:47.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:47.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:47.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:47.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:47.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:48.008 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:48.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:48.131 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:48.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:48.254 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:48.328 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:43:48.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:43:48.350 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:43:48.351 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:43:48.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:48.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:43:48.373 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:43:48.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:48.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:43:48.394 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:43:48.416 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:43:48.417 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:43:48.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:43:48.439 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:43:48.453 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:43:48.453 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:43:48.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:43:48.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:43:48.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:48.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:43:48.482 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:43:48.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3664076379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:48.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:48.504 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:43:48.504 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:43:48.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:43:48.526 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:43:48.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:43:48.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:43:48.569 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:43:48.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:43:48.579 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3664076379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:48.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:48.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:48.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:43:48.650 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:43:48.650 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-06-27T21:43:48.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:43:48.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:48.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:48.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3664076379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:43:48.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:48.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:48.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:48.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:48.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:49.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:49.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:43:49.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:49.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:49.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:49.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:49.370 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:49.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:49.439 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:49.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:49.498 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:49.562 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:49 smithi151 ceph-mon[28965]: pgmap v1981: 47 pgs: 47 active+clean; 17 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 690 op/s 2024-06-27T21:43:49.562 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:43:49.563 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:43:49.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:49.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:49.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:49 smithi190 ceph-mon[32247]: pgmap v1981: 47 pgs: 47 active+clean; 17 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 690 op/s 2024-06-27T21:43:49.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:43:49.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:43:49.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:49.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:49.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:43:49.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:49 smithi192 ceph-mon[32956]: pgmap v1981: 47 pgs: 47 active+clean; 17 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 690 op/s 2024-06-27T21:43:49.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:43:49.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:43:49.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:49.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:49.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:49.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:49.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:50.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:50.120 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:50.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:50.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:50.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:50.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:50.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:50.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:50 smithi190 ceph-mon[32247]: osdmap e791: 12 total, 12 up, 10 in 2024-06-27T21:43:50.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:50.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:50.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:50 smithi151 ceph-mon[28965]: osdmap e791: 12 total, 12 up, 10 in 2024-06-27T21:43:50.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:50.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:50.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:50.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:50.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:50 smithi192 ceph-mon[32956]: osdmap e791: 12 total, 12 up, 10 in 2024-06-27T21:43:50.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:50.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:50.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:50.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:50.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:51.020 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:161667 6.a e562/545 6:59b12ea2:::benchmark_data_smithi192_117994_object54073:head v 562'10079, mlcod=562'10077)", 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:08.117515+0000", 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 582.90034561499999, 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33725649000000002, 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.021 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117515+0000", 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117515+0000", 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117516+0000", 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.116e-06 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.117524+0000", 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1570000000000003e-06 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449716+0000", 2024-06-27T21:43:51.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33219267099999999 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449723+0000", 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8909999999999998e-06 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449783+0000", 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9747999999999998e-05 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.449809+0000", 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6126e-05 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.454734+0000", 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049245469999999996 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.454772+0000", 2024-06-27T21:43:51.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8234e-05 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:172585 6.a e601/545 6:508e154a:::benchmark_data_smithi192_117994_object64991:head v 601'10775, mlcod=601'10773)", 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:50.005917+0000", 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 481.01194362400003, 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51760722800000003, 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.026 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005917+0000", 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005917+0000", 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005918+0000", 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0419999999999999e-06 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.005925+0000", 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1470000000000004e-06 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519722+0000", 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51379602700000004 2024-06-27T21:43:51.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519729+0000", 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3320000000000003e-06 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519777+0000", 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7945999999999998e-05 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.519792+0000", 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5098e-05 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.523488+0000", 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036964519999999998 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:50.523525+0000", 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6183999999999998e-05 2024-06-27T21:43:51.030 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_backfill(progress 6.b e 603/603 lb 6:d3c288e1:::benchmark_data_smithi192_117994_object105693:head)", 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 478.21786702499998, 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28496729500000001, 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799994+0000", 2024-06-27T21:43:51.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0100000000000001e-07 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:52.799998+0000", 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.844e-06 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084805+0000", 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28480739300000002 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084813+0000", 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8970000000000008e-06 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084905+0000", 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1991999999999996e-05 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084917+0000", 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2104e-05 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.084961+0000", 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3863999999999999e-05 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e629/526 cost 1049576)", 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:37.697950+0000", 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 433.319910634, 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34383320699999997, 2024-06-27T21:43:51.035 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697950+0000", 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697950+0000", 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697951+0000", 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8400000000000003e-07 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:37.697962+0000", 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0906e-05 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041190+0000", 2024-06-27T21:43:51.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34322828100000002 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041215+0000", 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4195000000000001e-05 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041284+0000", 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9418999999999994e-05 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041783+0000", 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049952199999999997 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e629/526 cost 1049576)", 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:38.041150+0000", 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 432.97671053400001, 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37134862200000002, 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.040 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041150+0000", 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041150+0000", 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041152+0000", 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.311e-06 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.041165+0000", 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3446e-05 2024-06-27T21:43:51.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.411855+0000", 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37068976999999997 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.411861+0000", 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5769999999999999e-06 2024-06-27T21:43:51.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.411926+0000", 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4894999999999994e-05 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:38.412499+0000", 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057262299999999997 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.044 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:179811 6.b e629/612 6:d8080e2b:::benchmark_data_smithi192_117994_object72217:head v 629'10993, mlcod=629'10989)", 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:42.327863+0000", 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 428.689997608, 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42010233600000002, 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327863+0000", 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327863+0000", 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327866+0000", 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4439999999999998e-06 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.327871+0000", 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4099999999999999e-06 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745267+0000", 2024-06-27T21:43:51.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41739600700000001 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745276+0000", 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7309999999999999e-06 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745336+0000", 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9886000000000002e-05 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.745352+0000", 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5999999999999999e-05 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.747915+0000", 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00256347 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.747966+0000", 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0387999999999999e-05 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.049 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:179812 6.b e629/612 6:ddc36af4:::benchmark_data_smithi192_117994_object72218:head v 629'10994, mlcod=629'10990)", 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:42.332970+0000", 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 428.684890806, 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43448742499999998, 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332970+0000", 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332970+0000", 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332971+0000", 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4099999999999997e-07 2024-06-27T21:43:51.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.332977+0000", 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1310000000000003e-06 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761778+0000", 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42880081399999997 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761788+0000", 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0161e-05 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761853+0000", 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5106999999999999e-05 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.761868+0000", 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5330999999999999e-05 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.767414+0000", 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055453519999999999 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:42.767458+0000", 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3687999999999999e-05 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e660/651 cost 1049576)", 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 362.58604928400001, 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38737757699999997, 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431812+0000", 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5499999999999997e-07 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.431826+0000", 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3038e-05 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.818693+0000", 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38686758300000001 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.818699+0000", 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5430000000000001e-06 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.818749+0000", 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0827e-05 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:48.819189+0000", 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043973100000000001 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(6.c e669/651 cost 1049576)", 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:01.097374+0000", 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 349.92048666900001, 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35812077399999998, 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097374+0000", 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097374+0000", 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097375+0000", 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8400000000000003e-07 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.097384+0000", 2024-06-27T21:43:51.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9050000000000003e-06 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.454745+0000", 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35736057199999999 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.454757+0000", 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2823000000000001e-05 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.454898+0000", 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000140276 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:01.455495+0000", 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000597414 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:203244 6.1a e674/654 6:5cb63d6a:::benchmark_data_smithi192_117994_object95650:head v 674'12085, mlcod=674'12082)", 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:08.709463+0000", 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 342.30839778699999, 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26052489099999998, 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709463+0000", 2024-06-27T21:43:51.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709463+0000", 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709464+0000", 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0899999999999999e-06 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.709472+0000", 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5329999999999997e-06 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968046+0000", 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25857432699999999 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968054+0000", 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9500000000000001e-06 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968160+0000", 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105979 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.968187+0000", 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6894e-05 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.969941+0000", 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0017538370000000001 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:08.969988+0000", 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7280999999999997e-05 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.7 684/683 [PushOp(6:e38f712a:::benchmark_data_smithi192_117994_object97546:head, version: 499'5997, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e38f712a:::benchmark_data_smithi192_117994_object97546:head@499'5997, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:23.417763+0000", 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 327.60009811899999, 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33887941300000002, 2024-06-27T21:43:51.067 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417763+0000", 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417763+0000", 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417764+0000", 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.483e-06 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.417825+0000", 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0652000000000001e-05 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754466+0000", 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33664066300000001 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754480+0000", 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4008e-05 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754586+0000", 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010654600000000001 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.754596+0000", 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0299000000000001e-05 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:23.756642+0000", 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002045762 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.7 685/683 [PushOp(6:e2087deb:::benchmark_data_smithi192_117994_object102906:head, version: 502'6332, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e2087deb:::benchmark_data_smithi192_117994_object102906:head@502'6332, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:25.269923+0000", 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 325.74793777399998, 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24171029199999999, 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269923+0000", 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269923+0000", 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269924+0000", 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.269951+0000", 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7229e-05 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509327+0000", 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.239376003 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509337+0000", 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8949999999999993e-06 2024-06-27T21:43:51.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509404+0000", 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6612999999999994e-05 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.509413+0000", 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2560000000000008e-06 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:25.511633+0000", 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022206729999999998 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26020.0:207989 6.a e688/651 6:57913744:::benchmark_data_smithi192_117994_object100395:head v 688'12240, mlcod=688'12237)", 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:29.064520+0000", 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 321.95334104, 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055321000000002, 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.076 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064520+0000", 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064520+0000", 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064521+0000", 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.064525+0000", 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6609999999999999e-06 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362612+0000", 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29808666 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362625+0000", 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2726e-05 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362724+0000", 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9574000000000006e-05 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.362768+0000", 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3479000000000002e-05 2024-06-27T21:43:51.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.365040+0000", 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002272086 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.365073+0000", 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3293000000000001e-05 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e705,698 6:17d4eb95:::benchmark_data_smithi192_117994_object66222:head,607'10400)", 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 281.78876014100001, 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28567531600000001, 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:43:51.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229101+0000", 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0299999999999999e-07 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.229107+0000", 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.806e-06 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514622+0000", 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28551521499999999 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514630+0000", 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9049999999999997e-06 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514724+0000", 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3922e-05 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514731+0000", 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4740000000000002e-06 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.514776+0000", 2024-06-27T21:43:51.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5491000000000002e-05 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e706,698 6:10363e48:::benchmark_data_smithi192_117994_object81348:head,659'10912)", 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:11.552360+0000", 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 279.465500781, 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73961317900000001, 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.085 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552360+0000", 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552360+0000", 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552361+0000", 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6400000000000002e-07 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.552365+0000", 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4750000000000004e-06 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291750+0000", 2024-06-27T21:43:51.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73938458600000001 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291773+0000", 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2864999999999999e-05 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291907+0000", 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013467000000000001 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291915+0000", 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8529999999999992e-06 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.291973+0000", 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8165999999999998e-05 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e706,698 6:12477890:::benchmark_data_smithi192_117994_object82990:head,660'10965)", 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:12.440774+0000", 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 278.57708685799997, 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78605415199999995, 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440774+0000", 2024-06-27T21:43:51.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440774+0000", 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440781+0000", 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5200000000000003e-06 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.440790+0000", 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7459999999999995e-06 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226686+0000", 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78589590899999995 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226693+0000", 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9639999999999997e-06 2024-06-27T21:43:51.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226763+0000", 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9511000000000006e-05 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226770+0000", 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3470000000000001e-06 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.226828+0000", 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8155000000000001e-05 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e706,698 6:11d68537:::benchmark_data_smithi192_117994_object83145:head,660'10975)", 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 277.76291024199998, 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52791518199999998, 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.094 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254951+0000", 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7199999999999999e-07 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.254957+0000", 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8869999999999997e-06 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782724+0000", 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52776675900000003 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:51.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782731+0000", 2024-06-27T21:43:51.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4919999999999996e-06 2024-06-27T21:43:51.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782792+0000", 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0925999999999999e-05 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782802+0000", 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7699999999999996e-06 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.782866+0000", 2024-06-27T21:43:51.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3775999999999994e-05 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.8 e707,698 6:10e09e4c:::benchmark_data_smithi192_117994_object98629:head,686'11458)", 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:14.832210+0000", 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 276.18565084699998, 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.85005609900000001, 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:51.100 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832210+0000", 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832210+0000", 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832211+0000", 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.832214+0000", 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6610000000000002e-06 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682103+0000", 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84988828800000005 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682119+0000", 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6116999999999998e-05 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682203+0000", 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4399000000000003e-05 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682212+0000", 2024-06-27T21:43:51.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1689999999999998e-06 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:15.682266+0000", 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3881000000000001e-05 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:91171 7.2 e746/718 7:41ce5b32:::benchmark_data_smithi192_150129_object91170:head v 746'5663, mlcod=746'5660)", 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:41:01.987004+0000", 2024-06-27T21:43:51.104 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 169.030856661, 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16252371099999999, 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:01.987004+0000", 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:01.987004+0000", 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:01.987005+0000", 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.99e-07 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:01.987034+0000", 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9266000000000001e-05 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:02.146799+0000", 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15976469700000001 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:02.146805+0000", 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7919999999999998e-06 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:02.146873+0000", 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8071000000000006e-05 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:02.146895+0000", 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2127e-05 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:02.149495+0000", 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026001700000000002 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:02.149528+0000", 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2988999999999999e-05 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:128216 7.c e757/718 7:30e47c69:::benchmark_data_smithi192_150129_object24911:head v 757'7944, mlcod=757'7941)", 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:41:47.078221+0000", 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 123.93964035, 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21460217500000001, 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:51.109 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.078221+0000", 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.078221+0000", 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.078221+0000", 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2799999999999996e-07 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.078226+0000", 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1209999999999998e-06 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.289504+0000", 2024-06-27T21:43:51.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21127805599999999 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.289523+0000", 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8445999999999999e-05 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.289633+0000", 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110116 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:51.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.289651+0000", 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8136999999999999e-05 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.292784+0000", 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031331589999999999 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.292823+0000", 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8512e-05 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.113 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.114 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.114 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:51.114 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:51.114 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:43:51.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:51.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:51.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:51.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:43:51.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:51.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:51.382 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-06-27T21:43:51.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-06-27T21:43:51.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:51.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:51.529 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:43:51.530 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:43:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 791, 2024-06-27T21:43:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:43:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:43:51.530 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:43:49.362962+0000", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:23:43.806967+0000", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:43:10.855290+0000", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:43:51.531 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 27, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 7, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:43:51.533 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:43:51.534 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "789", 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "535", 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:43:51.535 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:43:51.536 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:43:51.537 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:43:51.538 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.3399999141693115, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.3399999141693115, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.82999998331069946, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.940000057220459, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.940000057220459, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.79288095235824585, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.79000002145767212, 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 7, 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_5", 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:39:33.628910+0000", 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:43:51.540 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-06-27T21:43:51.541 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "725", 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:43:51.542 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:43:51.543 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:43:51.544 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:43:51.545 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3.75, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.75, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.75, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.809999942779541, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.809999942779541, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.79288095235824585, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.79000002145767212, 2024-06-27T21:43:51.546 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.547 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 780, 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:43:51.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:43:51.553 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-06-27T21:43:51.554 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.1994171142578125, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 662, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:43:51.555 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.558 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:43:51.559 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 779, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.563 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:43:51.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.568 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.3845977783203125, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.569 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 718, 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:43:51.572 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.573 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:43:51.574 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:43:51.575 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.3238067626953125, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 31, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 347, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 362, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 779, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 348, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:43:51.578 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.579 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/3587059633", 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/3587059633", 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/3587059633", 2024-06-27T21:43:51.580 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/3587059633", 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.5222625732421875, 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.9023590087890625, 2024-06-27T21:43:51.581 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 779, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.584 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:43:51.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6059112548828125, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.0889892578125, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 479, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.588 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.589 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:43:51.590 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.591 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.593 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.594 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 686, 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.595 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.596 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:43:51.598 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:43:51.599 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.600 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 779, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.601 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.603 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.604 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:43:51.606 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6501922607421875, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 779, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.608 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.609 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.610 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:43:51.611 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:43:51.612 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:51.613 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.613 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.613 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 779, 2024-06-27T21:43:51.614 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:43:51.615 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.616 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.618 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:43:51.619 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:43:51.620 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 780, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.623 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:43:51.624 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.625 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:43:51.626 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:34.970836+0000", 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:27.723044+0000", 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:43:51.628 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:42:40.448789+0000", 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.629 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:48.777217+0000", 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:14.235028+0000", 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 347 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:42:38.482498+0000", 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.631 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:42:55.173983+0000", 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-06-27T21:43:51.632 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:32.656082+0000", 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:20.619726+0000", 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:06.634566+0000", 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:42:53.434125+0000", 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:43:51.635 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:43:28.324075+0000", 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.0", 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:43:51.636 INFO:teuthology.orchestra.run.smithi151.stdout: 10 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:43:51.637 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:43:51.638 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:43:51.639 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:43:51.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:51.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:51.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:51.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:43:51.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:51.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:51.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:52.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:52.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:51 smithi190 ceph-mon[32247]: pgmap v1983: 47 pgs: 47 active+clean; 577 KiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 511 op/s 2024-06-27T21:43:52.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4031604517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:43:52.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:52.109 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:52.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:52.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:52.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:51 smithi151 ceph-mon[28965]: pgmap v1983: 47 pgs: 47 active+clean; 577 KiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 511 op/s 2024-06-27T21:43:52.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4031604517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:43:52.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:52.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:51 smithi192 ceph-mon[32956]: pgmap v1983: 47 pgs: 47 active+clean; 577 KiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 511 op/s 2024-06-27T21:43:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4031604517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:43:52.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:52.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:52.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:52.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:52.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:52.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:52.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:52.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:52.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:52.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:52.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:53.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:53.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:53.036 INFO:tasks.thrashosds.thrasher:j is {'epoch': 791, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:43:49.362962+0000', 'last_up_change': '2024-06-27T21:23:43.806967+0000', 'last_in_change': '2024-06-27T21:43:10.855290+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '789', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '535', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3399999141693115, 'score_stable': 2.3399999141693115, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.7928809523582458, 'average_primary_affinity': 0.7900000214576721, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-06-27T21:39:33.628910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '725', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.75, 'score_stable': 3.75, 'optimal_score': 0.75, 'raw_score_acting': 2.809999942779541, 'raw_score_stable': 2.809999942779541, 'primary_affinity_weighted': 0.7928809523582458, 'average_primary_affinity': 0.7900000214576721, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.1994171142578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 662, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 0.3845977783203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3238067626953125, 'last_clean_begin': 31, 'last_clean_end': 347, 'up_from': 362, 'up_thru': 779, 'down_at': 348, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 3587059633}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 3587059633}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 3587059633}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 3587059633}]}, 'public_addr': '172.21.15.190:6801/3587059633', 'cluster_addr': '172.21.15.190:6803/3587059633', 'heartbeat_back_addr': '172.21.15.190:6807/3587059633', 'heartbeat_front_addr': '172.21.15.190:6805/3587059633', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 0.5222625732421875, 'primary_affinity': 0.9023590087890625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 0.6059112548828125, 'primary_affinity': 0.0889892578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 479, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 686, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 0.6501922607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:43:34.970836+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-27T21:43:27.723044+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:42:40.448789+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:43:48.777217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:43:14.235028+0000', 'dead_epoch': 347}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:42:38.482498+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:42:55.173983+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-06-27T21:43:32.656082+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:43:20.619726+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:43:06.634566+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:42:53.434125+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:43:28.324075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 9, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:43:53.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:43:53.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:53.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:53.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:53.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:53.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:53.406 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:53.479 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:53.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:53.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:53.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:53.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:53.725 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:43:53.725 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:43:53.725 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:43:53.726 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:43:53.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:53.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:53.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:53.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:53.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:53.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:53 smithi151 ceph-mon[28965]: pgmap v1984: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 394 op/s 2024-06-27T21:43:54.000 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:54.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:54.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:54.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:54.230 INFO:teuthology.orchestra.run.smithi151.stderr:2024-06-27T21:43:54.225+0000 7f4a63fff640 0 --2- 172.21.15.151:0/1494331786 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f4a6c102ca0 0x7f4a6c1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-06-27T21:43:54.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:53 smithi190 ceph-mon[32247]: pgmap v1984: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 394 op/s 2024-06-27T21:43:54.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:53 smithi192 ceph-mon[32956]: pgmap v1984: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 394 op/s 2024-06-27T21:43:54.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:54.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:43:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.406 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:54.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:54.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:54.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:54.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:54.654 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:54.659 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:54.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:54.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:54.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:54.881 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:43:54.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:54.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:54.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:54.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:55.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:55.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:55.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1494331786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:55.133 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:55.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:55.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1494331786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:55.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:55.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1494331786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:43:55.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:55.410 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:55.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:55.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:55.637 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:55.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:55.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:55.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:55.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:55.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:55.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:56.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:43:56.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:55 smithi190 ceph-mon[32247]: pgmap v1985: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 180 op/s 2024-06-27T21:43:56.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:55 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:56.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:55 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1494331786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:56.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:55 smithi190 ceph-mon[32247]: osdmap e792: 12 total, 12 up, 10 in 2024-06-27T21:43:56.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:55 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:43:56.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:43:56.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:55 smithi151 ceph-mon[28965]: pgmap v1985: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 180 op/s 2024-06-27T21:43:56.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:55 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:56.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:55 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1494331786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:56.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:55 smithi151 ceph-mon[28965]: osdmap e792: 12 total, 12 up, 10 in 2024-06-27T21:43:56.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:55 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:43:56.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:56.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:56.233 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:43:56.233 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:43:56.233 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:43:56.233 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:43:56.233 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:43:56.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:56.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:55 smithi192 ceph-mon[32956]: pgmap v1985: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 180 op/s 2024-06-27T21:43:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:55 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:43:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:55 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1494331786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:43:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:55 smithi192 ceph-mon[32956]: osdmap e792: 12 total, 12 up, 10 in 2024-06-27T21:43:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:55 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:43:56.375 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:56.376 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:56.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:56.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:56.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:43:56.532 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:43:56.587 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:43:56.588 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:43:56.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:56.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:56.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:56.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:56.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:43:56.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:56.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:56.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:57.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:57.049 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:57.114 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:43:57.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:57.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:57.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:57.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:57.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:57.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:57.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:57.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:57.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:57.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:57.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:57.795 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:57.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:57.923 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:57.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:57.934 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1987,"stamp":"2024-06-27T21:43:56.267415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206606,"num_write_kb":6611329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5577,"num_bytes_recovered":364923008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163837,"ondisk_log_size":163837,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":6004748,"kb_used_data":26644,"kb_used_omap":106,"kb_used_meta":5977877,"kb_avail":931446772,"statfs":{"total":959950356480,"available":953801494528,"internally_reserved":0,"allocated":27283456,"data_stored":24516065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":109496,"internal_metadata":6121346120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17235968,"num_objects":-263,"num_object_clones":0,"num_object_copies":-789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001723"},"pg_stats":[{"pgid":"7.d","version":"789'13186","reported_seq":13451,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021569+0000","last_change":"2024-06-27T21:42:45.305102+0000","last_active":"2024-06-27T21:43:55.021569+0000","last_peered":"2024-06-27T21:43:55.021569+0000","last_clean":"2024-06-27T21:43:55.021569+0000","last_became_active":"2024-06-27T21:39:34.794590+0000","last_became_peered":"2024-06-27T21:39:34.794590+0000","last_unstale":"2024-06-27T21:43:55.021569+0000","last_undegraded":"2024-06-27T21:43:55.021569+0000","last_fullsized":"2024-06-27T21:43:55.021569+0000","mapping_epoch":718,"log_start":"738'3100","ondisk_log_start":"738'3100","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'10576","last_scrub_stamp":"2024-06-27T21:42:45.304931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:45.304931+0000","objects_scrubbed":2627,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13186,"num_write_kb":421889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,9],"acting":[0,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":34,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157961+0000","last_change":"2024-06-27T21:43:11.868366+0000","last_active":"2024-06-27T21:43:50.157961+0000","last_peered":"2024-06-27T21:43:50.157961+0000","last_clean":"2024-06-27T21:43:50.157961+0000","last_became_active":"2024-06-27T21:43:11.867925+0000","last_became_peered":"2024-06-27T21:43:11.867925+0000","last_unstale":"2024-06-27T21:43:50.157961+0000","last_undegraded":"2024-06-27T21:43:50.157961+0000","last_fullsized":"2024-06-27T21:43:50.157961+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.045022+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:56.646935+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.045022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:18.903693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.a","version":"789'13492","reported_seq":13787,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:54.885359+0000","last_change":"2024-06-27T21:43:43.075310+0000","last_active":"2024-06-27T21:43:54.885359+0000","last_peered":"2024-06-27T21:43:54.885359+0000","last_clean":"2024-06-27T21:43:54.885359+0000","last_became_active":"2024-06-27T21:43:11.869383+0000","last_became_peered":"2024-06-27T21:43:11.869383+0000","last_unstale":"2024-06-27T21:43:54.885359+0000","last_undegraded":"2024-06-27T21:43:54.885359+0000","last_fullsized":"2024-06-27T21:43:54.885359+0000","mapping_epoch":779,"log_start":"739'3401","ondisk_log_start":"739'3401","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"789'13370","last_scrub_stamp":"2024-06-27T21:43:43.075134+0000","last_deep_scrub":"789'13370","last_deep_scrub_stamp":"2024-06-27T21:43:43.075134+0000","last_clean_scrub_stamp":"2024-06-27T21:43:43.075134+0000","objects_scrubbed":152,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:10.861146+0000","scrub_duration":1324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13340,"num_write_kb":426880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":718,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021490+0000","last_change":"2024-06-27T21:43:35.233857+0000","last_active":"2024-06-27T21:43:55.021490+0000","last_peered":"2024-06-27T21:43:55.021490+0000","last_clean":"2024-06-27T21:43:55.021490+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:43:55.021490+0000","last_undegraded":"2024-06-27T21:43:55.021490+0000","last_fullsized":"2024-06-27T21:43:55.021490+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:35.233787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:35.233787+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:02.828491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"789'13358","reported_seq":17505,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158045+0000","last_change":"2024-06-27T21:43:29.801743+0000","last_active":"2024-06-27T21:43:50.158045+0000","last_peered":"2024-06-27T21:43:50.158045+0000","last_clean":"2024-06-27T21:43:50.158045+0000","last_became_active":"2024-06-27T21:43:11.874242+0000","last_became_peered":"2024-06-27T21:43:11.874242+0000","last_unstale":"2024-06-27T21:43:50.158045+0000","last_undegraded":"2024-06-27T21:43:50.158045+0000","last_fullsized":"2024-06-27T21:43:50.158045+0000","mapping_epoch":779,"log_start":"739'3300","ondisk_log_start":"739'3300","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"786'12615","last_scrub_stamp":"2024-06-27T21:43:29.801398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:29.801398+0000","objects_scrubbed":748,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:52.607891+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13072,"num_write_kb":418304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":126681088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,4],"acting":[11,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":588,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.740573+0000","last_change":"2024-06-27T21:43:28.309291+0000","last_active":"2024-06-27T21:43:49.740573+0000","last_peered":"2024-06-27T21:43:49.740573+0000","last_clean":"2024-06-27T21:43:49.740573+0000","last_became_active":"2024-06-27T21:43:11.867436+0000","last_became_peered":"2024-06-27T21:43:11.867436+0000","last_unstale":"2024-06-27T21:43:49.740573+0000","last_undegraded":"2024-06-27T21:43:49.740573+0000","last_fullsized":"2024-06-27T21:43:49.740573+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:28.309166+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:43:28.309166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:34.923844+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,11],"acting":[8,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.8","version":"789'12838","reported_seq":13109,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369613+0000","last_change":"2024-06-27T21:43:37.394118+0000","last_active":"2024-06-27T21:43:49.369613+0000","last_peered":"2024-06-27T21:43:49.369613+0000","last_clean":"2024-06-27T21:43:49.369613+0000","last_became_active":"2024-06-27T21:39:34.792732+0000","last_became_peered":"2024-06-27T21:39:34.792732+0000","last_unstale":"2024-06-27T21:43:49.369613+0000","last_undegraded":"2024-06-27T21:43:49.369613+0000","last_fullsized":"2024-06-27T21:43:49.369613+0000","mapping_epoch":718,"log_start":"737'2800","ondisk_log_start":"737'2800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"788'12451","last_scrub_stamp":"2024-06-27T21:43:37.393892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:37.393892+0000","objects_scrubbed":388,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:05.025098+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12838,"num_write_kb":410816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,10],"acting":[2,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":706,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369446+0000","last_change":"2024-06-27T21:42:32.012085+0000","last_active":"2024-06-27T21:43:49.369446+0000","last_peered":"2024-06-27T21:43:49.369446+0000","last_clean":"2024-06-27T21:43:49.369446+0000","last_became_active":"2024-06-27T21:38:50.794172+0000","last_became_peered":"2024-06-27T21:38:50.794172+0000","last_unstale":"2024-06-27T21:43:49.369446+0000","last_undegraded":"2024-06-27T21:43:49.369446+0000","last_fullsized":"2024-06-27T21:43:49.369446+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:32.011989+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:42:32.011989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:49.807998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8],"acting":[4,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"789'15761","reported_seq":16089,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021232+0000","last_change":"2024-06-27T21:43:12.905876+0000","last_active":"2024-06-27T21:43:55.021232+0000","last_peered":"2024-06-27T21:43:55.021232+0000","last_clean":"2024-06-27T21:43:55.021232+0000","last_became_active":"2024-06-27T21:43:12.866451+0000","last_became_peered":"2024-06-27T21:43:12.866451+0000","last_unstale":"2024-06-27T21:43:55.021232+0000","last_undegraded":"2024-06-27T21:43:55.021232+0000","last_fullsized":"2024-06-27T21:43:55.021232+0000","mapping_epoch":780,"log_start":"746'5700","ondisk_log_start":"746'5700","created":718,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'12890","last_scrub_stamp":"2024-06-27T21:42:31.140335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.140335+0000","objects_scrubbed":2884,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:55.984138+0000","scrub_duration":788,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12900,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1327,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.778602+0000","last_change":"2024-06-27T21:43:41.736561+0000","last_active":"2024-06-27T21:43:49.778602+0000","last_peered":"2024-06-27T21:43:49.778602+0000","last_clean":"2024-06-27T21:43:49.778602+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:43:49.778602+0000","last_undegraded":"2024-06-27T21:43:49.778602+0000","last_fullsized":"2024-06-27T21:43:49.778602+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:41.736465+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:43:41.736465+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:53.976631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"789'12852","reported_seq":13078,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021459+0000","last_change":"2024-06-27T21:43:30.243502+0000","last_active":"2024-06-27T21:43:55.021459+0000","last_peered":"2024-06-27T21:43:55.021459+0000","last_clean":"2024-06-27T21:43:55.021459+0000","last_became_active":"2024-06-27T21:39:34.795560+0000","last_became_peered":"2024-06-27T21:39:34.795560+0000","last_unstale":"2024-06-27T21:43:55.021459+0000","last_undegraded":"2024-06-27T21:43:55.021459+0000","last_fullsized":"2024-06-27T21:43:55.021459+0000","mapping_epoch":718,"log_start":"737'2800","ondisk_log_start":"737'2800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"786'12177","last_scrub_stamp":"2024-06-27T21:43:30.243225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:30.243225+0000","objects_scrubbed":676,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:53.381960+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12852,"num_write_kb":411264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,4],"acting":[0,10,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":51,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157689+0000","last_change":"2024-06-27T21:43:28.615651+0000","last_active":"2024-06-27T21:43:50.157689+0000","last_peered":"2024-06-27T21:43:50.157689+0000","last_clean":"2024-06-27T21:43:50.157689+0000","last_became_active":"2024-06-27T21:43:11.867111+0000","last_became_peered":"2024-06-27T21:43:11.867111+0000","last_unstale":"2024-06-27T21:43:50.157689+0000","last_undegraded":"2024-06-27T21:43:50.157689+0000","last_fullsized":"2024-06-27T21:43:50.157689+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:28.615535+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:43:28.615535+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:29.068942+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.5","version":"789'13098","reported_seq":13367,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158263+0000","last_change":"2024-06-27T21:43:12.895534+0000","last_active":"2024-06-27T21:43:50.158263+0000","last_peered":"2024-06-27T21:43:50.158263+0000","last_clean":"2024-06-27T21:43:50.158263+0000","last_became_active":"2024-06-27T21:43:12.867649+0000","last_became_peered":"2024-06-27T21:43:12.867649+0000","last_unstale":"2024-06-27T21:43:50.158263+0000","last_undegraded":"2024-06-27T21:43:50.158263+0000","last_fullsized":"2024-06-27T21:43:50.158263+0000","mapping_epoch":780,"log_start":"737'3000","ondisk_log_start":"737'3000","created":718,"last_epoch_clean":781,"parent":"0.0","parent_split_bits":0,"last_scrub":"769'10505","last_scrub_stamp":"2024-06-27T21:42:43.842122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:43.842122+0000","objects_scrubbed":2595,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:58.387402+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13098,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":696,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369494+0000","last_change":"2024-06-27T21:43:45.673074+0000","last_active":"2024-06-27T21:43:49.369494+0000","last_peered":"2024-06-27T21:43:49.369494+0000","last_clean":"2024-06-27T21:43:49.369494+0000","last_became_active":"2024-06-27T21:36:51.808187+0000","last_became_peered":"2024-06-27T21:36:51.808187+0000","last_unstale":"2024-06-27T21:43:49.369494+0000","last_undegraded":"2024-06-27T21:43:49.369494+0000","last_fullsized":"2024-06-27T21:43:49.369494+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_clean_scrub_stamp":"2024-06-27T21:43:45.673000+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:15.485022+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10],"acting":[4,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"789'12928","reported_seq":13157,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021367+0000","last_change":"2024-06-27T21:43:32.245642+0000","last_active":"2024-06-27T21:43:55.021367+0000","last_peered":"2024-06-27T21:43:55.021367+0000","last_clean":"2024-06-27T21:43:55.021367+0000","last_became_active":"2024-06-27T21:39:34.791789+0000","last_became_peered":"2024-06-27T21:39:34.791789+0000","last_unstale":"2024-06-27T21:43:55.021367+0000","last_undegraded":"2024-06-27T21:43:55.021367+0000","last_fullsized":"2024-06-27T21:43:55.021367+0000","mapping_epoch":718,"log_start":"737'2900","ondisk_log_start":"737'2900","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"786'12339","last_scrub_stamp":"2024-06-27T21:43:32.245339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:32.245339+0000","objects_scrubbed":590,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:40.603634+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12928,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":578,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021410+0000","last_change":"2024-06-27T21:43:42.388740+0000","last_active":"2024-06-27T21:43:55.021410+0000","last_peered":"2024-06-27T21:43:55.021410+0000","last_clean":"2024-06-27T21:43:55.021410+0000","last_became_active":"2024-06-27T21:38:50.236296+0000","last_became_peered":"2024-06-27T21:38:50.236296+0000","last_unstale":"2024-06-27T21:43:55.021410+0000","last_undegraded":"2024-06-27T21:43:55.021410+0000","last_fullsized":"2024-06-27T21:43:55.021410+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:43:42.388603+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:05.761518+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"789'15354","reported_seq":15748,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.740832+0000","last_change":"2024-06-27T21:42:46.780469+0000","last_active":"2024-06-27T21:43:49.740832+0000","last_peered":"2024-06-27T21:43:49.740832+0000","last_clean":"2024-06-27T21:43:49.740832+0000","last_became_active":"2024-06-27T21:39:34.792965+0000","last_became_peered":"2024-06-27T21:39:34.792965+0000","last_unstale":"2024-06-27T21:43:49.740832+0000","last_undegraded":"2024-06-27T21:43:49.740832+0000","last_fullsized":"2024-06-27T21:43:49.740832+0000","mapping_epoch":718,"log_start":"744'5300","ondisk_log_start":"744'5300","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'12841","last_scrub_stamp":"2024-06-27T21:42:46.780247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:46.780247+0000","objects_scrubbed":2528,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12872,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":1119,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157707+0000","last_change":"2024-06-27T21:43:45.083175+0000","last_active":"2024-06-27T21:43:50.157707+0000","last_peered":"2024-06-27T21:43:50.157707+0000","last_clean":"2024-06-27T21:43:50.157707+0000","last_became_active":"2024-06-27T21:31:22.939580+0000","last_became_peered":"2024-06-27T21:31:22.939580+0000","last_unstale":"2024-06-27T21:43:50.157707+0000","last_undegraded":"2024-06-27T21:43:50.157707+0000","last_fullsized":"2024-06-27T21:43:50.157707+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:45.083066+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.083066+0000","last_clean_scrub_stamp":"2024-06-27T21:43:45.083066+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:07.037139+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"789'12788","reported_seq":16338,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157918+0000","last_change":"2024-06-27T21:43:11.920558+0000","last_active":"2024-06-27T21:43:50.157918+0000","last_peered":"2024-06-27T21:43:50.157918+0000","last_clean":"2024-06-27T21:43:50.157918+0000","last_became_active":"2024-06-27T21:43:11.876666+0000","last_became_peered":"2024-06-27T21:43:11.876666+0000","last_unstale":"2024-06-27T21:43:50.157918+0000","last_undegraded":"2024-06-27T21:43:50.157918+0000","last_fullsized":"2024-06-27T21:43:50.157918+0000","mapping_epoch":779,"log_start":"737'2700","ondisk_log_start":"737'2700","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'10295","last_scrub_stamp":"2024-06-27T21:42:46.488398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:46.488398+0000","objects_scrubbed":2498,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:02.942738+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12788,"num_write_kb":409216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1640,"num_bytes_recovered":107479040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,11],"acting":[10,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1031,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157832+0000","last_change":"2024-06-27T21:42:38.868502+0000","last_active":"2024-06-27T21:43:50.157832+0000","last_peered":"2024-06-27T21:43:50.157832+0000","last_clean":"2024-06-27T21:43:50.157832+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:43:50.157832+0000","last_undegraded":"2024-06-27T21:43:50.157832+0000","last_fullsized":"2024-06-27T21:43:50.157832+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:38.868396+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:42:38.868396+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:59.494271+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"789'12834","reported_seq":13101,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:54.885356+0000","last_change":"2024-06-27T21:42:31.383661+0000","last_active":"2024-06-27T21:43:54.885356+0000","last_peered":"2024-06-27T21:43:54.885356+0000","last_clean":"2024-06-27T21:43:54.885356+0000","last_became_active":"2024-06-27T21:39:34.922315+0000","last_became_peered":"2024-06-27T21:39:34.922315+0000","last_unstale":"2024-06-27T21:43:54.885356+0000","last_undegraded":"2024-06-27T21:43:54.885356+0000","last_fullsized":"2024-06-27T21:43:54.885356+0000","mapping_epoch":718,"log_start":"737'2800","ondisk_log_start":"737'2800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"766'9880","last_scrub_stamp":"2024-06-27T21:42:31.383542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.383542+0000","objects_scrubbed":2980,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:58.964672+0000","scrub_duration":828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12834,"num_write_kb":410688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1116,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157822+0000","last_change":"2024-06-27T21:42:43.954829+0000","last_active":"2024-06-27T21:43:50.157822+0000","last_peered":"2024-06-27T21:43:50.157822+0000","last_clean":"2024-06-27T21:43:50.157822+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:43:50.157822+0000","last_undegraded":"2024-06-27T21:43:50.157822+0000","last_fullsized":"2024-06-27T21:43:50.157822+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:43.954718+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:42:43.954718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:09.893759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.3","version":"789'12634","reported_seq":12898,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.047997+0000","last_change":"2024-06-27T21:43:44.084499+0000","last_active":"2024-06-27T21:43:50.047997+0000","last_peered":"2024-06-27T21:43:50.047997+0000","last_clean":"2024-06-27T21:43:50.047997+0000","last_became_active":"2024-06-27T21:39:34.790415+0000","last_became_peered":"2024-06-27T21:39:34.790415+0000","last_unstale":"2024-06-27T21:43:50.047997+0000","last_undegraded":"2024-06-27T21:43:50.047997+0000","last_fullsized":"2024-06-27T21:43:50.047997+0000","mapping_epoch":718,"log_start":"737'2600","ondisk_log_start":"737'2600","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"789'12573","last_scrub_stamp":"2024-06-27T21:43:44.084258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:44.084258+0000","objects_scrubbed":64,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:49.125916+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12634,"num_write_kb":404288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1126,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157639+0000","last_change":"2024-06-27T21:43:30.085069+0000","last_active":"2024-06-27T21:43:50.157639+0000","last_peered":"2024-06-27T21:43:50.157639+0000","last_clean":"2024-06-27T21:43:50.157639+0000","last_became_active":"2024-06-27T21:43:11.874833+0000","last_became_peered":"2024-06-27T21:43:11.874833+0000","last_unstale":"2024-06-27T21:43:50.157639+0000","last_undegraded":"2024-06-27T21:43:50.157639+0000","last_fullsized":"2024-06-27T21:43:50.157639+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:30.084942+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:43:30.084942+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:54.167876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.2","version":"789'12762","reported_seq":12992,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157705+0000","last_change":"2024-06-27T21:43:32.238195+0000","last_active":"2024-06-27T21:43:50.157705+0000","last_peered":"2024-06-27T21:43:50.157705+0000","last_clean":"2024-06-27T21:43:50.157705+0000","last_became_active":"2024-06-27T21:39:34.922345+0000","last_became_peered":"2024-06-27T21:39:34.922345+0000","last_unstale":"2024-06-27T21:43:50.157705+0000","last_undegraded":"2024-06-27T21:43:50.157705+0000","last_fullsized":"2024-06-27T21:43:50.157705+0000","mapping_epoch":718,"log_start":"737'2700","ondisk_log_start":"737'2700","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"786'12200","last_scrub_stamp":"2024-06-27T21:43:32.237832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:32.237832+0000","objects_scrubbed":564,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:51.352135+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12762,"num_write_kb":408384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":42,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158030+0000","last_change":"2024-06-27T21:43:41.489152+0000","last_active":"2024-06-27T21:43:50.158030+0000","last_peered":"2024-06-27T21:43:50.158030+0000","last_clean":"2024-06-27T21:43:50.158030+0000","last_became_active":"2024-06-27T21:43:11.870685+0000","last_became_peered":"2024-06-27T21:43:11.870685+0000","last_unstale":"2024-06-27T21:43:50.158030+0000","last_undegraded":"2024-06-27T21:43:50.158030+0000","last_fullsized":"2024-06-27T21:43:50.158030+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:41.488994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:41.488994+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:56.651709+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.c","version":"789'13473","reported_seq":17752,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158103+0000","last_change":"2024-06-27T21:43:27.028071+0000","last_active":"2024-06-27T21:43:50.158103+0000","last_peered":"2024-06-27T21:43:50.158103+0000","last_clean":"2024-06-27T21:43:50.158103+0000","last_became_active":"2024-06-27T21:43:11.873181+0000","last_became_peered":"2024-06-27T21:43:11.873181+0000","last_unstale":"2024-06-27T21:43:50.158103+0000","last_undegraded":"2024-06-27T21:43:50.158103+0000","last_fullsized":"2024-06-27T21:43:50.158103+0000","mapping_epoch":779,"log_start":"741'3400","ondisk_log_start":"741'3400","created":718,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"784'12628","last_scrub_stamp":"2024-06-27T21:43:27.027844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:27.027844+0000","objects_scrubbed":852,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:35.181251+0000","scrub_duration":432,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12874,"num_write_kb":411968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":129957888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,2],"acting":[11,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":635,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369401+0000","last_change":"2024-06-27T21:43:40.622401+0000","last_active":"2024-06-27T21:43:49.369401+0000","last_peered":"2024-06-27T21:43:49.369401+0000","last_clean":"2024-06-27T21:43:49.369401+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:43:49.369401+0000","last_undegraded":"2024-06-27T21:43:49.369401+0000","last_fullsized":"2024-06-27T21:43:49.369401+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:43:40.622325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:56.174741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"789'12860","reported_seq":13120,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021511+0000","last_change":"2024-06-27T21:42:41.603867+0000","last_active":"2024-06-27T21:43:55.021511+0000","last_peered":"2024-06-27T21:43:55.021511+0000","last_clean":"2024-06-27T21:43:55.021511+0000","last_became_active":"2024-06-27T21:39:34.794604+0000","last_became_peered":"2024-06-27T21:39:34.794604+0000","last_unstale":"2024-06-27T21:43:55.021511+0000","last_undegraded":"2024-06-27T21:43:55.021511+0000","last_fullsized":"2024-06-27T21:43:55.021511+0000","mapping_epoch":718,"log_start":"737'2800","ondisk_log_start":"737'2800","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"767'10263","last_scrub_stamp":"2024-06-27T21:42:41.603682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:42:41.603682+0000","objects_scrubbed":2597,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:07.852539+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12860,"num_write_kb":411520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":567,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:54.885355+0000","last_change":"2024-06-27T21:42:31.538654+0000","last_active":"2024-06-27T21:43:54.885355+0000","last_peered":"2024-06-27T21:43:54.885355+0000","last_clean":"2024-06-27T21:43:54.885355+0000","last_became_active":"2024-06-27T21:38:49.780122+0000","last_became_peered":"2024-06-27T21:38:49.780122+0000","last_unstale":"2024-06-27T21:43:54.885355+0000","last_undegraded":"2024-06-27T21:43:54.885355+0000","last_fullsized":"2024-06-27T21:43:54.885355+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:31.538537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.538537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:58.964642+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.e","version":"789'12768","reported_seq":13039,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369997+0000","last_change":"2024-06-27T21:43:43.481502+0000","last_active":"2024-06-27T21:43:49.369997+0000","last_peered":"2024-06-27T21:43:49.369997+0000","last_clean":"2024-06-27T21:43:49.369997+0000","last_became_active":"2024-06-27T21:39:34.794749+0000","last_became_peered":"2024-06-27T21:39:34.794749+0000","last_unstale":"2024-06-27T21:43:49.369997+0000","last_undegraded":"2024-06-27T21:43:49.369997+0000","last_fullsized":"2024-06-27T21:43:49.369997+0000","mapping_epoch":718,"log_start":"737'2700","ondisk_log_start":"737'2700","created":718,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":0,"last_scrub":"789'12677","last_scrub_stamp":"2024-06-27T21:43:43.481381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:39:34.324064+0000","last_clean_scrub_stamp":"2024-06-27T21:43:43.481381+0000","objects_scrubbed":97,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:55.610763+0000","scrub_duration":282,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1011,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369777+0000","last_change":"2024-06-27T21:42:30.174565+0000","last_active":"2024-06-27T21:43:49.369777+0000","last_peered":"2024-06-27T21:43:49.369777+0000","last_clean":"2024-06-27T21:43:49.369777+0000","last_became_active":"2024-06-27T21:36:51.804872+0000","last_became_peered":"2024-06-27T21:36:51.804872+0000","last_unstale":"2024-06-27T21:43:49.369777+0000","last_undegraded":"2024-06-27T21:43:49.369777+0000","last_fullsized":"2024-06-27T21:43:49.369777+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:30.174449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:30.174449+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:54.238505+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,4],"acting":[2,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":250,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158122+0000","last_change":"2024-06-27T21:42:37.883302+0000","last_active":"2024-06-27T21:43:50.158122+0000","last_peered":"2024-06-27T21:43:50.158122+0000","last_clean":"2024-06-27T21:43:50.158122+0000","last_became_active":"2024-06-27T21:38:49.781023+0000","last_became_peered":"2024-06-27T21:38:49.781023+0000","last_unstale":"2024-06-27T21:43:50.158122+0000","last_undegraded":"2024-06-27T21:43:50.158122+0000","last_fullsized":"2024-06-27T21:43:50.158122+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:37.883204+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:42:37.883204+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":717,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021324+0000","last_change":"2024-06-27T21:42:31.337909+0000","last_active":"2024-06-27T21:43:55.021324+0000","last_peered":"2024-06-27T21:43:55.021324+0000","last_clean":"2024-06-27T21:43:55.021324+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:43:55.021324+0000","last_undegraded":"2024-06-27T21:43:55.021324+0000","last_fullsized":"2024-06-27T21:43:55.021324+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:31.337766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:31.337766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":743,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.778625+0000","last_change":"2024-06-27T21:43:11.867679+0000","last_active":"2024-06-27T21:43:49.778625+0000","last_peered":"2024-06-27T21:43:49.778625+0000","last_clean":"2024-06-27T21:43:49.778625+0000","last_became_active":"2024-06-27T21:43:11.867292+0000","last_became_peered":"2024-06-27T21:43:11.867292+0000","last_unstale":"2024-06-27T21:43:49.778625+0000","last_undegraded":"2024-06-27T21:43:49.778625+0000","last_fullsized":"2024-06-27T21:43:49.778625+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:28.788214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:42:28.788214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:58.659458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":34,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157903+0000","last_change":"2024-06-27T21:43:11.921500+0000","last_active":"2024-06-27T21:43:50.157903+0000","last_peered":"2024-06-27T21:43:50.157903+0000","last_clean":"2024-06-27T21:43:50.157903+0000","last_became_active":"2024-06-27T21:43:11.874424+0000","last_became_peered":"2024-06-27T21:43:11.874424+0000","last_unstale":"2024-06-27T21:43:50.157903+0000","last_undegraded":"2024-06-27T21:43:50.157903+0000","last_fullsized":"2024-06-27T21:43:50.157903+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.286680+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.286680+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:01.356535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,0],"acting":[11,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":716,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369894+0000","last_change":"2024-06-27T21:43:38.265995+0000","last_active":"2024-06-27T21:43:49.369894+0000","last_peered":"2024-06-27T21:43:49.369894+0000","last_clean":"2024-06-27T21:43:49.369894+0000","last_became_active":"2024-06-27T21:38:49.776796+0000","last_became_peered":"2024-06-27T21:38:49.776796+0000","last_unstale":"2024-06-27T21:43:49.369894+0000","last_undegraded":"2024-06-27T21:43:49.369894+0000","last_fullsized":"2024-06-27T21:43:49.369894+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:43:38.265889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:42.379268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":439,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.740589+0000","last_change":"2024-06-27T21:43:32.372744+0000","last_active":"2024-06-27T21:43:49.740589+0000","last_peered":"2024-06-27T21:43:49.740589+0000","last_clean":"2024-06-27T21:43:49.740589+0000","last_became_active":"2024-06-27T21:38:49.782627+0000","last_became_peered":"2024-06-27T21:38:49.782627+0000","last_unstale":"2024-06-27T21:43:49.740589+0000","last_undegraded":"2024-06-27T21:43:49.740589+0000","last_fullsized":"2024-06-27T21:43:49.740589+0000","mapping_epoch":697,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":698,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:32.372638+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:43:32.372638+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:34.619143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,3],"acting":[8,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":455,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-06-27T21:43:55.021513+0000","last_change":"2024-06-27T21:43:34.188289+0000","last_active":"2024-06-27T21:43:55.021513+0000","last_peered":"2024-06-27T21:43:55.021513+0000","last_clean":"2024-06-27T21:43:55.021513+0000","last_became_active":"2024-06-27T21:43:11.867640+0000","last_became_peered":"2024-06-27T21:43:11.867640+0000","last_unstale":"2024-06-27T21:43:55.021513+0000","last_undegraded":"2024-06-27T21:43:55.021513+0000","last_fullsized":"2024-06-27T21:43:55.021513+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:34.188181+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:43:34.188181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:39.733378+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":379,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.740833+0000","last_change":"2024-06-27T21:43:38.420637+0000","last_active":"2024-06-27T21:43:49.740833+0000","last_peered":"2024-06-27T21:43:49.740833+0000","last_clean":"2024-06-27T21:43:49.740833+0000","last_became_active":"2024-06-27T21:36:51.808305+0000","last_became_peered":"2024-06-27T21:36:51.808305+0000","last_unstale":"2024-06-27T21:43:49.740833+0000","last_undegraded":"2024-06-27T21:43:49.740833+0000","last_fullsized":"2024-06-27T21:43:49.740833+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:38.420522+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:43:38.420522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:55.827489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":1055,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369677+0000","last_change":"2024-06-27T21:43:28.360416+0000","last_active":"2024-06-27T21:43:49.369677+0000","last_peered":"2024-06-27T21:43:49.369677+0000","last_clean":"2024-06-27T21:43:49.369677+0000","last_became_active":"2024-06-27T21:43:11.867791+0000","last_became_peered":"2024-06-27T21:43:11.867791+0000","last_unstale":"2024-06-27T21:43:49.369677+0000","last_undegraded":"2024-06-27T21:43:49.369677+0000","last_fullsized":"2024-06-27T21:43:49.369677+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:28.360278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:28.360278+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:56.645042+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,10],"acting":[2,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":579,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.157544+0000","last_change":"2024-06-27T21:43:31.233832+0000","last_active":"2024-06-27T21:43:50.157544+0000","last_peered":"2024-06-27T21:43:50.157544+0000","last_clean":"2024-06-27T21:43:50.157544+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:43:50.157544+0000","last_undegraded":"2024-06-27T21:43:50.157544+0000","last_fullsized":"2024-06-27T21:43:50.157544+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:31.233757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:31.233757+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:40.445361+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":647,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.369314+0000","last_change":"2024-06-27T21:43:11.868081+0000","last_active":"2024-06-27T21:43:49.369314+0000","last_peered":"2024-06-27T21:43:49.369314+0000","last_clean":"2024-06-27T21:43:49.369314+0000","last_became_active":"2024-06-27T21:43:11.867693+0000","last_became_peered":"2024-06-27T21:43:11.867693+0000","last_unstale":"2024-06-27T21:43:49.369314+0000","last_undegraded":"2024-06-27T21:43:49.369314+0000","last_fullsized":"2024-06-27T21:43:49.369314+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:34.044962+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:17:08.764871+0000","last_clean_scrub_stamp":"2024-06-27T21:42:34.044962+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:43:57.996342+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,8],"acting":[4,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":362,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.740564+0000","last_change":"2024-06-27T21:43:37.424635+0000","last_active":"2024-06-27T21:43:49.740564+0000","last_peered":"2024-06-27T21:43:49.740564+0000","last_clean":"2024-06-27T21:43:49.740564+0000","last_became_active":"2024-06-27T21:36:51.803452+0000","last_became_peered":"2024-06-27T21:36:51.803452+0000","last_unstale":"2024-06-27T21:43:49.740564+0000","last_undegraded":"2024-06-27T21:43:49.740564+0000","last_fullsized":"2024-06-27T21:43:49.740564+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:37.424523+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:43:37.424523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:38.566700+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":380,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:49.740645+0000","last_change":"2024-06-27T21:43:41.399150+0000","last_active":"2024-06-27T21:43:49.740645+0000","last_peered":"2024-06-27T21:43:49.740645+0000","last_clean":"2024-06-27T21:43:49.740645+0000","last_became_active":"2024-06-27T21:43:11.871918+0000","last_became_peered":"2024-06-27T21:43:11.871918+0000","last_unstale":"2024-06-27T21:43:49.740645+0000","last_undegraded":"2024-06-27T21:43:49.740645+0000","last_fullsized":"2024-06-27T21:43:49.740645+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:41.399047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:41.399047+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:00.956036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,11],"acting":[8,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":34,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158155+0000","last_change":"2024-06-27T21:43:11.876604+0000","last_active":"2024-06-27T21:43:50.158155+0000","last_peered":"2024-06-27T21:43:50.158155+0000","last_clean":"2024-06-27T21:43:50.158155+0000","last_became_active":"2024-06-27T21:43:11.874600+0000","last_became_peered":"2024-06-27T21:43:11.874600+0000","last_unstale":"2024-06-27T21:43:50.158155+0000","last_undegraded":"2024-06-27T21:43:50.158155+0000","last_fullsized":"2024-06-27T21:43:50.158155+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:42:52.064655+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:42:52.064655+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:04.854024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,10],"acting":[11,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":42,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-06-27T21:43:50.158190+0000","last_change":"2024-06-27T21:43:27.585656+0000","last_active":"2024-06-27T21:43:50.158190+0000","last_peered":"2024-06-27T21:43:50.158190+0000","last_clean":"2024-06-27T21:43:50.158190+0000","last_became_active":"2024-06-27T21:43:11.874396+0000","last_became_peered":"2024-06-27T21:43:11.874396+0000","last_unstale":"2024-06-27T21:43:50.158190+0000","last_undegraded":"2024-06-27T21:43:50.158190+0000","last_fullsized":"2024-06-27T21:43:50.158190+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:27.585537+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:43:27.585537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:52.654865+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206606,"num_write_kb":6611329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5573,"num_bytes_recovered":364118016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":83361792,"data_stored":83361792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160985,"ondisk_log_size":160985,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":804992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253280,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762320,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982832,"statfs":{"total":95995035648,"available":95214419968,"internally_reserved":0,"allocated":921600,"data_stored":643729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514916,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740112,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93005040,"statfs":{"total":95995035648,"available":95237160960,"internally_reserved":0,"allocated":921600,"data_stored":643729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":756939200},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743849,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623636,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121516,"statfs":{"total":95995035648,"available":95356432384,"internally_reserved":0,"allocated":1056768,"data_stored":777934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":637532608},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972781,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469192,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93275960,"statfs":{"total":95995035648,"available":95514583040,"internally_reserved":0,"allocated":716800,"data_stored":442481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":227633267116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528759,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654164,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93090988,"statfs":{"total":95995035648,"available":95325171712,"internally_reserved":0,"allocated":1056768,"data_stored":777934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757692,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":428936,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93316216,"statfs":{"total":95995035648,"available":95555805184,"internally_reserved":0,"allocated":716800,"data_stored":442481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161398,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430984,"kb_used_data":700,"kb_used_omap":92,"kb_used_meta":430179,"kb_avail":93314168,"statfs":{"total":95995035648,"available":95553708032,"internally_reserved":0,"allocated":716800,"data_stored":442481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":440503394},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455304,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":93289848,"statfs":{"total":95995035648,"available":95528804352,"internally_reserved":0,"allocated":716800,"data_stored":442481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":465500615},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411785,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659600,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085552,"statfs":{"total":95995035648,"available":95319605248,"internally_reserved":0,"allocated":921600,"data_stored":643729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738836,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780500,"kb_used_data":19080,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92964652,"statfs":{"total":95995035648,"available":95195803648,"internally_reserved":0,"allocated":19537920,"data_stored":19259086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16515072,"data_stored":16515072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":66846720,"data_stored":66846720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:43:57.937 INFO:tasks.thrashosds.thrasher:Setting 1.1 to [0, 9, 11] 2024-06-27T21:43:57.937 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1', '0', '9', '11'] 2024-06-27T21:43:57.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap 1.1 0 9 11 2024-06-27T21:43:58.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:43:58.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:43:58.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:57 smithi151 ceph-mon[28965]: pgmap v1987: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 32 op/s 2024-06-27T21:43:58.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:57 smithi151 ceph-mon[28965]: from='client.17403 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:43:58.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:43:58.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:43:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:58.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:43:58.252 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:57 smithi190 ceph-mon[32247]: pgmap v1987: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 32 op/s 2024-06-27T21:43:58.252 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:57 smithi190 ceph-mon[32247]: from='client.17403 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:43:58.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:43:58.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:43:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:57 smithi192 ceph-mon[32956]: pgmap v1987: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 32 op/s 2024-06-27T21:43:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:57 smithi192 ceph-mon[32956]: from='client.17403 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:43:58.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:58.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:58.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:58.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:58.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:58.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:58.750 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:43:58.750 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:43:58.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:43:58.750 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:43:58.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.750 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 562/526 [PushOp(6:3e0eb16b:::benchmark_data_smithi192_117994_object53939:head, version: 486'3456, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:3e0eb16b:::benchmark_data_smithi192_117994_object53939:head@486'3456, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:06.669052+0000", 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 592.07774963400004, 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.630691863, 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669052+0000", 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669052+0000", 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669054+0000", 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.062e-06 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669078+0000", 2024-06-27T21:43:58.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4839999999999999e-05 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669114+0000", 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5289e-05 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:06.669131+0000", 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7782e-05 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.276632+0000", 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.607500828 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.277207+0000", 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057437199999999997 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:08.299744+0000", 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.022537689999999999 2024-06-27T21:43:58.754 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 563/526 [PushOp(6:37131d4f:::benchmark_data_smithi192_117994_object54534:head, version: 486'3492, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:37131d4f:::benchmark_data_smithi192_117994_object54534:head@486'3492, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 586.94184729000006, 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8262880849999998, 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.804955+0000", 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:43:58.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.805071+0000", 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011569999999999999 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.805091+0000", 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0188999999999999e-05 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:11.805103+0000", 2024-06-27T21:43:58.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1243999999999999e-05 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.607611+0000", 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.802508198 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.608194+0000", 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058344199999999999 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.631243+0000", 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023048639999999999 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 563/526 [PushReplyOp(6:3fd23acf:::benchmark_data_smithi192_117994_object54296:head)])", 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:12.279687+0000", 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 586.46711491300005, 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.580862403, 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.759 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279687+0000", 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279687+0000", 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279688+0000", 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6300000000000004e-07 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.279693+0000", 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4280000000000004e-06 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.308287+0000", 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.028593997999999999 2024-06-27T21:43:58.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.308293+0000", 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6280000000000002e-06 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.833554+0000", 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5252613340000001 2024-06-27T21:43:58.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.860550+0000", 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026995252000000001 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 563/526 [PushReplyOp(6:3fd23acf:::benchmark_data_smithi192_117994_object54296:head)])", 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:12.281111+0000", 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 586.46569080899997, 2024-06-27T21:43:58.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5498509820000002, 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281111+0000", 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281111+0000", 2024-06-27T21:43:58.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281112+0000", 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281118+0000", 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1510000000000001e-06 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281134+0000", 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6079e-05 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:12.281140+0000", 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7799999999999997e-06 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.819872+0000", 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.538732019 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:13.830962+0000", 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011090368999999999 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 564/526 [PushOp(6:3f5493a1:::benchmark_data_smithi192_117994_object54907:head, version: 486'3518, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:3f5493a1:::benchmark_data_smithi192_117994_object54907:head@486'3518, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 583.11837141700005, 2024-06-27T21:43:58.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.825308699, 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:43:58.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628431+0000", 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0800000000000004e-07 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628460+0000", 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8890999999999998e-05 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628498+0000", 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7774999999999997e-05 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:15.628513+0000", 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4922999999999999e-05 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:17.429599+0000", 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.801085638 2024-06-27T21:43:58.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:17.430172+0000", 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057386999999999998 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:34:17.453739+0000", 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023566894000000001 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.14 589/550 [PushOp(6:2810808f:::benchmark_data_smithi192_117994_object70270:head, version: 491'4530, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:2810808f:::benchmark_data_smithi192_117994_object70270:head@491'4530, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:19.148860+0000", 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 519.59794197600002, 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.685495178, 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148860+0000", 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148860+0000", 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148861+0000", 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2099999999999995e-07 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148951+0000", 2024-06-27T21:43:58.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0097000000000002e-05 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148986+0000", 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5046000000000003e-05 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:19.148997+0000", 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0667e-05 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:20.834172+0000", 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6851757279999999 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:20.834183+0000", 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.082e-05 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:20.834355+0000", 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017199899999999999 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.775 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.c 595/526 [PushOp(6:37916176:::benchmark_data_smithi192_117994_object62965:head, version: 489'4052, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:37916176:::benchmark_data_smithi192_117994_object62965:head@489'4052, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 504.76312148, 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6254610199999999, 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983681+0000", 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7e-07 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983704+0000", 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3281e-05 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983735+0000", 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.082e-05 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:33.983745+0000", 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0020000000000001e-05 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587861+0000", 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6041156220000001 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:58.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.588341+0000", 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048023500000000003 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.609142+0000", 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020800572 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.781 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.14 595/550 [PushOp(6:2828d551:::benchmark_data_smithi192_117994_object77037:head, version: 493'4953, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:2828d551:::benchmark_data_smithi192_117994_object77037:head@493'4953, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 504.74508434099999, 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5858424599999998, 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001718+0000", 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8899999999999999e-07 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001810+0000", 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1711000000000003e-05 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001853+0000", 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2876000000000001e-05 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:34.001869+0000", 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6161999999999999e-05 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587317+0000", 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5854477760000001 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587331+0000", 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4338000000000001e-05 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:35.587560+0000", 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000228908 2024-06-27T21:43:58.785 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPushReply(6.c 604/526 [PushReplyOp(6:3c890e83:::benchmark_data_smithi192_117994_object64998:head)])", 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:35:53.074990+0000", 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 485.67181176299999, 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6298133859999999, 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:43:58.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074990+0000", 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074990+0000", 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074991+0000", 2024-06-27T21:43:58.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1999999999999999e-07 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.074995+0000", 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9999999999999998e-06 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.075019+0000", 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4428999999999999e-05 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:53.075025+0000", 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9669999999999998e-06 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.678270+0000", 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6032448320000001 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:35:54.704804+0000", 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026533537999999999 2024-06-27T21:43:58.789 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.790 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:43:58.790 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:43:58.790 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:43:58.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:58.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:58.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:58.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:59.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:43:59.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:43:59.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:43:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:59.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:43:59.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:59.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:59.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:43:59.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:43:59.456 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:43:59.492 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:43:59.492 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:43:59.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:43:59.514 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:43:59.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:43:59.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:43:59.536 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:43:59.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:43:59.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:43:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:43:59.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:43:59.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:43:59.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:43:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:43:59.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:43:59.780 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:43:59.781 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:43:59.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:43:59.802 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:43:59.803 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:43:59.807 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:43:59.824 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:43:59.824 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:43:59.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:43:59.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:43:59.867 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:43:59.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:43:59.869 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:43:59.891 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:43:59.892 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:43:59.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:43:59.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:43:59.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:43:59.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:43:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:43:59.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:43:59.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:43:59.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:43:59.998 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:59 smithi192 ceph-mon[32956]: pgmap v1988: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 9 op/s 2024-06-27T21:43:59.998 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:43:59.998 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:43:59.998 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:59 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:43:59.998 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:43:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/155929986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:00.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:00.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:00.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:00.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:59 smithi151 ceph-mon[28965]: pgmap v1988: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 9 op/s 2024-06-27T21:44:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:44:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:44:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:59 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:44:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:43:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/155929986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:59 smithi190 ceph-mon[32247]: pgmap v1988: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 9 op/s 2024-06-27T21:44:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:44:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:44:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:59 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:44:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:43:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/155929986' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:00.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:00.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:00.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:00.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:00.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:00.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:00.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:00.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:00.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.802 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:00.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:00.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:00.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:00.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:01.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:01.034 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:01.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_5' removed 2024-06-27T21:44:01.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:01.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:01.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:01.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:01.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:01.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:01 smithi151 ceph-mon[28965]: osdmap e793: 12 total, 12 up, 10 in 2024-06-27T21:44:01.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:01 smithi151 ceph-mon[28965]: pgmap v1990: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:01.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1100987068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:44:01.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:01.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:01.431 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:01.432 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:01.432 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:01 smithi192 ceph-mon[32956]: osdmap e793: 12 total, 12 up, 10 in 2024-06-27T21:44:01.432 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:01 smithi192 ceph-mon[32956]: pgmap v1990: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:01.432 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1100987068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:44:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:01.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:01 smithi190 ceph-mon[32247]: osdmap e793: 12 total, 12 up, 10 in 2024-06-27T21:44:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:01 smithi190 ceph-mon[32247]: pgmap v1990: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1100987068' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:44:01.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:01.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:01.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:01.669 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:01.669 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:01.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:01.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:01.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:01.902 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:44:01.903 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:44:01.903 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-06-27T21:44:01.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-06-27T21:44:01.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:01.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:01.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:02.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:02.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:02.130 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1100987068' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:44:02.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:02 smithi151 ceph-mon[28965]: osdmap e794: 12 total, 12 up, 10 in 2024-06-27T21:44:02.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:02.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:02.164 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:02.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:02.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:02.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:02.310 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1100987068' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:44:02.311 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:02 smithi192 ceph-mon[32956]: osdmap e794: 12 total, 12 up, 10 in 2024-06-27T21:44:02.311 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:02.311 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:02.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:44:02.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:02.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:02.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1100987068' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:44:02.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:02 smithi190 ceph-mon[32247]: osdmap e794: 12 total, 12 up, 10 in 2024-06-27T21:44:02.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:02.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:02.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:02.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:02.598 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:02.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:02.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:02.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:02.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:02.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:02.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:02.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:02.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:03.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:03.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:03.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:44:03.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:03.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:03.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:03.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:03.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:03.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:03.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:03 smithi151 ceph-mon[28965]: osdmap e795: 12 total, 12 up, 10 in 2024-06-27T21:44:03.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:03 smithi151 ceph-mon[28965]: pgmap v1993: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:03.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:03 smithi190 ceph-mon[32247]: osdmap e795: 12 total, 12 up, 10 in 2024-06-27T21:44:03.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:03 smithi190 ceph-mon[32247]: pgmap v1993: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:03.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:03.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:03 smithi192 ceph-mon[32956]: osdmap e795: 12 total, 12 up, 10 in 2024-06-27T21:44:03.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:03 smithi192 ceph-mon[32956]: pgmap v1993: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:03.555 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:03.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:03.664 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:03.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:03.693 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:03.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:03.802 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:03.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:03.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:03.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:03.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:04.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:04.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:04.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:04.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:04.286 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:04.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:04.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:04.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:04.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:04.560 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:04.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:04.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:04.785 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:04.786 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:04.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:04.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:04.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:04.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:04.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:05.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:05.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:05.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:05.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:05.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:05.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:05.332 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:44:05.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:05.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:05.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:05 smithi192 ceph-mon[32956]: pgmap v1994: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:05.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1389175764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:44:05.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:05.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:05.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:44:05.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:05 smithi151 ceph-mon[28965]: pgmap v1994: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:05.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1389175764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:44:05.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:05.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:05.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:05 smithi190 ceph-mon[32247]: pgmap v1994: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:05.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1389175764' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:44:05.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:05.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:05.859 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:05.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:05.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:05.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:06.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:06.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:06.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:06.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:06.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:06.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:06.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:06.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:06.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:06 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:06.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1389175764' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:44:06.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:06 smithi192 ceph-mon[32956]: osdmap e796: 12 total, 12 up, 10 in 2024-06-27T21:44:06.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:06.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:06.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:06 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:06.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1389175764' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:44:06.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:06 smithi151 ceph-mon[28965]: osdmap e796: 12 total, 12 up, 10 in 2024-06-27T21:44:06.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:06.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:06.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:06 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:06.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1389175764' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:44:06.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:06 smithi190 ceph-mon[32247]: osdmap e796: 12 total, 12 up, 10 in 2024-06-27T21:44:06.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:06.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:06.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:06.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:06.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:07.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:07.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:07.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:07.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:07.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:07.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:07.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:07.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:07.530 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:07.530 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:07.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:07 smithi192 ceph-mon[32956]: pgmap v1996: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:07.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:07 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:44:07.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:07 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:44:07.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:07 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:44:07.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:07 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:44:07.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:07.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:07.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:07 smithi151 ceph-mon[28965]: pgmap v1996: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:07.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:07 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:44:07.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:07 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:44:07.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:07 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:44:07.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:07 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:44:07.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:07.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:07.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:07 smithi190 ceph-mon[32247]: pgmap v1996: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:07.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:07 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:44:07.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:07 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:44:07.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:07 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:44:07.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:07 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:44:07.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:07.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:07.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:07.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:07.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:08.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:08.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:08.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:44:08.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:08.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:08.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:08.338 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:08.398 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:08.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:08.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:08.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:08.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:08.644 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:08 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:44:08.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:08 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:44:08.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1780464624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-27T21:44:08.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:08 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-27T21:44:08.649 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:08 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:44:08.649 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:08 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:44:08.649 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1780464624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-27T21:44:08.649 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:08 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-27T21:44:08.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:08.713 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:08.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:08 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:44:08.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:08 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:44:08.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1780464624' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-27T21:44:08.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:08 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-06-27T21:44:08.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:08.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:08.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:44:08.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:08.967 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:08.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:09.030 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_6' created 2024-06-27T21:44:09.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:09.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:09.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:09.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:09.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:09.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:09.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:09.465 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:09.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:09 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:44:09.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:09 smithi192 ceph-mon[32956]: pgmap v1997: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:09.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:09 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-27T21:44:09.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:09 smithi192 ceph-mon[32956]: osdmap e797: 12 total, 12 up, 10 in 2024-06-27T21:44:09.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:09.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:09.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:09 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:44:09.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:09 smithi151 ceph-mon[28965]: pgmap v1997: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:09.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:09 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-27T21:44:09.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:09 smithi151 ceph-mon[28965]: osdmap e797: 12 total, 12 up, 10 in 2024-06-27T21:44:09.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-06-27T21:44:09.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:09.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:09 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:44:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:09 smithi190 ceph-mon[32247]: pgmap v1997: 31 pgs: 31 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:09 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-06-27T21:44:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:09 smithi190 ceph-mon[32247]: osdmap e797: 12 total, 12 up, 10 in 2024-06-27T21:44:09.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:09.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:09.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:09.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:09.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:10.004 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:10.004 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:10.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:10.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:10.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:44:10.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:44:10.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:44:10.187 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:44:10.189 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:10.190 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:10.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:10.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:44:10.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:44:10.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:10.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:44:10.236 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:44:10.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:44:10.258 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:44:10.280 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:44:10.280 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:44:10.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:44:10.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:44:10.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:44:10.325 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:44:10.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:10.346 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:44:10.346 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:44:10.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:10.354 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.1 pg_upmap mapping to [0,9,11] 2024-06-27T21:44:10.373 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:44:10.373 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:44:10.396 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:44:10.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:44:10.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:44:10.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:44:10.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:10.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:10.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: osdmap e798: 12 total, 12 up, 10 in 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3052145194' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]: dispatch 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]: dispatch 2024-06-27T21:44:10.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3771242840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:10.602 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:10.607 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:44:10.607 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:44:10.607 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:44:10.607 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: osdmap e798: 12 total, 12 up, 10 in 2024-06-27T21:44:10.607 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3052145194' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]: dispatch 2024-06-27T21:44:10.607 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]: dispatch 2024-06-27T21:44:10.608 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3771242840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:10.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:44:10.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:10.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: osdmap e798: 12 total, 12 up, 10 in 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3052145194' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]: dispatch 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]: dispatch 2024-06-27T21:44:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3771242840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:10.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:10.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:10.882 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:10.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:10.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:10.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:11.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:11.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:11.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:11.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:11.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:11.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:11.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:11.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:11.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]': finished 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: osdmap e799: 12 total, 12 up, 10 in 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:44:11.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:44:11.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]': finished 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: osdmap e799: 12 total, 12 up, 10 in 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:44:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:44:11.698 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:44:11.698 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:44:11.698 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:44:11.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:11.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [0, 9, 11]}]': finished 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: osdmap e799: 12 total, 12 up, 10 in 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:44:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:44:11.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:44:11.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:11.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:11.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:11.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:11.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:12.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:12.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:12.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:12.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:12.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:12.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:12.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:12.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:12.544 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:44:12.544 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:44:12.544 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:44:12.544 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:44:12.544 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:44:12.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:12.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:12.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:44:12.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:44:12.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: osdmap e800: 12 total, 12 up, 10 in 2024-06-27T21:44:12.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:44:12.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:44:12.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:12.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:12 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:12.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.752 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:12.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:44:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:44:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: osdmap e800: 12 total, 12 up, 10 in 2024-06-27T21:44:12.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:44:12.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:44:12.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:12.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:12 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: osdmap e800: 12 total, 12 up, 10 in 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:12.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:12 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:12.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:12.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:12.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:12.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:13.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:13.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:13.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:44:13.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:13.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:13.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:13.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:13.369 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:13.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:13.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:13 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:44:13.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:13 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:44:13.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:13 smithi151 ceph-mon[28965]: pgmap v2003: 47 pgs: 1 peering, 1 creating+peering, 5 unknown, 40 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:13.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:13 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:44:13.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:13 smithi151 ceph-mon[28965]: osdmap e801: 12 total, 12 up, 10 in 2024-06-27T21:44:13.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:13.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:13.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:13.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:13.708 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:13 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:44:13.708 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:13 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:44:13.708 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:13 smithi192 ceph-mon[32956]: pgmap v2003: 47 pgs: 1 peering, 1 creating+peering, 5 unknown, 40 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:13.708 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:13 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:44:13.708 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:13 smithi192 ceph-mon[32956]: osdmap e801: 12 total, 12 up, 10 in 2024-06-27T21:44:13.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:13.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:13.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:13 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:44:13.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:13 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:44:13.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:13 smithi190 ceph-mon[32247]: pgmap v2003: 47 pgs: 1 peering, 1 creating+peering, 5 unknown, 40 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:13.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:13 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:44:13.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:13 smithi190 ceph-mon[32247]: osdmap e801: 12 total, 12 up, 10 in 2024-06-27T21:44:13.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:13.888 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:13.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:13.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:14.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:14.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:14.134 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:14.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:14.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:14.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:14.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:14.384 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:14.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:14 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:44:14.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:14 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:44:14.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:14 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:44:14.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:14 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:44:14.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:14.500 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:14 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:44:14.500 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:14 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:44:14.500 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:14 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:44:14.500 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:14 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:44:14.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:14.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:14.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:14.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:14.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:14.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:14 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:44:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:14 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:44:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:14 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:44:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:14 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:44:14.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:14.878 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:44:14.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:14.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:14.930 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:44:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:44:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:44:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:44:14.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.26020.0:175945 6.1 e617/552 ondisk, result = 0)", 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:16.057770+0000", 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 478.868821578, 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66511086900000005, 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057770+0000", 2024-06-27T21:44:14.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057770+0000", 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057771+0000", 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7899999999999997e-07 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.057773+0000", 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.694e-06 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722691+0000", 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.66491738499999997 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722698+0000", 2024-06-27T21:44:14.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0709999999999997e-06 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722766+0000", 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8103999999999997e-05 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722777+0000", 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1148000000000001e-05 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:16.722881+0000", 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103588 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26020.0:177152 6.4 e624/606 6:228c94ea:::benchmark_data_smithi192_117994_object69558:head v 624'10842, mlcod=624'10840)", 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:36:24.041473+0000", 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 470.88511808999999, 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78381791700000003, 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041473+0000", 2024-06-27T21:44:14.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041473+0000", 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041474+0000", 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.041479+0000", 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.361e-06 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.822955+0000", 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78147614200000004 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.822971+0000", 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5420000000000001e-05 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.823065+0000", 2024-06-27T21:44:14.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4021e-05 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.823084+0000", 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8881000000000001e-05 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.825254+0000", 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021705190000000001 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:36:24.825291+0000", 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6813000000000001e-05 2024-06-27T21:44:14.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.5 e707,698 6:a62b003a:::benchmark_data_smithi192_117994_object75931:head,643'10604)", 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:14.231833+0000", 2024-06-27T21:44:14.945 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 300.694758143, 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88407170000000002, 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231833+0000", 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231833+0000", 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231834+0000", 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8800000000000002e-07 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231837+0000", 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.63e-06 2024-06-27T21:44:14.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115556+0000", 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.883718958 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115566+0000", 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0165e-05 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115625+0000", 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8303000000000001e-05 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115905+0000", 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028005599999999999 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7073 7.a 7:5854dda9:::benchmark_data_smithi192_150129_object7072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.950 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.702641+0000", 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.22395039600002, 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0929825260000001, 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7073 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702641+0000", 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702641+0000", 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702642+0000", 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0410000000000001e-06 2024-06-27T21:44:14.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702663+0000", 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1299999999999999e-05 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702677+0000", 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3947999999999999e-05 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702684+0000", 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.861e-06 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702737+0000", 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2877999999999998e-05 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702943+0000", 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020602399999999999 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.703106+0000", 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016296600000000001 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.704211+0000", 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001105396 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.706633+0000", 2024-06-27T21:44:14.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002422113 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795514+0000", 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.088880246 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795562+0000", 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8405000000000002e-05 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795623+0000", 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1347999999999996e-05 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7081 7.a 7:589100a8:::benchmark_data_smithi192_150129_object7080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.705118+0000", 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.22147328099999, 2024-06-27T21:44:14.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0906754319999998, 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7081 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705118+0000", 2024-06-27T21:44:14.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705118+0000", 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705120+0000", 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7069999999999999e-06 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705149+0000", 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9462e-05 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705166+0000", 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.666e-05 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705174+0000", 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7179999999999996e-06 2024-06-27T21:44:14.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705226+0000", 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1969000000000003e-05 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705359+0000", 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000133857 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705538+0000", 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017908900000000001 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.706085+0000", 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054701999999999995 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.714773+0000", 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0086879260000000003 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795682+0000", 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0809086040000002 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795746+0000", 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4462999999999996e-05 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795793+0000", 2024-06-27T21:44:14.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6956999999999998e-05 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7113 7.a 7:518e8ad9:::benchmark_data_smithi192_150129_object7112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.722268+0000", 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.20432293699997, 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0741637069999999, 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7113 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722268+0000", 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722268+0000", 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722269+0000", 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.68e-07 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722291+0000", 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2226999999999999e-05 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722299+0000", 2024-06-27T21:44:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0129999999999993e-06 2024-06-27T21:44:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722305+0000", 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5999999999999997e-06 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722358+0000", 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2775999999999998e-05 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722499+0000", 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000140968 2024-06-27T21:44:14.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722636+0000", 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137942 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.723239+0000", 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060242800000000001 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.724713+0000", 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001473741 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796327+0000", 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.071613948 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796387+0000", 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0509e-05 2024-06-27T21:44:14.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796432+0000", 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4987000000000001e-05 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7144 7.a 7:58d52035:::benchmark_data_smithi192_150129_object7143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.734600+0000", 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.19199081300002, 2024-06-27T21:44:14.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0619978899999998, 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7144 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734600+0000", 2024-06-27T21:44:14.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734600+0000", 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734602+0000", 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.846e-06 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734627+0000", 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4635999999999999e-05 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734650+0000", 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3322000000000001e-05 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734659+0000", 2024-06-27T21:44:14.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5879999999999998e-06 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734720+0000", 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1525999999999993e-05 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734853+0000", 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013227900000000001 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.735030+0000", 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017699600000000001 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.735604+0000", 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00057416300000000004 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.736022+0000", 2024-06-27T21:44:14.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000417933 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796489+0000", 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.060467429 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796553+0000", 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3582000000000002e-05 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796598+0000", 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5590000000000002e-05 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7161 7.a 7:5c50f347:::benchmark_data_smithi192_150129_object7160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.18000053600002, 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0504606029999999, 2024-06-27T21:44:14.979 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7161 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4099999999999998e-07 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746608+0000", 2024-06-27T21:44:14.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6625000000000001e-05 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746618+0000", 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0011e-05 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746623+0000", 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3360000000000002e-06 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746663+0000", 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0018000000000002e-05 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746745+0000", 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1840000000000002e-05 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746828+0000", 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2491999999999996e-05 2024-06-27T21:44:14.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.747302+0000", 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047436100000000002 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.747576+0000", 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027408000000000001 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796916+0000", 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0493401699999998 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797001+0000", 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5148000000000005e-05 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797051+0000", 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9880999999999999e-05 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.985 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7163 7.a 7:551d8d3c:::benchmark_data_smithi192_150129_object7162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.751481+0000", 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.17511032800002, 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.045761766, 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.986 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7163 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751481+0000", 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751481+0000", 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751482+0000", 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.201e-06 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751512+0000", 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9422000000000001e-05 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751525+0000", 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3512999999999999e-05 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751533+0000", 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0190000000000006e-06 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751584+0000", 2024-06-27T21:44:14.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0909999999999999e-05 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751667+0000", 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3344999999999996e-05 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751758+0000", 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0295000000000002e-05 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.752302+0000", 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054424699999999998 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.752958+0000", 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00065651900000000005 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797146+0000", 2024-06-27T21:44:14.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0441872939999999 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797204+0000", 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8359000000000001e-05 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797243+0000", 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8642e-05 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:14.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7170 7.a 7:5e96af1b:::benchmark_data_smithi192_150129_object7169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.754627+0000", 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.17196457799997, 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0436859030000001, 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:14.993 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7170 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754627+0000", 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754627+0000", 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754628+0000", 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.16e-07 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:14.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754647+0000", 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9052999999999999e-05 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754657+0000", 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0349999999999999e-05 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754662+0000", 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.074e-06 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754696+0000", 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4245000000000002e-05 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754770+0000", 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3411999999999994e-05 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754850+0000", 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0724999999999997e-05 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.755252+0000", 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000401717 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.755609+0000", 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035673299999999997 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:14.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798066+0000", 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0424571569999999 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798271+0000", 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020472700000000001 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798313+0000", 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1894000000000002e-05 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:14.999 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7173 7.a 7:5e8032bc:::benchmark_data_smithi192_150129_object7172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.758350+0000", 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.16824120899997, 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0401055800000001, 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7173 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758350+0000", 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758350+0000", 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758351+0000", 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8400000000000003e-07 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758369+0000", 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7813999999999999e-05 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758379+0000", 2024-06-27T21:44:15.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0657e-05 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758384+0000", 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9599999999999999e-06 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758431+0000", 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6353000000000002e-05 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758548+0000", 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011780700000000001 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758687+0000", 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000139004 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.759099+0000", 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041104999999999999 2024-06-27T21:44:15.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.759403+0000", 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030414899999999998 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798361+0000", 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0389582499999999 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798414+0000", 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2806999999999998e-05 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798456+0000", 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1944999999999999e-05 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.006 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7194 7.a 7:54efe092:::benchmark_data_smithi192_150129_object7193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.804471+0000", 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 264.12212007199997, 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9944490020000001, 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7194 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804471+0000", 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804471+0000", 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804472+0000", 2024-06-27T21:44:15.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4000000000000001e-07 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804489+0000", 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6928000000000001e-05 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804498+0000", 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3740000000000004e-06 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804503+0000", 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5460000000000002e-06 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804540+0000", 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7283999999999997e-05 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804619+0000", 2024-06-27T21:44:15.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8466000000000004e-05 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804704+0000", 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5797999999999994e-05 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.805270+0000", 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056605399999999997 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.805570+0000", 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030003099999999997 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798830+0000", 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9932598270000002 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798883+0000", 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2920999999999997e-05 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798920+0000", 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7033000000000003e-05 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7203 7.a 7:54cc3b12:::benchmark_data_smithi192_150129_object7202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.932232+0000", 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.99435893100002, 2024-06-27T21:44:15.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.866820186, 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7203 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932232+0000", 2024-06-27T21:44:15.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932232+0000", 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932234+0000", 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.167e-06 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932254+0000", 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0332000000000001e-05 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932266+0000", 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1739000000000001e-05 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932272+0000", 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1979999999999998e-06 2024-06-27T21:44:15.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932317+0000", 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4972000000000002e-05 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932414+0000", 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6875999999999999e-05 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932500+0000", 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6761000000000005e-05 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.933047+0000", 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054685900000000002 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.933322+0000", 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027448799999999998 2024-06-27T21:44:15.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798967+0000", 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8656448829999999 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799020+0000", 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.363e-05 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799053+0000", 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2280999999999998e-05 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7206 7.a 7:5fcfa5ab:::benchmark_data_smithi192_150129_object7205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.968510+0000", 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.95808089600001, 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8313247289999999, 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7206 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968510+0000", 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968510+0000", 2024-06-27T21:44:15.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968512+0000", 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.669e-06 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968551+0000", 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8559999999999997e-05 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968562+0000", 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0910999999999999e-05 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968567+0000", 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2039999999999996e-06 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968603+0000", 2024-06-27T21:44:15.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6644999999999999e-05 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968704+0000", 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100657 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968799+0000", 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4724999999999999e-05 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.969300+0000", 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00050102600000000001 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.969718+0000", 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041863499999999999 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799711+0000", 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8299927789999999 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799791+0000", 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0276000000000002e-05 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799835+0000", 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3642e-05 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7212 7.a 7:59657ab7:::benchmark_data_smithi192_150129_object7211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.090909+0000", 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.83568221799999, 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7091218459999999, 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7212 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.027 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090909+0000", 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090909+0000", 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090910+0000", 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2600000000000001e-07 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090929+0000", 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9236e-05 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090940+0000", 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0614999999999999e-05 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090947+0000", 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9330000000000004e-06 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090998+0000", 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1405e-05 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.091083+0000", 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5008999999999999e-05 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.091170+0000", 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6833e-05 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.091624+0000", 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045440399999999999 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.092085+0000", 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00046061799999999999 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799883+0000", 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7077979029999999 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799991+0000", 2024-06-27T21:44:15.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108422 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800031+0000", 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9542000000000002e-05 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7223 7.a 7:59acbc88:::benchmark_data_smithi192_150129_object7222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.227352+0000", 2024-06-27T21:44:15.033 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.69923962799999, 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5728116769999998, 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7223 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227352+0000", 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227352+0000", 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227353+0000", 2024-06-27T21:44:15.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4020000000000001e-06 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227373+0000", 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0409000000000001e-05 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227385+0000", 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1406999999999999e-05 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227392+0000", 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1350000000000003e-06 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227440+0000", 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7726999999999999e-05 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227529+0000", 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8769000000000003e-05 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227615+0000", 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6230000000000006e-05 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.228120+0000", 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00050544700000000001 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.228405+0000", 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028437900000000002 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800072+0000", 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.571667862 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800126+0000", 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3245000000000003e-05 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800163+0000", 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7664999999999999e-05 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7236 7.a 7:5bae2729:::benchmark_data_smithi192_150129_object7235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.314672+0000", 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.61191911600002, 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.485620113, 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.040 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7236 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314672+0000", 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314672+0000", 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314673+0000", 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9199999999999999e-07 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314692+0000", 2024-06-27T21:44:15.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9357000000000001e-05 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314703+0000", 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0854e-05 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314709+0000", 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3990000000000003e-06 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314760+0000", 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0980000000000003e-05 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314848+0000", 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8475999999999998e-05 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314935+0000", 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6965999999999994e-05 2024-06-27T21:44:15.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.315411+0000", 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047577600000000002 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.316018+0000", 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060737699999999998 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800204+0000", 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.484185246 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800257+0000", 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3349e-05 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800292+0000", 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5441000000000002e-05 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.046 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7275 7.a 7:56cae157:::benchmark_data_smithi192_150129_object7274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e730)", 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.900012+0000", 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 263.02657942799999, 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9010095310000001, 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.047 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7275 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900012+0000", 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900012+0000", 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900013+0000", 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900056+0000", 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3597999999999998e-05 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900067+0000", 2024-06-27T21:44:15.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1022000000000001e-05 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900073+0000", 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2599999999999996e-06 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900113+0000", 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0071999999999999e-05 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900200+0000", 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7625000000000002e-05 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900284+0000", 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3646000000000005e-05 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900850+0000", 2024-06-27T21:44:15.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056577099999999996 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.901121+0000", 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027103400000000002 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800754+0000", 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8996332389999999 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800954+0000", 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019964599999999999 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801021+0000", 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7858e-05 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7276 7.a 7:5d7fcb60:::benchmark_data_smithi192_150129_object7275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e730)", 2024-06-27T21:44:15.053 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:52.017209+0000", 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 262.909382747, 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7839677919999999, 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7276 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017209+0000", 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017209+0000", 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017210+0000", 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.421e-06 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017254+0000", 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3893999999999997e-05 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017267+0000", 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3006999999999999e-05 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017275+0000", 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0320000000000003e-06 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017331+0000", 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5615999999999997e-05 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017446+0000", 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115633 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017547+0000", 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101346 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.018283+0000", 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00073507599999999998 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.018792+0000", 2024-06-27T21:44:15.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000509075 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801070+0000", 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7822779560000002 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801123+0000", 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3492999999999998e-05 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801176+0000", 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3242999999999998e-05 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.26656.0:199169 7.e e787/718 7:7f1ee011:::benchmark_data_smithi192_150129_object95864:head v 787'12319, mlcod=787'12316)", 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:43:35.324439+0000", 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 39.602151999999997, 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27772121500000002, 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:15.060 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.324439+0000", 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.324439+0000", 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.324440+0000", 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6300000000000004e-07 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.324446+0000", 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999997e-06 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.598315+0000", 2024-06-27T21:44:15.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.27386922499999999 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.598321+0000", 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0190000000000003e-06 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.598372+0000", 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0695000000000002e-05 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:15.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.598389+0000", 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7198e-05 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.602118+0000", 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037284029999999999 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:35.602161+0000", 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2951999999999998e-05 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.064 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:15.065 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:15.065 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:44:15.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:15.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:15.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:15.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:15.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:15.343 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:15.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.467 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:15.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:15.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:44:15.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: pgmap v2005: 47 pgs: 1 peering, 46 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:15.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/258661280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:15.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:44:15.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:44:15.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:44:15.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/258661280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:15.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: osdmap e802: 12 total, 12 up, 10 in 2024-06-27T21:44:15.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4128206299' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:44:15.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:15.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:15.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:15.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:15.729 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: pgmap v2005: 47 pgs: 1 peering, 46 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:15.729 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/258661280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:15.729 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:44:15.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:44:15.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:44:15.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/258661280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:15.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: osdmap e802: 12 total, 12 up, 10 in 2024-06-27T21:44:15.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4128206299' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:44:15.749 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:15.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: pgmap v2005: 47 pgs: 1 peering, 46 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/258661280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/258661280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: osdmap e802: 12 total, 12 up, 10 in 2024-06-27T21:44:15.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4128206299' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:44:15.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:15.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:15.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:15.888 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-06-27T21:44:15.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:16.046 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.067 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:16.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:16.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:16.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:16.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:16.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:16.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:16.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:44:16.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:44:16.439 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-06-27T21:44:16.439 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-06-27T21:44:16.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 3 1 2024-06-27T21:44:16.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:16.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:16.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:16.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:16.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:16.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:16.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:16.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:16.952 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:16.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:17.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:17.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:17.120 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4128206299' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:44:17.120 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:16 smithi192 ceph-mon[32956]: osdmap e803: 12 total, 12 up, 10 in 2024-06-27T21:44:17.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:17.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:17.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4128206299' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:44:17.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:16 smithi151 ceph-mon[28965]: osdmap e803: 12 total, 12 up, 10 in 2024-06-27T21:44:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4128206299' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:44:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:16 smithi190 ceph-mon[32247]: osdmap e803: 12 total, 12 up, 10 in 2024-06-27T21:44:17.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:17.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:17.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:17.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:17.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:17.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:17.599 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:44:17.600 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:44:17.600 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:44:17.600 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:44:17.614 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:17.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:44:17.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:17.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:17.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:17.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:17.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:17.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:17.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:18.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:18.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:18.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:18.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:17 smithi151 ceph-mon[28965]: pgmap v2008: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:18.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:17 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:44:18.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:17 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:44:18.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:17 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:44:18.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:18.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:17 smithi192 ceph-mon[32956]: pgmap v2008: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:18.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:17 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:44:18.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:17 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:44:18.203 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:17 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:44:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:18.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:18.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:44:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:17 smithi190 ceph-mon[32247]: pgmap v2008: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:17 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:44:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:17 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:44:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:17 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:44:18.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:18.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:18.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:18.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:18.481 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:18.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:18.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:18.622 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-06-27T21:44:18.625 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:44:18.625 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:44:18.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:18.680 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:44:18.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:44:18.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_167666 2024-06-27T21:44:18.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:44:18.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:44:18.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:18.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:18.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:18.899 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:44:18.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:44:18.905 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:18.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:44:18.927 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:44:18.936 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:18.953 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:44:18.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:44:18.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:18.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:18.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:44:18.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:44:19.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:44:19.001 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:44:19.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:44:19.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:44:19.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:18 smithi151 ceph-mon[28965]: 1.1a deep-scrub starts 2024-06-27T21:44:19.025 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:18 smithi151 ceph-mon[28965]: 1.1a deep-scrub ok 2024-06-27T21:44:19.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1619854550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:19.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:44:19.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:44:19.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:19.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:44:19.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:44:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:19.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:19.101 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:44:19.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:44:19.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:44:19.126 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:44:19.150 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:44:19.150 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:44:19.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:44:19.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:44:19.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:19.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:18 smithi190 ceph-mon[32247]: 1.1a deep-scrub starts 2024-06-27T21:44:19.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:18 smithi190 ceph-mon[32247]: 1.1a deep-scrub ok 2024-06-27T21:44:19.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1619854550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:19.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:18 smithi192 ceph-mon[32956]: 1.1a deep-scrub starts 2024-06-27T21:44:19.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:18 smithi192 ceph-mon[32956]: 1.1a deep-scrub ok 2024-06-27T21:44:19.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1619854550' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:19.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:19.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:19.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:19.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:19.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:19.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:19.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 1738 1722 107.619 107.625 0.00351517 0.00922454 2024-06-27T21:44:19.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:19.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:19.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:19.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:19.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:19.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:19.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:20.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:19 smithi192 ceph-mon[32956]: pgmap v2009: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:20.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:19 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:20.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:19 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:20.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:19 smithi192 ceph-mon[32956]: osdmap e804: 12 total, 12 up, 10 in 2024-06-27T21:44:20.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:19 smithi192 ceph-mon[32956]: osdmap e805: 12 total, 12 up, 10 in 2024-06-27T21:44:20.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:20.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:20.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:20.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:19 smithi151 ceph-mon[28965]: pgmap v2009: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:20.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:19 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:20.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:19 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:20.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:19 smithi151 ceph-mon[28965]: osdmap e804: 12 total, 12 up, 10 in 2024-06-27T21:44:20.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:19 smithi151 ceph-mon[28965]: osdmap e805: 12 total, 12 up, 10 in 2024-06-27T21:44:20.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:20.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:19 smithi190 ceph-mon[32247]: pgmap v2009: 47 pgs: 47 active+clean; 577 KiB data, 5.7 GiB used, 888 GiB / 894 GiB avail 2024-06-27T21:44:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:19 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:19 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:19 smithi190 ceph-mon[32247]: osdmap e804: 12 total, 12 up, 10 in 2024-06-27T21:44:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:19 smithi190 ceph-mon[32247]: osdmap e805: 12 total, 12 up, 10 in 2024-06-27T21:44:20.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:20.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:20.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:20.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:20.550 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:20.551 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:20.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:20.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:20.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 3564 3548 110.864 114.125 0.0180727 0.00859177 2024-06-27T21:44:20.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:20.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:20.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:20.873 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:20.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:20 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:44:20.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:20 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:44:20.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:20 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:20.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:20 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:20.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:20.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:20.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:21.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:21.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:21.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:20 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:44:21.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:20 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:44:21.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:20 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:21.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:20 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:21.230 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:21.263 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:20 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:44:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:20 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:44:21.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:20 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:21.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:20 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:21.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:21.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:21.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:21.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:21.609 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:44:21.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:21.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 15 5519 5504 114.655 122.25 0.00548719 0.00870118 2024-06-27T21:44:21.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:21.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:21.807 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:21.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:44:21.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:21.912 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-06-27T21:44:21.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:21.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:21.940 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:22.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:22.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:22.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.183 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:22.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:21 smithi151 ceph-mon[28965]: pgmap v2012: 47 pgs: 47 active+clean; 59 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 11 MiB/s wr, 173 op/s 2024-06-27T21:44:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:21 smithi151 ceph-mon[28965]: osdmap e806: 12 total, 12 up, 10 in 2024-06-27T21:44:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3288228421' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:44:22.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:22.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:22.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:21 smithi192 ceph-mon[32956]: pgmap v2012: 47 pgs: 47 active+clean; 59 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 11 MiB/s wr, 173 op/s 2024-06-27T21:44:22.206 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:21 smithi192 ceph-mon[32956]: osdmap e806: 12 total, 12 up, 10 in 2024-06-27T21:44:22.206 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3288228421' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:44:22.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:21 smithi190 ceph-mon[32247]: pgmap v2012: 47 pgs: 47 active+clean; 59 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 11 MiB/s wr, 173 op/s 2024-06-27T21:44:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:21 smithi190 ceph-mon[32247]: osdmap e806: 12 total, 12 up, 10 in 2024-06-27T21:44:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3288228421' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:44:22.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:22.306 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:22.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:22.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:22.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:22.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:22.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:22.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 6472 6456 100.865 59.5 0.00791624 0.00989766 2024-06-27T21:44:22.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:22.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:22.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:22.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:22.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:22.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:23.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:23.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:23.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3288228421' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:44:23.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:22 smithi192 ceph-mon[32956]: osdmap e807: 12 total, 12 up, 10 in 2024-06-27T21:44:23.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:23.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3288228421' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:44:23.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:22 smithi151 ceph-mon[28965]: osdmap e807: 12 total, 12 up, 10 in 2024-06-27T21:44:23.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:23.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3288228421' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:44:23.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:22 smithi190 ceph-mon[32247]: osdmap e807: 12 total, 12 up, 10 in 2024-06-27T21:44:23.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:23.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:23.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:23.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:23.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:23.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:23.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 16 8509 8493 106.153 127.312 0.0028208 0.00940315 2024-06-27T21:44:23.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:23.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:23.757 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:23.757 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:23.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:23.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:23.930 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:44:23.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:23.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:23.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:24.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:23 smithi192 ceph-mon[32956]: pgmap v2015: 47 pgs: 47 active+clean; 90 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 22 MiB/s wr, 358 op/s 2024-06-27T21:44:24.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1278176631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:44:24.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:23 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:44:24.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:24.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:24.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:24.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:23 smithi151 ceph-mon[28965]: pgmap v2015: 47 pgs: 47 active+clean; 90 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 22 MiB/s wr, 358 op/s 2024-06-27T21:44:24.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1278176631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:44:24.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:23 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:44:24.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.228 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:24.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:23 smithi190 ceph-mon[32247]: pgmap v2015: 47 pgs: 47 active+clean; 90 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 22 MiB/s wr, 358 op/s 2024-06-27T21:44:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1278176631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:44:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:23 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:44:24.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:24.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:24.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:24.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:24.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:24.616 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:24.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 16 9854 9838 102.47 84.0625 0.00507888 0.00974513 2024-06-27T21:44:24.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:24.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:24.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:24.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:24.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:24.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:44:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:24.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:25.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:25.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:25.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:24 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:25.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:24 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:44:25.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:24 smithi151 ceph-mon[28965]: osdmap e808: 12 total, 12 up, 10 in 2024-06-27T21:44:25.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:24 smithi151 ceph-mon[28965]: osdmap e809: 12 total, 12 up, 10 in 2024-06-27T21:44:25.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:25.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:25.259 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:24 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:24 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:44:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:24 smithi190 ceph-mon[32247]: osdmap e808: 12 total, 12 up, 10 in 2024-06-27T21:44:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:24 smithi190 ceph-mon[32247]: osdmap e809: 12 total, 12 up, 10 in 2024-06-27T21:44:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:24 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:24 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:44:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:24 smithi192 ceph-mon[32956]: osdmap e808: 12 total, 12 up, 10 in 2024-06-27T21:44:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:24 smithi192 ceph-mon[32956]: osdmap e809: 12 total, 12 up, 10 in 2024-06-27T21:44:25.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:25.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:25.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:25.499 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:25.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:25.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:25.681 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 11403 11387 101.661 96.8125 0.00341845 0.0098231 2024-06-27T21:44:25.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:25.760 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:25.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:25.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:25.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:25.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:26.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:26.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:26.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:25 smithi151 ceph-mon[28965]: pgmap v2017: 47 pgs: 47 active+clean; 258 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 45 MiB/s wr, 725 op/s 2024-06-27T21:44:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:25 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:25 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:26.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:25 smithi190 ceph-mon[32247]: pgmap v2017: 47 pgs: 47 active+clean; 258 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 45 MiB/s wr, 725 op/s 2024-06-27T21:44:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:25 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:25 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:26.272 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:25 smithi192 ceph-mon[32956]: pgmap v2017: 47 pgs: 47 active+clean; 258 MiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 45 MiB/s wr, 725 op/s 2024-06-27T21:44:26.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:25 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:26.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:25 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:26.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:26.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:26.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:26.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:26.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:26.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:26.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 13074 13058 102.007 104.438 0.0763336 0.00976804 2024-06-27T21:44:26.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:26.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:26.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:26.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:44:26.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:44:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:26.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:26.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:26.900 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:44:26.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:44:26.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:44:26.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:44:26.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:44:26.944 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:44:26.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:44:26.971 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:44:26.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:44:26.995 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:44:26.995 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:44:26.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:27.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:27.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:44:27.022 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:44:27.044 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:44:27.045 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:44:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:26 smithi192 ceph-mon[32956]: osdmap e810: 12 total, 12 up, 10 in 2024-06-27T21:44:27.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2550501755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:27.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:44:27.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:44:27.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:44:27.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:44:27.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:44:27.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:44:27.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:27.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:44:27.139 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:44:27.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:26 smithi151 ceph-mon[28965]: osdmap e810: 12 total, 12 up, 10 in 2024-06-27T21:44:27.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2550501755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:27.140 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:27.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:27.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:26 smithi190 ceph-mon[32247]: osdmap e810: 12 total, 12 up, 10 in 2024-06-27T21:44:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2550501755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:27.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:27.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:27.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:27.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:27.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:27.646 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:27.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 14284 14268 99.0745 75.625 0.0234039 0.0100685 2024-06-27T21:44:27.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:27.786 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:27.840 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11] out_osds: [1, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:44:27.840 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:44:27.840 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-06-27T21:44:27.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 1 2024-06-27T21:44:27.888 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:27.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:27.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:27.923 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:44:27.923 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:44:27.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.923 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:14738 8.6 8:6ad84202:::benchmark_data_smithi192_167666_object14737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-06-27T21:44:27.923 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.898706+0000", 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019485402999999998, 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019539668999999999, 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 14738 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898706+0000", 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898706+0000", 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898707+0000", 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.06e-06 2024-06-27T21:44:27.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898727+0000", 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0173000000000002e-05 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898738+0000", 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0696000000000001e-05 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898745+0000", 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8120000000000002e-06 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898795+0000", 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9960000000000003e-05 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.898929+0000", 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134442 2024-06-27T21:44:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.899096+0000", 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016723700000000001 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.901108+0000", 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002011966 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:27.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.909066+0000", 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.007957736 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:14761 8.3 8:c3be4912:::benchmark_data_smithi192_167666_object14760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.915142+0000", 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.003049211, 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031814619999999999, 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.929 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 14761 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915142+0000", 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915142+0000", 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915143+0000", 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6400000000000001e-07 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915161+0000", 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8505e-05 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915171+0000", 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5349999999999993e-06 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915176+0000", 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3000000000000001e-06 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915228+0000", 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1620999999999999e-05 2024-06-27T21:44:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915356+0000", 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012811699999999999 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,6", 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915501+0000", 2024-06-27T21:44:27.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000145372 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917358+0000", 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001856246 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917547+0000", 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000189626 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:14740 8.6 8:672a3604:::benchmark_data_smithi192_167666_object14739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.899959+0000", 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.01823284, 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018432674, 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:27.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 14740 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.899959+0000", 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.899959+0000", 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.899960+0000", 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.099e-06 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.899988+0000", 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8518e-05 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.900004+0000", 2024-06-27T21:44:27.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6022e-05 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.900016+0000", 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1905e-05 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.900209+0000", 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019260999999999999 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.900339+0000", 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012995199999999999 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.900474+0000", 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013576800000000001 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.901440+0000", 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096561900000000003 2024-06-27T21:44:27.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.909105+0000", 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.007664789 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:14762 8.6 8:64e36ec0:::benchmark_data_smithi192_167666_object14761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.915590+0000", 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0026010019999999998, 2024-06-27T21:44:27.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028701289999999999, 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 14762 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915590+0000", 2024-06-27T21:44:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915590+0000", 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915591+0000", 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3900000000000004e-07 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915613+0000", 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1401999999999999e-05 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915620+0000", 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.96e-06 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915628+0000", 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4800000000000004e-06 2024-06-27T21:44:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915690+0000", 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1849999999999999e-05 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.915817+0000", 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012723500000000001 2024-06-27T21:44:27.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.916025+0000", 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020803400000000001 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.916828+0000", 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080291800000000001 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:14766 8.6 8:674ef1d9:::benchmark_data_smithi192_167666_object14765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.917621+0000", 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00057055799999999998, 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000904994, 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:44:27.946 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 14766 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917621+0000", 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917621+0000", 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917622+0000", 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0500000000000002e-07 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917642+0000", 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0313e-05 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917655+0000", 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2775000000000001e-05 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917660+0000", 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.181e-06 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917744+0000", 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3854e-05 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.917870+0000", 2024-06-27T21:44:27.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012584 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,10", 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.918005+0000", 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000135313 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.918937+0000", 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00093238800000000003 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.27031.0:14766 8.6 e810/797 ondisk, result = 0)", 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.919070+0000", 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9991212, 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037426799999999997, 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919070+0000", 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919070+0000", 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919071+0000", 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4899999999999995e-07 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919074+0000", 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.456e-06 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919094+0000", 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9771000000000001e-05 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919099+0000", 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.724e-06 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919216+0000", 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000116913 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919222+0000", 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9730000000000002e-06 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.954 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:14769 8.7 8.20a2b2b7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:27.919400+0000", 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9987917, 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100338, 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 14769 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:44:27.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919400+0000", 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919400+0000", 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919401+0000", 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.142e-06 2024-06-27T21:44:27.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919424+0000", 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3549000000000001e-05 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919438+0000", 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3926e-05 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:44:27.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:27.919446+0000", 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7810000000000005e-06 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 7 2024-06-27T21:44:27.958 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:44:28.002 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:28.021 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:28.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:28.127 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:27 smithi151 ceph-mon[28965]: pgmap v2020: 47 pgs: 47 active+clean; 574 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-27T21:44:28.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:28.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:27 smithi190 ceph-mon[32247]: pgmap v2020: 47 pgs: 47 active+clean; 574 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-27T21:44:28.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:28.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:27 smithi192 ceph-mon[32956]: pgmap v2020: 47 pgs: 47 active+clean; 574 MiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-06-27T21:44:28.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:28.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:28.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:28.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:28.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:44:28.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 15748 15732 98.3163 91.5 0.0015304 0.0101603 2024-06-27T21:44:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:28.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:28.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:28.804 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:28.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:28.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:28.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:28.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:29.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:29.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:29.161 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:29.190 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:29.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:29.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:29.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:29.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:29.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.575 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:29.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:29.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:29.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 17027 17011 96.6449 79.9375 0.0060919 0.0103366 2024-06-27T21:44:29.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:29.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:29.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:29.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:29.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:29.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:29.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:44:30.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:30.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:30.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:30.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:29 smithi190 ceph-mon[32247]: pgmap v2021: 47 pgs: 47 active+clean; 720 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-27T21:44:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:30.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:29 smithi190 ceph-mon[32247]: osdmap e811: 12 total, 12 up, 10 in 2024-06-27T21:44:30.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:29 smithi151 ceph-mon[28965]: pgmap v2021: 47 pgs: 47 active+clean; 720 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-27T21:44:30.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:30.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:30.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:29 smithi151 ceph-mon[28965]: osdmap e811: 12 total, 12 up, 10 in 2024-06-27T21:44:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:29 smithi192 ceph-mon[32956]: pgmap v2021: 47 pgs: 47 active+clean; 720 MiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-06-27T21:44:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:29 smithi192 ceph-mon[32956]: osdmap e811: 12 total, 12 up, 10 in 2024-06-27T21:44:30.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:30.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:30.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:30.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:30.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:30.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:30.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 18197 18181 94.6848 73.125 0.00318755 0.0105536 2024-06-27T21:44:30.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:30.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:30.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:30.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:30.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:30.963 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:30.963 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:44:30.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:30.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:31.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:31.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:31.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:31.229 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1979926580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:31.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:31.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1979926580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:31.269 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1979926580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:31.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:31.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:31.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:31.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:31.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:31.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 19576 19560 94.0306 86.1875 0.00446102 0.0106257 2024-06-27T21:44:31.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:31.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:31.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:31.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:31.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:31.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:31.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:32.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:32.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:32.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:32.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:31 smithi190 ceph-mon[32247]: pgmap v2023: 47 pgs: 47 active+clean; 970 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-06-27T21:44:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:31 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1979926580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:32.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:31 smithi190 ceph-mon[32247]: osdmap e812: 12 total, 12 up, 10 in 2024-06-27T21:44:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:31 smithi192 ceph-mon[32956]: pgmap v2023: 47 pgs: 47 active+clean; 970 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-06-27T21:44:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:31 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1979926580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:31 smithi192 ceph-mon[32956]: osdmap e812: 12 total, 12 up, 10 in 2024-06-27T21:44:32.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:32.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:31 smithi151 ceph-mon[28965]: pgmap v2023: 47 pgs: 47 active+clean; 970 MiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-06-27T21:44:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:31 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1979926580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:31 smithi151 ceph-mon[28965]: osdmap e812: 12 total, 12 up, 10 in 2024-06-27T21:44:32.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:32.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:32.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:32.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:32.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 20692 20676 92.2959 69.75 0.00764548 0.0108239 2024-06-27T21:44:32.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:32.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:32.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:32.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:32.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:32.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:33.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:33.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:33.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:33.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:33.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:33.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:33.248 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:32 smithi151 ceph-mon[28965]: osdmap e813: 12 total, 12 up, 10 in 2024-06-27T21:44:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:32 smithi190 ceph-mon[32247]: osdmap e813: 12 total, 12 up, 10 in 2024-06-27T21:44:33.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:33.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:32 smithi192 ceph-mon[32956]: osdmap e813: 12 total, 12 up, 10 in 2024-06-27T21:44:33.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:33.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:33.463 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:44:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:44:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:44:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:44:33.463 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:44:33.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:33.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:33.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:33.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:33.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 15 21742 21727 90.5216 65.6875 0.0103104 0.0110353 2024-06-27T21:44:33.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:33.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:33.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:33.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:33.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:33.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:34.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:34.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:34.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:34.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:34.246 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:33 smithi192 ceph-mon[32956]: pgmap v2026: 47 pgs: 47 active+clean; 1023 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:44:34.247 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:34.247 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:34.247 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:33 smithi192 ceph-mon[32956]: osdmap e814: 12 total, 12 up, 10 in 2024-06-27T21:44:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:33 smithi190 ceph-mon[32247]: pgmap v2026: 47 pgs: 47 active+clean; 1023 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:44:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:34.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:33 smithi190 ceph-mon[32247]: osdmap e814: 12 total, 12 up, 10 in 2024-06-27T21:44:34.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:34.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:33 smithi151 ceph-mon[28965]: pgmap v2026: 47 pgs: 47 active+clean; 1023 MiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:44:34.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:34.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:34.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:33 smithi151 ceph-mon[28965]: osdmap e814: 12 total, 12 up, 10 in 2024-06-27T21:44:34.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:44:34.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:34.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:34.512 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:34.537 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:34.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:34.647 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:44:34.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:34.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 23156 23140 90.3831 88.3125 0.0165365 0.0110257 2024-06-27T21:44:34.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:34 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:44:34.518+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e814 definitely_dead 0 2024-06-27T21:44:34.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:34.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:34.880 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.1. 2024-06-27T21:44:34.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:34.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:34.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:35.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:35.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:35.046 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2089146038' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-27T21:44:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2089146038' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-06-27T21:44:35.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:34 smithi192 ceph-mon[32956]: osdmap e815: 12 total, 12 up, 11 in 2024-06-27T21:44:35.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2089146038' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-27T21:44:35.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2089146038' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-06-27T21:44:35.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:34 smithi151 ceph-mon[28965]: osdmap e815: 12 total, 12 up, 11 in 2024-06-27T21:44:35.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:35.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:44:35.161 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:35.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2089146038' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-06-27T21:44:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2089146038' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-06-27T21:44:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:34 smithi190 ceph-mon[32247]: osdmap e815: 12 total, 12 up, 11 in 2024-06-27T21:44:35.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:35.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:35.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:35.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:35.511 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:35.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:35.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:44:35.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:44:35.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:35.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:44:35.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:44:35.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:35.665 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:44:35.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:44:35.682 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 23430 23414 86.0736 17.125 0.00201708 0.0111782 2024-06-27T21:44:35.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:44:35.686 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:44:35.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:44:35.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:44:35.730 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:44:35.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:44:35.752 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:44:35.752 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:44:35.760 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:35.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:44:35.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:44:35.782 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:35.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:44:35.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:44:35.818 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:44:35.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:44:35.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:44:35.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:44:35.864 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:44:35.864 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:44:35.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:35.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:35.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:36.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:36.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:36.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:35 smithi192 ceph-mon[32956]: pgmap v2028: 47 pgs: 47 active+clean; 1.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 47 MiB/s wr, 745 op/s 2024-06-27T21:44:36.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3779320424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:36.047 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:35 smithi192 ceph-mon[32956]: osdmap e816: 12 total, 12 up, 11 in 2024-06-27T21:44:36.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:36.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:35 smithi190 ceph-mon[32247]: pgmap v2028: 47 pgs: 47 active+clean; 1.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 47 MiB/s wr, 745 op/s 2024-06-27T21:44:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3779320424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:35 smithi190 ceph-mon[32247]: osdmap e816: 12 total, 12 up, 11 in 2024-06-27T21:44:36.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:36.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:36.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:35 smithi151 ceph-mon[28965]: pgmap v2028: 47 pgs: 47 active+clean; 1.1 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 47 MiB/s wr, 745 op/s 2024-06-27T21:44:36.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3779320424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:36.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:35 smithi151 ceph-mon[28965]: osdmap e816: 12 total, 12 up, 11 in 2024-06-27T21:44:36.362 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-06-27T21:44:36.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:36.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:36.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:36.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:36.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:36.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:36.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:36.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 16 23431 23415 81.295 0.0625 0.764772 0.0112104 2024-06-27T21:44:36.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.797 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:36.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:36.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:36.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:36.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:37.010 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:37.011 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:37.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:37.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:37.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:37.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:37.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:37.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:37.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:37.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:37.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:37.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.549 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:37.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:37.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:37.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:37.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 23602 23586 77.5787 10.6875 0.00217193 0.012352 2024-06-27T21:44:37.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:37.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:37.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:37 smithi192 ceph-mon[32956]: pgmap v2031: 47 pgs: 47 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:44:37.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:37 smithi192 ceph-mon[32956]: osdmap e817: 12 total, 12 up, 11 in 2024-06-27T21:44:37.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:37.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:37.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:37.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:44:38.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:38.087 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:38.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:37 smithi151 ceph-mon[28965]: pgmap v2031: 47 pgs: 47 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:44:38.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:37 smithi151 ceph-mon[28965]: osdmap e817: 12 total, 12 up, 11 in 2024-06-27T21:44:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.224 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:38.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:38.254 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:44:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:37 smithi190 ceph-mon[32247]: pgmap v2031: 47 pgs: 47 active+clean; 1.4 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-06-27T21:44:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:37 smithi190 ceph-mon[32247]: osdmap e817: 12 total, 12 up, 11 in 2024-06-27T21:44:38.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:38.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:38.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:38.495 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:38.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:38.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:38.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:44:38.681742+0000 min lat: 0.00103148 max lat: 2.88254 avg lat: 0.0127619 2024-06-27T21:44:38.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:44:38.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 25075 25059 78.3025 92.0625 0.00834937 0.0127619 2024-06-27T21:44:38.724 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:38.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:38.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:38.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:38.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:38.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:39.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:39.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:44:39.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:39.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:39.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:39.263 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:39.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:39.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:39.406 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:39.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:39.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:39.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:39.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:39.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 15 26405 26390 78.5348 83.1875 0.0130919 0.0127257 2024-06-27T21:44:39.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:39.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:39.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:39.893 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:44:39.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:39.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:40.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:40.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:40.040 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:40.041 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:40.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:39 smithi190 ceph-mon[32247]: pgmap v2033: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 1.4 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s; 1391/70023 objects misplaced (1.986%); 398 KiB/s, 6 objects/s recovering 2024-06-27T21:44:40.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3927627170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:44:40.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:40.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:40.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:39 smithi151 ceph-mon[28965]: pgmap v2033: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 1.4 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s; 1391/70023 objects misplaced (1.986%); 398 KiB/s, 6 objects/s recovering 2024-06-27T21:44:40.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3927627170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:44:40.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:39 smithi192 ceph-mon[32956]: pgmap v2033: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 1.4 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s; 1391/70023 objects misplaced (1.986%); 398 KiB/s, 6 objects/s recovering 2024-06-27T21:44:40.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3927627170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:44:40.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:40.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:40.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:40.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:40.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:40.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:40.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:40.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 27679 27663 78.5812 79.5625 0.0183319 0.0127082 2024-06-27T21:44:40.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:44:40.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:40.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:40.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:40.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:40.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:40 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3927627170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:44:41.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:40 smithi190 ceph-mon[32247]: osdmap e818: 12 total, 12 up, 11 in 2024-06-27T21:44:41.026 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:44:41.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:41.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:41.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:41.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:41.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:40 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:41.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3927627170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:44:41.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:40 smithi151 ceph-mon[28965]: osdmap e818: 12 total, 12 up, 11 in 2024-06-27T21:44:41.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:40 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:44:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3927627170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:44:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:40 smithi192 ceph-mon[32956]: osdmap e818: 12 total, 12 up, 11 in 2024-06-27T21:44:41.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:41.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11, 1] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:44:41.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:44:41.364 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-06-27T21:44:41.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 7 2024-06-27T21:44:41.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:41.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:41.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:41.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:41.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:41.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 28751 28735 78.0774 67 0.0133473 0.0127989 2024-06-27T21:44:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.696 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:41.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:41.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:41.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:41.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:41.941 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:41.956 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:41.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:42.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:42.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:42.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:41 smithi151 ceph-mon[28965]: pgmap v2035: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 40 MiB/s wr, 632 op/s; 4421/75708 objects degraded (5.840%); 3609/75708 objects misplaced (4.767%); 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:44:42.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:41 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 4421/75708 objects degraded (5.840%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:42.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:42.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.224 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:42.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:41 smithi190 ceph-mon[32247]: pgmap v2035: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 40 MiB/s wr, 632 op/s; 4421/75708 objects degraded (5.840%); 3609/75708 objects misplaced (4.767%); 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:44:42.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:41 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 4421/75708 objects degraded (5.840%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:41 smithi192 ceph-mon[32956]: pgmap v2035: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 40 MiB/s wr, 632 op/s; 4421/75708 objects degraded (5.840%); 3609/75708 objects misplaced (4.767%); 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:44:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:41 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 4421/75708 objects degraded (5.840%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:42.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:42.360 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:42.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:42.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:42.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:42.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:42.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 15 30238 30223 78.6989 93 0.0159396 0.0126992 2024-06-27T21:44:42.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:42.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:42.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:42.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:42.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:42.906 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:44:42.906 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:44:42.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:43.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:43.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:44:43.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:43.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:43.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:43.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:43.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:43.416 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:43.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:43.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:43.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:44:43.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:43.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:43.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 31754 31738 79.3381 94.6875 0.01983 0.0125966 2024-06-27T21:44:43.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:43.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:44:43.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:44:43.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:43.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:44:43.818 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:44:43.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:44:43.845 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:44:43.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:44:43.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:44:43.890 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:44:43.902 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:44:43.902 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:44:43.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:43.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:44:43.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:44:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:43.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:43.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:43.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:44:43.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:44:44.022 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:44.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:44.084 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:44.154 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:43 smithi151 ceph-mon[28965]: pgmap v2036: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 35 MiB/s wr, 558 op/s; 4421/77337 objects degraded (5.717%); 3609/77337 objects misplaced (4.667%); 6.7 MiB/s, 106 objects/s recovering 2024-06-27T21:44:44.155 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1129208579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:44.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:43 smithi190 ceph-mon[32247]: pgmap v2036: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 35 MiB/s wr, 558 op/s; 4421/77337 objects degraded (5.717%); 3609/77337 objects misplaced (4.667%); 6.7 MiB/s, 106 objects/s recovering 2024-06-27T21:44:44.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1129208579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:44.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:44.196 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:44:44.197 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:44:44.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:44.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:44:44.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:44:44.260 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:44:44.261 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:44:44.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:44:44.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:44:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:43 smithi192 ceph-mon[32956]: pgmap v2036: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 35 MiB/s wr, 558 op/s; 4421/77337 objects degraded (5.717%); 3609/77337 objects misplaced (4.667%); 6.7 MiB/s, 106 objects/s recovering 2024-06-27T21:44:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1129208579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:44.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:44.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:44:44.322 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:44:44.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:44.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:44.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:44.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:44.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:44.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 32976 32960 79.2239 76.375 0.0104823 0.0126016 2024-06-27T21:44:44.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:44.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:44.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:44.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:44.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:44.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:44.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=3532926258,config=2353167246,health=756870642,kv=3737896932,logm=3632933611}) 2024-06-27T21:44:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=8576485}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=961806470}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3324637015}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1850268179}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001801003}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891832641}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408009228}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646485091}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161380945}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698700928}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603840655}) 2024-06-27T21:44:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912861862}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051935271}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527957599}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582192447}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628422289}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750823936}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168816337}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809853797}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676646364}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406313211}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4292172411}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131065231}) 2024-06-27T21:44:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008654823}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1691414870}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3914475416}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158595599}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161212951}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296467024}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1757289789}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971325395}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2648232307}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127317706}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3037844595}) 2024-06-27T21:44:45.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729794501}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360448753}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=978512819}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096758268}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=511055593}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3065679334}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027325025}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4120578301}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857107}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2974736924}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=419942932}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094676745}) 2024-06-27T21:44:45.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456773067}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148814463}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1777100851}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611523882}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038950171}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1831361130}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581674315}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656889890}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788481503}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640884379}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519735671}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=905907020}) 2024-06-27T21:44:45.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568400152}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717530161}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2729099666}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001913576}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629182757}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2476981702}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911299908}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=181415168}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3616828978}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539450871}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881495522}) 2024-06-27T21:44:45.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22210925}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2327142028}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264282220}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701638809}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3343264190}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=399644094,mdsmap=2339142170,mgr=2659146539}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=82} crc {mgr=4178315723,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=457800991,mon_config_key=601977549}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1546514009}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4243126457}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=9,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=3891349713,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=1527823373,osdmap=272794785}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=82} crc {osdmap=1075293361}) 2024-06-27T21:44:45.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:44 smithi190 ceph-mon[32247]: osdmap e819: 12 total, 12 up, 11 in 2024-06-27T21:44:45.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:45.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=3532926258,config=2353167246,health=756870642,kv=3737896932,logm=3632933611}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=8576485}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=961806470}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3324637015}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1850268179}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001801003}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891832641}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408009228}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646485091}) 2024-06-27T21:44:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161380945}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698700928}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603840655}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912861862}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051935271}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527957599}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582192447}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628422289}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750823936}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168816337}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809853797}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676646364}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406313211}) 2024-06-27T21:44:45.202 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4292172411}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131065231}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008654823}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1691414870}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3914475416}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158595599}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161212951}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296467024}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1757289789}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971325395}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2648232307}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127317706}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3037844595}) 2024-06-27T21:44:45.203 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729794501}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360448753}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=978512819}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096758268}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=511055593}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3065679334}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027325025}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4120578301}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857107}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2974736924}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=419942932}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094676745}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456773067}) 2024-06-27T21:44:45.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148814463}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1777100851}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611523882}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038950171}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1831361130}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581674315}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656889890}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788481503}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640884379}) 2024-06-27T21:44:45.205 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519735671}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=905907020}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568400152}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717530161}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2729099666}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001913576}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629182757}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2476981702}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911299908}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=181415168}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3616828978}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539450871}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881495522}) 2024-06-27T21:44:45.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22210925}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2327142028}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264282220}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701638809}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3343264190}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=399644094,mdsmap=2339142170,mgr=2659146539}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=82} crc {mgr=4178315723,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=457800991,mon_config_key=601977549}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1546514009}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4243126457}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=9,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=3891349713,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=1527823373,osdmap=272794785}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=82} crc {osdmap=1075293361}) 2024-06-27T21:44:45.207 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:44 smithi151 ceph-mon[28965]: osdmap e819: 12 total, 12 up, 11 in 2024-06-27T21:44:45.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:45.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:45.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=54,logm=6} crc {auth=3532926258,config=2353167246,health=756870642,kv=3737896932,logm=3632933611}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=8576485}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=961806470}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3324637015}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1850268179}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001801003}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891832641}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408009228}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646485091}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161380945}) 2024-06-27T21:44:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698700928}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603840655}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912861862}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051935271}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527957599}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582192447}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628422289}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750823936}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168816337}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809853797}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676646364}) 2024-06-27T21:44:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406313211}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4292172411}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131065231}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008654823}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1691414870}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3914475416}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=158595599}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161212951}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296467024}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1757289789}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971325395}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2648232307}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3127317706}) 2024-06-27T21:44:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3037844595}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1729794501}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360448753}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=978512819}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3096758268}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=511055593}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3065679334}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027325025}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4120578301}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179857107}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2974736924}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=419942932}) 2024-06-27T21:44:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094676745}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456773067}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148814463}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1777100851}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2611523882}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3038950171}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1831361130}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581674315}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2656889890}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788481503}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640884379}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519735671}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=905907020}) 2024-06-27T21:44:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568400152}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717530161}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2729099666}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1001913576}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629182757}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2476981702}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1911299908}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=181415168}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3616828978}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=539450871}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881495522}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=22210925}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2327142028}) 2024-06-27T21:44:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264282220}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2701638809}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3343264190}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=84,mdsmap=3,mgr=13} crc {logm=399644094,mdsmap=2339142170,mgr=2659146539}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=82} crc {mgr=4178315723,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=457800991,mon_config_key=601977549}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1546514009}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4243126457}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=9,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=3891349713,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=1527823373,osdmap=272794785}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=82} crc {osdmap=1075293361}) 2024-06-27T21:44:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:44 smithi192 ceph-mon[32956]: osdmap e819: 12 total, 12 up, 11 in 2024-06-27T21:44:45.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:45.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:45.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:45.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:45.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:45.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:45.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:45.649 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:44:45.649 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:44:45.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:44:45.649 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.15 636/550 [PushOp(6:adc7d4ea:::benchmark_data_smithi192_117994_object88705:head, version: 496'5501, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:adc7d4ea:::benchmark_data_smithi192_117994_object88705:head@496'5501, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 466.56993179800003, 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.545111978, 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074404+0000", 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074406+0000", 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7090000000000001e-06 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.074452+0000", 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5967999999999998e-05 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617875+0000", 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54342337500000004 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617887+0000", 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.236e-05 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617955+0000", 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7934000000000004e-05 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.617964+0000", 2024-06-27T21:44:45.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8130000000000002e-06 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:36:59.619516+0000", 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015518190000000001 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.21 657/654 [PushOp(6:86e4cd7e:::benchmark_data_smithi192_117994_object88829:head, version: 496'5366, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:86e4cd7e:::benchmark_data_smithi192_117994_object88829:head@496'5366, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:44:45.654 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:38.954212+0000", 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 426.690123234, 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44072743800000003, 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954212+0000", 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954212+0000", 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954213+0000", 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4300000000000002e-07 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:38.954267+0000", 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3646000000000001e-05 2024-06-27T21:44:45.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393751+0000", 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43948427699999998 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393764+0000", 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2349e-05 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393850+0000", 2024-06-27T21:44:45.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6161000000000004e-05 2024-06-27T21:44:45.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.393859+0000", 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8990000000000007e-06 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:39.394940+0000", 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010813629999999999 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.21 658/654 [PushOp(6:86f4a616:::benchmark_data_smithi192_117994_object106008:head, version: 503'6416, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:86f4a616:::benchmark_data_smithi192_117994_object106008:head@503'6416, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:43.012631+0000", 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 422.63170500000001, 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52881486700000002, 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.660 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012631+0000", 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012631+0000", 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012632+0000", 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8700000000000005e-07 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.012685+0000", 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3356000000000002e-05 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540285+0000", 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52760042299999999 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540302+0000", 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6521e-05 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540375+0000", 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2912999999999998e-05 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.540383+0000", 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9230000000000002e-06 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:43.541446+0000", 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010629439999999999 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.21 e659,654 6:8749a36d:::benchmark_data_smithi192_117994_object5123:head,507'6835)", 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:37:45.334891+0000", 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 420.309444735, 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43374579200000002, 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.665 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334891+0000", 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334891+0000", 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334892+0000", 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5600000000000005e-07 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.334896+0000", 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7709999999999997e-06 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768507+0000", 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43361023900000001 2024-06-27T21:44:45.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768515+0000", 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8449999999999992e-06 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768576+0000", 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0411000000000001e-05 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768586+0000", 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9499999999999996e-06 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:37:45.768637+0000", 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.092e-05 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.26020.0:208043 6.6 6:60c693c3:::benchmark_data_smithi192_117994_object100449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:38:29.151456+0000", 2024-06-27T21:44:45.669 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 376.49287927199998, 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35514869399999999, 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.26020", 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/1199121583", 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 208043 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151456+0000", 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151456+0000", 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151458+0000", 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1570000000000001e-06 2024-06-27T21:44:45.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151462+0000", 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7330000000000003e-06 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151475+0000", 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.314e-05 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151484+0000", 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4069999999999993e-06 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151536+0000", 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2608000000000002e-05 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151615+0000", 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8022000000000006e-05 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 7,11", 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.151904+0000", 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028929699999999999 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.153958+0000", 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020541629999999999 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.154599+0000", 2024-06-27T21:44:45.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064101700000000002 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.506497+0000", 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35189832500000001 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.506554+0000", 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6774000000000002e-05 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:38:29.506605+0000", 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1051000000000003e-05 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e705,698 6:601ec4c4:::benchmark_data_smithi192_117994_object69452:head,624'10423)", 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:09.763980+0000", 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 335.88035554200002, 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54553509600000005, 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.676 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763980+0000", 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763980+0000", 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763981+0000", 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9500000000000002e-07 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.763987+0000", 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2149999999999998e-06 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309162+0000", 2024-06-27T21:44:45.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.545174661 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309171+0000", 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1020000000000003e-06 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309223+0000", 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1712999999999998e-05 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.309515+0000", 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029271000000000001 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e706,698 6:643d305c:::benchmark_data_smithi192_117994_object73633:head,641'10548)", 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 334.76565887300001, 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41167358999999998, 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.680 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878677+0000", 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6599999999999996e-07 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:10.878681+0000", 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1990000000000002e-06 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290033+0000", 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41135234700000001 2024-06-27T21:44:45.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290043+0000", 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6709999999999994e-06 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290092+0000", 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9014000000000002e-05 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:11.290350+0000", 2024-06-27T21:44:45.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000258793 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(6.6 e707,698 6:616e8dab:::benchmark_data_smithi192_117994_object96457:head,675'11245)", 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 331.61484096300001, 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37847040300000001, 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:44:45.684 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029495+0000", 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9200000000000001e-07 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.029507+0000", 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1748e-05 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407628+0000", 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37812072299999999 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407644+0000", 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6121999999999999e-05 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407766+0000", 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000122533 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.407965+0000", 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019868500000000001 2024-06-27T21:44:45.687 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.688 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.688 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.689 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:128206 7.8 e757/718 7:18395da9:::benchmark_data_smithi192_150129_object24901:head v 757'7922, mlcod=757'7919)", 2024-06-27T21:44:45.689 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:41:47.067343+0000", 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 178.576993211, 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.235923413, 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.067343+0000", 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.067343+0000", 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.067343+0000", 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8199999999999999e-07 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.067349+0000", 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3129999999999998e-06 2024-06-27T21:44:45.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.300411+0000", 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23306254500000001 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.300415+0000", 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2180000000000001e-06 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.300487+0000", 2024-06-27T21:44:45.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1431000000000001e-05 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.300501+0000", 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4497e-05 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.303234+0000", 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027330549999999999 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:41:47.303266+0000", 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1671999999999998e-05 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:164264 7.b e767/718 7:d9f3fd60:::benchmark_data_smithi192_150129_object60959:head v 767'10396, mlcod=767'10393)", 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:42:36.451757+0000", 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 129.19257830000001, 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.245519718, 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.694 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.695 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 15 34208 34193 79.1436 77.0625 0.00887867 0.0126293 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.451757+0000", 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.451757+0000", 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.451758+0000", 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.95e-07 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.451763+0000", 2024-06-27T21:44:45.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4839999999999998e-06 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.693498+0000", 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24173482299999999 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.693506+0000", 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1480000000000006e-06 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.693590+0000", 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4068999999999998e-05 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.693617+0000", 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6838999999999998e-05 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.697241+0000", 2024-06-27T21:44:45.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036240489999999998 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:42:36.697277+0000", 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6310999999999998e-05 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:183017 7.3 e781/718 7:c4242f40:::benchmark_data_smithi192_150129_object79712:head v 781'11154, mlcod=781'11150)", 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:43:14.415251+0000", 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 91.229084960999998, 2024-06-27T21:44:45.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10796175099999999, 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.415251+0000", 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.415251+0000", 2024-06-27T21:44:45.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.415251+0000", 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.75e-07 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.415255+0000", 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4010000000000001e-06 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.519654+0000", 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104399751 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.519660+0000", 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9900000000000002e-06 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.519709+0000", 2024-06-27T21:44:45.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8711000000000002e-05 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.519725+0000", 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5637999999999998e-05 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.523183+0000", 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003458488 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.523213+0000", 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9196999999999998e-05 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:183022 7.3 e781/718 7:c8e1546b:::benchmark_data_smithi192_150129_object79717:head v 781'11155, mlcod=781'11151)", 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:43:14.421640+0000", 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 91.222695400000006, 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13078693499999999, 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.705 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.421640+0000", 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.421640+0000", 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.421641+0000", 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5000000000000002e-07 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.421646+0000", 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8300000000000003e-06 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.548293+0000", 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.12664716500000001 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.548298+0000", 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8470000000000003e-06 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.548348+0000", 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0163000000000001e-05 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.548366+0000", 2024-06-27T21:44:45.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7808999999999998e-05 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.552400+0000", 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00403448 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:14.552427+0000", 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6991e-05 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:201852 7.6 e788/718 7:682df53f:::benchmark_data_smithi192_150129_object98547:head v 788'12553, mlcod=788'12550)", 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:43:38.900958+0000", 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 66.743377382000006, 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.11323024900000001, 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:38.900958+0000", 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:38.900958+0000", 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:38.900959+0000", 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8999999999999996e-07 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:38.900965+0000", 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3249999999999996e-06 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.010769+0000", 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.109803568 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.010774+0000", 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4280000000000004e-06 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.010824+0000", 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9567999999999997e-05 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.010857+0000", 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3416e-05 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.014156+0000", 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032986909999999999 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.014189+0000", 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2563e-05 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.26656.0:201939 7.6 e788/718 7:6f36ec76:::benchmark_data_smithi192_150129_object98634:head v 788'12555, mlcod=788'12552)", 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:43:39.075156+0000", 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 66.569179415999997, 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.11350546, 2024-06-27T21:44:45.714 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.075156+0000", 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.075156+0000", 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.075157+0000", 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0100000000000003e-07 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.075162+0000", 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8409999999999999e-06 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.185121+0000", 2024-06-27T21:44:45.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10995927899999999 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.185134+0000", 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.226e-05 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.185196+0000", 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2485999999999998e-05 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.185211+0000", 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.452e-05 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.188625+0000", 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034147259999999999 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:43:39.188662+0000", 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6446999999999998e-05 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.718 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:16931 8.2 e810/797 8:45eb235e:::benchmark_data_smithi192_167666_object16930:head v 810'1078, mlcod=810'1075)", 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:29.443376+0000", 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.200959684000001, 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18152353800000001, 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.443376+0000", 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.443376+0000", 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.443377+0000", 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0299999999999999e-07 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.443428+0000", 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1289000000000003e-05 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.623873+0000", 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.180445562 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.623891+0000", 2024-06-27T21:44:45.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8012999999999998e-05 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.624043+0000", 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015109999999999999 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.624081+0000", 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8816000000000002e-05 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.624866+0000", 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00078448999999999999 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:29.624900+0000", 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3764999999999999e-05 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.723 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:18801 8.5 e812/797 8:ac8091dd:::benchmark_data_smithi192_167666_object18800:head v 812'1126, mlcod=811'1122)", 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:31.027687+0000", 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.616648375, 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.098384417000000002, 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.027687+0000", 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.027687+0000", 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.027688+0000", 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0399999999999996e-07 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.027736+0000", 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8581000000000001e-05 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.125152+0000", 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.097415350999999997 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.125161+0000", 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3840000000000003e-06 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.125220+0000", 2024-06-27T21:44:45.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8786000000000003e-05 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.125239+0000", 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8961999999999999e-05 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.126033+0000", 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079425000000000003 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:31.126072+0000", 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8599000000000001e-05 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:23124 8.b e814/797 8:d889794c:::benchmark_data_smithi192_167666_object23123:head v 814'1409, mlcod=814'1406)", 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:34.605151+0000", 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.039185208999999, 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.134741678, 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.728 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.605151+0000", 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.605151+0000", 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.605151+0000", 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6499999999999999e-07 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.605173+0000", 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2178e-05 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.738944+0000", 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13377117399999999 2024-06-27T21:44:45.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.738949+0000", 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6090000000000003e-06 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.739031+0000", 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1897999999999994e-05 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.739052+0000", 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1089000000000001e-05 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.739860+0000", 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080811399999999997 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.739892+0000", 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2150999999999998e-05 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.732 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:23153 8.f e814/797 8:ffaae4d7:::benchmark_data_smithi192_167666_object23152:head v 814'1475, mlcod=814'1472)", 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:34.646165+0000", 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.998170563, 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.095937568000000001, 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.646165+0000", 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.646165+0000", 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.646166+0000", 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1099999999999995e-07 2024-06-27T21:44:45.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.646197+0000", 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1299e-05 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.741280+0000", 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.095082948 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.741285+0000", 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8450000000000002e-06 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.741350+0000", 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4936000000000003e-05 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.741364+0000", 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3709e-05 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.742072+0000", 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000708098 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.742103+0000", 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1121999999999998e-05 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.737 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:23402 8.0 e815/797 8:0cf7414f:::benchmark_data_smithi192_167666_object23401:head v 815'1399, mlcod=814'1396)", 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:34.897033+0000", 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.747302317000001, 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27473579100000001, 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897033+0000", 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897033+0000", 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897034+0000", 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9500000000000002e-07 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897149+0000", 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011548 2024-06-27T21:44:45.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170582+0000", 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27343230699999999 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170594+0000", 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2294e-05 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170749+0000", 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000155322 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170790+0000", 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0772999999999997e-05 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.171735+0000", 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00094516900000000004 2024-06-27T21:44:45.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.171769+0000", 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3850999999999998e-05 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:23416 8.2 e815/797 8:4f4be8b9:::benchmark_data_smithi192_167666_object23415:head v 815'1485, mlcod=814'1482)", 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:35.040870+0000", 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.603465525000001, 2024-06-27T21:44:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31619940000000002, 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040870+0000", 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040870+0000", 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040871+0000", 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0890000000000001e-06 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040902+0000", 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.042e-05 2024-06-27T21:44:45.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356031+0000", 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31512962300000003 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356037+0000", 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5720000000000002e-06 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:44:45.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356119+0000", 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1866999999999994e-05 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356135+0000", 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6730000000000001e-05 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.357034+0000", 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00089823599999999996 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:44:45.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.357070+0000", 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5862999999999999e-05 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:44:45.747 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:44:45.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:45.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:45.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:45.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:45.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:46.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:46.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:46.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:45 smithi151 ceph-mon[28965]: pgmap v2037: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 45 MiB/s wr, 720 op/s; 4421/84771 objects degraded (5.215%); 3108/84771 objects misplaced (3.666%); 9.3 MiB/s, 147 objects/s recovering 2024-06-27T21:44:46.207 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:45 smithi190 ceph-mon[32247]: pgmap v2037: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 45 MiB/s wr, 720 op/s; 4421/84771 objects degraded (5.215%); 3108/84771 objects misplaced (3.666%); 9.3 MiB/s, 147 objects/s recovering 2024-06-27T21:44:46.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:45 smithi192 ceph-mon[32956]: pgmap v2037: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 1.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 45 MiB/s wr, 720 op/s; 4421/84771 objects degraded (5.215%); 3108/84771 objects misplaced (3.666%); 9.3 MiB/s, 147 objects/s recovering 2024-06-27T21:44:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:46.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:46.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:46.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:46.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:46.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:46.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:46.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 35525 35509 79.2543 82.25 0.00678265 0.01261 2024-06-27T21:44:46.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:46.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:44:46.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:46.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:46.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:46.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:46.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:47.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:47.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:47.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:46 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 4421/97059 objects degraded (4.555%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:47.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:47.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:46 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 4421/97059 objects degraded (4.555%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:46 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 4421/97059 objects degraded (4.555%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:47.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:47.388 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:47.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:47.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:47.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:47.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:47.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 36627 36611 78.8962 68.875 0.0373124 0.0126592 2024-06-27T21:44:47.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:47.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:47.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:47.910 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:47.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:48.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:48.040 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:48.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:48.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:47 smithi151 ceph-mon[28965]: pgmap v2039: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 4421/97059 objects degraded (4.555%); 2015/97059 objects misplaced (2.076%); 18 MiB/s, 279 objects/s recovering 2024-06-27T21:44:48.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:47 smithi190 ceph-mon[32247]: pgmap v2039: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 4421/97059 objects degraded (4.555%); 2015/97059 objects misplaced (2.076%); 18 MiB/s, 279 objects/s recovering 2024-06-27T21:44:48.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:47 smithi192 ceph-mon[32956]: pgmap v2039: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 4421/97059 objects degraded (4.555%); 2015/97059 objects misplaced (2.076%); 18 MiB/s, 279 objects/s recovering 2024-06-27T21:44:48.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:48.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:48.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:48.534 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:48.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:48.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:48.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 37760 37744 78.6266 70.8125 0.0200358 0.0127104 2024-06-27T21:44:48.699 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:48.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:48.811 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:48.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:48.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:48.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:48.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:48.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:48.994 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:44:49.001 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.7. 2024-06-27T21:44:49.056 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:49.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:49.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:44:48.890+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e819 definitely_dead 0 2024-06-27T21:44:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:44:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:44:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3164269877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:49.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4063982381' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:44:49.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:48 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:44:49.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:48 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:44:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:48 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:44:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3164269877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:48 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4063982381' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:44:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:48 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:44:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:48 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:44:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:48 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:44:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3164269877' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:48 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:44:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4063982381' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:44:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:48 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-06-27T21:44:49.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:49.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:49.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:49.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:49.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:49.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:49.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:49.683 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 38269 38253 77.1163 31.8125 0.00496716 0.0127198 2024-06-27T21:44:49.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:49.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.824 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:49.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:49.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:49.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:49.951 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:49.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:50.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:50.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:50.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:50.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:50.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:44:50.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:49 smithi151 ceph-mon[28965]: pgmap v2040: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 2.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s; 4421/102942 objects degraded (4.295%); 1549/102942 objects misplaced (1.505%); 15 MiB/s, 245 objects/s recovering 2024-06-27T21:44:50.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:49 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:44:50.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:50.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-27T21:44:50.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:49 smithi151 ceph-mon[28965]: osdmap e820: 12 total, 12 up, 12 in 2024-06-27T21:44:50.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:49 smithi151 ceph-mon[28965]: osdmap e821: 12 total, 12 up, 12 in 2024-06-27T21:44:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:49 smithi190 ceph-mon[32247]: pgmap v2040: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 2.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s; 4421/102942 objects degraded (4.295%); 1549/102942 objects misplaced (1.505%); 15 MiB/s, 245 objects/s recovering 2024-06-27T21:44:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:49 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:44:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-27T21:44:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:49 smithi190 ceph-mon[32247]: osdmap e820: 12 total, 12 up, 12 in 2024-06-27T21:44:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:49 smithi190 ceph-mon[32247]: osdmap e821: 12 total, 12 up, 12 in 2024-06-27T21:44:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:49 smithi192 ceph-mon[32956]: pgmap v2040: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 41 active+clean; 2.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s; 4421/102942 objects degraded (4.295%); 1549/102942 objects misplaced (1.505%); 15 MiB/s, 245 objects/s recovering 2024-06-27T21:44:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:49 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:44:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:44:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-06-27T21:44:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:49 smithi192 ceph-mon[32956]: osdmap e820: 12 total, 12 up, 12 in 2024-06-27T21:44:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:49 smithi192 ceph-mon[32956]: osdmap e821: 12 total, 12 up, 12 in 2024-06-27T21:44:50.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:50.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:50.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:50.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:50.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:50.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:50.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:50.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 16 38304 38288 74.7748 2.1875 0.00275622 0.0128396 2024-06-27T21:44:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:50.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:50.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:50.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:50.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:50.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:50.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:51.028 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-06-27T21:44:51.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:51.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:51.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:51.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:44:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:44:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:50 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:44:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:50 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:44:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:50 smithi190 ceph-mon[32247]: osdmap e822: 12 total, 12 up, 12 in 2024-06-27T21:44:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:44:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:44:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:50 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:44:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:50 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:44:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:50 smithi192 ceph-mon[32956]: osdmap e822: 12 total, 12 up, 12 in 2024-06-27T21:44:51.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:51.337 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:51.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:44:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:44:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:50 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:44:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:50 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:44:51.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:50 smithi151 ceph-mon[28965]: osdmap e822: 12 total, 12 up, 12 in 2024-06-27T21:44:51.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:51.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:51.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:51.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 39037 39021 73.897 45.8125 0.0455256 0.0135242 2024-06-27T21:44:51.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:51.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:51.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:44:51.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:44:51.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:51.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:51.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:52.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:52.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:52 smithi151 ceph-mon[28965]: pgmap v2043: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 37 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 1467/111906 objects degraded (1.311%); 3111/111906 objects misplaced (2.780%); 31 MiB/s, 491 objects/s recovering 2024-06-27T21:44:52.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:52 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:44:52.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:52 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:44:52.103 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:52.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:52.126 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:44:52.126 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:44:52.127 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:44:52.127 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:44:52.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:52.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:52 smithi190 ceph-mon[32247]: pgmap v2043: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 37 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 1467/111906 objects degraded (1.311%); 3111/111906 objects misplaced (2.780%); 31 MiB/s, 491 objects/s recovering 2024-06-27T21:44:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:52 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:44:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:52 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:44:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:52 smithi192 ceph-mon[32956]: pgmap v2043: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 37 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 94 MiB/s wr, 1.51k op/s; 1467/111906 objects degraded (1.311%); 3111/111906 objects misplaced (2.780%); 31 MiB/s, 491 objects/s recovering 2024-06-27T21:44:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:52 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:44:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:52 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:44:52.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:52.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:52.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:52.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:52.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:52.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:52.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 39574 39558 72.7106 33.5625 0.00302723 0.013546 2024-06-27T21:44:52.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:52.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:52.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:52.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:52.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:52.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:52.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:53.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:53.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:53.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:53.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:53.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:53 smithi192 ceph-mon[32956]: osdmap e823: 12 total, 12 up, 12 in 2024-06-27T21:44:53.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:53.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:53.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:53 smithi151 ceph-mon[28965]: osdmap e823: 12 total, 12 up, 12 in 2024-06-27T21:44:53.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:53.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:53.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:53 smithi190 ceph-mon[32247]: osdmap e823: 12 total, 12 up, 12 in 2024-06-27T21:44:53.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:53.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:44:53.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:53.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 15 40521 40506 72.3259 59.25 0.00829328 0.0138177 2024-06-27T21:44:53.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:53.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:53.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:53.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:53.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:53.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:53.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:54.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:54.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:54.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:54.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:44:54.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:54 smithi190 ceph-mon[32247]: pgmap v2046: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 37 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s; 1467/113277 objects degraded (1.295%); 3111/113277 objects misplaced (2.746%); 22 MiB/s, 347 objects/s recovering 2024-06-27T21:44:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 1467/113277 objects degraded (1.295%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:54 smithi190 ceph-mon[32247]: osdmap e824: 12 total, 12 up, 12 in 2024-06-27T21:44:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:54 smithi192 ceph-mon[32956]: pgmap v2046: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 37 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s; 1467/113277 objects degraded (1.295%); 3111/113277 objects misplaced (2.746%); 22 MiB/s, 347 objects/s recovering 2024-06-27T21:44:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 1467/113277 objects degraded (1.295%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:54 smithi192 ceph-mon[32956]: osdmap e824: 12 total, 12 up, 12 in 2024-06-27T21:44:54.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:54.376 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:54 smithi151 ceph-mon[28965]: pgmap v2046: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 6 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 37 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 54 MiB/s wr, 861 op/s; 1467/113277 objects degraded (1.295%); 3111/113277 objects misplaced (2.746%); 22 MiB/s, 347 objects/s recovering 2024-06-27T21:44:54.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 1467/113277 objects degraded (1.295%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:54.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:54 smithi151 ceph-mon[28965]: osdmap e824: 12 total, 12 up, 12 in 2024-06-27T21:44:54.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:54.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:54.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:54.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:54.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:54.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 41309 41293 71.6831 49.1875 0.0451542 0.0139409 2024-06-27T21:44:54.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:54.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:54.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:54.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:54.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:54.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:55.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:55.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:55.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:55.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:55.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:55.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:55.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:55.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:55.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:55.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:55.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:44:55.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:44:55.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 42260 42244 71.352 59.4375 0.0193049 0.0140081 2024-06-27T21:44:55.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:55.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:44:55.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:55.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:55.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:55.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:55.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:44:55.946 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:44:55.977 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:44:55.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:44:56.008 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:44:56.008 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:44:56.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:56.029 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11, 1, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:44:56.029 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:44:56.030 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-06-27T21:44:56.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:44:56.035 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:44:56.035 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:44:56.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:56.065 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:44:56.065 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:44:56.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:44:56.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:44:56.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:44:56.126 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:44:56.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:56.157 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:44:56.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:44:56.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:55 smithi151 ceph-mon[28965]: pgmap v2048: 47 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 peering, 3 active+recovering+undersized+remapped, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 346 op/s; 2/116484 objects degraded (0.002%); 6266/116484 objects misplaced (5.379%); 10 MiB/s, 164 objects/s recovering 2024-06-27T21:44:56.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:55 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:44:56.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:55 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:44:56.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:55 smithi151 ceph-mon[28965]: osdmap e825: 12 total, 12 up, 12 in 2024-06-27T21:44:56.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:56.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:44:56.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:44:56.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:44:56.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:44:56.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:44:56.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:44:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:55 smithi190 ceph-mon[32247]: pgmap v2048: 47 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 peering, 3 active+recovering+undersized+remapped, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 346 op/s; 2/116484 objects degraded (0.002%); 6266/116484 objects misplaced (5.379%); 10 MiB/s, 164 objects/s recovering 2024-06-27T21:44:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:55 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:44:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:55 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:44:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:55 smithi190 ceph-mon[32247]: osdmap e825: 12 total, 12 up, 12 in 2024-06-27T21:44:56.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:44:56.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:44:56.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:55 smithi192 ceph-mon[32956]: pgmap v2048: 47 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 peering, 3 active+recovering+undersized+remapped, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 346 op/s; 2/116484 objects degraded (0.002%); 6266/116484 objects misplaced (5.379%); 10 MiB/s, 164 objects/s recovering 2024-06-27T21:44:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:55 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:44:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:55 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:44:56.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:55 smithi192 ceph-mon[32956]: osdmap e825: 12 total, 12 up, 12 in 2024-06-27T21:44:56.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:56.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:56.425 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:56.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:56.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:56.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:44:56.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 43335 43319 71.2423 67.1875 0.00378151 0.0140296 2024-06-27T21:44:56.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:44:56.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:56.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:56.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:56.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:56.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:56.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2373402126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:57.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:44:57.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:44:57.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:57.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2373402126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:57.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2373402126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:44:57.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:57.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:57.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:57.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:44:57.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:44:57.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:57.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 44421 44405 71.1558 67.875 0.00624941 0.0140459 2024-06-27T21:44:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.712 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:57.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:57.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:57.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:57.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:57.982 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:57.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:58.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:58.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:58.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:57 smithi151 ceph-mon[28965]: pgmap v2050: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 614 op/s; 2458/122649 objects degraded (2.004%); 7468/122649 objects misplaced (6.089%); 21 MiB/s, 341 objects/s recovering 2024-06-27T21:44:58.201 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:44:58.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:58.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:58.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:57 smithi190 ceph-mon[32247]: pgmap v2050: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 614 op/s; 2458/122649 objects degraded (2.004%); 7468/122649 objects misplaced (6.089%); 21 MiB/s, 341 objects/s recovering 2024-06-27T21:44:58.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:57 smithi192 ceph-mon[32956]: pgmap v2050: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 614 op/s; 2458/122649 objects degraded (2.004%); 7468/122649 objects misplaced (6.089%); 21 MiB/s, 341 objects/s recovering 2024-06-27T21:44:58.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:58.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:58.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:44:58.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:44:58.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:58.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:58.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:44:58.683435+0000 min lat: 0.00103148 max lat: 2.88254 avg lat: 0.0139756 2024-06-27T21:44:58.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:44:58.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 45723 45707 71.411 81.375 0.0290804 0.0139756 2024-06-27T21:44:58.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:44:58.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:44:58.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:58.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:58.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:58.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:58.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:58 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2458/127416 objects degraded (1.929%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:59.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:59.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:44:59.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:44:59.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:44:59.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:59.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:58 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2458/127416 objects degraded (1.929%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:59.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:58 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2458/127416 objects degraded (1.929%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:44:59.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:44:59.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:44:59.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:44:59.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:44:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:44:59.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:59.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:59.684 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 16 47009 46993 71.6295 80.375 0.0335934 0.0139533 2024-06-27T21:44:59.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:44:59.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:44:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:44:59.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:44:59.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:44:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:44:59.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:00.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:00.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:00.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:00.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:44:59 smithi151 ceph-mon[28965]: pgmap v2051: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 748 op/s; 2458/127416 objects degraded (1.929%); 7101/127416 objects misplaced (5.573%); 22 MiB/s, 351 objects/s recovering 2024-06-27T21:45:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.207 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:00.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:00.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:44:59 smithi192 ceph-mon[32956]: pgmap v2051: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 748 op/s; 2458/127416 objects degraded (1.929%); 7101/127416 objects misplaced (5.573%); 22 MiB/s, 351 objects/s recovering 2024-06-27T21:45:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:44:59 smithi190 ceph-mon[32247]: pgmap v2051: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 748 op/s; 2458/127416 objects degraded (1.929%); 7101/127416 objects misplaced (5.573%); 22 MiB/s, 351 objects/s recovering 2024-06-27T21:45:00.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:00.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:00.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:00.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:00.467 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:45:00.468 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:45:00.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:00.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:00.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 48021 48005 71.4298 63.25 0.00985001 0.0139914 2024-06-27T21:45:00.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:00.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:00.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:00.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:00.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:00.978 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:00.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:01.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:01.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:01.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:01.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:01.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:01.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:01.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:01.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:01.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:01.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:01.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 49220 49204 71.5113 74.9375 0.00470347 0.013977 2024-06-27T21:45:01.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:01.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:01.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:01.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:01.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:01.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:02.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:02.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:02.196 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:01 smithi151 ceph-mon[28965]: pgmap v2052: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 987 op/s; 2458/136980 objects degraded (1.794%); 5355/136980 objects misplaced (3.909%); 31 MiB/s, 495 objects/s recovering 2024-06-27T21:45:02.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:02.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:01 smithi190 ceph-mon[32247]: pgmap v2052: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 987 op/s; 2458/136980 objects degraded (1.794%); 5355/136980 objects misplaced (3.909%); 31 MiB/s, 495 objects/s recovering 2024-06-27T21:45:02.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:01 smithi192 ceph-mon[32956]: pgmap v2052: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 987 op/s; 2458/136980 objects degraded (1.794%); 5355/136980 objects misplaced (3.909%); 31 MiB/s, 495 objects/s recovering 2024-06-27T21:45:02.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:02.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:02.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:02.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:02.607 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:45:02.608 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:45:02.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:02.655 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:02.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 50479 50463 71.6742 78.6875 0.00604473 0.0139451 2024-06-27T21:45:02.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:02.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:02.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:02.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:02.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:03.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:03.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:03.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:03.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:03.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:03.318 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:03.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:03.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:03.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:03.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:03.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:03.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:03.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 50990 50974 70.7911 31.9375 0.00698929 0.013955 2024-06-27T21:45:03.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:03.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:03.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:03.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:03.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:03.965 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:45:04.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:04.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:04.069 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2053,"stamp":"2024-06-27T21:45:02.297508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3041395232,"num_objects":46401,"num_object_clones":0,"num_object_copies":139203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2458,"num_objects_misplaced":5355,"num_objects_unfound":0,"num_objects_dirty":46401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46399,"num_write_kb":2969536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8101,"num_bytes_recovered":531585696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49251,"ondisk_log_size":49251,"up":141,"acting":135,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":150,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16633176,"kb_used_data":9130872,"kb_used_omap":110,"kb_used_meta":7502097,"kb_avail":1108308648,"statfs":{"total":1151940427776,"available":1134908055552,"internally_reserved":0,"allocated":9350012928,"data_stored":9344877216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":7682148311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":539754496,"num_objects":8236,"num_object_clones":0,"num_object_copies":24708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":991,"num_objects_misplaced":2244,"num_objects_unfound":0,"num_objects_dirty":8236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8236,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3961,"num_bytes_recovered":259588096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.284828"},"pg_stats":[{"pgid":"8.2","version":"825'3007","reported_seq":3073,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:59.782103+0000","last_change":"2024-06-27T21:44:09.369368+0000","last_active":"2024-06-27T21:44:59.782103+0000","last_peered":"2024-06-27T21:44:59.782103+0000","last_clean":"2024-06-27T21:44:59.782103+0000","last_became_active":"2024-06-27T21:44:09.368432+0000","last_became_peered":"2024-06-27T21:44:09.368432+0000","last_unstale":"2024-06-27T21:44:59.782103+0000","last_undegraded":"2024-06-27T21:44:59.782103+0000","last_fullsized":"2024-06-27T21:44:59.782103+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":3007,"log_dups_size":0,"ondisk_log_size":3007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:34.566329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197066752,"num_objects":3007,"num_object_clones":0,"num_object_copies":9021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3007,"num_write_kb":192448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":110,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.889135+0000","last_change":"2024-06-27T21:44:19.389009+0000","last_active":"2024-06-27T21:44:54.889135+0000","last_peered":"2024-06-27T21:44:54.889135+0000","last_clean":"2024-06-27T21:44:54.889135+0000","last_became_active":"2024-06-27T21:43:11.867925+0000","last_became_peered":"2024-06-27T21:43:11.867925+0000","last_unstale":"2024-06-27T21:44:54.889135+0000","last_undegraded":"2024-06-27T21:44:54.889135+0000","last_fullsized":"2024-06-27T21:44:54.889135+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:19.388901+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:56.646935+0000","last_clean_scrub_stamp":"2024-06-27T21:44:19.388901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:29.655005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.5","version":"825'2795","reported_seq":5670,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:59.777502+0000","last_change":"2024-06-27T21:44:49.897778+0000","last_active":"2024-06-27T21:44:59.777502+0000","last_peered":"2024-06-27T21:44:59.777502+0000","last_clean":"2024-06-27T21:44:59.777502+0000","last_became_active":"2024-06-27T21:44:49.892618+0000","last_became_peered":"2024-06-27T21:44:49.892618+0000","last_unstale":"2024-06-27T21:44:59.777502+0000","last_undegraded":"2024-06-27T21:44:59.777502+0000","last_fullsized":"2024-06-27T21:44:59.777502+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2795,"log_dups_size":0,"ondisk_log_size":2795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:17.006980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183173120,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":178880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1400,"num_bytes_recovered":91750400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":784,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.925161+0000","last_change":"2024-06-27T21:43:35.233857+0000","last_active":"2024-06-27T21:44:54.925161+0000","last_peered":"2024-06-27T21:44:54.925161+0000","last_clean":"2024-06-27T21:44:54.925161+0000","last_became_active":"2024-06-27T21:32:59.709361+0000","last_became_peered":"2024-06-27T21:32:59.709361+0000","last_unstale":"2024-06-27T21:44:54.925161+0000","last_undegraded":"2024-06-27T21:44:54.925161+0000","last_fullsized":"2024-06-27T21:44:54.925161+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:35.233787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:35.233787+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:02.828491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"825'2862","reported_seq":2928,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:57.698742+0000","last_change":"2024-06-27T21:44:09.359654+0000","last_active":"2024-06-27T21:44:57.698742+0000","last_peered":"2024-06-27T21:44:57.698742+0000","last_clean":"2024-06-27T21:44:57.698742+0000","last_became_active":"2024-06-27T21:44:09.359363+0000","last_became_peered":"2024-06-27T21:44:09.359363+0000","last_unstale":"2024-06-27T21:44:57.698742+0000","last_undegraded":"2024-06-27T21:44:57.698742+0000","last_fullsized":"2024-06-27T21:44:57.698742+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:32.659982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":183168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,11],"acting":[5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":19,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.900011+0000","last_change":"2024-06-27T21:44:49.893971+0000","last_active":"2024-06-27T21:44:54.900011+0000","last_peered":"2024-06-27T21:44:54.900011+0000","last_clean":"2024-06-27T21:44:54.900011+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:44:54.900011+0000","last_undegraded":"2024-06-27T21:44:54.900011+0000","last_fullsized":"2024-06-27T21:44:54.900011+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:28.309166+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:43:28.309166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"825'2949","reported_seq":5219,"reported_epoch":825,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:44:59.751138+0000","last_change":"2024-06-27T21:44:48.042765+0000","last_active":"2024-06-27T21:44:59.751138+0000","last_peered":"2024-06-27T21:44:59.751138+0000","last_clean":"2024-06-27T21:44:34.875798+0000","last_became_active":"2024-06-27T21:44:37.191582+0000","last_became_peered":"2024-06-27T21:44:37.191582+0000","last_unstale":"2024-06-27T21:44:59.751138+0000","last_undegraded":"2024-06-27T21:44:59.751138+0000","last_fullsized":"2024-06-27T21:44:36.899084+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193265664,"num_objects":2949,"num_object_clones":0,"num_object_copies":8847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":368,"num_objects_unfound":0,"num_objects_dirty":2949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2949,"num_write_kb":188736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1100,"num_bytes_recovered":72089600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,10],"avail_no_missing":["11","2","10"],"object_location_counts":[{"shards":"2,10,11","objects":368},{"shards":"10,11","objects":2581}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":787,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.886628+0000","last_change":"2024-06-27T21:44:49.903426+0000","last_active":"2024-06-27T21:44:54.886628+0000","last_peered":"2024-06-27T21:44:54.886628+0000","last_clean":"2024-06-27T21:44:54.886628+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:44:54.886628+0000","last_undegraded":"2024-06-27T21:44:54.886628+0000","last_fullsized":"2024-06-27T21:44:54.886628+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:13.712411+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:44:13.712411+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:35.562783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"825'2900","reported_seq":4230,"reported_epoch":825,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:44:59.750683+0000","last_change":"2024-06-27T21:44:51.191239+0000","last_active":"2024-06-27T21:44:59.750683+0000","last_peered":"2024-06-27T21:44:59.750683+0000","last_clean":"2024-06-27T21:44:48.991120+0000","last_became_active":"2024-06-27T21:44:51.129910+0000","last_became_peered":"2024-06-27T21:44:51.129910+0000","last_unstale":"2024-06-27T21:44:59.750683+0000","last_undegraded":"2024-06-27T21:44:59.750683+0000","last_fullsized":"2024-06-27T21:44:50.892340+0000","mapping_epoch":821,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2900,"log_dups_size":0,"ondisk_log_size":2900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190054400,"num_objects":2900,"num_object_clones":0,"num_object_copies":8700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1695,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2900,"num_write_kb":185600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":41287680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10],"avail_no_missing":["11","2","10"],"object_location_counts":[{"shards":"2,10,11","objects":1695},{"shards":"10,11","objects":1205}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1404,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.928564+0000","last_change":"2024-06-27T21:44:54.724898+0000","last_active":"2024-06-27T21:44:54.928564+0000","last_peered":"2024-06-27T21:44:54.928564+0000","last_clean":"2024-06-27T21:44:54.928564+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:44:54.928564+0000","last_undegraded":"2024-06-27T21:44:54.928564+0000","last_fullsized":"2024-06-27T21:44:54.928564+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:54.724815+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:44:54.724815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:46:04.534025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"825'2926","reported_seq":2992,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:59.759774+0000","last_change":"2024-06-27T21:44:09.348636+0000","last_active":"2024-06-27T21:44:59.759774+0000","last_peered":"2024-06-27T21:44:59.759774+0000","last_clean":"2024-06-27T21:44:59.759774+0000","last_became_active":"2024-06-27T21:44:09.348307+0000","last_became_peered":"2024-06-27T21:44:09.348307+0000","last_unstale":"2024-06-27T21:44:59.759774+0000","last_undegraded":"2024-06-27T21:44:59.759774+0000","last_fullsized":"2024-06-27T21:44:59.759774+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2926,"log_dups_size":0,"ondisk_log_size":2926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:16.928152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2926,"num_write_kb":187264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3],"acting":[10,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":372,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.887000+0000","last_change":"2024-06-27T21:44:48.018800+0000","last_active":"2024-06-27T21:44:54.887000+0000","last_peered":"2024-06-27T21:44:54.887000+0000","last_clean":"2024-06-27T21:44:54.887000+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:44:54.887000+0000","last_undegraded":"2024-06-27T21:44:54.887000+0000","last_fullsized":"2024-06-27T21:44:54.887000+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:48.018665+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:44:48.018665+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:46:02.756434+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.a","version":"825'2782","reported_seq":2848,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:58.083943+0000","last_change":"2024-06-27T21:44:09.349122+0000","last_active":"2024-06-27T21:44:58.083943+0000","last_peered":"2024-06-27T21:44:58.083943+0000","last_clean":"2024-06-27T21:44:58.083943+0000","last_became_active":"2024-06-27T21:44:09.348805+0000","last_became_peered":"2024-06-27T21:44:09.348805+0000","last_unstale":"2024-06-27T21:44:58.083943+0000","last_undegraded":"2024-06-27T21:44:58.083943+0000","last_fullsized":"2024-06-27T21:44:58.083943+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2782,"log_dups_size":0,"ondisk_log_size":2782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:30.271355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182321152,"num_objects":2782,"num_object_clones":0,"num_object_copies":8346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2782,"num_write_kb":178048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,3],"acting":[8,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":769,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.886728+0000","last_change":"2024-06-27T21:44:35.891622+0000","last_active":"2024-06-27T21:44:54.886728+0000","last_peered":"2024-06-27T21:44:54.886728+0000","last_clean":"2024-06-27T21:44:54.886728+0000","last_became_active":"2024-06-27T21:44:35.890923+0000","last_became_peered":"2024-06-27T21:44:35.890923+0000","last_unstale":"2024-06-27T21:44:54.886728+0000","last_undegraded":"2024-06-27T21:44:54.886728+0000","last_fullsized":"2024-06-27T21:44:54.886728+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_clean_scrub_stamp":"2024-06-27T21:43:45.673000+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:10.944162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"825'2766","reported_seq":5690,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:58.094681+0000","last_change":"2024-06-27T21:44:53.024197+0000","last_active":"2024-06-27T21:44:58.094681+0000","last_peered":"2024-06-27T21:44:58.094681+0000","last_clean":"2024-06-27T21:44:58.094681+0000","last_became_active":"2024-06-27T21:44:53.021846+0000","last_became_peered":"2024-06-27T21:44:53.021846+0000","last_unstale":"2024-06-27T21:44:58.094681+0000","last_undegraded":"2024-06-27T21:44:58.094681+0000","last_fullsized":"2024-06-27T21:44:58.094681+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2766,"log_dups_size":0,"ondisk_log_size":2766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:26.698471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181272576,"num_objects":2766,"num_object_clones":0,"num_object_copies":8298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2766,"num_write_kb":177024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1424,"num_bytes_recovered":93323264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":19,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.899813+0000","last_change":"2024-06-27T21:44:49.893054+0000","last_active":"2024-06-27T21:44:54.899813+0000","last_peered":"2024-06-27T21:44:54.899813+0000","last_clean":"2024-06-27T21:44:54.899813+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:44:54.899813+0000","last_undegraded":"2024-06-27T21:44:54.899813+0000","last_fullsized":"2024-06-27T21:44:54.899813+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:43:42.388603+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"825'2987","reported_seq":3053,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:45:00.277170+0000","last_change":"2024-06-27T21:44:09.346431+0000","last_active":"2024-06-27T21:45:00.277170+0000","last_peered":"2024-06-27T21:45:00.277170+0000","last_clean":"2024-06-27T21:45:00.277170+0000","last_became_active":"2024-06-27T21:44:09.346083+0000","last_became_peered":"2024-06-27T21:44:09.346083+0000","last_unstale":"2024-06-27T21:45:00.277170+0000","last_undegraded":"2024-06-27T21:45:00.277170+0000","last_fullsized":"2024-06-27T21:45:00.277170+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2987,"log_dups_size":0,"ondisk_log_size":2987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:11.405898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195756032,"num_objects":2987,"num_object_clones":0,"num_object_copies":8961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":191168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,11],"acting":[9,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":71,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.925071+0000","last_change":"2024-06-27T21:44:12.263340+0000","last_active":"2024-06-27T21:44:54.925071+0000","last_peered":"2024-06-27T21:44:54.925071+0000","last_clean":"2024-06-27T21:44:54.925071+0000","last_became_active":"2024-06-27T21:44:11.419424+0000","last_became_peered":"2024-06-27T21:44:11.419424+0000","last_unstale":"2024-06-27T21:44:54.925071+0000","last_undegraded":"2024-06-27T21:44:54.925071+0000","last_fullsized":"2024-06-27T21:44:54.925071+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:12.263224+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.083066+0000","last_clean_scrub_stamp":"2024-06-27T21:44:12.263224+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:34.285540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"825'3098","reported_seq":3164,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:45:01.200897+0000","last_change":"2024-06-27T21:44:09.368722+0000","last_active":"2024-06-27T21:45:01.200897+0000","last_peered":"2024-06-27T21:45:01.200897+0000","last_clean":"2024-06-27T21:45:01.200897+0000","last_became_active":"2024-06-27T21:44:09.368389+0000","last_became_peered":"2024-06-27T21:44:09.368389+0000","last_unstale":"2024-06-27T21:45:01.200897+0000","last_undegraded":"2024-06-27T21:45:01.200897+0000","last_fullsized":"2024-06-27T21:45:01.200897+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":3098,"log_dups_size":0,"ondisk_log_size":3098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:19.150795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203030528,"num_objects":3098,"num_object_clones":0,"num_object_copies":9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3098,"num_write_kb":198272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1107,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.890606+0000","last_change":"2024-06-27T21:44:10.967719+0000","last_active":"2024-06-27T21:44:54.890606+0000","last_peered":"2024-06-27T21:44:54.890606+0000","last_clean":"2024-06-27T21:44:54.890606+0000","last_became_active":"2024-06-27T21:32:59.706698+0000","last_became_peered":"2024-06-27T21:32:59.706698+0000","last_unstale":"2024-06-27T21:44:54.890606+0000","last_undegraded":"2024-06-27T21:44:54.890606+0000","last_fullsized":"2024-06-27T21:44:54.890606+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:10.967643+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:44:10.967643+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:21.826452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,2],"acting":[10,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"825'3026","reported_seq":5058,"reported_epoch":825,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-06-27T21:44:59.754058+0000","last_change":"2024-06-27T21:44:48.739481+0000","last_active":"2024-06-27T21:44:59.754058+0000","last_peered":"2024-06-27T21:44:59.754058+0000","last_clean":"2024-06-27T21:44:34.881444+0000","last_became_active":"2024-06-27T21:44:37.192097+0000","last_became_peered":"2024-06-27T21:44:37.192097+0000","last_unstale":"2024-06-27T21:44:59.754058+0000","last_undegraded":"2024-06-27T21:44:36.898090+0000","last_fullsized":"2024-06-27T21:44:36.886092+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":3026,"log_dups_size":0,"ondisk_log_size":3026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198311936,"num_objects":3026,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":504,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":193664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":64356352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2],"avail_no_missing":["11","2","4"],"object_location_counts":[{"shards":"2,4,11","objects":504},{"shards":"2,11","objects":2521}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1192,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.890495+0000","last_change":"2024-06-27T21:44:14.677715+0000","last_active":"2024-06-27T21:44:54.890495+0000","last_peered":"2024-06-27T21:44:54.890495+0000","last_clean":"2024-06-27T21:44:54.890495+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:44:54.890495+0000","last_undegraded":"2024-06-27T21:44:54.890495+0000","last_fullsized":"2024-06-27T21:44:54.890495+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:14.677595+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:44:14.677595+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:44.558207+0000","scrub_duration":772,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.c","version":"825'2817","reported_seq":3877,"reported_epoch":825,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:44:58.097695+0000","last_change":"2024-06-27T21:44:51.135241+0000","last_active":"2024-06-27T21:44:58.097695+0000","last_peered":"2024-06-27T21:44:58.097695+0000","last_clean":"2024-06-27T21:44:48.984542+0000","last_became_active":"2024-06-27T21:44:51.114339+0000","last_became_peered":"2024-06-27T21:44:51.114339+0000","last_unstale":"2024-06-27T21:44:58.097695+0000","last_undegraded":"2024-06-27T21:44:58.097695+0000","last_fullsized":"2024-06-27T21:44:50.889843+0000","mapping_epoch":821,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2817,"log_dups_size":0,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184614912,"num_objects":2817,"num_object_clones":0,"num_object_copies":8451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1919,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":180288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":495,"num_bytes_recovered":32440320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7],"acting":[8,3],"avail_no_missing":["8","3","11"],"object_location_counts":[{"shards":"3,8","objects":898},{"shards":"3,8,11","objects":1919}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1195,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.890614+0000","last_change":"2024-06-27T21:43:30.085069+0000","last_active":"2024-06-27T21:44:54.890614+0000","last_peered":"2024-06-27T21:44:54.890614+0000","last_clean":"2024-06-27T21:44:54.890614+0000","last_became_active":"2024-06-27T21:43:11.874833+0000","last_became_peered":"2024-06-27T21:43:11.874833+0000","last_unstale":"2024-06-27T21:44:54.890614+0000","last_undegraded":"2024-06-27T21:44:54.890614+0000","last_fullsized":"2024-06-27T21:44:54.890614+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:30.084942+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:43:30.084942+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.d","version":"825'3011","reported_seq":3080,"reported_epoch":825,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-06-27T21:44:59.740056+0000","last_change":"2024-06-27T21:44:51.139245+0000","last_active":"2024-06-27T21:44:59.740056+0000","last_peered":"2024-06-27T21:44:59.740056+0000","last_clean":"2024-06-27T21:44:48.976572+0000","last_became_active":"2024-06-27T21:44:51.131826+0000","last_became_peered":"2024-06-27T21:44:51.131826+0000","last_unstale":"2024-06-27T21:44:59.740056+0000","last_undegraded":"2024-06-27T21:44:50.909300+0000","last_fullsized":"2024-06-27T21:44:50.892764+0000","mapping_epoch":821,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":3011,"log_dups_size":0,"ondisk_log_size":3011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197328896,"num_objects":3011,"num_object_clones":0,"num_object_copies":9033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3011,"num_write_kb":192704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,2],"avail_no_missing":["11","2","5"],"object_location_counts":[{"shards":"2,5,11","objects":2456},{"shards":"2,11","objects":555}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":116,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.889167+0000","last_change":"2024-06-27T21:44:49.896620+0000","last_active":"2024-06-27T21:44:54.889167+0000","last_peered":"2024-06-27T21:44:54.889167+0000","last_clean":"2024-06-27T21:44:54.889167+0000","last_became_active":"2024-06-27T21:44:49.896227+0000","last_became_peered":"2024-06-27T21:44:49.896227+0000","last_unstale":"2024-06-27T21:44:54.889167+0000","last_undegraded":"2024-06-27T21:44:54.889167+0000","last_fullsized":"2024-06-27T21:44:54.889167+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:41.488994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:41.488994+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:02.555174+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.3","version":"825'2915","reported_seq":5928,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:59.736708+0000","last_change":"2024-06-27T21:44:49.896840+0000","last_active":"2024-06-27T21:44:59.736708+0000","last_peered":"2024-06-27T21:44:59.736708+0000","last_clean":"2024-06-27T21:44:59.736708+0000","last_became_active":"2024-06-27T21:44:49.894444+0000","last_became_peered":"2024-06-27T21:44:49.894444+0000","last_unstale":"2024-06-27T21:44:59.736708+0000","last_undegraded":"2024-06-27T21:44:59.736708+0000","last_fullsized":"2024-06-27T21:44:59.736708+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:26.309223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":8745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2915,"num_write_kb":186560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1468,"num_bytes_recovered":96206848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":703,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.886645+0000","last_change":"2024-06-27T21:43:40.622401+0000","last_active":"2024-06-27T21:44:54.886645+0000","last_peered":"2024-06-27T21:44:54.886645+0000","last_clean":"2024-06-27T21:44:54.886645+0000","last_became_active":"2024-06-27T21:32:59.709365+0000","last_became_peered":"2024-06-27T21:32:59.709365+0000","last_unstale":"2024-06-27T21:44:54.886645+0000","last_undegraded":"2024-06-27T21:44:54.886645+0000","last_fullsized":"2024-06-27T21:44:54.886645+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:43:40.622325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:56.174741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"825'2680","reported_seq":2746,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:57.335127+0000","last_change":"2024-06-27T21:44:09.578872+0000","last_active":"2024-06-27T21:44:57.335127+0000","last_peered":"2024-06-27T21:44:57.335127+0000","last_clean":"2024-06-27T21:44:57.335127+0000","last_became_active":"2024-06-27T21:44:09.578583+0000","last_became_peered":"2024-06-27T21:44:09.578583+0000","last_unstale":"2024-06-27T21:44:57.335127+0000","last_undegraded":"2024-06-27T21:44:57.335127+0000","last_fullsized":"2024-06-27T21:44:57.335127+0000","mapping_epoch":797,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2680,"log_dups_size":0,"ondisk_log_size":2680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:09.250372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175636480,"num_objects":2680,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2680,"num_write_kb":171520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":27,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.899783+0000","last_change":"2024-06-27T21:44:51.580191+0000","last_active":"2024-06-27T21:44:54.899783+0000","last_peered":"2024-06-27T21:44:54.899783+0000","last_clean":"2024-06-27T21:44:54.899783+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:44:54.899783+0000","last_undegraded":"2024-06-27T21:44:54.899783+0000","last_fullsized":"2024-06-27T21:44:54.899783+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:51.580089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:44:51.580089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:55.297449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1","version":"825'2878","reported_seq":4141,"reported_epoch":825,"state":"active+recovering+undersized+remapped","last_fresh":"2024-06-27T21:44:58.517948+0000","last_change":"2024-06-27T21:44:51.667936+0000","last_active":"2024-06-27T21:44:58.517948+0000","last_peered":"2024-06-27T21:44:58.517948+0000","last_clean":"2024-06-27T21:44:34.882838+0000","last_became_active":"2024-06-27T21:44:37.754712+0000","last_became_peered":"2024-06-27T21:44:37.754712+0000","last_unstale":"2024-06-27T21:44:58.517948+0000","last_undegraded":"2024-06-27T21:44:58.517948+0000","last_fullsized":"2024-06-27T21:44:36.885247+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":797,"last_epoch_clean":798,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:44:08.332154+0000","last_clean_scrub_stamp":"2024-06-27T21:44:08.332154+0000","objects_scrubbed":0,"log_size":2878,"log_dups_size":0,"ondisk_log_size":2878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188612608,"num_objects":2878,"num_object_clones":0,"num_object_copies":8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":869,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2878,"num_write_kb":184192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":39124992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,10],"acting":[2,10],"avail_no_missing":["2","0","10"],"object_location_counts":[{"shards":"0,2,10","objects":869},{"shards":"2,10","objects":2009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1088,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.886803+0000","last_change":"2024-06-27T21:44:06.358598+0000","last_active":"2024-06-27T21:44:54.886803+0000","last_peered":"2024-06-27T21:44:54.886803+0000","last_clean":"2024-06-27T21:44:54.886803+0000","last_became_active":"2024-06-27T21:36:51.804872+0000","last_became_peered":"2024-06-27T21:36:51.804872+0000","last_unstale":"2024-06-27T21:44:54.886803+0000","last_undegraded":"2024-06-27T21:44:54.886803+0000","last_fullsized":"2024-06-27T21:44:54.886803+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:06.358481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:44:06.358481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:11.083773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,4],"acting":[2,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":27,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.899786+0000","last_change":"2024-06-27T21:44:50.599216+0000","last_active":"2024-06-27T21:44:54.899786+0000","last_peered":"2024-06-27T21:44:54.899786+0000","last_clean":"2024-06-27T21:44:54.899786+0000","last_became_active":"2024-06-27T21:44:49.890705+0000","last_became_peered":"2024-06-27T21:44:49.890705+0000","last_unstale":"2024-06-27T21:44:54.899786+0000","last_undegraded":"2024-06-27T21:44:54.899786+0000","last_fullsized":"2024-06-27T21:44:54.899786+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:50.599088+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:44:50.599088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:46:16.365886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":791,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.925039+0000","last_change":"2024-06-27T21:44:13.257652+0000","last_active":"2024-06-27T21:44:54.925039+0000","last_peered":"2024-06-27T21:44:54.925039+0000","last_clean":"2024-06-27T21:44:54.925039+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:44:54.925039+0000","last_undegraded":"2024-06-27T21:44:54.925039+0000","last_fullsized":"2024-06-27T21:44:54.925039+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:13.257557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:44:13.257557+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:18.529790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":825,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.928523+0000","last_change":"2024-06-27T21:44:35.891024+0000","last_active":"2024-06-27T21:44:54.928523+0000","last_peered":"2024-06-27T21:44:54.928523+0000","last_clean":"2024-06-27T21:44:54.928523+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:44:54.928523+0000","last_undegraded":"2024-06-27T21:44:54.928523+0000","last_fullsized":"2024-06-27T21:44:54.928523+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:06.605134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:44:06.605134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:15.208470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":116,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.889087+0000","last_change":"2024-06-27T21:44:35.891573+0000","last_active":"2024-06-27T21:44:54.889087+0000","last_peered":"2024-06-27T21:44:54.889087+0000","last_clean":"2024-06-27T21:44:54.889087+0000","last_became_active":"2024-06-27T21:44:35.891196+0000","last_became_peered":"2024-06-27T21:44:35.891196+0000","last_unstale":"2024-06-27T21:44:54.889087+0000","last_undegraded":"2024-06-27T21:44:54.889087+0000","last_fullsized":"2024-06-27T21:44:54.889087+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:11.419818+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:44:11.419818+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:25.152210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":797,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.887139+0000","last_change":"2024-06-27T21:44:49.890298+0000","last_active":"2024-06-27T21:44:54.887139+0000","last_peered":"2024-06-27T21:44:54.887139+0000","last_clean":"2024-06-27T21:44:54.887139+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:44:54.887139+0000","last_undegraded":"2024-06-27T21:44:54.887139+0000","last_fullsized":"2024-06-27T21:44:54.887139+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:43:38.265889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:02.523947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":19,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.899985+0000","last_change":"2024-06-27T21:44:49.892132+0000","last_active":"2024-06-27T21:44:54.899985+0000","last_peered":"2024-06-27T21:44:54.899985+0000","last_clean":"2024-06-27T21:44:54.899985+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:44:54.899985+0000","last_undegraded":"2024-06-27T21:44:54.899985+0000","last_fullsized":"2024-06-27T21:44:54.899985+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:32.372638+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:43:32.372638+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":29,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.891045+0000","last_change":"2024-06-27T21:44:35.890335+0000","last_active":"2024-06-27T21:44:54.891045+0000","last_peered":"2024-06-27T21:44:54.891045+0000","last_clean":"2024-06-27T21:44:54.891045+0000","last_became_active":"2024-06-27T21:44:35.890025+0000","last_became_peered":"2024-06-27T21:44:35.890025+0000","last_unstale":"2024-06-27T21:44:54.891045+0000","last_undegraded":"2024-06-27T21:44:54.891045+0000","last_fullsized":"2024-06-27T21:44:54.891045+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:34.188181+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:43:34.188181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":448,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.900362+0000","last_change":"2024-06-27T21:43:38.420637+0000","last_active":"2024-06-27T21:44:54.900362+0000","last_peered":"2024-06-27T21:44:54.900362+0000","last_clean":"2024-06-27T21:44:54.900362+0000","last_became_active":"2024-06-27T21:36:51.808305+0000","last_became_peered":"2024-06-27T21:36:51.808305+0000","last_unstale":"2024-06-27T21:44:54.900362+0000","last_undegraded":"2024-06-27T21:44:54.900362+0000","last_fullsized":"2024-06-27T21:44:54.900362+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:38.420522+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:43:38.420522+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:44:55.827489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":1137,"reported_epoch":825,"state":"active+recovery_wait+degraded","last_fresh":"2024-06-27T21:44:54.886498+0000","last_change":"2024-06-27T21:44:49.892908+0000","last_active":"2024-06-27T21:44:54.886498+0000","last_peered":"2024-06-27T21:44:54.886498+0000","last_clean":"2024-06-27T21:44:44.889307+0000","last_became_active":"2024-06-27T21:44:49.892516+0000","last_became_peered":"2024-06-27T21:44:49.892516+0000","last_unstale":"2024-06-27T21:44:54.886498+0000","last_undegraded":"2024-06-27T21:44:49.887412+0000","last_fullsized":"2024-06-27T21:44:54.886498+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:28.360278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:28.360278+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1006240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":["2","1","11"],"object_location_counts":[{"shards":"1,2,11","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":647,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.891192+0000","last_change":"2024-06-27T21:43:31.233832+0000","last_active":"2024-06-27T21:44:54.891192+0000","last_peered":"2024-06-27T21:44:54.891192+0000","last_clean":"2024-06-27T21:44:54.891192+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:44:54.891192+0000","last_undegraded":"2024-06-27T21:44:54.891192+0000","last_fullsized":"2024-06-27T21:44:54.891192+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:31.233757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:31.233757+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":734,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.886531+0000","last_change":"2024-06-27T21:44:49.894578+0000","last_active":"2024-06-27T21:44:54.886531+0000","last_peered":"2024-06-27T21:44:54.886531+0000","last_clean":"2024-06-27T21:44:54.886531+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:44:54.886531+0000","last_undegraded":"2024-06-27T21:44:54.886531+0000","last_fullsized":"2024-06-27T21:44:54.886531+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:17.659266+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:44:17.659266+0000","last_clean_scrub_stamp":"2024-06-27T21:44:17.659266+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:34.694322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":431,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.900048+0000","last_change":"2024-06-27T21:43:37.424635+0000","last_active":"2024-06-27T21:44:54.900048+0000","last_peered":"2024-06-27T21:44:54.900048+0000","last_clean":"2024-06-27T21:44:54.900048+0000","last_became_active":"2024-06-27T21:36:51.803452+0000","last_became_peered":"2024-06-27T21:36:51.803452+0000","last_unstale":"2024-06-27T21:44:54.900048+0000","last_undegraded":"2024-06-27T21:44:54.900048+0000","last_fullsized":"2024-06-27T21:44:54.900048+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:37.424523+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:43:37.424523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":455,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.900882+0000","last_change":"2024-06-27T21:44:35.890655+0000","last_active":"2024-06-27T21:44:54.900882+0000","last_peered":"2024-06-27T21:44:54.900882+0000","last_clean":"2024-06-27T21:44:54.900882+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:44:54.900882+0000","last_undegraded":"2024-06-27T21:44:54.900882+0000","last_fullsized":"2024-06-27T21:44:54.900882+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:41.399047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:43:41.399047+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":116,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.891402+0000","last_change":"2024-06-27T21:44:35.892043+0000","last_active":"2024-06-27T21:44:54.891402+0000","last_peered":"2024-06-27T21:44:54.891402+0000","last_clean":"2024-06-27T21:44:54.891402+0000","last_became_active":"2024-06-27T21:44:35.891422+0000","last_became_peered":"2024-06-27T21:44:35.891422+0000","last_unstale":"2024-06-27T21:44:54.891402+0000","last_undegraded":"2024-06-27T21:44:54.891402+0000","last_fullsized":"2024-06-27T21:44:54.891402+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:44:06.601108+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:44:06.601108+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:45:16.628990+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":116,"reported_epoch":825,"state":"active+clean","last_fresh":"2024-06-27T21:44:54.889200+0000","last_change":"2024-06-27T21:44:49.895064+0000","last_active":"2024-06-27T21:44:54.889200+0000","last_peered":"2024-06-27T21:44:54.889200+0000","last_clean":"2024-06-27T21:44:54.889200+0000","last_became_active":"2024-06-27T21:44:49.894533+0000","last_became_peered":"2024-06-27T21:44:49.894533+0000","last_unstale":"2024-06-27T21:44:54.889200+0000","last_undegraded":"2024-06-27T21:44:54.889200+0000","last_fullsized":"2024-06-27T21:44:54.889200+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:43:27.585537+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:43:27.585537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3040804864,"num_objects":46399,"num_object_clones":0,"num_object_copies":139197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2457,"num_objects_misplaced":5355,"num_objects_unfound":0,"num_objects_dirty":46399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46399,"num_write_kb":2969536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8096,"num_bytes_recovered":530579456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9477488640,"data_stored":9477488640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46399,"ondisk_log_size":46399,"up":48,"acting":42,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1006240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253293,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2404328,"kb_used_data":1570720,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":91340824,"statfs":{"total":95995035648,"available":93533003776,"internally_reserved":0,"allocated":1608417280,"data_stored":1607986640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":853604805},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2263072,"kb_used_data":1479448,"kb_used_omap":1,"kb_used_meta":783614,"kb_avail":91482080,"statfs":{"total":95995035648,"available":93677649920,"internally_reserved":0,"allocated":1514954752,"data_stored":1514527664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":802421184},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826336,"kb_used_data":191256,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92918816,"statfs":{"total":95995035648,"available":95148867584,"internally_reserved":0,"allocated":195846144,"data_stored":195419056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":650312128},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972794,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031648,"kb_used_data":527064,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92713504,"statfs":{"total":95995035648,"available":94938628096,"internally_reserved":0,"allocated":539713536,"data_stored":539286448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267130,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":797856,"kb_used_data":157208,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92947296,"statfs":{"total":95995035648,"available":95178031104,"internally_reserved":0,"allocated":160980992,"data_stored":160553904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":918240,"kb_used_data":260952,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":92826912,"statfs":{"total":95995035648,"available":95054757888,"internally_reserved":0,"allocated":267214848,"data_stored":266787760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673053126},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757705,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955296,"kb_used_data":509016,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92789856,"statfs":{"total":95995035648,"available":95016812544,"internally_reserved":0,"allocated":521232384,"data_stored":520805296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161411,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1414048,"kb_used_data":969880,"kb_used_omap":92,"kb_used_meta":444067,"kb_avail":92331104,"statfs":{"total":95995035648,"available":94547050496,"internally_reserved":0,"allocated":993157120,"data_stored":992730032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":454724706},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160096,"kb_used_data":695768,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92585056,"statfs":{"total":95995035648,"available":94807097344,"internally_reserved":0,"allocated":712466432,"data_stored":712039344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":475462087},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1839528,"kb_used_data":1140576,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91905624,"statfs":{"total":95995035648,"available":94111358976,"internally_reserved":0,"allocated":1167949824,"data_stored":1167519184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575324,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1568296,"kb_used_data":945696,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92176856,"statfs":{"total":95995035648,"available":94389100544,"internally_reserved":0,"allocated":968392704,"data_stored":967962064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738849,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1454432,"kb_used_data":683288,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92290720,"statfs":{"total":95995035648,"available":94505697280,"internally_reserved":0,"allocated":699686912,"data_stored":699259824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":699006976,"data_stored":699006976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":991494144,"data_stored":991494144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1193738240,"data_stored":1193738240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":714014720,"data_stored":714014720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1014562816,"data_stored":1014562816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":529334272,"data_stored":529334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":267649024,"data_stored":267649024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":169476096,"data_stored":169476096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":548536320,"data_stored":548536320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":195756032,"data_stored":195756032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1541799936,"data_stored":1541799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1612120064,"data_stored":1612120064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:45:04.072 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-06-27T21:45:04.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-06-27T21:45:04.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.175 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:04.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: pgmap v2053: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 887 op/s; 2458/139203 objects degraded (1.766%); 5355/139203 objects misplaced (3.847%); 27 MiB/s, 426 objects/s recovering 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: from='client.17529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: osdmap e826: 12 total, 12 up, 12 in 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:45:04.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1186344745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:45:04.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: pgmap v2053: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 887 op/s; 2458/139203 objects degraded (1.766%); 5355/139203 objects misplaced (3.847%); 27 MiB/s, 426 objects/s recovering 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: from='client.17529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: osdmap e826: 12 total, 12 up, 12 in 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:45:04.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1186344745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:45:04.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: pgmap v2053: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 40 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 887 op/s; 2458/139203 objects degraded (1.766%); 5355/139203 objects misplaced (3.847%); 27 MiB/s, 426 objects/s recovering 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: from='client.17529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: osdmap e826: 12 total, 12 up, 12 in 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:45:04.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1186344745' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:45:04.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:04.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:04.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:04.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:04.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:04.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:04.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 51310 51294 69.6869 20 0.00513436 0.0142047 2024-06-27T21:45:04.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:04.802 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:04.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:04.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:04.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:05.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:05.054 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:05.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:05.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:05.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:04 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:05.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1186344745' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:45:05.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:04 smithi151 ceph-mon[28965]: osdmap e827: 12 total, 12 up, 12 in 2024-06-27T21:45:05.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:04 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2458/139203 objects degraded (1.766%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:04 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1186344745' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:45:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:04 smithi190 ceph-mon[32247]: osdmap e827: 12 total, 12 up, 12 in 2024-06-27T21:45:05.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:04 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2458/139203 objects degraded (1.766%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:05.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:04 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1186344745' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:45:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:04 smithi192 ceph-mon[32956]: osdmap e827: 12 total, 12 up, 12 in 2024-06-27T21:45:05.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:04 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2458/139203 objects degraded (1.766%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:05.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:05.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:05.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:05.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:05.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:05.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:05.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 52180 52164 69.361 54.375 0.0190724 0.0144106 2024-06-27T21:45:05.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:05.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:05.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:05.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:05.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:45:05.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:05.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:06.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.103 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:06.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:06.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:05 smithi151 ceph-mon[28965]: pgmap v2056: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s; 2457/145890 objects degraded (1.684%); 4433/145890 objects misplaced (3.039%); 24 MiB/s, 379 objects/s recovering 2024-06-27T21:45:06.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:06.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:06.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:05 smithi151 ceph-mon[28965]: osdmap e828: 12 total, 12 up, 12 in 2024-06-27T21:45:06.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.233 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:06.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:05 smithi190 ceph-mon[32247]: pgmap v2056: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s; 2457/145890 objects degraded (1.684%); 4433/145890 objects misplaced (3.039%); 24 MiB/s, 379 objects/s recovering 2024-06-27T21:45:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:05 smithi190 ceph-mon[32247]: osdmap e828: 12 total, 12 up, 12 in 2024-06-27T21:45:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:05 smithi192 ceph-mon[32956]: pgmap v2056: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 41 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s; 2457/145890 objects degraded (1.684%); 4433/145890 objects misplaced (3.039%); 24 MiB/s, 379 objects/s recovering 2024-06-27T21:45:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:06.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:05 smithi192 ceph-mon[32956]: osdmap e828: 12 total, 12 up, 12 in 2024-06-27T21:45:06.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:06.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:06.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:06.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:06.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:06.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:06.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 53430 53414 69.5435 78.125 0.0239042 0.0143734 2024-06-27T21:45:06.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:06.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:06.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:06.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:06.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:06.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:07.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:07.022 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:07.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:45:07.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:07.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:07.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.267 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:07.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:07 smithi192 ceph-mon[32956]: pgmap v2058: 47 pgs: 1 peering, 4 active+recovering+undersized+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s; 5500/153660 objects misplaced (3.579%); 24 MiB/s, 385 objects/s recovering 2024-06-27T21:45:07.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:07 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2457/145890 objects degraded (1.684%), 2 pgs degraded) 2024-06-27T21:45:07.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:07 smithi192 ceph-mon[32956]: osdmap e829: 12 total, 12 up, 12 in 2024-06-27T21:45:07.298 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:07 smithi151 ceph-mon[28965]: pgmap v2058: 47 pgs: 1 peering, 4 active+recovering+undersized+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s; 5500/153660 objects misplaced (3.579%); 24 MiB/s, 385 objects/s recovering 2024-06-27T21:45:07.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:07 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2457/145890 objects degraded (1.684%), 2 pgs degraded) 2024-06-27T21:45:07.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:07 smithi151 ceph-mon[28965]: osdmap e829: 12 total, 12 up, 12 in 2024-06-27T21:45:07.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:07.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:07.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:07.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:07 smithi190 ceph-mon[32247]: pgmap v2058: 47 pgs: 1 peering, 4 active+recovering+undersized+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s; 5500/153660 objects misplaced (3.579%); 24 MiB/s, 385 objects/s recovering 2024-06-27T21:45:07.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:07 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2457/145890 objects degraded (1.684%), 2 pgs degraded) 2024-06-27T21:45:07.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:07 smithi190 ceph-mon[32247]: osdmap e829: 12 total, 12 up, 12 in 2024-06-27T21:45:07.517 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:07.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:07.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:07.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 16 54068 54052 68.9379 39.875 0.00960683 0.0144228 2024-06-27T21:45:07.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:07.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:07.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:07.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:07.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:07.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:08.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:08.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:08.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:08.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:08.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:08.318 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:08.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:08.398 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:08.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:08.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:08.562 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:08.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:08.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 55014 54998 68.7415 59.125 0.0216124 0.0145396 2024-06-27T21:45:08.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:08.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:08.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:08.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:08.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:08.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:09.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:09.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:09.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:09.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:08 smithi190 ceph-mon[32247]: osdmap e830: 12 total, 12 up, 12 in 2024-06-27T21:45:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:45:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:45:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:45:09.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:08 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:45:09.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:08 smithi192 ceph-mon[32956]: osdmap e830: 12 total, 12 up, 12 in 2024-06-27T21:45:09.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:45:09.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:45:09.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:45:09.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:08 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:45:09.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:09.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:09.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:08 smithi151 ceph-mon[28965]: osdmap e830: 12 total, 12 up, 12 in 2024-06-27T21:45:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:45:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:45:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:45:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:08 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:45:09.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:09.493 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:09.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:09.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:09.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:09.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 56155 56139 68.7918 71.3125 0.106936 0.0145069 2024-06-27T21:45:09.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:09.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:09.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:09.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:09.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:45:09.937 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:45:09.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:45:09.964 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:45:09.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:09.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:45:09.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:45:09.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:09.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:09.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:10.017 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:45:10.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:45:10.046 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:45:10.047 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:45:10.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:45:10.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:45:10.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:10.101 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:45:10.101 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:45:10.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:10.129 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:45:10.129 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:45:10.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:45:10.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:45:10.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:10.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:10.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:10.372 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:10.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:45:10.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:45:10.421 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:45:10.421 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:45:10.456 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:45:10.456 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:45:10.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:10.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:10.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:10.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:10.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 57175 57159 68.6948 63.75 0.0425973 0.0145455 2024-06-27T21:45:10.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:10.768 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:10.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:10.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:10.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:10.912 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 8 pg_num to 26 2024-06-27T21:45:10.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:11.026 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:11.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:11.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:11.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:10 smithi151 ceph-mon[28965]: osdmap e831: 12 total, 12 up, 12 in 2024-06-27T21:45:11.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/409942443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:11.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/886708999' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:45:11.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:10 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:45:11.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:11.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:10 smithi190 ceph-mon[32247]: osdmap e831: 12 total, 12 up, 12 in 2024-06-27T21:45:11.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/409942443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:11.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/886708999' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:45:11.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:10 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:45:11.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:11.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:10 smithi192 ceph-mon[32956]: osdmap e831: 12 total, 12 up, 12 in 2024-06-27T21:45:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/409942443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/886708999' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:45:11.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:10 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]: dispatch 2024-06-27T21:45:11.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:11.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:11.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:11.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:11.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:11.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:11.685 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 58043 58027 68.4221 54.25 0.0116778 0.0146074 2024-06-27T21:45:11.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:11.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:11.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:11.927 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:11.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:12.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:12.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:12.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:12.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:11 smithi151 ceph-mon[28965]: pgmap v2063: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 511 op/s; 4620/160272 objects misplaced (2.883%); 10 MiB/s, 165 objects/s recovering 2024-06-27T21:45:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:11 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-27T21:45:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:11 smithi151 ceph-mon[28965]: osdmap e832: 12 total, 12 up, 12 in 2024-06-27T21:45:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:45:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:45:12.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:45:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:11 smithi190 ceph-mon[32247]: pgmap v2063: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 511 op/s; 4620/160272 objects misplaced (2.883%); 10 MiB/s, 165 objects/s recovering 2024-06-27T21:45:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:11 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-27T21:45:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:11 smithi190 ceph-mon[32247]: osdmap e832: 12 total, 12 up, 12 in 2024-06-27T21:45:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:45:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:45:12.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:45:12.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:11 smithi192 ceph-mon[32956]: pgmap v2063: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 511 op/s; 4620/160272 objects misplaced (2.883%); 10 MiB/s, 165 objects/s recovering 2024-06-27T21:45:12.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:11 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-06-27T21:45:12.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:11 smithi192 ceph-mon[32956]: osdmap e832: 12 total, 12 up, 12 in 2024-06-27T21:45:12.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:45:12.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:45:12.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:45:12.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:12.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:12.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:12.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:12.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:12.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:12.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 59056 59040 68.3274 63.3125 0.00663793 0.0146298 2024-06-27T21:45:12.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:12.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:12.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:12.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:12.985 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:12.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:13.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:13.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:13.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:13.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:45:13.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:13.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:13.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:45:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-06-27T21:45:13.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:13.354 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:13.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:13.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:13.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:13.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:13.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 59346 59330 67.4146 18.125 0.389604 0.0146337 2024-06-27T21:45:13.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:13.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:13.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:13.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:13.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:13.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:13.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:14.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:14.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:14.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:13 smithi151 ceph-mon[28965]: pgmap v2065: 47 pgs: 3 active+recovering+undersized+remapped, 44 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 3767/166821 objects misplaced (2.258%); 20 MiB/s, 325 objects/s recovering 2024-06-27T21:45:14.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-27T21:45:14.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:13 smithi151 ceph-mon[28965]: osdmap e833: 12 total, 12 up, 12 in 2024-06-27T21:45:14.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:14.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:13 smithi190 ceph-mon[32247]: pgmap v2065: 47 pgs: 3 active+recovering+undersized+remapped, 44 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 3767/166821 objects misplaced (2.258%); 20 MiB/s, 325 objects/s recovering 2024-06-27T21:45:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-27T21:45:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:13 smithi190 ceph-mon[32247]: osdmap e833: 12 total, 12 up, 12 in 2024-06-27T21:45:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:13 smithi192 ceph-mon[32956]: pgmap v2065: 47 pgs: 3 active+recovering+undersized+remapped, 44 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 3767/166821 objects misplaced (2.258%); 20 MiB/s, 325 objects/s recovering 2024-06-27T21:45:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-06-27T21:45:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:13 smithi192 ceph-mon[32956]: osdmap e833: 12 total, 12 up, 12 in 2024-06-27T21:45:14.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:14.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:14.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:14.474 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:14.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:14.600 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:14.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:45:14.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 59652 59636 66.5523 19.125 0.00204805 0.0148791 2024-06-27T21:45:14.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:14.723 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:14.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:14.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:14.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:14.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:14.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:14.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:15.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:15.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:15.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: osdmap e834: 12 total, 12 up, 12 in 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4103994090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4103994090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:45:15.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:14 smithi151 ceph-mon[28965]: osdmap e835: 12 total, 12 up, 12 in 2024-06-27T21:45:15.217 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: osdmap e834: 12 total, 12 up, 12 in 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4103994090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4103994090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:45:15.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:14 smithi190 ceph-mon[32247]: osdmap e835: 12 total, 12 up, 12 in 2024-06-27T21:45:15.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: osdmap e834: 12 total, 12 up, 12 in 2024-06-27T21:45:15.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:15.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-06-27T21:45:15.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:15.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4103994090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:45:15.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:15.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-06-27T21:45:15.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4103994090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:45:15.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:14 smithi192 ceph-mon[32956]: osdmap e835: 12 total, 12 up, 12 in 2024-06-27T21:45:15.316 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:45:15.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.342 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:15.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:15.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:15.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:15.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:15.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:15.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 59675 59659 65.4099 1.4375 0.00220204 0.0148858 2024-06-27T21:45:15.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:15.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:15.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:15.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:15.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:15.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:15.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:16.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:16.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:16.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: pgmap v2068: 57 pgs: 1 peering, 10 unknown, 3 active+recovering+undersized+remapped, 43 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s; 3117/172542 objects misplaced (1.807%); 21 MiB/s, 340 objects/s recovering 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: 8.3 scrub starts 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: 8.4 scrub starts 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:16.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: 8.8 scrub starts 2024-06-27T21:45:16.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:15 smithi151 ceph-mon[28965]: osdmap e836: 12 total, 12 up, 12 in 2024-06-27T21:45:16.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: pgmap v2068: 57 pgs: 1 peering, 10 unknown, 3 active+recovering+undersized+remapped, 43 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s; 3117/172542 objects misplaced (1.807%); 21 MiB/s, 340 objects/s recovering 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: 8.3 scrub starts 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: 8.4 scrub starts 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:16.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:16.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: 8.8 scrub starts 2024-06-27T21:45:16.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:15 smithi190 ceph-mon[32247]: osdmap e836: 12 total, 12 up, 12 in 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: pgmap v2068: 57 pgs: 1 peering, 10 unknown, 3 active+recovering+undersized+remapped, 43 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s; 3117/172542 objects misplaced (1.807%); 21 MiB/s, 340 objects/s recovering 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: 8.3 scrub starts 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: 8.4 scrub starts 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: 8.8 scrub starts 2024-06-27T21:45:16.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:15 smithi192 ceph-mon[32956]: osdmap e836: 12 total, 12 up, 12 in 2024-06-27T21:45:16.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.344 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:16.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:16.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:16.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:16.569 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:16.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:16.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 59675 59659 64.2821 0 - 0.0148858 2024-06-27T21:45:16.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:16.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:16.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:16.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:16.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:16.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:17.016 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:45:17.016 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:45:17.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:45:17.016 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164221 7.6 7:6148d821:::benchmark_data_smithi192_150129_object60916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.362114+0000", 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.64533889200001, 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3802650190000003, 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.017 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164221 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.362114+0000", 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.362114+0000", 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.362114+0000", 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7299999999999995e-07 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.362120+0000", 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0140000000000004e-06 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.362127+0000", 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7989999999999996e-06 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.362131+0000", 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3370000000000001e-06 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.367122+0000", 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049904459999999999 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.367264+0000", 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142471 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.367395+0000", 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013071299999999999 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.372446+0000", 2024-06-27T21:45:17.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050507709999999999 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.372709+0000", 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026284399999999998 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.742277+0000", 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3695681239999997 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.742334+0000", 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7595e-05 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.742379+0000", 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4231999999999999e-05 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164230 7.4 7:21b76a38:::benchmark_data_smithi192_150129_object60925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.374527+0000", 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.63292521899999, 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3680296580000002, 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164230 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374527+0000", 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374527+0000", 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374529+0000", 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3629999999999999e-06 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374535+0000", 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2369999999999997e-06 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374548+0000", 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3567000000000001e-05 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374556+0000", 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4900000000000003e-06 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374610+0000", 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4073999999999998e-05 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374726+0000", 2024-06-27T21:45:17.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115667 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.374837+0000", 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011168 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.380029+0000", 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051916059999999997 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.386077+0000", 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060480070000000002 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.742493+0000", 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.356416576 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.742526+0000", 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2909e-05 2024-06-27T21:45:17.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.742557+0000", 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0482000000000001e-05 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164238 7.4 7:24ad967c:::benchmark_data_smithi192_150129_object60933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.390089+0000", 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.61736299699999, 2024-06-27T21:45:17.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3560875980000002, 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164238 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390089+0000", 2024-06-27T21:45:17.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390089+0000", 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390092+0000", 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3159999999999999e-06 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390099+0000", 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6620000000000005e-06 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390117+0000", 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7499e-05 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390126+0000", 2024-06-27T21:45:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2329999999999995e-06 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390190+0000", 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3414e-05 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390273+0000", 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3228999999999999e-05 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.390344+0000", 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1328999999999994e-05 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.394444+0000", 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041001409999999999 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.401686+0000", 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072419449999999996 2024-06-27T21:45:17.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.746048+0000", 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.344362061 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.746119+0000", 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1184000000000002e-05 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.746177+0000", 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7584999999999998e-05 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164244 7.4 7:24a35dc8:::benchmark_data_smithi192_150129_object60939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.405297+0000", 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.60215588599999, 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.348226736, 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.037 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164244 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405297+0000", 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405297+0000", 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405298+0000", 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2780000000000001e-06 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405307+0000", 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7050000000000005e-06 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405318+0000", 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1384e-05 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405323+0000", 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4260000000000002e-06 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405370+0000", 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6180000000000002e-05 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405460+0000", 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0115e-05 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.405535+0000", 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4941e-05 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.409261+0000", 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037264450000000001 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.415131+0000", 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005870119 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.753396+0000", 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3382648049999997 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.753473+0000", 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6941999999999996e-05 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.753523+0000", 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0396000000000003e-05 2024-06-27T21:45:17.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164257 7.6 7:68a4a832:::benchmark_data_smithi192_150129_object60952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.439428+0000", 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.56802490199999, 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.316140614, 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.045 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164257 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439428+0000", 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439428+0000", 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439429+0000", 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0449999999999999e-06 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:17.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439434+0000", 2024-06-27T21:45:17.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0900000000000004e-06 2024-06-27T21:45:17.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439443+0000", 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2480000000000002e-06 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439450+0000", 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8800000000000002e-06 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439509+0000", 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9485999999999999e-05 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439656+0000", 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000146265 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.439804+0000", 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014808599999999999 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.445340+0000", 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055362919999999999 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.445758+0000", 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041852099999999999 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.755481+0000", 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3097225200000002 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.755531+0000", 2024-06-27T21:45:17.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9605999999999999e-05 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.755568+0000", 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7574999999999999e-05 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164268 7.4 7:27ca62b7:::benchmark_data_smithi192_150129_object60963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.457182+0000", 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.550270665, 2024-06-27T21:45:17.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3001977509999998, 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164268 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457182+0000", 2024-06-27T21:45:17.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457182+0000", 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457182+0000", 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9500000000000002e-07 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457188+0000", 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5690000000000004e-06 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457198+0000", 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0046e-05 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457203+0000", 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0860000000000001e-06 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457255+0000", 2024-06-27T21:45:17.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1418999999999997e-05 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457336+0000", 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1165e-05 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.457409+0000", 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3532e-05 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.461164+0000", 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003754575 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.464291+0000", 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031273820000000002 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757278+0000", 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2929863309999998 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757332+0000", 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4248e-05 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757380+0000", 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7703000000000001e-05 2024-06-27T21:45:17.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164279 7.4 7:28d10813:::benchmark_data_smithi192_150129_object60974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.491027+0000", 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.51642514299999, 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2665036860000001, 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.063 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164279 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491027+0000", 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491027+0000", 2024-06-27T21:45:17.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491029+0000", 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.421e-06 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491034+0000", 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5779999999999998e-06 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491043+0000", 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8759999999999994e-06 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491050+0000", 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4529999999999999e-06 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491103+0000", 2024-06-27T21:45:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3502999999999999e-05 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491235+0000", 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132099 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.491361+0000", 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125955 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496064+0000", 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047023530000000003 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.509737+0000", 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013673009999999999 2024-06-27T21:45:17.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757426+0000", 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2476895020000001 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757487+0000", 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0841000000000002e-05 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757531+0000", 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4094999999999997e-05 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164280 7.6 7:630bdb0f:::benchmark_data_smithi192_150129_object60975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.496507+0000", 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.510945851, 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2667287209999998, 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.071 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164280 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496507+0000", 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496507+0000", 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496508+0000", 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2200000000000002e-07 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496512+0000", 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.442e-06 2024-06-27T21:45:17.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496523+0000", 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0848e-05 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496530+0000", 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.61e-06 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496576+0000", 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5890000000000003e-05 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496664+0000", 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7589000000000005e-05 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.496744+0000", 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9957999999999996e-05 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.500479+0000", 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037353500000000001 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.501238+0000", 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00075914099999999996 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763003+0000", 2024-06-27T21:45:17.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2617648089999998 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763186+0000", 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018260499999999999 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763235+0000", 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9557e-05 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164284 7.4 7:2d14e778:::benchmark_data_smithi192_150129_object60979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.549493+0000", 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.45795975600001, 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2081656020000002, 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.078 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164284 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549493+0000", 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549493+0000", 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549494+0000", 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5659999999999999e-06 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549500+0000", 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999999e-06 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549514+0000", 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3994e-05 2024-06-27T21:45:17.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549524+0000", 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0113e-05 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549585+0000", 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0992000000000002e-05 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549703+0000", 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117361 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.549822+0000", 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119659 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.554534+0000", 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047115500000000001 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.608252+0000", 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.053718360999999999 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757579+0000", 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1493265030000002 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757624+0000", 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4869e-05 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.757658+0000", 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4523999999999997e-05 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164300 7.4 7:2d11e108:::benchmark_data_smithi192_150129_object60995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.084 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.720336+0000", 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.28711645999999, 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0382981830000002, 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164300 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720336+0000", 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720336+0000", 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720338+0000", 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6339999999999999e-06 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720347+0000", 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6930000000000002e-06 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720364+0000", 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6494000000000002e-05 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720371+0000", 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8870000000000003e-06 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720457+0000", 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5828999999999994e-05 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720551+0000", 2024-06-27T21:45:17.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4208000000000003e-05 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.720630+0000", 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9414999999999998e-05 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.724563+0000", 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039327399999999997 2024-06-27T21:45:17.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.738094+0000", 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013530916 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.758547+0000", 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0204528890000004 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.758596+0000", 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9586999999999998e-05 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.758634+0000", 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7891e-05 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164308 7.6 7:696a40d6:::benchmark_data_smithi192_150129_object61003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.091 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.823153+0000", 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.18429938400001, 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.94033195, 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164308 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823153+0000", 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823153+0000", 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823154+0000", 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.409e-06 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823160+0000", 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3510000000000001e-06 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823173+0000", 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2892e-05 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823180+0000", 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1690000000000003e-06 2024-06-27T21:45:17.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823237+0000", 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7127999999999999e-05 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823333+0000", 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6311000000000007e-05 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.823415+0000", 2024-06-27T21:45:17.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1605000000000002e-05 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.827256+0000", 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038407570000000002 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.827710+0000", 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045446400000000002 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763401+0000", 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9356911080000003 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763449+0000", 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.757e-05 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763485+0000", 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6186000000000002e-05 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164310 7.6 7:62920566:::benchmark_data_smithi192_150129_object61005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.861665+0000", 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.14578757300001, 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9019962799999997, 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.098 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164310 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861665+0000", 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861665+0000", 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861666+0000", 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.003e-06 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861671+0000", 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1560000000000003e-06 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861681+0000", 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6560000000000004e-06 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861687+0000", 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3110000000000002e-06 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861730+0000", 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3498000000000003e-05 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861804+0000", 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3833999999999998e-05 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.861872+0000", 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7565999999999997e-05 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.865643+0000", 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037714530000000001 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.866107+0000", 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046346899999999999 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763584+0000", 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8974773000000003 2024-06-27T21:45:17.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763627+0000", 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2419000000000002e-05 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763661+0000", 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4615e-05 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164311 7.6 7:6f16057d:::benchmark_data_smithi192_150129_object61006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.868091+0000", 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.13936119900001, 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8956826759999998, 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164311 2024-06-27T21:45:17.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868091+0000", 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868091+0000", 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868092+0000", 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.238e-06 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868100+0000", 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3429999999999998e-06 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868109+0000", 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5670000000000001e-06 2024-06-27T21:45:17.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868114+0000", 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9799999999999998e-06 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868151+0000", 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6814999999999999e-05 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868225+0000", 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3329000000000002e-05 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.868294+0000", 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9733000000000005e-05 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.871437+0000", 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031425099999999998 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.871573+0000", 2024-06-27T21:45:17.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013663299999999999 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763699+0000", 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8921256130000002 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763742+0000", 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2602999999999999e-05 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763774+0000", 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2311999999999998e-05 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164318 7.6 7:6b21a1d0:::benchmark_data_smithi192_150129_object61013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:36.936930+0000", 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.070522383, 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8269623929999996, 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.112 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164318 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936930+0000", 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936930+0000", 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936931+0000", 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.291e-06 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936937+0000", 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2449999999999998e-06 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936948+0000", 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1644999999999999e-05 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936954+0000", 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2909999999999998e-06 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.936995+0000", 2024-06-27T21:45:17.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1056000000000001e-05 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.937089+0000", 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4190000000000005e-05 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.937164+0000", 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5414999999999995e-05 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:36.940987+0000", 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038223900000000002 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.007807+0000", 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.066820743000000002 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763813+0000", 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7560053309999999 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763859+0000", 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6319000000000002e-05 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.763892+0000", 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3476999999999998e-05 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164325 7.6 7:6279dfeb:::benchmark_data_smithi192_150129_object61020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:37.003296+0000", 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 160.00415664499999, 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7671408929999997, 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164325 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.120 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003296+0000", 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003296+0000", 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003297+0000", 2024-06-27T21:45:17.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.362e-06 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003304+0000", 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7190000000000004e-06 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003316+0000", 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2298e-05 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003322+0000", 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0589999999999999e-06 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003365+0000", 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2796999999999997e-05 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003452+0000", 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7289999999999999e-05 2024-06-27T21:45:17.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.003527+0000", 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4262000000000002e-05 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.007190+0000", 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036631480000000002 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.064321+0000", 2024-06-27T21:45:17.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.057131517 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.770347+0000", 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7060256029999996 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.770392+0000", 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5250000000000002e-05 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.770437+0000", 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4588000000000001e-05 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:164333 7.6 7:60b848eb:::benchmark_data_smithi192_150129_object61028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:42:37.060105+0000", 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 159.94734774400001, 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7104485609999998, 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.126 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164333 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060105+0000", 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060105+0000", 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060106+0000", 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6130000000000001e-06 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060113+0000", 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4370000000000004e-06 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060127+0000", 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3842999999999999e-05 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060134+0000", 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0990000000000001e-06 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060182+0000", 2024-06-27T21:45:17.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8557000000000003e-05 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060270+0000", 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8250000000000004e-05 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.060348+0000", 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7261000000000004e-05 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.064227+0000", 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038793289999999999 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:37.705872+0000", 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.64164478199999997 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.770479+0000", 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0646068230000001 2024-06-27T21:45:17.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.770521+0000", 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2369e-05 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:42:43.770553+0000", 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2197999999999999e-05 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:181686 7.5 7.37ee3015 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e779)", 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:43:10.948584+0000", 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 126.058868881, 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91446650799999996, 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 181686 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.133 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948584+0000", 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948584+0000", 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948586+0000", 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9750000000000001e-06 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948593+0000", 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5129999999999999e-06 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948611+0000", 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7649e-05 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948619+0000", 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2560000000000002e-06 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948692+0000", 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2750000000000007e-05 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:10.948704+0000", 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2683e-05 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:11.863036+0000", 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91433139500000005 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:11.863050+0000", 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4287000000000001e-05 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26656.0:198262 7.d 7:bebd5faa:::benchmark_data_smithi192_150129_object94957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e787)", 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:43:33.848686+0000", 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 103.15876688, 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21772090099999999, 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26656", 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 198262 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.138 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848686+0000", 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848686+0000", 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848686+0000", 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8699999999999995e-07 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848690+0000", 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6320000000000001e-06 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848698+0000", 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5210000000000002e-06 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848708+0000", 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5750000000000007e-06 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848751+0000", 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2861000000000002e-05 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.848891+0000", 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014042899999999999 2024-06-27T21:45:17.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.849002+0000", 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110825 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.853840+0000", 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048377660000000003 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.853886+0000", 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6430000000000001e-05 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.066190+0000", 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21230407600000001 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.066244+0000", 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4119999999999997e-05 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.066406+0000", 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000162079 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26656.0:198262 7.d e787/718 ondisk, result = 0)", 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:43:33.856087+0000", 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 103.151365016, 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21033044000000001, 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.144 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.856087+0000", 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.856087+0000", 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.856088+0000", 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4600000000000004e-07 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:33.856092+0000", 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.416e-06 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.061114+0000", 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.205022127 2024-06-27T21:45:17.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.061121+0000", 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8349999999999998e-06 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.066176+0000", 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050555169999999998 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.066184+0000", 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8669999999999994e-06 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:43:34.066418+0000", 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023393200000000001 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:23389 8.e 8:7289a1e9:::benchmark_data_smithi192_167666_object23388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e815)", 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:44:34.881663+0000", 2024-06-27T21:45:17.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.125789671, 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25644687399999999, 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 23389 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:45:17.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881663+0000", 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881663+0000", 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881665+0000", 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9860000000000001e-06 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881736+0000", 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1073000000000002e-05 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881748+0000", 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1909000000000001e-05 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881755+0000", 2024-06-27T21:45:17.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2869999999999999e-06 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881809+0000", 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3967e-05 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op must wait for map", 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.881814+0000", 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3059999999999997e-06 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.884201+0000", 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023867839999999999 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.884351+0000", 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015016899999999999 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.884500+0000", 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014825 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.885883+0000", 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001383876 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:34.887225+0000", 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013412350000000001 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:35.138033+0000", 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25080879499999997 2024-06-27T21:45:17.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:35.138077+0000", 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4008999999999998e-05 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:35.138110+0000", 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2227999999999997e-05 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:45:17.155 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:45:17.156 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:45:17.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:16 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:17.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:16 smithi151 ceph-mon[28965]: 8.3 scrub starts 2024-06-27T21:45:17.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:16 smithi151 ceph-mon[28965]: 8.4 scrub starts 2024-06-27T21:45:17.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:16 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:17.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:16 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:17.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:16 smithi151 ceph-mon[28965]: osdmap e837: 12 total, 12 up, 12 in 2024-06-27T21:45:17.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:17.204 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:16 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:16 smithi190 ceph-mon[32247]: 8.3 scrub starts 2024-06-27T21:45:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:16 smithi190 ceph-mon[32247]: 8.4 scrub starts 2024-06-27T21:45:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:16 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:16 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:17.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:16 smithi190 ceph-mon[32247]: osdmap e837: 12 total, 12 up, 12 in 2024-06-27T21:45:17.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:16 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:17.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:16 smithi192 ceph-mon[32956]: 8.3 scrub starts 2024-06-27T21:45:17.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:16 smithi192 ceph-mon[32956]: 8.4 scrub starts 2024-06-27T21:45:17.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:16 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:17.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:16 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:17.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:16 smithi192 ceph-mon[32956]: osdmap e837: 12 total, 12 up, 12 in 2024-06-27T21:45:17.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:17.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:17.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:17.471 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.472 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:17.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:17.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:17.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 60346 60330 63.9033 20.9688 0.00391171 0.0156295 2024-06-27T21:45:17.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:17.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:17.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:17.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:17.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:18.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:18.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:18.104 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11, 1, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:45:18.104 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:45:18.105 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 3, 4, 5, 6, 9, 10, 2, 8, 11, 1, 7] 2024-06-27T21:45:18.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd out 11 2024-06-27T21:45:18.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:17 smithi151 ceph-mon[28965]: pgmap v2071: 57 pgs: 2 active+clean+scrubbing, 1 peering, 2 remapped+peering, 6 unknown, 3 active+recovering+undersized+remapped, 43 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 56 op/s; 1980/167505 objects misplaced (1.182%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:45:18.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:17 smithi151 ceph-mon[28965]: 8.7 scrub starts 2024-06-27T21:45:18.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:17 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:18.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:18.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:17 smithi190 ceph-mon[32247]: pgmap v2071: 57 pgs: 2 active+clean+scrubbing, 1 peering, 2 remapped+peering, 6 unknown, 3 active+recovering+undersized+remapped, 43 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 56 op/s; 1980/167505 objects misplaced (1.182%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:45:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:17 smithi190 ceph-mon[32247]: 8.7 scrub starts 2024-06-27T21:45:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:17 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:18.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:17 smithi192 ceph-mon[32956]: pgmap v2071: 57 pgs: 2 active+clean+scrubbing, 1 peering, 2 remapped+peering, 6 unknown, 3 active+recovering+undersized+remapped, 43 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s wr, 56 op/s; 1980/167505 objects misplaced (1.182%); 13 MiB/s, 209 objects/s recovering 2024-06-27T21:45:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:17 smithi192 ceph-mon[32956]: 8.7 scrub starts 2024-06-27T21:45:18.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:17 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:18.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:18.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:18.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:18.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:18.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:18.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:18.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:45:18.685272+0000 min lat: 0.00103148 max lat: 4.14194 avg lat: 0.0156592 2024-06-27T21:45:18.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:45:18.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 61292 61276 63.8235 59.125 0.00292852 0.0156592 2024-06-27T21:45:18.705 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:18.807 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:18.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:18.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:18.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:45:18.870 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:18.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:18.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:18.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:19.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:19.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:18 smithi151 ceph-mon[28965]: 8.6 scrub starts 2024-06-27T21:45:19.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:19.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:19.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:19.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:18 smithi190 ceph-mon[32247]: 8.6 scrub starts 2024-06-27T21:45:19.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:18 smithi192 ceph-mon[32956]: 8.6 scrub starts 2024-06-27T21:45:19.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:19.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:19.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:19.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:19.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:19.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:19.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 62369 62353 63.8806 67.3125 0.0172335 0.0156472 2024-06-27T21:45:19.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:19.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:19.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:19.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:19.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:19.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:19.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:45:20.008 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:20.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:20.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:20.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:19 smithi190 ceph-mon[32247]: pgmap v2073: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 1 peering, 2 remapped+peering, 4 unknown, 3 active+recovering+undersized+remapped, 44 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1831/168756 objects degraded (1.085%); 9065/168756 objects misplaced (5.372%); 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:45:20.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:19 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 1831/168756 objects degraded (1.085%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:45:20.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:20.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:19 smithi151 ceph-mon[28965]: pgmap v2073: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 1 peering, 2 remapped+peering, 4 unknown, 3 active+recovering+undersized+remapped, 44 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1831/168756 objects degraded (1.085%); 9065/168756 objects misplaced (5.372%); 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:45:20.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:19 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 1831/168756 objects degraded (1.085%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:45:20.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:20.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:19 smithi192 ceph-mon[32956]: pgmap v2073: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 1 peering, 2 remapped+peering, 4 unknown, 3 active+recovering+undersized+remapped, 44 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1831/168756 objects degraded (1.085%); 9065/168756 objects misplaced (5.372%); 7.9 MiB/s, 126 objects/s recovering 2024-06-27T21:45:20.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:19 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 1831/168756 objects degraded (1.085%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:45:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:20.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:20.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:20.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:20.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:20.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:20.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 63343 63327 63.8321 60.875 0.0103175 0.0156571 2024-06-27T21:45:20.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:20.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:20.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:20.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:20.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:20.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:21.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:21.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:21.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:21.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:20 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:21.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:20 smithi151 ceph-mon[28965]: 8.8 scrub starts 2024-06-27T21:45:21.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:21.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:20 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:21.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:20 smithi190 ceph-mon[32247]: 8.8 scrub starts 2024-06-27T21:45:21.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:20 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:21.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:20 smithi192 ceph-mon[32956]: 8.8 scrub starts 2024-06-27T21:45:21.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:21.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:21.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:21.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:21.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:21.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:21.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:21.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 16 64364 64348 63.8317 63.8125 0.00742936 0.015659 2024-06-27T21:45:21.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:21.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:21.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:21.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:21.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:21.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:21.958 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:22.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:22.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:22.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:22.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:21 smithi151 ceph-mon[28965]: pgmap v2074: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 6 active+recovering+undersized+remapped, 45 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 295 op/s; 5593/177867 objects degraded (3.144%); 23003/177867 objects misplaced (12.933%); 23 MiB/s, 374 objects/s recovering 2024-06-27T21:45:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:21 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:21 smithi151 ceph-mon[28965]: 8.3 scrub starts 2024-06-27T21:45:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:21 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:22.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:21 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:21 smithi190 ceph-mon[32247]: pgmap v2074: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 6 active+recovering+undersized+remapped, 45 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 295 op/s; 5593/177867 objects degraded (3.144%); 23003/177867 objects misplaced (12.933%); 23 MiB/s, 374 objects/s recovering 2024-06-27T21:45:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:21 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:21 smithi190 ceph-mon[32247]: 8.3 scrub starts 2024-06-27T21:45:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:21 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:22.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:21 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:22.269 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:21 smithi192 ceph-mon[32956]: pgmap v2074: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 6 active+recovering+undersized+remapped, 45 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 295 op/s; 5593/177867 objects degraded (3.144%); 23003/177867 objects misplaced (12.933%); 23 MiB/s, 374 objects/s recovering 2024-06-27T21:45:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:21 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:21 smithi192 ceph-mon[32956]: 8.3 scrub starts 2024-06-27T21:45:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:21 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:21 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.305 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:22.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:22.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:22.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:22.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:22.564 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:22.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:22.686 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 65292 65276 63.7405 58 0.019308 0.0156819 2024-06-27T21:45:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:22.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:22.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:22.832 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:22.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:45:22.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:45:22.916 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:45:22.916 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:45:22.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:22.955 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:45:22.955 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:45:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:22.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:22.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:22.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:45:22.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:45:23.031 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:45:23.032 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:45:23.074 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:45:23.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:45:23.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:23.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:23.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:45:23.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:45:23.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:45:23.162 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:45:23.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:45:23.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:45:23.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:22 smithi151 ceph-mon[28965]: 8.4 scrub starts 2024-06-27T21:45:23.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:22 smithi151 ceph-mon[28965]: 8.7 scrub starts 2024-06-27T21:45:23.206 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/487252877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:23.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:23.246 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:45:23.246 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:45:23.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:22 smithi190 ceph-mon[32247]: 8.4 scrub starts 2024-06-27T21:45:23.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:22 smithi190 ceph-mon[32247]: 8.7 scrub starts 2024-06-27T21:45:23.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/487252877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:23.281 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:45:23.281 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:45:23.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:22 smithi192 ceph-mon[32956]: 8.4 scrub starts 2024-06-27T21:45:23.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:22 smithi192 ceph-mon[32956]: 8.7 scrub starts 2024-06-27T21:45:23.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/487252877' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:23.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:45:23.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:45:23.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:23.391 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:23.493 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:23.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:23.561 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:23 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:45:23.450+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e837 definitely_dead 0 2024-06-27T21:45:23.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:23.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:23.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 66281 66265 63.7108 61.8125 0.00770491 0.0156894 2024-06-27T21:45:23.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:23.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:23.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:23.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:23.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:23.987 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.11. 2024-06-27T21:45:24.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:24.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:24.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:24.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:23 smithi151 ceph-mon[28965]: pgmap v2075: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 46 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 366 op/s; 11128/185751 objects degraded (5.991%); 32139/185751 objects misplaced (17.302%); 13 MiB/s, 201 objects/s recovering 2024-06-27T21:45:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:23 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:24.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2805206050' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:45:24.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:23 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:45:24.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:23 smithi151 ceph-mon[28965]: Client client.admin marked osd.11 out, while it was still marked up 2024-06-27T21:45:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:23 smithi190 ceph-mon[32247]: pgmap v2075: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 46 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 366 op/s; 11128/185751 objects degraded (5.991%); 32139/185751 objects misplaced (17.302%); 13 MiB/s, 201 objects/s recovering 2024-06-27T21:45:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:23 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2805206050' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:45:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:23 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:45:24.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:23 smithi190 ceph-mon[32247]: Client client.admin marked osd.11 out, while it was still marked up 2024-06-27T21:45:24.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:24.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:24.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:23 smithi192 ceph-mon[32956]: pgmap v2075: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 46 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 366 op/s; 11128/185751 objects degraded (5.991%); 32139/185751 objects misplaced (17.302%); 13 MiB/s, 201 objects/s recovering 2024-06-27T21:45:24.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:23 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:24.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2805206050' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:45:24.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:23 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["11"]}]: dispatch 2024-06-27T21:45:24.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:23 smithi192 ceph-mon[32956]: Client client.admin marked osd.11 out, while it was still marked up 2024-06-27T21:45:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:24.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:24.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:24.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:24.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:24.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:24.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:24.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 66626 66610 63.0721 21.5625 0.00240619 0.0157213 2024-06-27T21:45:24.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:24.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:24.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:24.927 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:24.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:25.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:25.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:25.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:25.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:25 smithi190 ceph-mon[32247]: 8.6 scrub starts 2024-06-27T21:45:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:25 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-06-27T21:45:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:25 smithi190 ceph-mon[32247]: osdmap e838: 12 total, 12 up, 11 in 2024-06-27T21:45:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:25 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 11128/185751 objects degraded (5.991%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:25.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:25 smithi190 ceph-mon[32247]: osdmap e839: 12 total, 12 up, 11 in 2024-06-27T21:45:25.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:25 smithi192 ceph-mon[32956]: 8.6 scrub starts 2024-06-27T21:45:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:25 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-06-27T21:45:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:25 smithi192 ceph-mon[32956]: osdmap e838: 12 total, 12 up, 11 in 2024-06-27T21:45:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:25 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 11128/185751 objects degraded (5.991%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:25.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:25 smithi192 ceph-mon[32956]: osdmap e839: 12 total, 12 up, 11 in 2024-06-27T21:45:25.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:25.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:25.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:25.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:25 smithi151 ceph-mon[28965]: 8.6 scrub starts 2024-06-27T21:45:25.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:25 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-06-27T21:45:25.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:25 smithi151 ceph-mon[28965]: osdmap e838: 12 total, 12 up, 11 in 2024-06-27T21:45:25.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:25 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 11128/185751 objects degraded (5.991%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:25.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:25 smithi151 ceph-mon[28965]: osdmap e839: 12 total, 12 up, 11 in 2024-06-27T21:45:25.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:25.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:25.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.675 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:25.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:25.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 16 66643 66627 62.1466 1.0625 0.00323999 0.0157797 2024-06-27T21:45:25.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:25.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:25.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:25.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:25.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:26.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:26.055 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:26.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:26.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:26 smithi151 ceph-mon[28965]: pgmap v2077: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 46 active+clean; 3.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 11128/192405 objects degraded (5.784%); 30362/192405 objects misplaced (15.780%); 23 MiB/s, 374 objects/s recovering 2024-06-27T21:45:26.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:26 smithi151 ceph-mon[28965]: osdmap e840: 12 total, 12 up, 11 in 2024-06-27T21:45:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:26 smithi190 ceph-mon[32247]: pgmap v2077: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 46 active+clean; 3.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 11128/192405 objects degraded (5.784%); 30362/192405 objects misplaced (15.780%); 23 MiB/s, 374 objects/s recovering 2024-06-27T21:45:26.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:26 smithi190 ceph-mon[32247]: osdmap e840: 12 total, 12 up, 11 in 2024-06-27T21:45:26.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:26.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:26 smithi192 ceph-mon[32956]: pgmap v2077: 57 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 46 active+clean; 3.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 11128/192405 objects degraded (5.784%); 30362/192405 objects misplaced (15.780%); 23 MiB/s, 374 objects/s recovering 2024-06-27T21:45:26.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:26 smithi192 ceph-mon[32956]: osdmap e840: 12 total, 12 up, 11 in 2024-06-27T21:45:26.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:26.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:26.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:26.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:26.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:26.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:26.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 16 66695 66679 61.2805 3.25 0.316131 0.0160635 2024-06-27T21:45:26.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:26.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:26.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:26.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:26.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:27.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:27.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:27.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:27.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:27 smithi190 ceph-mon[32247]: 8.6 starting backfill to osd.2 from (0'0,819'2324] MIN to 837'3886 2024-06-27T21:45:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:27 smithi190 ceph-mon[32247]: 8.7 starting backfill to osd.2 from (0'0,814'1468] MIN to 837'3976 2024-06-27T21:45:27.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:27 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:27.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:27 smithi192 ceph-mon[32956]: 8.6 starting backfill to osd.2 from (0'0,819'2324] MIN to 837'3886 2024-06-27T21:45:27.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:27 smithi192 ceph-mon[32956]: 8.7 starting backfill to osd.2 from (0'0,814'1468] MIN to 837'3976 2024-06-27T21:45:27.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:27 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:27.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:27.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:27.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:27.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:27 smithi151 ceph-mon[28965]: 8.6 starting backfill to osd.2 from (0'0,819'2324] MIN to 837'3886 2024-06-27T21:45:27.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:27 smithi151 ceph-mon[28965]: 8.7 starting backfill to osd.2 from (0'0,814'1468] MIN to 837'3976 2024-06-27T21:45:27.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:27 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:27.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:27.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:27.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:27.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:27.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 16 66714 66698 60.4095 1.1875 0.00335332 0.0160662 2024-06-27T21:45:27.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:27.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:27.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:27.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:27.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:27.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:28.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:28.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:28.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:28.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:28 smithi190 ceph-mon[32247]: pgmap v2080: 57 pgs: 3 remapped+peering, 2 unknown, 3 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 40 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 11128/197562 objects degraded (5.633%); 25804/197562 objects misplaced (13.061%); 23 MiB/s, 365 objects/s recovering 2024-06-27T21:45:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:28 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:28 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:28 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:45:28.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:28 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:28 smithi192 ceph-mon[32956]: pgmap v2080: 57 pgs: 3 remapped+peering, 2 unknown, 3 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 40 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 11128/197562 objects degraded (5.633%); 25804/197562 objects misplaced (13.061%); 23 MiB/s, 365 objects/s recovering 2024-06-27T21:45:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:28 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:28 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:28 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:45:28.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:28 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:28.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:28.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:28.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:28.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:28 smithi151 ceph-mon[28965]: pgmap v2080: 57 pgs: 3 remapped+peering, 2 unknown, 3 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 40 active+clean; 4.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 11128/197562 objects degraded (5.633%); 25804/197562 objects misplaced (13.061%); 23 MiB/s, 365 objects/s recovering 2024-06-27T21:45:28.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:28 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:28.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:28 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:28.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:28 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:45:28.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:28 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:28.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:28.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:28.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:28.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:28.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 66714 66698 59.5465 0 - 0.0160662 2024-06-27T21:45:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:28.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:28.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:28.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:28.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:29.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:29.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:29.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:29.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:29.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:29.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:29.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:29.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:29.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:29.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.493 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:29.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:29.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:29.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:29.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:29.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:29.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 15 67240 67225 59.1717 16.4688 0.00861097 0.0168938 2024-06-27T21:45:29.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:29.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:29.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:29.887 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:29.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:29.978 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:29.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:30.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:30.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:30.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:30.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:30 smithi151 ceph-mon[28965]: pgmap v2081: 57 pgs: 3 remapped+peering, 2 unknown, 3 peering, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 37 active+clean; 4.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 46 MiB/s wr, 740 op/s; 9297/199077 objects degraded (4.670%); 30508/199077 objects misplaced (15.325%); 29 MiB/s, 462 objects/s recovering 2024-06-27T21:45:30.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:30 smithi151 ceph-mon[28965]: osdmap e841: 12 total, 12 up, 11 in 2024-06-27T21:45:30.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:30 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:30.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:30 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 9297/199077 objects degraded (4.670%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:30.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:30.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:30 smithi192 ceph-mon[32956]: pgmap v2081: 57 pgs: 3 remapped+peering, 2 unknown, 3 peering, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 37 active+clean; 4.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 46 MiB/s wr, 740 op/s; 9297/199077 objects degraded (4.670%); 30508/199077 objects misplaced (15.325%); 29 MiB/s, 462 objects/s recovering 2024-06-27T21:45:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:30 smithi192 ceph-mon[32956]: osdmap e841: 12 total, 12 up, 11 in 2024-06-27T21:45:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:30 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:30 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 9297/199077 objects degraded (4.670%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:30.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:30.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:30.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:30 smithi190 ceph-mon[32247]: pgmap v2081: 57 pgs: 3 remapped+peering, 2 unknown, 3 peering, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 37 active+clean; 4.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 46 MiB/s wr, 740 op/s; 9297/199077 objects degraded (4.670%); 30508/199077 objects misplaced (15.325%); 29 MiB/s, 462 objects/s recovering 2024-06-27T21:45:30.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:30 smithi190 ceph-mon[32247]: osdmap e841: 12 total, 12 up, 11 in 2024-06-27T21:45:30.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:30 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:30.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:30 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 9297/199077 objects degraded (4.670%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:30.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:30.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:30.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:30.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 68088 68072 59.085 52.9375 0.00703887 0.0169178 2024-06-27T21:45:30.688 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:30.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:30.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:30.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:30.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:30.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:31.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:31.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:31.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:31.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:31.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:31.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:31 smithi151 ceph-mon[28965]: pgmap v2083: 57 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 38 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 28 MiB/s wr, 454 op/s; 17845/200589 objects degraded (8.896%); 36149/200589 objects misplaced (18.021%); 34 MiB/s, 547 objects/s recovering 2024-06-27T21:45:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:31 smithi151 ceph-mon[28965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:45:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:31 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-27T21:45:31.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:31 smithi151 ceph-mon[28965]: osdmap e842: 12 total, 12 up, 11 in 2024-06-27T21:45:31.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:31.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:31.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:31 smithi190 ceph-mon[32247]: pgmap v2083: 57 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 38 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 28 MiB/s wr, 454 op/s; 17845/200589 objects degraded (8.896%); 36149/200589 objects misplaced (18.021%); 34 MiB/s, 547 objects/s recovering 2024-06-27T21:45:31.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:31.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:31.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:31 smithi190 ceph-mon[32247]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:45:31.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:31 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-27T21:45:31.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:31 smithi190 ceph-mon[32247]: osdmap e842: 12 total, 12 up, 11 in 2024-06-27T21:45:31.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:31 smithi192 ceph-mon[32956]: pgmap v2083: 57 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 2 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 38 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 28 MiB/s wr, 454 op/s; 17845/200589 objects degraded (8.896%); 36149/200589 objects misplaced (18.021%); 34 MiB/s, 547 objects/s recovering 2024-06-27T21:45:31.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:31.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:31.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:31 smithi192 ceph-mon[32956]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-06-27T21:45:31.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:31 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-27T21:45:31.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:31 smithi192 ceph-mon[32956]: osdmap e842: 12 total, 12 up, 11 in 2024-06-27T21:45:31.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:31.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 68895 68879 58.9665 50.4375 0.0297185 0.0169534 2024-06-27T21:45:31.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:31.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:31.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:31.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:31.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:31.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:31.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:32.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:45:32.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:32.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:32.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:32.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:32.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:32.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:32.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:32 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:32.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:32 smithi151 ceph-mon[28965]: osdmap e843: 12 total, 12 up, 11 in 2024-06-27T21:45:32.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:32.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:32.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:32 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:32.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:32 smithi192 ceph-mon[32956]: osdmap e843: 12 total, 12 up, 11 in 2024-06-27T21:45:32.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:32.664 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:32.687 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 69667 69651 58.8217 48.25 0.0127256 0.0169936 2024-06-27T21:45:32.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:32 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:32.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:32 smithi190 ceph-mon[32247]: osdmap e843: 12 total, 12 up, 11 in 2024-06-27T21:45:32.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:32.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:32.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:32.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:32.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:33.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:33.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:33.166 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:33.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:33.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:33.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:33.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:33.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:33.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:33.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:33.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 15 70558 70543 58.7806 55.75 0.0118068 0.0170053 2024-06-27T21:45:33.689 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:33.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:33 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:33.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:33 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:33.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:33.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:33.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:33 smithi151 ceph-mon[28965]: pgmap v2086: 57 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 38 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 14 MiB/s wr, 219 op/s; 19659/201522 objects degraded (9.755%); 35364/201522 objects misplaced (17.548%); 26 MiB/s, 416 objects/s recovering 2024-06-27T21:45:33.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:33 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:33 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:33 smithi190 ceph-mon[32247]: pgmap v2086: 57 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 38 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 14 MiB/s wr, 219 op/s; 19659/201522 objects degraded (9.755%); 35364/201522 objects misplaced (17.548%); 26 MiB/s, 416 objects/s recovering 2024-06-27T21:45:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:33 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:33 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:33 smithi192 ceph-mon[32956]: pgmap v2086: 57 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 38 active+clean; 4.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 14 MiB/s wr, 219 op/s; 19659/201522 objects degraded (9.755%); 35364/201522 objects misplaced (17.548%); 26 MiB/s, 416 objects/s recovering 2024-06-27T21:45:33.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:33.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:33.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:33.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:34.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:34.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:34.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:34.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:34.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:34.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:34.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:34.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:34.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:34.633 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:34.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 71322 71306 58.6346 47.6875 0.00346107 0.0170469 2024-06-27T21:45:34.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:34 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:34.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:34.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:34 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:34.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:34.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:34 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:34.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:34.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:34.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:35.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:35.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:35.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:35.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:35.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:35.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:35.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:35.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:35.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:35.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:35.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:35.666 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:35 smithi190 ceph-mon[32247]: pgmap v2087: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 29 MiB/s wr, 458 op/s; 19657/207336 objects degraded (9.481%); 31948/207336 objects misplaced (15.409%); 39 MiB/s, 619 objects/s recovering 2024-06-27T21:45:35.667 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:35 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 19657/207336 objects degraded (9.481%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:35.679 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:35.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 71997 71981 58.421 42.1875 0.0147082 0.0171089 2024-06-27T21:45:35.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:35 smithi151 ceph-mon[28965]: pgmap v2087: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 29 MiB/s wr, 458 op/s; 19657/207336 objects degraded (9.481%); 31948/207336 objects misplaced (15.409%); 39 MiB/s, 619 objects/s recovering 2024-06-27T21:45:35.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:35 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 19657/207336 objects degraded (9.481%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:35.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:35.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:35 smithi192 ceph-mon[32956]: pgmap v2087: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 29 MiB/s wr, 458 op/s; 19657/207336 objects degraded (9.481%); 31948/207336 objects misplaced (15.409%); 39 MiB/s, 619 objects/s recovering 2024-06-27T21:45:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:35 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 19657/207336 objects degraded (9.481%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:35.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:35.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:35.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:35.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:36.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:36.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:36.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:36.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:36.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:36.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:36.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:36.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:36.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:36.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 72779 72763 58.2985 48.875 0.0166955 0.0171435 2024-06-27T21:45:36.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:36 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:36.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:36.741 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:36 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:36.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:36.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:36 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:36.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:36.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:36.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:36.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:37.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:37.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:37.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:37.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:37.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:37.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:37.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:37.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:37.530 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:37.609 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:45:37.609 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:45:37.633 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:37.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:37.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 73523 73507 58.1492 46.5 0.0100044 0.0171899 2024-06-27T21:45:37.694 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:37 smithi151 ceph-mon[28965]: pgmap v2088: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 36 MiB/s wr, 581 op/s; 19657/212721 objects degraded (9.241%); 30504/212721 objects misplaced (14.340%); 36 MiB/s, 574 objects/s recovering 2024-06-27T21:45:37.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:37.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:37 smithi190 ceph-mon[32247]: pgmap v2088: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 36 MiB/s wr, 581 op/s; 19657/212721 objects degraded (9.241%); 30504/212721 objects misplaced (14.340%); 36 MiB/s, 574 objects/s recovering 2024-06-27T21:45:37.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:37 smithi192 ceph-mon[32956]: pgmap v2088: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 36 MiB/s wr, 581 op/s; 19657/212721 objects degraded (9.241%); 30504/212721 objects misplaced (14.340%); 36 MiB/s, 574 objects/s recovering 2024-06-27T21:45:37.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:37.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:37.930 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:37.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:38.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:38.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:38.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:38.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:38.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.320 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:38.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:38.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:38.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:38.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:38.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:38.656 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:38 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:38.656 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:38 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:38.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:45:38.686987+0000 min lat: 0.00103148 max lat: 5.1397 avg lat: 0.017223 2024-06-27T21:45:38.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:45:38.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 74304 74288 58.0324 48.8125 0.0391702 0.017223 2024-06-27T21:45:38.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:38.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:38.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:38 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:38.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:38 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:38.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:38 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:38.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:38 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:38.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:38.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:38.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:38.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:38.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:39.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:39.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:39.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:39.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:39.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:39.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:39.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:39.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:39.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:39.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:39.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 75137 75121 57.9587 52.0625 0.00347867 0.0172451 2024-06-27T21:45:39.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:39 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:39.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:39 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:39.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:39 smithi151 ceph-mon[28965]: pgmap v2089: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.4 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 42 MiB/s wr, 664 op/s; 19657/216540 objects degraded (9.078%); 28487/216540 objects misplaced (13.156%); 36 MiB/s, 580 objects/s recovering 2024-06-27T21:45:39.761 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:39.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:39 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:39.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:39 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:39.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:39 smithi190 ceph-mon[32247]: pgmap v2089: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.4 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 42 MiB/s wr, 664 op/s; 19657/216540 objects degraded (9.078%); 28487/216540 objects misplaced (13.156%); 36 MiB/s, 580 objects/s recovering 2024-06-27T21:45:39.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:39.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:39 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:39 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:39 smithi192 ceph-mon[32956]: pgmap v2089: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.4 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 42 MiB/s wr, 664 op/s; 19657/216540 objects degraded (9.078%); 28487/216540 objects misplaced (13.156%); 36 MiB/s, 580 objects/s recovering 2024-06-27T21:45:39.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:39.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:39.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:40.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:40.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:40.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:40.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:40.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:40.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:45:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:40.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:40.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:40.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:40.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:40.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:40.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:40.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 75573 75557 57.5841 27.25 0.0108075 0.0172993 2024-06-27T21:45:40.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:40.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:40 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 19657/216540 objects degraded (9.078%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:40.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:40 smithi151 ceph-mon[28965]: osdmap e844: 12 total, 12 up, 11 in 2024-06-27T21:45:40.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:40 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 19657/216540 objects degraded (9.078%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:40.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:40 smithi190 ceph-mon[32247]: osdmap e844: 12 total, 12 up, 11 in 2024-06-27T21:45:40.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:40.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:40 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 19657/216540 objects degraded (9.078%), 7 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:40.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:40 smithi192 ceph-mon[32956]: osdmap e844: 12 total, 12 up, 11 in 2024-06-27T21:45:40.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:40.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:40.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:40.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:41.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:41.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:41.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:41.211 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:41.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:41.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:41.409 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:41.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:41.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:41.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:41.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:41.689 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 76266 76250 57.4122 43.3125 0.00534485 0.0174054 2024-06-27T21:45:41.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:41.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:41.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:41.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:41.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:41.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:41 smithi151 ceph-mon[28965]: pgmap v2091: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s; 19657/221955 objects degraded (8.856%); 26331/221955 objects misplaced (11.863%); 43 MiB/s, 680 objects/s recovering 2024-06-27T21:45:41.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:41 smithi151 ceph-mon[28965]: osdmap e845: 12 total, 12 up, 11 in 2024-06-27T21:45:41.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:41 smithi151 ceph-mon[28965]: 8.15 scrub starts 2024-06-27T21:45:41.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:41 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:41.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:41.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:42.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:42.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:42.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:42.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:41 smithi190 ceph-mon[32247]: pgmap v2091: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s; 19657/221955 objects degraded (8.856%); 26331/221955 objects misplaced (11.863%); 43 MiB/s, 680 objects/s recovering 2024-06-27T21:45:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:41 smithi190 ceph-mon[32247]: osdmap e845: 12 total, 12 up, 11 in 2024-06-27T21:45:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:41 smithi190 ceph-mon[32247]: 8.15 scrub starts 2024-06-27T21:45:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:41 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:41 smithi192 ceph-mon[32956]: pgmap v2091: 57 pgs: 1 active+remapped+backfilling, 7 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.5 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s; 19657/221955 objects degraded (8.856%); 26331/221955 objects misplaced (11.863%); 43 MiB/s, 680 objects/s recovering 2024-06-27T21:45:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:41 smithi192 ceph-mon[32956]: osdmap e845: 12 total, 12 up, 11 in 2024-06-27T21:45:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:41 smithi192 ceph-mon[32956]: 8.15 scrub starts 2024-06-27T21:45:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:41 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:42.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:42.377 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:42.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:42.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:42.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:42.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:42.689 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 76989 76973 57.2666 45.1875 0.0773258 0.0174533 2024-06-27T21:45:42.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:42.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:42.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:42.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:42.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:43.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:43.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:43.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:43.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:43.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:42 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:43.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:42 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:43.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:42 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:43.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:43.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:43.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:43.538 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:43.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:43.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:43.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 77464 77448 56.9421 29.6875 0.235631 0.0174816 2024-06-27T21:45:43.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:43.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:43.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:43.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:43.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:44.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:44.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:44.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:44.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:43 smithi151 ceph-mon[28965]: pgmap v2093: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 43 MiB/s wr, 693 op/s; 17738/223992 objects degraded (7.919%); 26991/223992 objects misplaced (12.050%); 31 MiB/s, 502 objects/s recovering 2024-06-27T21:45:44.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:43 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:45:44.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:43 smithi151 ceph-mon[28965]: osdmap e846: 12 total, 12 up, 11 in 2024-06-27T21:45:44.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:43 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:43 smithi190 ceph-mon[32247]: pgmap v2093: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 43 MiB/s wr, 693 op/s; 17738/223992 objects degraded (7.919%); 26991/223992 objects misplaced (12.050%); 31 MiB/s, 502 objects/s recovering 2024-06-27T21:45:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:43 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:45:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:43 smithi190 ceph-mon[32247]: osdmap e846: 12 total, 12 up, 11 in 2024-06-27T21:45:44.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:43 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:44.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:43 smithi192 ceph-mon[32956]: pgmap v2093: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 6 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 40 active+clean; 4.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 43 MiB/s wr, 693 op/s; 17738/223992 objects degraded (7.919%); 26991/223992 objects misplaced (12.050%); 31 MiB/s, 502 objects/s recovering 2024-06-27T21:45:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:43 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:45:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:43 smithi192 ceph-mon[32956]: osdmap e846: 12 total, 12 up, 11 in 2024-06-27T21:45:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:43 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:44.304 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:44.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.429 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:44.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:44.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:44.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:44.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:44.688 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 77620 77604 56.3933 9.75 0.0495559 0.0177224 2024-06-27T21:45:44.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:44.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.828 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:44.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:44.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:44.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:44.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:45.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:45.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:45.184 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 1, 7] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:45:45.184 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:45:45.185 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-06-27T21:45:45.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd in 11 2024-06-27T21:45:45.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: osdmap e847: 12 total, 12 up, 11 in 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:45.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:45.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 17738/223992 objects degraded (7.919%), 6 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:45.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: osdmap e847: 12 total, 12 up, 11 in 2024-06-27T21:45:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:45.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 17738/223992 objects degraded (7.919%), 6 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: osdmap e847: 12 total, 12 up, 11 in 2024-06-27T21:45:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 17738/223992 objects degraded (7.919%), 6 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:45.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:45.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:45.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:45.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:45.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:45.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:45.689 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 77979 77963 56.003 22.4375 0.250473 0.0177759 2024-06-27T21:45:45.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:45.727 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:45.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:45.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:45.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:45.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:45.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:45.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:46.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:46.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:45 smithi151 ceph-mon[28965]: pgmap v2096: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 39 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 34 MiB/s wr, 548 op/s; 10233/229200 objects degraded (4.465%); 31248/229200 objects misplaced (13.634%); 35 MiB/s, 565 objects/s recovering 2024-06-27T21:45:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:45 smithi151 ceph-mon[28965]: osdmap e848: 12 total, 12 up, 11 in 2024-06-27T21:45:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:45 smithi151 ceph-mon[28965]: 8.11 scrub starts 2024-06-27T21:45:46.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:46.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:45 smithi190 ceph-mon[32247]: pgmap v2096: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 39 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 34 MiB/s wr, 548 op/s; 10233/229200 objects degraded (4.465%); 31248/229200 objects misplaced (13.634%); 35 MiB/s, 565 objects/s recovering 2024-06-27T21:45:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:45 smithi190 ceph-mon[32247]: osdmap e848: 12 total, 12 up, 11 in 2024-06-27T21:45:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:45 smithi190 ceph-mon[32247]: 8.11 scrub starts 2024-06-27T21:45:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:45 smithi192 ceph-mon[32956]: pgmap v2096: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 1 creating+peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 39 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 34 MiB/s wr, 548 op/s; 10233/229200 objects degraded (4.465%); 31248/229200 objects misplaced (13.634%); 35 MiB/s, 565 objects/s recovering 2024-06-27T21:45:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:45 smithi192 ceph-mon[32956]: osdmap e848: 12 total, 12 up, 11 in 2024-06-27T21:45:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:45 smithi192 ceph-mon[32956]: 8.11 scrub starts 2024-06-27T21:45:46.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:46.365 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:46.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:46.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:46.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:46.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:46.689 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 78281 78265 55.5811 18.875 0.31493 0.0179239 2024-06-27T21:45:46.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:46.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:46.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:46.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:46.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:46.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:46.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:47.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:47.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:47.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:47.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:47.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:47.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:47.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:46 smithi190 ceph-mon[32247]: osdmap e849: 12 total, 12 up, 11 in 2024-06-27T21:45:47.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:46 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:47.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:46 smithi190 ceph-mon[32247]: 8.15 scrub starts 2024-06-27T21:45:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:46 smithi192 ceph-mon[32956]: osdmap e849: 12 total, 12 up, 11 in 2024-06-27T21:45:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:46 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:47.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:46 smithi192 ceph-mon[32956]: 8.15 scrub starts 2024-06-27T21:45:47.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:47.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:47.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:47.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:46 smithi151 ceph-mon[28965]: osdmap e849: 12 total, 12 up, 11 in 2024-06-27T21:45:47.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:46 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:47.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:46 smithi151 ceph-mon[28965]: 8.15 scrub starts 2024-06-27T21:45:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:47.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:47.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:47.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:47.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:47.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:47.689 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 16 78750 78734 55.2859 29.3125 0.017787 0.0180808 2024-06-27T21:45:47.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:47.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:47.831 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:47.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:47.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:47.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:48.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:48.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:47 smithi151 ceph-mon[28965]: pgmap v2099: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 45 MiB/s wr, 721 op/s; 10233/232647 objects degraded (4.399%); 28003/232647 objects misplaced (12.037%); 61 MiB/s, 973 objects/s recovering 2024-06-27T21:45:48.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:47 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-27T21:45:48.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:47 smithi151 ceph-mon[28965]: osdmap e850: 12 total, 12 up, 11 in 2024-06-27T21:45:48.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:47 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:48.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:48.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:48.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:48.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:48.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:48.251 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:47 smithi192 ceph-mon[32956]: pgmap v2099: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 45 MiB/s wr, 721 op/s; 10233/232647 objects degraded (4.399%); 28003/232647 objects misplaced (12.037%); 61 MiB/s, 973 objects/s recovering 2024-06-27T21:45:48.251 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:47 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-27T21:45:48.251 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:47 smithi192 ceph-mon[32956]: osdmap e850: 12 total, 12 up, 11 in 2024-06-27T21:45:48.252 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:47 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:48.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:47 smithi190 ceph-mon[32247]: pgmap v2099: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 41 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 45 MiB/s wr, 721 op/s; 10233/232647 objects degraded (4.399%); 28003/232647 objects misplaced (12.037%); 61 MiB/s, 973 objects/s recovering 2024-06-27T21:45:48.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:47 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-06-27T21:45:48.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:47 smithi190 ceph-mon[32247]: osdmap e850: 12 total, 12 up, 11 in 2024-06-27T21:45:48.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:47 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:48.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:48.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:48.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:48.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:48.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:48.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:45:48.662 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:45:48.689 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 9 79435 79426 55.1521 43.25 0.07878 0.0181248 2024-06-27T21:45:48.706 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:45:48.706 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:45:48.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:48.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:45:48.733 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:45:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:48.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:45:48.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:45:48.782 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:45:48.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:45:48.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:48.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:48.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:48.964 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:45:48.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:49.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:49.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:45:49.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:45:49.059 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:45:49.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:45:49.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:45:49.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:45:49.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:49.120 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:45:49.120 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:45:49.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:49.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:45:49.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:45:49.178 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:45:49.178 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:45:49.208 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:45:49.209 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:45:49.224 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:48 smithi151 ceph-mon[28965]: 8.3 scrub starts 2024-06-27T21:45:49.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:48 smithi151 ceph-mon[28965]: from='client.17565 172.21.15.151:0/4080366052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:45:49.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:48 smithi151 ceph-mon[28965]: from='client.17565 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:45:49.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:48 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:49.225 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/409816017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:49.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:49.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:48 smithi190 ceph-mon[32247]: 8.3 scrub starts 2024-06-27T21:45:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:48 smithi190 ceph-mon[32247]: from='client.17565 172.21.15.151:0/4080366052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:45:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:48 smithi190 ceph-mon[32247]: from='client.17565 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:45:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:48 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:49.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/409816017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:48 smithi192 ceph-mon[32956]: 8.3 scrub starts 2024-06-27T21:45:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:48 smithi192 ceph-mon[32956]: from='client.17565 172.21.15.151:0/4080366052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:45:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:48 smithi192 ceph-mon[32956]: from='client.17565 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:45:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:48 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:49.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/409816017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:45:49.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:49.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:49.490 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:49.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:49.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:49.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:49.690 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.7122 2024-06-27T21:45:49.690 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 79435 2024-06-27T21:45:49.690 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:45:49.690 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:45:49.690 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 54.7301 2024-06-27T21:45:49.690 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 30.296 2024-06-27T21:45:49.691 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 127.312 2024-06-27T21:45:49.691 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 0 2024-06-27T21:45:49.691 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 875 2024-06-27T21:45:49.691 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 484.75 2024-06-27T21:45:49.691 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 2037 2024-06-27T21:45:49.691 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 0 2024-06-27T21:45:49.692 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0181475 2024-06-27T21:45:49.692 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0886967 2024-06-27T21:45:49.692 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 5.1397 2024-06-27T21:45:49.692 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.00103148 2024-06-27T21:45:49.699 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:45:49.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:49.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:49.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:49.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:49.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:50.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:50.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:50.090 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:45:50.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:50.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:50.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: pgmap v2101: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 42 active+clean; 4.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 30 MiB/s wr, 485 op/s; 10233/234441 objects degraded (4.365%); 25179/234441 objects misplaced (10.740%); 50 MiB/s, 803 objects/s recovering 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: from='client.17565 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: osdmap e851: 12 total, 12 up, 11 in 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: 8.7 scrub starts 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: 8.7 scrub ok 2024-06-27T21:45:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 10233/234441 objects degraded (4.365%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:50.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: pgmap v2101: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 42 active+clean; 4.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 30 MiB/s wr, 485 op/s; 10233/234441 objects degraded (4.365%); 25179/234441 objects misplaced (10.740%); 50 MiB/s, 803 objects/s recovering 2024-06-27T21:45:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: from='client.17565 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:45:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: osdmap e851: 12 total, 12 up, 11 in 2024-06-27T21:45:50.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:45:50.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: 8.7 scrub starts 2024-06-27T21:45:50.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: 8.7 scrub ok 2024-06-27T21:45:50.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 10233/234441 objects degraded (4.365%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:50.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:50.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:50.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: pgmap v2101: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 42 active+clean; 4.8 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 30 MiB/s wr, 485 op/s; 10233/234441 objects degraded (4.365%); 25179/234441 objects misplaced (10.740%); 50 MiB/s, 803 objects/s recovering 2024-06-27T21:45:50.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:45:50.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: from='client.17565 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:45:50.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: osdmap e851: 12 total, 12 up, 11 in 2024-06-27T21:45:50.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:45:50.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: 8.7 scrub starts 2024-06-27T21:45:50.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: 8.7 scrub ok 2024-06-27T21:45:50.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 10233/234441 objects degraded (4.365%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:50.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:50.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:50.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:50.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:50.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:50.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:50.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:50.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:50.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:51.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:51.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:51.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:51.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:50 smithi151 ceph-mon[28965]: osdmap e852: 12 total, 12 up, 11 in 2024-06-27T21:45:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:50 smithi151 ceph-mon[28965]: 8.1 scrub starts 2024-06-27T21:45:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:50 smithi151 ceph-mon[28965]: 8.11 scrub starts 2024-06-27T21:45:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:50 smithi151 ceph-mon[28965]: 8.11 scrub ok 2024-06-27T21:45:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:50 smithi151 ceph-mon[28965]: 8.1 scrub ok 2024-06-27T21:45:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:50 smithi190 ceph-mon[32247]: osdmap e852: 12 total, 12 up, 11 in 2024-06-27T21:45:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:50 smithi190 ceph-mon[32247]: 8.1 scrub starts 2024-06-27T21:45:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:50 smithi190 ceph-mon[32247]: 8.11 scrub starts 2024-06-27T21:45:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:50 smithi190 ceph-mon[32247]: 8.11 scrub ok 2024-06-27T21:45:51.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:50 smithi190 ceph-mon[32247]: 8.1 scrub ok 2024-06-27T21:45:51.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:50 smithi192 ceph-mon[32956]: osdmap e852: 12 total, 12 up, 11 in 2024-06-27T21:45:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:50 smithi192 ceph-mon[32956]: 8.1 scrub starts 2024-06-27T21:45:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:50 smithi192 ceph-mon[32956]: 8.11 scrub starts 2024-06-27T21:45:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:50 smithi192 ceph-mon[32956]: 8.11 scrub ok 2024-06-27T21:45:51.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:50 smithi192 ceph-mon[32956]: 8.1 scrub ok 2024-06-27T21:45:51.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:51.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:51.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:51.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:51.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:51.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:51.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:51.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:51.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:51.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:51.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:51.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:51.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:52.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:45:52.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:52.079 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:52.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:52.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:51 smithi151 ceph-mon[28965]: pgmap v2104: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 43 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 234 B/s rd, 24 MiB/s wr, 387 op/s; 10233/236763 objects degraded (4.322%); 24016/236763 objects misplaced (10.143%); 33 MiB/s, 520 objects/s recovering 2024-06-27T21:45:52.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:51 smithi151 ceph-mon[28965]: 8.0 scrub ok 2024-06-27T21:45:52.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:51 smithi151 ceph-mon[28965]: 8.4 scrub starts 2024-06-27T21:45:52.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:51 smithi151 ceph-mon[28965]: osdmap e853: 12 total, 12 up, 11 in 2024-06-27T21:45:52.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:51 smithi151 ceph-mon[28965]: 8.9 scrub starts 2024-06-27T21:45:52.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:51 smithi190 ceph-mon[32247]: pgmap v2104: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 43 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 234 B/s rd, 24 MiB/s wr, 387 op/s; 10233/236763 objects degraded (4.322%); 24016/236763 objects misplaced (10.143%); 33 MiB/s, 520 objects/s recovering 2024-06-27T21:45:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:51 smithi190 ceph-mon[32247]: 8.0 scrub ok 2024-06-27T21:45:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:51 smithi190 ceph-mon[32247]: 8.4 scrub starts 2024-06-27T21:45:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:51 smithi190 ceph-mon[32247]: osdmap e853: 12 total, 12 up, 11 in 2024-06-27T21:45:52.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:51 smithi190 ceph-mon[32247]: 8.9 scrub starts 2024-06-27T21:45:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:51 smithi192 ceph-mon[32956]: pgmap v2104: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 43 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 234 B/s rd, 24 MiB/s wr, 387 op/s; 10233/236763 objects degraded (4.322%); 24016/236763 objects misplaced (10.143%); 33 MiB/s, 520 objects/s recovering 2024-06-27T21:45:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:51 smithi192 ceph-mon[32956]: 8.0 scrub ok 2024-06-27T21:45:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:51 smithi192 ceph-mon[32956]: 8.4 scrub starts 2024-06-27T21:45:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:51 smithi192 ceph-mon[32956]: osdmap e853: 12 total, 12 up, 11 in 2024-06-27T21:45:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:51 smithi192 ceph-mon[32956]: 8.9 scrub starts 2024-06-27T21:45:52.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:52.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:52.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:52.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:52.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:45:52.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:52.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:52.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:52.716 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:52.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:52.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:52.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:52.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:52.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:53.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:53.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:53.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:52 smithi151 ceph-mon[28965]: osdmap e854: 12 total, 12 up, 11 in 2024-06-27T21:45:53.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.15 scrub starts 2024-06-27T21:45:53.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.5 scrub starts 2024-06-27T21:45:53.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.5 scrub ok 2024-06-27T21:45:53.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.9 scrub ok 2024-06-27T21:45:53.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:53.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:52 smithi190 ceph-mon[32247]: osdmap e854: 12 total, 12 up, 11 in 2024-06-27T21:45:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:52 smithi190 ceph-mon[32247]: 8.15 scrub starts 2024-06-27T21:45:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:52 smithi190 ceph-mon[32247]: 8.5 scrub starts 2024-06-27T21:45:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:52 smithi190 ceph-mon[32247]: 8.5 scrub ok 2024-06-27T21:45:53.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:52 smithi190 ceph-mon[32247]: 8.9 scrub ok 2024-06-27T21:45:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:52 smithi192 ceph-mon[32956]: osdmap e854: 12 total, 12 up, 11 in 2024-06-27T21:45:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:52 smithi192 ceph-mon[32956]: 8.15 scrub starts 2024-06-27T21:45:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:52 smithi192 ceph-mon[32956]: 8.5 scrub starts 2024-06-27T21:45:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:52 smithi192 ceph-mon[32956]: 8.5 scrub ok 2024-06-27T21:45:53.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:52 smithi192 ceph-mon[32956]: 8.9 scrub ok 2024-06-27T21:45:53.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:53.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:53.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:53.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:53.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:53.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:53.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:53.756 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:53.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:53.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:53.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:53.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:54.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:54.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:54.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:54.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: 8.4 scrub ok 2024-06-27T21:45:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: pgmap v2107: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 43 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 255 B/s rd, 17 MiB/s wr, 309 op/s; 10183/237312 objects degraded (4.291%); 22452/237312 objects misplaced (9.461%); 16 MiB/s, 259 objects/s recovering 2024-06-27T21:45:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: 8.14 scrub starts 2024-06-27T21:45:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: 8.15 scrub ok 2024-06-27T21:45:54.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: osdmap e855: 12 total, 12 up, 11 in 2024-06-27T21:45:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: 8.10 scrub starts 2024-06-27T21:45:54.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:53 smithi190 ceph-mon[32247]: 8.2 scrub starts 2024-06-27T21:45:54.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: 8.4 scrub ok 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: pgmap v2107: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 43 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 255 B/s rd, 17 MiB/s wr, 309 op/s; 10183/237312 objects degraded (4.291%); 22452/237312 objects misplaced (9.461%); 16 MiB/s, 259 objects/s recovering 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: 8.14 scrub starts 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: 8.15 scrub ok 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: osdmap e855: 12 total, 12 up, 11 in 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: 8.10 scrub starts 2024-06-27T21:45:54.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:53 smithi192 ceph-mon[32956]: 8.2 scrub starts 2024-06-27T21:45:54.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:54.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.4 scrub ok 2024-06-27T21:45:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: pgmap v2107: 57 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 43 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 255 B/s rd, 17 MiB/s wr, 309 op/s; 10183/237312 objects degraded (4.291%); 22452/237312 objects misplaced (9.461%); 16 MiB/s, 259 objects/s recovering 2024-06-27T21:45:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.14 scrub starts 2024-06-27T21:45:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.15 scrub ok 2024-06-27T21:45:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: osdmap e855: 12 total, 12 up, 11 in 2024-06-27T21:45:54.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.10 scrub starts 2024-06-27T21:45:54.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:53 smithi151 ceph-mon[28965]: 8.2 scrub starts 2024-06-27T21:45:54.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:54.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:54.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:54.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:54.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:54.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:54.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:54.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:54.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:55.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:55.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:55.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: 8.14 scrub ok 2024-06-27T21:45:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: 8.3 scrub starts 2024-06-27T21:45:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: 8.10 scrub ok 2024-06-27T21:45:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: 8.18 scrub starts 2024-06-27T21:45:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: 8.2 scrub ok 2024-06-27T21:45:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: 8.18 scrub ok 2024-06-27T21:45:55.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:55 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 10183/237312 objects degraded (4.291%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:55.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:55.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:55.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: 8.14 scrub ok 2024-06-27T21:45:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: 8.3 scrub starts 2024-06-27T21:45:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: 8.10 scrub ok 2024-06-27T21:45:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: 8.18 scrub starts 2024-06-27T21:45:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: 8.2 scrub ok 2024-06-27T21:45:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: 8.18 scrub ok 2024-06-27T21:45:55.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:55 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 10183/237312 objects degraded (4.291%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:55.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: 8.14 scrub ok 2024-06-27T21:45:55.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: 8.3 scrub starts 2024-06-27T21:45:55.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: 8.10 scrub ok 2024-06-27T21:45:55.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: 8.18 scrub starts 2024-06-27T21:45:55.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: 8.2 scrub ok 2024-06-27T21:45:55.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: 8.18 scrub ok 2024-06-27T21:45:55.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:55 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 10183/237312 objects degraded (4.291%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:45:55.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:55.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:55.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:55.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:55.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:55.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:55.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:55.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:55.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:55.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:55.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:56.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:56.062 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:56.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:56.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:56 smithi190 ceph-mon[32247]: pgmap v2109: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 46 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 6052/235719 objects degraded (2.567%); 22404/235719 objects misplaced (9.505%) 2024-06-27T21:45:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:56 smithi190 ceph-mon[32247]: 8.3 scrub ok 2024-06-27T21:45:56.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:45:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:56 smithi192 ceph-mon[32956]: pgmap v2109: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 46 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 6052/235719 objects degraded (2.567%); 22404/235719 objects misplaced (9.505%) 2024-06-27T21:45:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:56 smithi192 ceph-mon[32956]: 8.3 scrub ok 2024-06-27T21:45:56.317 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:45:56.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.447 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:56.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:56.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:56 smithi151 ceph-mon[28965]: pgmap v2109: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 46 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 6052/235719 objects degraded (2.567%); 22404/235719 objects misplaced (9.505%) 2024-06-27T21:45:56.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:56 smithi151 ceph-mon[28965]: 8.3 scrub ok 2024-06-27T21:45:56.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:56.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:56.579 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:45:56.579 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:45:56.579 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:45:56.579 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:45:56.579 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:45:56.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:56.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:56.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:56.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:56.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:56.957 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:56.962 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:57.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:57.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:57.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:57.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:57 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:57 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:57.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:45:57.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:45:57.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:57 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:57.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:57 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:57.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:57.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:57.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:57 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:57.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:57 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:57.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:57.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:57.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:57.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:57.834 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.857 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:57.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:57.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:45:57.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:57.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:45:58.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:58.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:58.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:58.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:58.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:58.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:58 smithi190 ceph-mon[32247]: pgmap v2110: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 102/233373 objects degraded (0.044%); 26843/233373 objects misplaced (11.502%) 2024-06-27T21:45:58.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:58 smithi190 ceph-mon[32247]: osdmap e856: 12 total, 12 up, 11 in 2024-06-27T21:45:58.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:58 smithi192 ceph-mon[32956]: pgmap v2110: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 102/233373 objects degraded (0.044%); 26843/233373 objects misplaced (11.502%) 2024-06-27T21:45:58.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:58 smithi192 ceph-mon[32956]: osdmap e856: 12 total, 12 up, 11 in 2024-06-27T21:45:58.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:58.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:58.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:58 smithi151 ceph-mon[28965]: pgmap v2110: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 102/233373 objects degraded (0.044%); 26843/233373 objects misplaced (11.502%) 2024-06-27T21:45:58.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:58 smithi151 ceph-mon[28965]: osdmap e856: 12 total, 12 up, 11 in 2024-06-27T21:45:58.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:58.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:58.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:58.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:58.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:45:58.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:45:58.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:45:58.864 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:45:58.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:58.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:58.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:59.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:45:59.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:45:59.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:45:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:59.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:45:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:45:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:45:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:45:59 smithi192 ceph-mon[32956]: osdmap e857: 12 total, 12 up, 11 in 2024-06-27T21:45:59.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:45:59.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:45:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:45:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:45:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:45:59 smithi151 ceph-mon[28965]: osdmap e857: 12 total, 12 up, 11 in 2024-06-27T21:45:59.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:45:59.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:45:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:45:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:45:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:45:59 smithi190 ceph-mon[32247]: osdmap e857: 12 total, 12 up, 11 in 2024-06-27T21:45:59.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:45:59.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:45:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:45:59.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:45:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:45:59.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:45:59.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:45:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:45:59.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:45:59.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:45:59.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:00.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:00.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:00.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:00.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:00.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:00 smithi192 ceph-mon[32956]: pgmap v2113: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 102/231918 objects degraded (0.044%); 26843/231918 objects misplaced (11.574%) 2024-06-27T21:46:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:00 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 102/231918 objects degraded (0.044%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:00.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:00.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:00.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:00 smithi151 ceph-mon[28965]: pgmap v2113: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 102/231918 objects degraded (0.044%); 26843/231918 objects misplaced (11.574%) 2024-06-27T21:46:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:00 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 102/231918 objects degraded (0.044%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:00.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:00.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:00 smithi190 ceph-mon[32247]: pgmap v2113: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 6 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 102/231918 objects degraded (0.044%); 26843/231918 objects misplaced (11.574%) 2024-06-27T21:46:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:00 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 102/231918 objects degraded (0.044%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:00.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.632 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:00.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:00.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:00.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:00.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:00.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:00.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:00.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:00.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:01.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:01.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:01.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:01.234 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:01.256 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:01.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:01.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:01 smithi151 ceph-mon[28965]: pgmap v2114: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 128/230277 objects degraded (0.056%); 24453/230277 objects misplaced (10.619%) 2024-06-27T21:46:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:01 smithi151 ceph-mon[28965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:46:01.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:01.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:01.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:01 smithi190 ceph-mon[32247]: pgmap v2114: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 128/230277 objects degraded (0.056%); 24453/230277 objects misplaced (10.619%) 2024-06-27T21:46:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:01 smithi190 ceph-mon[32247]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:46:01.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:01 smithi192 ceph-mon[32956]: pgmap v2114: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 128/230277 objects degraded (0.056%); 24453/230277 objects misplaced (10.619%) 2024-06-27T21:46:01.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:01 smithi192 ceph-mon[32956]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-06-27T21:46:01.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:01.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:01.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:01.754 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:01.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:01.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:01.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:01.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:02.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:02.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:02.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:02.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:02.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:02.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:02.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:02.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:02.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:02.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:02.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:02.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:02.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:02.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:02.882 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:02.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:02.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:03.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:03.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:03.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:03.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:03.261 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:03.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:03.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:03.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.514 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:03.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:03.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:03.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:03.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:03 smithi151 ceph-mon[28965]: pgmap v2115: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 128/230064 objects degraded (0.056%); 22559/230064 objects misplaced (9.806%) 2024-06-27T21:46:03.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:03.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:03.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:03 smithi190 ceph-mon[32247]: pgmap v2115: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 128/230064 objects degraded (0.056%); 22559/230064 objects misplaced (9.806%) 2024-06-27T21:46:03.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:03 smithi192 ceph-mon[32956]: pgmap v2115: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 128/230064 objects degraded (0.056%); 22559/230064 objects misplaced (9.806%) 2024-06-27T21:46:03.866 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:03.889 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:03.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:03.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:04.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:04.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:04.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:04.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:04.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:04.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:04.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:04.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:04.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:04.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:04.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:04.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:04.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:04.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:04.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:04.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:05.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.043 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:05.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:05.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:05.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:05.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:05.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:05.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:05.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:05.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:05.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:05.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:05.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:05.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:05 smithi151 ceph-mon[28965]: pgmap v2116: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 128/230019 objects degraded (0.056%); 20876/230019 objects misplaced (9.076%) 2024-06-27T21:46:05.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:05 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 128/230019 objects degraded (0.056%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:05.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:05 smithi190 ceph-mon[32247]: pgmap v2116: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 128/230019 objects degraded (0.056%); 20876/230019 objects misplaced (9.076%) 2024-06-27T21:46:05.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:05 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 128/230019 objects degraded (0.056%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:05.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:05.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:05 smithi192 ceph-mon[32956]: pgmap v2116: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 128/230019 objects degraded (0.056%); 20876/230019 objects misplaced (9.076%) 2024-06-27T21:46:05.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:05 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 128/230019 objects degraded (0.056%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:05.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:05.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:05.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:05.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:06.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:06.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:06.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:06.180 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:06.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.307 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:06.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:06.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:06.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:06.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:06.559 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:06.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:06.683 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:06.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:06.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:06 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:06.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:06 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:06.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:06 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:06.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:06 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:06.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:06.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:06 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:06.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:06 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:06.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:06.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:06.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:06.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:07.035 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.058 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:07.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:07.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:07.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:07.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:07.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:07.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:07.444 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:07.546 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:07.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:07.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:07.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:07.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:07 smithi151 ceph-mon[28965]: pgmap v2117: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 128/228954 objects degraded (0.056%); 17899/228954 objects misplaced (7.818%) 2024-06-27T21:46:07.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:07 smithi151 ceph-mon[28965]: osdmap e858: 12 total, 12 up, 11 in 2024-06-27T21:46:07.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:07 smithi190 ceph-mon[32247]: pgmap v2117: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 128/228954 objects degraded (0.056%); 17899/228954 objects misplaced (7.818%) 2024-06-27T21:46:07.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:07 smithi190 ceph-mon[32247]: osdmap e858: 12 total, 12 up, 11 in 2024-06-27T21:46:07.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:07 smithi192 ceph-mon[32956]: pgmap v2117: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 128/228954 objects degraded (0.056%); 17899/228954 objects misplaced (7.818%) 2024-06-27T21:46:07.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:07 smithi192 ceph-mon[32956]: osdmap e858: 12 total, 12 up, 11 in 2024-06-27T21:46:07.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:07.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:07.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:07.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:07.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:08.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:08.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:08.188 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:08.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:08.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:08.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:08.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:08.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:08.573 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:08.595 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:08.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:08.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:08 smithi151 ceph-mon[28965]: osdmap e859: 12 total, 12 up, 11 in 2024-06-27T21:46:08.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:08.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:08 smithi190 ceph-mon[32247]: osdmap e859: 12 total, 12 up, 11 in 2024-06-27T21:46:08.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:08 smithi192 ceph-mon[32956]: osdmap e859: 12 total, 12 up, 11 in 2024-06-27T21:46:08.823 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:08.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:08.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:08.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:08.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:09.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:09.101 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:09.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:09.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:09.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:09.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:09.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:09.487 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:09.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:46:09.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:09.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:09.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:09 smithi151 ceph-mon[28965]: 8.8 scrub starts 2024-06-27T21:46:09.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:09 smithi151 ceph-mon[28965]: pgmap v2120: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 133 op/s; 128/227082 objects degraded (0.056%); 17899/227082 objects misplaced (7.882%); 33 MiB/s, 533 objects/s recovering 2024-06-27T21:46:09.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:09 smithi151 ceph-mon[28965]: 8.8 scrub ok 2024-06-27T21:46:09.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:09.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:09 smithi190 ceph-mon[32247]: 8.8 scrub starts 2024-06-27T21:46:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:09 smithi190 ceph-mon[32247]: pgmap v2120: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 133 op/s; 128/227082 objects degraded (0.056%); 17899/227082 objects misplaced (7.882%); 33 MiB/s, 533 objects/s recovering 2024-06-27T21:46:09.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:09 smithi190 ceph-mon[32247]: 8.8 scrub ok 2024-06-27T21:46:09.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:09 smithi192 ceph-mon[32956]: 8.8 scrub starts 2024-06-27T21:46:09.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:09 smithi192 ceph-mon[32956]: pgmap v2120: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 133 op/s; 128/227082 objects degraded (0.056%); 17899/227082 objects misplaced (7.882%); 33 MiB/s, 533 objects/s recovering 2024-06-27T21:46:09.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:09 smithi192 ceph-mon[32956]: 8.8 scrub ok 2024-06-27T21:46:09.838 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:09.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:09.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:09.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:09.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:10.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:10.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:10.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:10.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:10.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:10.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:10.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:10.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:10.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:10.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:10.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:10 smithi151 ceph-mon[28965]: osdmap e860: 12 total, 12 up, 11 in 2024-06-27T21:46:10.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:10 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 128/227082 objects degraded (0.056%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:10.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:10 smithi151 ceph-mon[28965]: osdmap e861: 12 total, 12 up, 11 in 2024-06-27T21:46:10.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:10.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:10 smithi190 ceph-mon[32247]: osdmap e860: 12 total, 12 up, 11 in 2024-06-27T21:46:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:10 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 128/227082 objects degraded (0.056%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:10.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:10 smithi190 ceph-mon[32247]: osdmap e861: 12 total, 12 up, 11 in 2024-06-27T21:46:10.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:10 smithi192 ceph-mon[32956]: osdmap e860: 12 total, 12 up, 11 in 2024-06-27T21:46:10.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:10 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 128/227082 objects degraded (0.056%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:10.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:10 smithi192 ceph-mon[32956]: osdmap e861: 12 total, 12 up, 11 in 2024-06-27T21:46:10.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:10.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:10.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:10.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:10.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:11.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:11.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:11.172 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:11.184 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:11.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:11.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:11.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:11.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:11.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:11.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:11.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:11.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:11.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:11 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:46:11.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:11 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:46:11.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:11 smithi151 ceph-mon[28965]: pgmap v2123: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 107/223266 objects degraded (0.048%); 15610/223266 objects misplaced (6.992%); 27 MiB/s, 444 objects/s recovering 2024-06-27T21:46:11.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:11 smithi151 ceph-mon[28965]: 8.13 scrub starts 2024-06-27T21:46:11.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:11 smithi151 ceph-mon[28965]: osdmap e862: 12 total, 12 up, 11 in 2024-06-27T21:46:11.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:11.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:11 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:46:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:11 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:46:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:11 smithi190 ceph-mon[32247]: pgmap v2123: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 107/223266 objects degraded (0.048%); 15610/223266 objects misplaced (6.992%); 27 MiB/s, 444 objects/s recovering 2024-06-27T21:46:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:11 smithi190 ceph-mon[32247]: 8.13 scrub starts 2024-06-27T21:46:11.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:11 smithi190 ceph-mon[32247]: osdmap e862: 12 total, 12 up, 11 in 2024-06-27T21:46:11.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:11 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:46:11.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:11 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:46:11.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:11 smithi192 ceph-mon[32956]: pgmap v2123: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 48 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 107/223266 objects degraded (0.048%); 15610/223266 objects misplaced (6.992%); 27 MiB/s, 444 objects/s recovering 2024-06-27T21:46:11.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:11 smithi192 ceph-mon[32956]: 8.13 scrub starts 2024-06-27T21:46:11.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:11 smithi192 ceph-mon[32956]: osdmap e862: 12 total, 12 up, 11 in 2024-06-27T21:46:11.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:11.876 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:11.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:11.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:11.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:12.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:12.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:12.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:12.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:12.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:12.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:12.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:12.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:12.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:12.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:12.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: 8.13 scrub ok 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:12.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:12 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:12.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: 8.13 scrub ok 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:12.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:12 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:12.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: 8.13 scrub ok 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:12.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:12 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:12.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:12.908 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:12.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:13.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:13.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:13.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:13.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:13.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:13.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:13.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:13.413 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:13.516 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:13.538 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:13.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:13.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:13.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:13.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:13.798 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:13.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:13.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:13.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:14.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:14.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:14.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:14 smithi190 ceph-mon[32247]: pgmap v2125: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 49 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 107/221829 objects degraded (0.048%); 13708/221829 objects misplaced (6.180%); 7.0 MiB/s, 116 objects/s recovering 2024-06-27T21:46:14.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:14 smithi190 ceph-mon[32247]: osdmap e863: 12 total, 12 up, 11 in 2024-06-27T21:46:14.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:14 smithi192 ceph-mon[32956]: pgmap v2125: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 49 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 107/221829 objects degraded (0.048%); 13708/221829 objects misplaced (6.180%); 7.0 MiB/s, 116 objects/s recovering 2024-06-27T21:46:14.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:14 smithi192 ceph-mon[32956]: osdmap e863: 12 total, 12 up, 11 in 2024-06-27T21:46:14.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:14.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:14.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:14.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:14 smithi151 ceph-mon[28965]: pgmap v2125: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 49 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 107/221829 objects degraded (0.048%); 13708/221829 objects misplaced (6.180%); 7.0 MiB/s, 116 objects/s recovering 2024-06-27T21:46:14.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:14 smithi151 ceph-mon[28965]: osdmap e863: 12 total, 12 up, 11 in 2024-06-27T21:46:14.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:14.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:14.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:14.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:14.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:14.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:14.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:14.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:14.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:15.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:15.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:15.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:15.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:15.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:15.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:15.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:15.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:15.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:15.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:15.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:15 smithi151 ceph-mon[28965]: osdmap e864: 12 total, 12 up, 11 in 2024-06-27T21:46:15.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:15 smithi151 ceph-mon[28965]: pgmap v2128: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 51 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 57/217470 objects degraded (0.026%); 12328/217470 objects misplaced (5.669%) 2024-06-27T21:46:15.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:15 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 107/221829 objects degraded (0.048%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:15.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:15 smithi151 ceph-mon[28965]: osdmap e865: 12 total, 12 up, 11 in 2024-06-27T21:46:15.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:15 smithi190 ceph-mon[32247]: osdmap e864: 12 total, 12 up, 11 in 2024-06-27T21:46:15.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:15 smithi190 ceph-mon[32247]: pgmap v2128: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 51 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 57/217470 objects degraded (0.026%); 12328/217470 objects misplaced (5.669%) 2024-06-27T21:46:15.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:15 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 107/221829 objects degraded (0.048%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:15.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:15 smithi190 ceph-mon[32247]: osdmap e865: 12 total, 12 up, 11 in 2024-06-27T21:46:15.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:15 smithi192 ceph-mon[32956]: osdmap e864: 12 total, 12 up, 11 in 2024-06-27T21:46:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:15 smithi192 ceph-mon[32956]: pgmap v2128: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 4 active+recovering+undersized+remapped, 51 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 57/217470 objects degraded (0.026%); 12328/217470 objects misplaced (5.669%) 2024-06-27T21:46:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:15 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 107/221829 objects degraded (0.048%), 2 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:15 smithi192 ceph-mon[32956]: osdmap e865: 12 total, 12 up, 11 in 2024-06-27T21:46:15.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:15.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:15.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:15.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:15.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:15.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:15.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:15.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:16.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.071 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:16.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:16.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:16.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:16.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:16.330 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:16.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:16.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:16 smithi151 ceph-mon[28965]: 8.12 scrub starts 2024-06-27T21:46:16.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:16 smithi151 ceph-mon[28965]: 8.12 scrub ok 2024-06-27T21:46:16.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:16.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:16 smithi190 ceph-mon[32247]: 8.12 scrub starts 2024-06-27T21:46:16.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:16 smithi190 ceph-mon[32247]: 8.12 scrub ok 2024-06-27T21:46:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:16 smithi192 ceph-mon[32956]: 8.12 scrub starts 2024-06-27T21:46:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:16 smithi192 ceph-mon[32956]: 8.12 scrub ok 2024-06-27T21:46:16.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:16.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:16.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:16.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:16.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:16.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:16.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:16.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:16.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:17.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:17.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:17.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:17 smithi151 ceph-mon[28965]: pgmap v2130: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 4.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 6 op/s; 57/214722 objects degraded (0.027%); 11358/214722 objects misplaced (5.290%) 2024-06-27T21:46:17.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:17.226 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:17.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:17.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:17.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:17.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:17.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:17 smithi190 ceph-mon[32247]: pgmap v2130: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 4.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 6 op/s; 57/214722 objects degraded (0.027%); 11358/214722 objects misplaced (5.290%) 2024-06-27T21:46:17.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:17 smithi192 ceph-mon[32956]: pgmap v2130: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 4.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 6 op/s; 57/214722 objects degraded (0.027%); 11358/214722 objects misplaced (5.290%) 2024-06-27T21:46:17.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.609 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:17.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:17.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:17.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:17.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:17.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:17.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:17.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:17.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:18.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:18.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:18.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:18.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:18.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:18.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:18.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:18.483 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:18.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:18.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:18.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:18.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:18.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:18.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:18.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:18.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:18.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:19.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:19.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:19.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:19.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:19.323 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:19.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:19.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:19.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:19.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:19.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:19.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:19.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:19 smithi151 ceph-mon[28965]: pgmap v2131: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 4.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 152 op/s; 57/211215 objects degraded (0.027%); 11358/211215 objects misplaced (5.377%) 2024-06-27T21:46:19.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:19.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:19 smithi190 ceph-mon[32247]: pgmap v2131: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 4.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 152 op/s; 57/211215 objects degraded (0.027%); 11358/211215 objects misplaced (5.377%) 2024-06-27T21:46:19.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:19.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:19 smithi192 ceph-mon[32956]: pgmap v2131: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 51 active+clean; 4.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 152 op/s; 57/211215 objects degraded (0.027%); 11358/211215 objects misplaced (5.377%) 2024-06-27T21:46:19.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:19.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:19.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:20.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.030 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:20.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:20.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:20.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:20.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:20.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:20.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:20.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:20.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:20.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:20.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:20.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:20.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:46:20.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:46:20.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:46:20.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:20 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:20.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:20 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:20.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:20 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 57/211215 objects degraded (0.027%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:20.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:20.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:46:20.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:46:20.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:46:20.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:20 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:20.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:20 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:20.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:20 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 57/211215 objects degraded (0.027%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:20.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:20.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:46:20.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:46:20.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:46:20.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:20 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:20.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:20 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:20.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:20 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 57/211215 objects degraded (0.027%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:20.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:20.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:20.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:21.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:21.042 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:21.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:21.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:21.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:21.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:21.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:21.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:21.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:21.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:21.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:21.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:21.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:21.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:21 smithi151 ceph-mon[28965]: pgmap v2132: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 57/206301 objects degraded (0.028%); 9892/206301 objects misplaced (4.795%) 2024-06-27T21:46:21.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:21 smithi190 ceph-mon[32247]: pgmap v2132: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 57/206301 objects degraded (0.028%); 9892/206301 objects misplaced (4.795%) 2024-06-27T21:46:21.785 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:21.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:21 smithi192 ceph-mon[32956]: pgmap v2132: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 57/206301 objects degraded (0.028%); 9892/206301 objects misplaced (4.795%) 2024-06-27T21:46:21.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:21.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:21.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:21.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:22.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:22.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:22.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:22.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:22.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:22.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:22.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:22.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:22.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:22.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:22.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:22.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:22.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:22.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:22.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:22.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:22.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:22.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:23.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:23.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:23.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:23.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:23.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:23.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:23.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:23.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:23.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:23.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:23.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:23.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: pgmap v2133: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 225 op/s; 57/204495 objects degraded (0.028%); 9292/204495 objects misplaced (4.544%) 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:23.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:23 smithi190 ceph-mon[32247]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: pgmap v2133: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 225 op/s; 57/204495 objects degraded (0.028%); 9292/204495 objects misplaced (4.544%) 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:23.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:23.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:23 smithi192 ceph-mon[32956]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:23.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:23.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:23.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: pgmap v2133: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 225 op/s; 57/204495 objects degraded (0.028%); 9292/204495 objects misplaced (4.544%) 2024-06-27T21:46:23.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:23.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:23.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:23.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:23.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:23.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:23.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:23 smithi151 ceph-mon[28965]: from='mgr.14150 ' entity='mgr.y' 2024-06-27T21:46:23.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:23.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:23.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:24.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:24.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:24.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:24.228 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:24.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:24.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:24.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:24.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:24.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:24.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:24.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:24.733 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:24.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:24.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:24.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:24.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:24.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:25.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:25.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:25.247 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:25.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:25.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:25.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:25.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:25.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:25.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:25.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:25.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:25 smithi190 ceph-mon[32247]: pgmap v2134: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 447 op/s; 57/195708 objects degraded (0.029%); 8054/195708 objects misplaced (4.115%) 2024-06-27T21:46:25.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:25 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 57/195708 objects degraded (0.029%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:25.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:25.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:25 smithi192 ceph-mon[32956]: pgmap v2134: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 447 op/s; 57/195708 objects degraded (0.029%); 8054/195708 objects misplaced (4.115%) 2024-06-27T21:46:25.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:25 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 57/195708 objects degraded (0.029%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:25.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:25.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:25.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:25.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:25.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:25 smithi151 ceph-mon[28965]: pgmap v2134: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 52 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 447 op/s; 57/195708 objects degraded (0.029%); 8054/195708 objects misplaced (4.115%) 2024-06-27T21:46:25.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:25 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 57/195708 objects degraded (0.029%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:26.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:26.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:26.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:26.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:26.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:26.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:26.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:26.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:26.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:26.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:26.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:26.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:26.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:26.799 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:26.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:26.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:26.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:27.031 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:27.056 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:27.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:27.183 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:27.285 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:27.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:27.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:27.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:27.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:27.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.568 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:27.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:27.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:27.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:27 smithi190 ceph-mon[32247]: pgmap v2135: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 479 op/s; 57/191313 objects degraded (0.030%); 6635/191313 objects misplaced (3.468%); 2.7 MiB/s, 43 objects/s recovering 2024-06-27T21:46:27.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:27.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:27 smithi192 ceph-mon[32956]: pgmap v2135: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 479 op/s; 57/191313 objects degraded (0.030%); 6635/191313 objects misplaced (3.468%); 2.7 MiB/s, 43 objects/s recovering 2024-06-27T21:46:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:27.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:27 smithi151 ceph-mon[28965]: pgmap v2135: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 3.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 479 op/s; 57/191313 objects degraded (0.030%); 6635/191313 objects misplaced (3.468%); 2.7 MiB/s, 43 objects/s recovering 2024-06-27T21:46:27.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:27.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:27.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:28.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:28.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:28.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:28.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:28.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:28.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:28.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:28.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:28.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:28.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:28.697 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:28.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:28.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:28.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:28 smithi190 ceph-mon[32247]: osdmap e866: 12 total, 12 up, 11 in 2024-06-27T21:46:28.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:28.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:28.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:28 smithi192 ceph-mon[32956]: osdmap e866: 12 total, 12 up, 11 in 2024-06-27T21:46:28.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:28.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:28.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:28.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:28.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:28.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:28 smithi151 ceph-mon[28965]: osdmap e866: 12 total, 12 up, 11 in 2024-06-27T21:46:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:28.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:29.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:29.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:29.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:29.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:29.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:29.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:29.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:29.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:29.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:29.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:29.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:29.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:29 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:46:29.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:29 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:46:29.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:29 smithi190 ceph-mon[32247]: pgmap v2137: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 57/186237 objects degraded (0.031%); 6635/186237 objects misplaced (3.563%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:46:29.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:29 smithi190 ceph-mon[32247]: osdmap e867: 12 total, 12 up, 11 in 2024-06-27T21:46:29.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:29 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:46:29.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:29 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:46:29.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:29 smithi192 ceph-mon[32956]: pgmap v2137: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 57/186237 objects degraded (0.031%); 6635/186237 objects misplaced (3.563%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:46:29.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:29 smithi192 ceph-mon[32956]: osdmap e867: 12 total, 12 up, 11 in 2024-06-27T21:46:29.825 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:29.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:29 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:46:29.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:29 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:46:29.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:29 smithi151 ceph-mon[28965]: pgmap v2137: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 52 active+clean; 3.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 57/186237 objects degraded (0.031%); 6635/186237 objects misplaced (3.563%); 1.2 MiB/s, 19 objects/s recovering 2024-06-27T21:46:29.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:29 smithi151 ceph-mon[28965]: osdmap e867: 12 total, 12 up, 11 in 2024-06-27T21:46:29.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:29.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:29.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:30.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.113 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:30.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:30.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:30.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:30.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:30.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:30.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:30.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:30.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:30.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:30 smithi151 ceph-mon[28965]: osdmap e868: 12 total, 12 up, 11 in 2024-06-27T21:46:30.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:30 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 57/186237 objects degraded (0.031%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:30.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:30 smithi151 ceph-mon[28965]: osdmap e869: 12 total, 12 up, 11 in 2024-06-27T21:46:30.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:30.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:30 smithi190 ceph-mon[32247]: osdmap e868: 12 total, 12 up, 11 in 2024-06-27T21:46:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:30 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 57/186237 objects degraded (0.031%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:30 smithi190 ceph-mon[32247]: osdmap e869: 12 total, 12 up, 11 in 2024-06-27T21:46:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:30.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:30 smithi192 ceph-mon[32956]: osdmap e868: 12 total, 12 up, 11 in 2024-06-27T21:46:30.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:30 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 57/186237 objects degraded (0.031%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-06-27T21:46:30.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:30 smithi192 ceph-mon[32956]: osdmap e869: 12 total, 12 up, 11 in 2024-06-27T21:46:30.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:30.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:30.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:30.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:31.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:31.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:31.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:31.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:31.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:31.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:31.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:31.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:31.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:31.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:31.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:31.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:31 smithi151 ceph-mon[28965]: 8.17 scrub starts 2024-06-27T21:46:31.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:31 smithi151 ceph-mon[28965]: pgmap v2141: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 52 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 57/179013 objects degraded (0.032%); 4678/179013 objects misplaced (2.613%); 18 MiB/s, 288 objects/s recovering 2024-06-27T21:46:31.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:31 smithi151 ceph-mon[28965]: 8.17 scrub ok 2024-06-27T21:46:31.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:31 smithi151 ceph-mon[28965]: 8.19 scrub starts 2024-06-27T21:46:31.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:31.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:31 smithi190 ceph-mon[32247]: 8.17 scrub starts 2024-06-27T21:46:31.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:31 smithi190 ceph-mon[32247]: pgmap v2141: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 52 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 57/179013 objects degraded (0.032%); 4678/179013 objects misplaced (2.613%); 18 MiB/s, 288 objects/s recovering 2024-06-27T21:46:31.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:31 smithi190 ceph-mon[32247]: 8.17 scrub ok 2024-06-27T21:46:31.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:31 smithi190 ceph-mon[32247]: 8.19 scrub starts 2024-06-27T21:46:31.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:31.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:31 smithi192 ceph-mon[32956]: 8.17 scrub starts 2024-06-27T21:46:31.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:31 smithi192 ceph-mon[32956]: pgmap v2141: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 52 active+clean; 3.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 57/179013 objects degraded (0.032%); 4678/179013 objects misplaced (2.613%); 18 MiB/s, 288 objects/s recovering 2024-06-27T21:46:31.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:31 smithi192 ceph-mon[32956]: 8.17 scrub ok 2024-06-27T21:46:31.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:31 smithi192 ceph-mon[32956]: 8.19 scrub starts 2024-06-27T21:46:31.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:31.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:31.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:32.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:32.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:32.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:32.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:32.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:32.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:32.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:32.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:32.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:32.528 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:32.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:32.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:32.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:32 smithi190 ceph-mon[32247]: 8.19 scrub ok 2024-06-27T21:46:32.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:32 smithi190 ceph-mon[32247]: 8.6 scrub starts 2024-06-27T21:46:32.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:32 smithi190 ceph-mon[32247]: osdmap e870: 12 total, 12 up, 11 in 2024-06-27T21:46:32.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:32.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:32 smithi151 ceph-mon[28965]: 8.19 scrub ok 2024-06-27T21:46:32.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:32 smithi151 ceph-mon[28965]: 8.6 scrub starts 2024-06-27T21:46:32.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:32 smithi151 ceph-mon[28965]: osdmap e870: 12 total, 12 up, 11 in 2024-06-27T21:46:32.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:32.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:32 smithi192 ceph-mon[32956]: 8.19 scrub ok 2024-06-27T21:46:32.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:32 smithi192 ceph-mon[32956]: 8.6 scrub starts 2024-06-27T21:46:32.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:32 smithi192 ceph-mon[32956]: osdmap e870: 12 total, 12 up, 11 in 2024-06-27T21:46:32.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:32.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:32.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:33.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:33.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:32 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:46:32.936+0000 7f15d6b1f640 -1 mon.a@0(leader).osd e871 definitely_dead 0 2024-06-27T21:46:33.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:33.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:33.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:33.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:33.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:33.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:33.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:33.514 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.11. 2024-06-27T21:46:33.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:33.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:33.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:33.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:46:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:46:33.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: pgmap v2143: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 52 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 57/177270 objects degraded (0.032%); 4678/177270 objects misplaced (2.639%); 18 MiB/s, 288 objects/s recovering 2024-06-27T21:46:33.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: osdmap e871: 12 total, 12 up, 11 in 2024-06-27T21:46:33.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:46:33.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:46:33.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4169192102' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-06-27T21:46:33.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: pgmap v2143: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 52 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 57/177270 objects degraded (0.032%); 4678/177270 objects misplaced (2.639%); 18 MiB/s, 288 objects/s recovering 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: osdmap e871: 12 total, 12 up, 11 in 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:46:33.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4169192102' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-06-27T21:46:33.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:33.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:46:33.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:46:33.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: pgmap v2143: 57 pgs: 1 active+remapped+backfilling, 1 creating+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 52 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 57/177270 objects degraded (0.032%); 4678/177270 objects misplaced (2.639%); 18 MiB/s, 288 objects/s recovering 2024-06-27T21:46:33.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: osdmap e871: 12 total, 12 up, 11 in 2024-06-27T21:46:33.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:46:33.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:46:33.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4169192102' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["11"]}]: dispatch 2024-06-27T21:46:33.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:33.933 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:33.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:34.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:34.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:34.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:34.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:34.284 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:34.310 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:34.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:34.437 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:34.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:34.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:34.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:34.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:34.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:34 smithi151 ceph-mon[28965]: 8.6 scrub starts 2024-06-27T21:46:34.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4169192102' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-06-27T21:46:34.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:34 smithi151 ceph-mon[28965]: osdmap e872: 12 total, 12 up, 12 in 2024-06-27T21:46:34.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:34.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:34 smithi192 ceph-mon[32956]: 8.6 scrub starts 2024-06-27T21:46:34.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4169192102' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-06-27T21:46:34.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:34 smithi192 ceph-mon[32956]: osdmap e872: 12 total, 12 up, 12 in 2024-06-27T21:46:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:34.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:34.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:34.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:34.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:35.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:34 smithi190 ceph-mon[32247]: 8.6 scrub starts 2024-06-27T21:46:35.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4169192102' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-06-27T21:46:35.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:34 smithi190 ceph-mon[32247]: osdmap e872: 12 total, 12 up, 12 in 2024-06-27T21:46:35.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:35.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:35.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:35.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:35.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:35.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:35.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:35.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:35.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:35.520 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:46:35.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:35.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:35.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:35.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: pgmap v2146: 57 pgs: 2 remapped+peering, 1 peering, 1 active+remapped+backfilling, 53 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 86/169563 objects misplaced (0.051%) 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/177270 objects degraded (0.032%), 1 pg degraded, 3 pgs undersized) 2024-06-27T21:46:35.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: osdmap e873: 12 total, 12 up, 12 in 2024-06-27T21:46:35.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1284764843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:46:35.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:46:35.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:46:35.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1284764843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:46:35.798 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:35 smithi192 ceph-mon[32956]: osdmap e874: 12 total, 12 up, 12 in 2024-06-27T21:46:35.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:35.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:35.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:46:35.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:46:35.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: pgmap v2146: 57 pgs: 2 remapped+peering, 1 peering, 1 active+remapped+backfilling, 53 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 86/169563 objects misplaced (0.051%) 2024-06-27T21:46:35.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/177270 objects degraded (0.032%), 1 pg degraded, 3 pgs undersized) 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: osdmap e873: 12 total, 12 up, 12 in 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1284764843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1284764843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:46:35.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:35 smithi151 ceph-mon[28965]: osdmap e874: 12 total, 12 up, 12 in 2024-06-27T21:46:35.916 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:46:35.916 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: pgmap v2146: 57 pgs: 2 remapped+peering, 1 peering, 1 active+remapped+backfilling, 53 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 86/169563 objects misplaced (0.051%) 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/177270 objects degraded (0.032%), 1 pg degraded, 3 pgs undersized) 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: osdmap e873: 12 total, 12 up, 12 in 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1284764843' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1284764843' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:46:35.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:35 smithi190 ceph-mon[32247]: osdmap e874: 12 total, 12 up, 12 in 2024-06-27T21:46:35.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:35.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:35.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:36.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:36.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:36.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:36.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:36.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:36.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:36.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:36.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:36.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:36.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:36.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:36.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:36.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:36.901 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:36.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:37.003 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:37.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:37.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:37.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:37.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:37.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:37.329 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:46:37.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:46:37.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:46:37.356 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:46:37.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:37.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:46:37.384 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:46:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:37.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:37.410 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:46:37.410 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:46:37.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:46:37.434 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:46:37.456 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:46:37.456 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:46:37.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:46:37.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:46:37.504 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:46:37.505 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:46:37.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:37.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:46:37.536 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:46:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:37.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:37.561 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:46:37.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:46:37.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:46:37.589 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:46:37.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:37 smithi151 ceph-mon[28965]: pgmap v2149: 57 pgs: 3 remapped+peering, 1 peering, 1 active+remapped+backfilling, 52 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86/167115 objects misplaced (0.051%) 2024-06-27T21:46:37.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:37 smithi151 ceph-mon[28965]: 8.e scrub starts 2024-06-27T21:46:37.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:37 smithi151 ceph-mon[28965]: 1.1b deep-scrub starts 2024-06-27T21:46:37.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:37 smithi151 ceph-mon[28965]: 1.1b deep-scrub ok 2024-06-27T21:46:37.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:37 smithi151 ceph-mon[28965]: 8.e scrub ok 2024-06-27T21:46:37.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2540711904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:46:37.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:46:37.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:46:37.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:37.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:37.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:37.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:37.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:37 smithi192 ceph-mon[32956]: pgmap v2149: 57 pgs: 3 remapped+peering, 1 peering, 1 active+remapped+backfilling, 52 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86/167115 objects misplaced (0.051%) 2024-06-27T21:46:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:37 smithi192 ceph-mon[32956]: 8.e scrub starts 2024-06-27T21:46:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:37 smithi192 ceph-mon[32956]: 1.1b deep-scrub starts 2024-06-27T21:46:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:37 smithi192 ceph-mon[32956]: 1.1b deep-scrub ok 2024-06-27T21:46:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:37 smithi192 ceph-mon[32956]: 8.e scrub ok 2024-06-27T21:46:37.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2540711904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:46:37.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:37.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:37.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:38.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:37 smithi190 ceph-mon[32247]: pgmap v2149: 57 pgs: 3 remapped+peering, 1 peering, 1 active+remapped+backfilling, 52 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86/167115 objects misplaced (0.051%) 2024-06-27T21:46:38.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:37 smithi190 ceph-mon[32247]: 8.e scrub starts 2024-06-27T21:46:38.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:37 smithi190 ceph-mon[32247]: 1.1b deep-scrub starts 2024-06-27T21:46:38.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:37 smithi190 ceph-mon[32247]: 1.1b deep-scrub ok 2024-06-27T21:46:38.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:37 smithi190 ceph-mon[32247]: 8.e scrub ok 2024-06-27T21:46:38.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2540711904' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:46:38.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:38.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:38.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:38.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:38.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:38.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:38.308 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:46:38.308 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-06-27T21:46:38.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-06-27T21:46:38.308 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:46:38.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.308 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(6.14 657/550 [PushReplyOp(6:2b5356dc:::benchmark_data_smithi192_117994_object107310:head)])", 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 538.08790636699996, 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70161021400000001, 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103399+0000", 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.03e-07 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.103401+0000", 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5680000000000001e-06 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.804560+0000", 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70115915399999995 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.804568+0000", 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0080000000000002e-06 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.804629+0000", 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1503999999999999e-05 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:37:39.805009+0000", 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037987699999999998 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.312 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(8.b 821/816 [PushOp(8:d6ab895e:::benchmark_data_smithi192_167666_object19687:head, version: 812'1187, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:d6ab895e:::benchmark_data_smithi192_167666_object19687:head@812'1187, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:44:50.213108+0000", 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 106.978197166, 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.480963645, 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:46:38.313 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.213108+0000", 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.213108+0000", 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.213109+0000", 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9900000000000001e-07 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.213158+0000", 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9721e-05 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.692800+0000", 2024-06-27T21:46:38.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47964193700000002 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.692832+0000", 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1952000000000002e-05 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.693029+0000", 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019670399999999999 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.693039+0000", 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0518e-05 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:50.694072+0000", 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010324139999999999 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(8.1 823/816 [PushOp(8:8b81ff1a:::benchmark_data_smithi192_167666_object2118:head, version: 805'129, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:8b81ff1a:::benchmark_data_smithi192_167666_object2118:head@805'129, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:44:52.628994+0000", 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 104.56231169599999, 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39029348899999999, 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:52.628994+0000", 2024-06-27T21:46:38.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:52.628994+0000", 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:52.628994+0000", 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3799999999999997e-07 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:52.629017+0000", 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2514000000000002e-05 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:53.018210+0000", 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.389193709 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:53.018225+0000", 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4346e-05 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:53.018302+0000", 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7628999999999998e-05 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:53.018312+0000", 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.166e-06 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:44:53.019287+0000", 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097558699999999998 2024-06-27T21:46:38.321 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27031.0:59345 8.f e833/827 8:fe6be046:::benchmark_data_smithi192_167666_object59344:head v 833'3779, mlcod=832'3776)", 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:45:13.140889+0000", 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.050416686000005, 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79631746800000003, 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.322 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.140889+0000", 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.140889+0000", 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.140890+0000", 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8400000000000003e-07 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.140970+0000", 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0555000000000004e-05 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.933321+0000", 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.792350518 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.933331+0000", 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0292000000000001e-05 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.933384+0000", 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3381999999999998e-05 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.933407+0000", 2024-06-27T21:46:38.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2459999999999998e-05 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.937151+0000", 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037439999999999999 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:13.937206+0000", 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5377000000000002e-05 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.326 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.13 855/839 [PushReplyOp(8:cc7cd534:::benchmark_data_smithi192_167666_object49199:head)])", 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:45:54.717130+0000", 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.474175799999998, 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45173322999999999, 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:54.717130+0000", 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:54.717130+0000", 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:54.717130+0000", 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7199999999999998e-07 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:54.717135+0000", 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9169999999999997e-06 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:55.166677+0000", 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44954179300000002 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:55.166689+0000", 2024-06-27T21:46:38.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1568e-05 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:55.166767+0000", 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8419999999999998e-05 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:55.168863+0000", 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020958600000000002 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.330 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:82216 8.13 8:c93b94e7:::benchmark_data_smithi192_167666_object2778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:45:57.666567+0000", 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.524738835000001, 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37739517099999997, 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.331 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 82216 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666567+0000", 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666567+0000", 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666568+0000", 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7630000000000001e-06 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666576+0000", 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0460000000000006e-06 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666592+0000", 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5790999999999998e-05 2024-06-27T21:46:38.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666613+0000", 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1047e-05 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666700+0000", 2024-06-27T21:46:38.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6897999999999994e-05 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.666878+0000", 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017779199999999999 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8,9", 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.667043+0000", 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000165124 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.668468+0000", 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00142454 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.669064+0000", 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00059666600000000004 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.669113+0000", 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8513999999999997e-05 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.042998+0000", 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37388553600000002 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.043078+0000", 2024-06-27T21:46:38.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9610000000000005e-05 2024-06-27T21:46:38.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.043962+0000", 2024-06-27T21:46:38.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088384399999999995 2024-06-27T21:46:38.338 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.13 856/839 [PushReplyOp(8:ce57223d:::benchmark_data_smithi192_167666_object53398:head)])", 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:45:57.669263+0000", 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.522041940999998, 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36933135700000003, 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.339 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.669263+0000", 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.669263+0000", 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.669264+0000", 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3200000000000004e-07 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:57.669272+0000", 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3939999999999998e-06 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.036535+0000", 2024-06-27T21:46:38.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36726327399999997 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.036544+0000", 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7660000000000004e-06 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.036624+0000", 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9882000000000005e-05 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:58.038595+0000", 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019712089999999998 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.17 857/836 [PushReplyOp(8:eac94f6a:::benchmark_data_smithi192_167666_object21214:head)])", 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:45:59.050060+0000", 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 38.141244909000001, 2024-06-27T21:46:38.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83391207000000001, 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.050060+0000", 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.050060+0000", 2024-06-27T21:46:38.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.050062+0000", 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0890000000000001e-06 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.050067+0000", 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1610000000000002e-06 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.882558+0000", 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83249171499999997 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.882568+0000", 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3780000000000007e-06 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.882638+0000", 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0486000000000002e-05 2024-06-27T21:46:38.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.883973+0000", 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013342409999999999 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.13 857/839 [PushReplyOp(8:ccf3ae6b:::benchmark_data_smithi192_167666_object56812:head)])", 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:45:59.054435+0000", 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 38.136870311000003, 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40317004099999998, 2024-06-27T21:46:38.347 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.054435+0000", 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.054435+0000", 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.054436+0000", 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.392e-06 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.054442+0000", 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8880000000000002e-06 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.456195+0000", 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40175256799999998 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.456205+0000", 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0376e-05 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.456294+0000", 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9110000000000006e-05 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:45:59.457605+0000", 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013107069999999999 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.13 857/839 [PushReplyOp(8:cbf3273f:::benchmark_data_smithi192_167666_object57947:head)])", 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:00.567931+0000", 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.623374814999998, 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.76201257600000005, 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.351 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.567931+0000", 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.567931+0000", 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.567931+0000", 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5400000000000001e-07 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.567936+0000", 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7450000000000003e-06 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.328885+0000", 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.76094924200000003 2024-06-27T21:46:38.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.328894+0000", 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9749999999999996e-06 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.328988+0000", 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4111999999999996e-05 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.329943+0000", 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000954848 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.17 857/836 [PushReplyOp(8:e8de900f:::benchmark_data_smithi192_167666_object23194:head)])", 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:00.748778+0000", 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.442526962000002, 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58578131099999997, 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.355 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.748778+0000", 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.748778+0000", 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.748779+0000", 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2099999999999995e-07 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.748788+0000", 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5690000000000005e-06 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.333514+0000", 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58472631900000005 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.333521+0000", 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2080000000000003e-06 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.333567+0000", 2024-06-27T21:46:38.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5731e-05 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.334560+0000", 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099256299999999991 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.17 857/836 [PushReplyOp(8:e8de900f:::benchmark_data_smithi192_167666_object23194:head)])", 2024-06-27T21:46:38.359 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:00.749016+0000", 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.442288855000001, 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55585089799999998, 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.749016+0000", 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.749016+0000", 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.749017+0000", 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3799999999999997e-07 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:00.749021+0000", 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.245e-06 2024-06-27T21:46:38.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.303703+0000", 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.55468149600000005 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.303730+0000", 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6868000000000001e-05 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.303796+0000", 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6223000000000006e-05 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:01.304867+0000", 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010714279999999999 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.13 857/839 [PushReplyOp(8:ca40e184:::benchmark_data_smithi192_167666_object5808:head)])", 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:03.398978+0000", 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.792327839000002, 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40133443699999999, 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.398978+0000", 2024-06-27T21:46:38.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.398978+0000", 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.398979+0000", 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.319e-06 2024-06-27T21:46:38.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.398988+0000", 2024-06-27T21:46:38.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6910000000000003e-06 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.799400+0000", 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40041255100000001 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.799411+0000", 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0438999999999999e-05 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.799480+0000", 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9178999999999996e-05 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:03.800312+0000", 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000832258 2024-06-27T21:46:38.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83318 8.13 8:ce5f8d77:::benchmark_data_smithi192_167666_object3880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:05.799321+0000", 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.391983944, 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35669821600000001, 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83318 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.799321+0000", 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.799321+0000", 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.799322+0000", 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9299999999999997e-07 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.799327+0000", 2024-06-27T21:46:38.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3420000000000001e-06 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.799335+0000", 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7280000000000001e-06 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.799342+0000", 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1810000000000004e-06 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.801596+0000", 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022537350000000002 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.801675+0000", 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8617999999999998e-05 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8,9", 2024-06-27T21:46:38.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.801769+0000", 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3933999999999999e-05 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.802759+0000", 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099043500000000001 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.803008+0000", 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000248743 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:05.805131+0000", 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021229230000000001 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.155493+0000", 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35036188899999998 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.155581+0000", 2024-06-27T21:46:38.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7873000000000004e-05 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.156020+0000", 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043902200000000001 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83612 8.17 8:e852b530:::benchmark_data_smithi192_167666_object4174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e858)", 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:06.430836+0000", 2024-06-27T21:46:38.376 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.760469181000001, 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57987192499999995, 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83612 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430836+0000", 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430836+0000", 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430837+0000", 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.15e-06 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430842+0000", 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2200000000000003e-06 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430852+0000", 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0346000000000001e-05 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430857+0000", 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2850000000000002e-06 2024-06-27T21:46:38.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.430926+0000", 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8584999999999995e-05 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.431077+0000", 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015136300000000001 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7,8", 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.431222+0000", 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014444999999999999 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.432757+0000", 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001535909 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.433129+0000", 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000371467 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.433562+0000", 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043261800000000001 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.009887+0000", 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57632573799999998 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.009972+0000", 2024-06-27T21:46:38.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5033999999999999e-05 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.010708+0000", 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073576000000000002 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83634 8.13 8:cb6506b3:::benchmark_data_smithi192_167666_object4196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e858)", 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:06.492543+0000", 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.698762836, 2024-06-27T21:46:38.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51847781699999995, 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83634 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492543+0000", 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492543+0000", 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492545+0000", 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2910000000000002e-06 2024-06-27T21:46:38.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492553+0000", 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9910000000000002e-06 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492575+0000", 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2554000000000001e-05 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492584+0000", 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.884e-06 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492657+0000", 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2455999999999999e-05 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492812+0000", 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000154902 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8,9", 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.492964+0000", 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015229499999999999 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.494424+0000", 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001460238 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.494579+0000", 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015491600000000001 2024-06-27T21:46:38.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.494950+0000", 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037113599999999999 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.010527+0000", 2024-06-27T21:46:38.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51557725700000001 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.010630+0000", 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103046 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011020+0000", 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038985099999999998 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.390 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83636 8.13 8:cf0009eb:::benchmark_data_smithi192_167666_object4198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e858)", 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:06.509350+0000", 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.681954859000001, 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50203868600000001, 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.391 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83636 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509350+0000", 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509350+0000", 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509353+0000", 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1639999999999999e-06 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509361+0000", 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.439e-06 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509382+0000", 2024-06-27T21:46:38.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1229e-05 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509393+0000", 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0542e-05 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509468+0000", 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4949000000000004e-05 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509627+0000", 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000159171 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8,9", 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.509772+0000", 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014499599999999999 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.511089+0000", 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001317066 2024-06-27T21:46:38.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.511142+0000", 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3217999999999998e-05 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.511790+0000", 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064766300000000004 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011069+0000", 2024-06-27T21:46:38.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.49927862499999998 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011139+0000", 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0487999999999993e-05 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011389+0000", 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025013599999999998 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.398 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83639 8.13 8:ce88965a:::benchmark_data_smithi192_167666_object4201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e858)", 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:06.513672+0000", 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.677633655000001, 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.498094021, 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83639 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513672+0000", 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513672+0000", 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513673+0000", 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.31e-06 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513678+0000", 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1730000000000003e-06 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513690+0000", 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1592e-05 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513696+0000", 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.083e-06 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513761+0000", 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4906999999999994e-05 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.513943+0000", 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000182626 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8,9", 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.514112+0000", 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016877799999999999 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.515533+0000", 2024-06-27T21:46:38.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014213190000000001 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.515614+0000", 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0199999999999998e-05 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.522604+0000", 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0069903680000000003 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011446+0000", 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.488841676 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011509+0000", 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3613000000000002e-05 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011766+0000", 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000256376 2024-06-27T21:46:38.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83643 8.17 8:ee3e4c80:::benchmark_data_smithi192_167666_object4205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e858)", 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:06.534539+0000", 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.656766005000001, 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47682817599999999, 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.406 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83643 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534539+0000", 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534539+0000", 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534545+0000", 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8690000000000001e-06 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534555+0000", 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4069999999999999e-06 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534577+0000", 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2591000000000001e-05 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534589+0000", 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1311999999999999e-05 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534669+0000", 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0970000000000006e-05 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.534828+0000", 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015814400000000001 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7,8", 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.535006+0000", 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000178059 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.536466+0000", 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00146075 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.536990+0000", 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052386999999999996 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.537102+0000", 2024-06-27T21:46:38.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111352 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.010767+0000", 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47366488699999998 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.010813+0000", 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6378000000000002e-05 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011368+0000", 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055458700000000005 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:83648 8.13 8:ce15b79e:::benchmark_data_smithi192_167666_object4210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e858)", 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:46:06.633755+0000", 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.557550267, 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37839896699999997, 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:46:38.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 83648 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633755+0000", 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:46:38.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633755+0000", 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633757+0000", 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.838e-06 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633768+0000", 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0709e-05 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633785+0000", 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7345000000000001e-05 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633794+0000", 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0149999999999992e-06 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633859+0000", 2024-06-27T21:46:38.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4607e-05 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.633991+0000", 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013217 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8,9", 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.634090+0000", 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8818000000000002e-05 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.635687+0000", 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015975900000000001 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.635709+0000", 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2065999999999998e-05 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:06.636169+0000", 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046023700000000001 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011813+0000", 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37564377500000001 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.011874+0000", 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0897000000000003e-05 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:46:38.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:46:07.012154+0000", 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027989999999999997 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:46:38.420 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:46:38.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:38.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:38.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:46:38.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:38.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:38.583 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:38 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:46:38.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:38.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:38.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:38 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:46:38.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:38 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:46:38.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:38.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.843 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:38.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:38.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:38 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:46:38.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:38.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:39.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:38 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:46:39.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:38 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:46:39.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:39.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:39.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:39.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:39.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:39.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:39.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:39.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:39.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:39.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:39.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:39.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:39 smithi192 ceph-mon[32956]: pgmap v2150: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 1 peering, 1 active+remapped+backfilling, 50 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2523/164361 objects degraded (1.535%); 1683/164361 objects misplaced (1.024%) 2024-06-27T21:46:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:39 smithi192 ceph-mon[32956]: Health check failed: Degraded data redundancy: 2523/164361 objects degraded (1.535%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:46:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:39 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:46:39.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:39 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:46:39.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:39.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:39.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:39 smithi151 ceph-mon[28965]: pgmap v2150: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 1 peering, 1 active+remapped+backfilling, 50 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2523/164361 objects degraded (1.535%); 1683/164361 objects misplaced (1.024%) 2024-06-27T21:46:39.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:39 smithi151 ceph-mon[28965]: Health check failed: Degraded data redundancy: 2523/164361 objects degraded (1.535%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:46:39.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:39 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:46:39.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:39 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:46:39.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:39.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:39.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:40.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:39 smithi190 ceph-mon[32247]: pgmap v2150: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 1 peering, 1 active+remapped+backfilling, 50 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2523/164361 objects degraded (1.535%); 1683/164361 objects misplaced (1.024%) 2024-06-27T21:46:40.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:39 smithi190 ceph-mon[32247]: Health check failed: Degraded data redundancy: 2523/164361 objects degraded (1.535%), 1 pg degraded (PG_DEGRADED) 2024-06-27T21:46:40.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:39 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:46:40.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:39 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:46:40.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:40.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:40.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:40.252 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:40.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:40.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:40.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:40.512 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:40.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:40.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:40.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:40.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:40.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:40.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:40.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:40.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:41.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:41.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:41.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:41.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:40 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:46:41.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:40 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:46:41.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:40 smithi151 ceph-mon[28965]: osdmap e875: 12 total, 12 up, 12 in 2024-06-27T21:46:41.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:40 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:46:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:40 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:46:41.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:40 smithi190 ceph-mon[32247]: osdmap e875: 12 total, 12 up, 12 in 2024-06-27T21:46:41.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:40 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:46:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:40 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:46:41.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:40 smithi192 ceph-mon[32956]: osdmap e875: 12 total, 12 up, 12 in 2024-06-27T21:46:41.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:41.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:41.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:41.530 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:41.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:41.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:41.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:41.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:41.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:41.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:41.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:42.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:42.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:42.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:42.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:42.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:41 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:46:42.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:41 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:46:42.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:41 smithi151 ceph-mon[28965]: pgmap v2152: 57 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 48 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 9912/157626 objects degraded (6.288%); 7300/157626 objects misplaced (4.631%); 8.3 MiB/s, 134 objects/s recovering 2024-06-27T21:46:42.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:41 smithi151 ceph-mon[28965]: 1.1 deep-scrub starts 2024-06-27T21:46:42.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:41 smithi151 ceph-mon[28965]: 1.1 deep-scrub ok 2024-06-27T21:46:42.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:41 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:46:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:41 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:46:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:41 smithi190 ceph-mon[32247]: pgmap v2152: 57 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 48 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 9912/157626 objects degraded (6.288%); 7300/157626 objects misplaced (4.631%); 8.3 MiB/s, 134 objects/s recovering 2024-06-27T21:46:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:41 smithi190 ceph-mon[32247]: 1.1 deep-scrub starts 2024-06-27T21:46:42.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:41 smithi190 ceph-mon[32247]: 1.1 deep-scrub ok 2024-06-27T21:46:42.292 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:41 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:46:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:41 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:46:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:41 smithi192 ceph-mon[32956]: pgmap v2152: 57 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 48 active+clean; 3.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 9912/157626 objects degraded (6.288%); 7300/157626 objects misplaced (4.631%); 8.3 MiB/s, 134 objects/s recovering 2024-06-27T21:46:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:41 smithi192 ceph-mon[32956]: 1.1 deep-scrub starts 2024-06-27T21:46:42.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:41 smithi192 ceph-mon[32956]: 1.1 deep-scrub ok 2024-06-27T21:46:42.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:42.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:42.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:42.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:42.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:42.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:42.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:42.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:42.849 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-06-27T21:46:42.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:42.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:42.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:43.048 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:43.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:43.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:43.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:42 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:46:43.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:42 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:46:43.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:46:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:43.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:43.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:42 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:46:43.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:42 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:46:43.269 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:42 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:46:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:42 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:46:43.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:43.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:43.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:43.472 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:43.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:43.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:43.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:43.729 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:43.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:43.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:43.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:43.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:43.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:44.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:44.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:44.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:43 smithi151 ceph-mon[28965]: pgmap v2153: 57 pgs: 2 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 12329/156027 objects degraded (7.902%); 7300/156027 objects misplaced (4.679%); 6.4 MiB/s, 103 objects/s recovering 2024-06-27T21:46:44.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:44.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:43 smithi190 ceph-mon[32247]: pgmap v2153: 57 pgs: 2 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 12329/156027 objects degraded (7.902%); 7300/156027 objects misplaced (4.679%); 6.4 MiB/s, 103 objects/s recovering 2024-06-27T21:46:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:43 smithi192 ceph-mon[32956]: pgmap v2153: 57 pgs: 2 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 12329/156027 objects degraded (7.902%); 7300/156027 objects misplaced (4.679%); 6.4 MiB/s, 103 objects/s recovering 2024-06-27T21:46:44.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:44.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:44.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:44.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:44.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:44.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:44.676 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:44.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:44.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:46:44.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:44.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:44.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:44.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:44.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:44.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:45.002 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:44 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 12329/156027 objects degraded (7.902%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:45.003 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:44 smithi151 ceph-mon[28965]: osdmap e876: 12 total, 12 up, 12 in 2024-06-27T21:46:45.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:45.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:45.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:45.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:45.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:44 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 12329/156027 objects degraded (7.902%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:45.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:44 smithi190 ceph-mon[32247]: osdmap e876: 12 total, 12 up, 12 in 2024-06-27T21:46:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:44 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 12329/156027 objects degraded (7.902%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:44 smithi192 ceph-mon[32956]: osdmap e876: 12 total, 12 up, 12 in 2024-06-27T21:46:45.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:45.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:45.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:45.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:45.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:45.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:45.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:45.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:45.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:45.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:45.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:45.930 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:46:45.976 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:46.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:46.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:46.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:45 smithi151 ceph-mon[28965]: pgmap v2154: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 12317/146454 objects degraded (8.410%); 6470/146454 objects misplaced (4.418%); 11 MiB/s, 273 objects/s recovering 2024-06-27T21:46:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2436334710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:46:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:45 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:46:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2436334710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:46:46.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:45 smithi151 ceph-mon[28965]: osdmap e877: 12 total, 12 up, 12 in 2024-06-27T21:46:46.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:45 smithi190 ceph-mon[32247]: pgmap v2154: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 12317/146454 objects degraded (8.410%); 6470/146454 objects misplaced (4.418%); 11 MiB/s, 273 objects/s recovering 2024-06-27T21:46:46.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2436334710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:46:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:45 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:46:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2436334710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:46:46.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:45 smithi190 ceph-mon[32247]: osdmap e877: 12 total, 12 up, 12 in 2024-06-27T21:46:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:46.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:45 smithi192 ceph-mon[32956]: pgmap v2154: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 800 op/s; 12317/146454 objects degraded (8.410%); 6470/146454 objects misplaced (4.418%); 11 MiB/s, 273 objects/s recovering 2024-06-27T21:46:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2436334710' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:46:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:45 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:46:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2436334710' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:46:46.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:45 smithi192 ceph-mon[32956]: osdmap e877: 12 total, 12 up, 12 in 2024-06-27T21:46:46.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:46.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:46.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:46.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:46.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:46.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:46.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:46.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:46.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:46.828 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:46.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:46.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:46.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:47.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:47.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:47.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:47.173 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:46 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:46:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:46 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:46:47.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:47.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:46 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:46:47.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:47.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:47.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:47.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:47.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:47.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:47.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:47.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:47.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:47.850 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 1, 7, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:46:47.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 0 1 2024-06-27T21:46:47.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:47.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:47.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:48.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:48.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:47 smithi151 ceph-mon[28965]: pgmap v2157: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 10127/143628 objects degraded (7.051%); 7725/143628 objects misplaced (5.378%); 9.6 MiB/s, 274 objects/s recovering 2024-06-27T21:46:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:47 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:48.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:47 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:48.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:48.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:48.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:47 smithi190 ceph-mon[32247]: pgmap v2157: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 10127/143628 objects degraded (7.051%); 7725/143628 objects misplaced (5.378%); 9.6 MiB/s, 274 objects/s recovering 2024-06-27T21:46:48.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:47 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:48.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:47 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:47 smithi192 ceph-mon[32956]: pgmap v2157: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 728 op/s; 10127/143628 objects degraded (7.051%); 7725/143628 objects misplaced (5.378%); 9.6 MiB/s, 274 objects/s recovering 2024-06-27T21:46:48.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:47 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:48.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:47 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:48.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:48.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:48.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:48.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:48.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:48.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:48.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:48.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:48.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:46:48.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:48.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:48.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:48.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:49.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:49.080 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:48 smithi151 ceph-mon[28965]: osdmap e878: 12 total, 12 up, 12 in 2024-06-27T21:46:49.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:49.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:49.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:49.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:48 smithi190 ceph-mon[32247]: osdmap e878: 12 total, 12 up, 12 in 2024-06-27T21:46:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:49.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:49.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:48 smithi192 ceph-mon[32956]: osdmap e878: 12 total, 12 up, 12 in 2024-06-27T21:46:49.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:49.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:49.469 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:49.508 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:46:49.509 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:46:49.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:49.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:49.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:49.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:49.781 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:49.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:49.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:49.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:49.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:50.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:50.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:50.143 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:49 smithi151 ceph-mon[28965]: pgmap v2159: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 10124/139743 objects degraded (7.245%); 7069/139743 objects misplaced (5.059%); 17 MiB/s, 402 objects/s recovering 2024-06-27T21:46:50.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2761737232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:46:50.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:49 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 10124/139743 objects degraded (7.245%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:50.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:49 smithi151 ceph-mon[28965]: osdmap e879: 12 total, 12 up, 12 in 2024-06-27T21:46:50.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:50.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:49 smithi190 ceph-mon[32247]: pgmap v2159: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 10124/139743 objects degraded (7.245%); 7069/139743 objects misplaced (5.059%); 17 MiB/s, 402 objects/s recovering 2024-06-27T21:46:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2761737232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:46:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:49 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 10124/139743 objects degraded (7.245%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:50.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:49 smithi190 ceph-mon[32247]: osdmap e879: 12 total, 12 up, 12 in 2024-06-27T21:46:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:49 smithi192 ceph-mon[32956]: pgmap v2159: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 10124/139743 objects degraded (7.245%); 7069/139743 objects misplaced (5.059%); 17 MiB/s, 402 objects/s recovering 2024-06-27T21:46:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2761737232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:46:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:49 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 10124/139743 objects degraded (7.245%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:50.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:49 smithi192 ceph-mon[32956]: osdmap e879: 12 total, 12 up, 12 in 2024-06-27T21:46:50.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:50.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:50.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:50.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:50.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:46:50.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:46:50.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:46:50.534 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:46:50.535 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:46:50.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:50.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:50.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:50.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:50.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:50.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:50.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:50.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:50.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:51.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:51.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:51.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:51.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:51.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:51.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:51.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:51.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:51.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:51.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:51.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:51.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:51.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:51.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:51.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:51.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:51.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:51.989 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:51.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:52.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:52.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:52.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.255 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:52.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:51 smithi190 ceph-mon[32247]: pgmap v2161: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 10120/133086 objects degraded (7.604%); 6781/133086 objects misplaced (5.095%); 13 MiB/s, 208 objects/s recovering 2024-06-27T21:46:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:51 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:46:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:51 smithi192 ceph-mon[32956]: pgmap v2161: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 10120/133086 objects degraded (7.604%); 6781/133086 objects misplaced (5.095%); 13 MiB/s, 208 objects/s recovering 2024-06-27T21:46:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:51 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:46:52.313 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:51 smithi151 ceph-mon[28965]: pgmap v2161: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 10120/133086 objects degraded (7.604%); 6781/133086 objects misplaced (5.095%); 13 MiB/s, 208 objects/s recovering 2024-06-27T21:46:52.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:51 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:46:52.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:52.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:52.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:52.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:52.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:52.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:52.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:52.783 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:52.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:52.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:52.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:53.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:53.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:53.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:53.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:53.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:53.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:53.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:53.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:53.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:53.480 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:53.490 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:46:53.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:53.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:53.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:53.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:53.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:53.818 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:53.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:53.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:53.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:54.052 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:54.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:46:54.080 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:54.095 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:53 smithi151 ceph-mon[28965]: pgmap v2162: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 10120/131562 objects degraded (7.692%); 6285/131562 objects misplaced (4.777%); 14 MiB/s, 229 objects/s recovering 2024-06-27T21:46:54.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:54.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:53 smithi190 ceph-mon[32247]: pgmap v2162: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 10120/131562 objects degraded (7.692%); 6285/131562 objects misplaced (4.777%); 14 MiB/s, 229 objects/s recovering 2024-06-27T21:46:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:53 smithi192 ceph-mon[32956]: pgmap v2162: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 10120/131562 objects degraded (7.692%); 6285/131562 objects misplaced (4.777%); 14 MiB/s, 229 objects/s recovering 2024-06-27T21:46:54.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:54.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:54.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:54.462 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:54.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:54.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:54.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:54.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:54.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:54.836 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:54.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:54.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:54.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:55.006 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-06-27T21:46:55.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:55.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:55.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4166560995' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:46:55.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:54 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:46:55.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:54 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 10120/131562 objects degraded (7.692%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:55.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:46:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.231 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:55.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:46:55.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4166560995' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:46:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:54 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:46:55.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:54 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 10120/131562 objects degraded (7.692%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4166560995' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:46:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:54 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:46:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:54 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 10120/131562 objects degraded (7.692%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:46:55.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:55.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:55.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:55.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:55.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:46:55.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:46:55.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:46:55.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:46:55.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:55.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:55.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:55.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:55.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:56.101 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:56.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:56.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:56.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:56 smithi190 ceph-mon[32247]: pgmap v2163: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 10105/121350 objects degraded (8.327%); 5778/121350 objects misplaced (4.761%); 15 MiB/s, 327 objects/s recovering 2024-06-27T21:46:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:56 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:46:56.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:56 smithi190 ceph-mon[32247]: osdmap e880: 12 total, 12 up, 12 in 2024-06-27T21:46:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:56 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:56.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:56 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:56 smithi192 ceph-mon[32956]: pgmap v2163: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 10105/121350 objects degraded (8.327%); 5778/121350 objects misplaced (4.761%); 15 MiB/s, 327 objects/s recovering 2024-06-27T21:46:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:56 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:46:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:56 smithi192 ceph-mon[32956]: osdmap e880: 12 total, 12 up, 12 in 2024-06-27T21:46:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:56 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:56.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:56 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:56.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:56.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:56.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:56 smithi151 ceph-mon[28965]: pgmap v2163: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 10105/121350 objects degraded (8.327%); 5778/121350 objects misplaced (4.761%); 15 MiB/s, 327 objects/s recovering 2024-06-27T21:46:56.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:56 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:46:56.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:56 smithi151 ceph-mon[28965]: osdmap e880: 12 total, 12 up, 12 in 2024-06-27T21:46:56.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:56 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:56.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:56 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:56.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:56.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:56.550 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:46:56.550 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:46:56.550 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:46:56.550 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:46:56.550 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:46:56.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:56.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:56.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:56.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:56.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:46:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:56.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:56.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:46:56.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:57.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:57.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:57.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:57.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:46:57.259 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.260 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:46:57.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:57 smithi190 ceph-mon[32247]: osdmap e881: 12 total, 12 up, 12 in 2024-06-27T21:46:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:57 smithi192 ceph-mon[32956]: osdmap e881: 12 total, 12 up, 12 in 2024-06-27T21:46:57.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:57.389 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:57.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:57 smithi151 ceph-mon[28965]: osdmap e881: 12 total, 12 up, 12 in 2024-06-27T21:46:57.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:57.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:57.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:57.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:57.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:57.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:57.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:57.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:57.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:58.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:58.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:58.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:58.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:58.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:58.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:58 smithi192 ceph-mon[32956]: pgmap v2166: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 10105/117657 objects degraded (8.589%); 5425/117657 objects misplaced (4.611%); 13 MiB/s, 312 objects/s recovering 2024-06-27T21:46:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:58 smithi192 ceph-mon[32956]: osdmap e882: 12 total, 12 up, 12 in 2024-06-27T21:46:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:58 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:46:58.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:58.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:58.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:58 smithi151 ceph-mon[28965]: pgmap v2166: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 10105/117657 objects degraded (8.589%); 5425/117657 objects misplaced (4.611%); 13 MiB/s, 312 objects/s recovering 2024-06-27T21:46:58.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:58 smithi151 ceph-mon[28965]: osdmap e882: 12 total, 12 up, 12 in 2024-06-27T21:46:58.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:58 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:46:58.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:58 smithi190 ceph-mon[32247]: pgmap v2166: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 49 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 10105/117657 objects degraded (8.589%); 5425/117657 objects misplaced (4.611%); 13 MiB/s, 312 objects/s recovering 2024-06-27T21:46:58.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:58 smithi190 ceph-mon[32247]: osdmap e882: 12 total, 12 up, 12 in 2024-06-27T21:46:58.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:58 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:46:58.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:58.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:58.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:58.682 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:58.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:46:58.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:46:58.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:58.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:58.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:59.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:59.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:59.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:46:59.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:46:59.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:46:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:46:59.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: osdmap e883: 12 total, 12 up, 12 in 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:59.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:46:59 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:59.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:46:59.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:59.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: osdmap e883: 12 total, 12 up, 12 in 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:59.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:46:59 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: osdmap e883: 12 total, 12 up, 12 in 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:46:59.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:46:59.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:46:59.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:46:59 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:46:59.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:46:59.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:46:59.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:46:59.692 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:46:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:46:59.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:46:59.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:46:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:46:59.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:46:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:46:59.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:46:59.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:00.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:00.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:00.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:00.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:47:00.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:00 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 10105/117657 objects degraded (8.589%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:00.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:00.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:00.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:47:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:00 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 10105/117657 objects degraded (8.589%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:00.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:00.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:47:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:00 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 10105/117657 objects degraded (8.589%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:00.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:00.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:00.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 3 1 2024-06-27T21:47:00.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:00.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:00.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:00.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:00.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:00.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:00.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:01.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:01.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:01.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:47:01.182 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:01.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:01.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:01.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:01.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:01.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:01 smithi151 ceph-mon[28965]: osdmap e884: 12 total, 12 up, 12 in 2024-06-27T21:47:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:01.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:01 smithi192 ceph-mon[32956]: osdmap e884: 12 total, 12 up, 12 in 2024-06-27T21:47:01.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:01.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:01.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:01.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:01 smithi190 ceph-mon[32247]: osdmap e884: 12 total, 12 up, 12 in 2024-06-27T21:47:01.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:01.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:01.644 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:01.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:01.758 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:01.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:01.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:01.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:01.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:02.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:02.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:02.150 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:02 smithi151 ceph-mon[28965]: pgmap v2171: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 8206/108378 objects degraded (7.572%); 6369/108378 objects misplaced (5.877%); 3.0 MiB/s, 279 objects/s recovering 2024-06-27T21:47:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:02.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:02.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:02.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:02 smithi192 ceph-mon[32956]: pgmap v2171: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 8206/108378 objects degraded (7.572%); 6369/108378 objects misplaced (5.877%); 3.0 MiB/s, 279 objects/s recovering 2024-06-27T21:47:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:02.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:02.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:02.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:02 smithi190 ceph-mon[32247]: pgmap v2171: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 8206/108378 objects degraded (7.572%); 6369/108378 objects misplaced (5.877%); 3.0 MiB/s, 279 objects/s recovering 2024-06-27T21:47:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:02.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:02.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:02.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:02.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:02.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:02.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:02.815 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:02.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:02.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:02.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:03.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:03.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:03.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:03.198 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.199 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:03.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:03.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:03.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:03 smithi151 ceph-mon[28965]: osdmap e885: 12 total, 12 up, 12 in 2024-06-27T21:47:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:03 smithi151 ceph-mon[28965]: pgmap v2173: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 8206/107076 objects degraded (7.664%); 5963/107076 objects misplaced (5.569%); 7.2 MiB/s, 345 objects/s recovering 2024-06-27T21:47:03.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:03.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:03 smithi190 ceph-mon[32247]: osdmap e885: 12 total, 12 up, 12 in 2024-06-27T21:47:03.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:03 smithi190 ceph-mon[32247]: pgmap v2173: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 8206/107076 objects degraded (7.664%); 5963/107076 objects misplaced (5.569%); 7.2 MiB/s, 345 objects/s recovering 2024-06-27T21:47:03.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:03 smithi192 ceph-mon[32956]: osdmap e885: 12 total, 12 up, 12 in 2024-06-27T21:47:03.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:03 smithi192 ceph-mon[32956]: pgmap v2173: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 8206/107076 objects degraded (7.664%); 5963/107076 objects misplaced (5.569%); 7.2 MiB/s, 345 objects/s recovering 2024-06-27T21:47:03.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:03.588 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:03.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:03.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:03.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:03.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:03.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:03.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:03.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:04.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:04.091 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:04.142 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:47:04.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:04.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:04.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:04.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:04.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:04.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:04.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:04 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:47:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:04 smithi151 ceph-mon[28965]: osdmap e886: 12 total, 12 up, 12 in 2024-06-27T21:47:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3186696518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:04.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:04.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:04 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:47:04.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:04 smithi190 ceph-mon[32247]: osdmap e886: 12 total, 12 up, 12 in 2024-06-27T21:47:04.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3186696518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:04.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:04.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:04.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:04 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:47:04.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:04 smithi192 ceph-mon[32956]: osdmap e886: 12 total, 12 up, 12 in 2024-06-27T21:47:04.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3186696518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:04.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:04.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:04.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:04.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:04.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:04.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:04.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:04.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:04.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:04.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:05.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:05.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:05.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:05.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:05 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:47:05.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:05 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:47:05.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:05 smithi151 ceph-mon[28965]: osdmap e887: 12 total, 12 up, 12 in 2024-06-27T21:47:05.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:05 smithi151 ceph-mon[28965]: pgmap v2176: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 8198/97779 objects degraded (8.384%); 4846/97779 objects misplaced (4.956%); 11 MiB/s, 357 objects/s recovering 2024-06-27T21:47:05.223 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:05 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 8206/107076 objects degraded (7.664%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:05.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:05.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:05.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:05.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:05 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:47:05.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:05 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:47:05.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:05 smithi190 ceph-mon[32247]: osdmap e887: 12 total, 12 up, 12 in 2024-06-27T21:47:05.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:05 smithi190 ceph-mon[32247]: pgmap v2176: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 8198/97779 objects degraded (8.384%); 4846/97779 objects misplaced (4.956%); 11 MiB/s, 357 objects/s recovering 2024-06-27T21:47:05.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:05 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 8206/107076 objects degraded (7.664%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:05.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:05.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:05.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:05 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:47:05.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:05 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:47:05.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:05 smithi192 ceph-mon[32956]: osdmap e887: 12 total, 12 up, 12 in 2024-06-27T21:47:05.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:05 smithi192 ceph-mon[32956]: pgmap v2176: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 8198/97779 objects degraded (8.384%); 4846/97779 objects misplaced (4.956%); 11 MiB/s, 357 objects/s recovering 2024-06-27T21:47:05.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:05 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 8206/107076 objects degraded (7.664%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:05.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:05.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:05.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:05.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:05.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:05.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:05.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:05.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:06.021 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:06.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:06.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:06.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:06.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:06.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:06.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:06.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:06 smithi151 ceph-mon[28965]: osdmap e888: 12 total, 12 up, 12 in 2024-06-27T21:47:06.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:06.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:06 smithi190 ceph-mon[32247]: osdmap e888: 12 total, 12 up, 12 in 2024-06-27T21:47:06.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:06.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:06 smithi192 ceph-mon[32956]: osdmap e888: 12 total, 12 up, 12 in 2024-06-27T21:47:06.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:06.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:06.773 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:06.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:06.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:06.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:06.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:47:06.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:07.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:07.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:07.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:07.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:07.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:07.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:07.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:07.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:07 smithi151 ceph-mon[28965]: osdmap e889: 12 total, 12 up, 12 in 2024-06-27T21:47:07.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:07 smithi151 ceph-mon[28965]: pgmap v2179: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 7971/94230 objects degraded (8.459%); 4846/94230 objects misplaced (5.143%); 5.7 MiB/s, 415 objects/s recovering 2024-06-27T21:47:07.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:07.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:07 smithi190 ceph-mon[32247]: osdmap e889: 12 total, 12 up, 12 in 2024-06-27T21:47:07.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:07 smithi190 ceph-mon[32247]: pgmap v2179: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 7971/94230 objects degraded (8.459%); 4846/94230 objects misplaced (5.143%); 5.7 MiB/s, 415 objects/s recovering 2024-06-27T21:47:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:07.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:07.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:07 smithi192 ceph-mon[32956]: osdmap e889: 12 total, 12 up, 12 in 2024-06-27T21:47:07.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:07 smithi192 ceph-mon[32956]: pgmap v2179: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 7971/94230 objects degraded (8.459%); 4846/94230 objects misplaced (5.143%); 5.7 MiB/s, 415 objects/s recovering 2024-06-27T21:47:07.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:07.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:07.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:07.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:07.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:07.924 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:07.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:08.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:08.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:08.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:08.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:08.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:08.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:08.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.425 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:08.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:08.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:08.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:08.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:08 smithi151 ceph-mon[28965]: osdmap e890: 12 total, 12 up, 12 in 2024-06-27T21:47:08.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:08.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:08.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:08 smithi190 ceph-mon[32247]: osdmap e890: 12 total, 12 up, 12 in 2024-06-27T21:47:08.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:08.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:08.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:08.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:08 smithi192 ceph-mon[32956]: osdmap e890: 12 total, 12 up, 12 in 2024-06-27T21:47:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.548 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:08.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:08.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:08.677 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:08.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:08.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:08.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:08.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:08.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:09.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:09.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:09.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:09.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:09.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:09.321 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:09.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:09.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:09 smithi151 ceph-mon[28965]: pgmap v2181: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 51 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 6221/90651 objects degraded (6.863%); 5740/90651 objects misplaced (6.332%); 1.7 MiB/s, 337 objects/s recovering 2024-06-27T21:47:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:09.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:09 smithi190 ceph-mon[32247]: pgmap v2181: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 51 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 6221/90651 objects degraded (6.863%); 5740/90651 objects misplaced (6.332%); 1.7 MiB/s, 337 objects/s recovering 2024-06-27T21:47:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:09 smithi192 ceph-mon[32956]: pgmap v2181: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 51 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 6221/90651 objects degraded (6.863%); 5740/90651 objects misplaced (6.332%); 1.7 MiB/s, 337 objects/s recovering 2024-06-27T21:47:09.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:09.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:09.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:09.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:09.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:09.848 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:09.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:09.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:09.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:10.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:10.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:10.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:10.223 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-06-27T21:47:10.238 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:10.340 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:10.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:47:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: osdmap e891: 12 total, 12 up, 12 in 2024-06-27T21:47:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2785311509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:47:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:47:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:10 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6221/90651 objects degraded (6.863%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:10.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:10.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:47:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: osdmap e891: 12 total, 12 up, 12 in 2024-06-27T21:47:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2785311509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:47:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:47:10.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:10 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6221/90651 objects degraded (6.863%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:10.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:47:10.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:10.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:10.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: osdmap e891: 12 total, 12 up, 12 in 2024-06-27T21:47:10.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2785311509' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:47:10.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:47:10.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:10 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6221/90651 objects degraded (6.863%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:10.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:10.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:10.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:10.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:10.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:10.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:10.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:10.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:10.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:11.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:11.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:11.212 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.240 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:11.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:11.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:11.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:11.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:11 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:47:11.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:11 smithi151 ceph-mon[28965]: osdmap e892: 12 total, 12 up, 12 in 2024-06-27T21:47:11.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:11 smithi151 ceph-mon[28965]: pgmap v2184: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 6214/85317 objects degraded (7.283%); 4996/85317 objects misplaced (5.856%); 12 MiB/s, 241 objects/s recovering 2024-06-27T21:47:11.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:11 smithi151 ceph-mon[28965]: osdmap e893: 12 total, 12 up, 12 in 2024-06-27T21:47:11.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:11.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:11 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:47:11.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:11 smithi190 ceph-mon[32247]: osdmap e892: 12 total, 12 up, 12 in 2024-06-27T21:47:11.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:11 smithi190 ceph-mon[32247]: pgmap v2184: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 6214/85317 objects degraded (7.283%); 4996/85317 objects misplaced (5.856%); 12 MiB/s, 241 objects/s recovering 2024-06-27T21:47:11.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:11 smithi190 ceph-mon[32247]: osdmap e893: 12 total, 12 up, 12 in 2024-06-27T21:47:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:11 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:47:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:11 smithi192 ceph-mon[32956]: osdmap e892: 12 total, 12 up, 12 in 2024-06-27T21:47:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:11 smithi192 ceph-mon[32956]: pgmap v2184: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 6214/85317 objects degraded (7.283%); 4996/85317 objects misplaced (5.856%); 12 MiB/s, 241 objects/s recovering 2024-06-27T21:47:11.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:11 smithi192 ceph-mon[32956]: osdmap e893: 12 total, 12 up, 12 in 2024-06-27T21:47:11.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:11.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:11.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:11.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:11.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:11.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:11.963 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:11.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:11.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:12.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:12.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:12.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:12.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:12.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:12.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:12.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:12.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:12.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:12.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:12.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:12.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:12.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:12.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:12.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:13.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:13.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:13.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:13.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:13.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:13.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:13.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:13.406 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:13.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:13 smithi151 ceph-mon[28965]: osdmap e894: 12 total, 12 up, 12 in 2024-06-27T21:47:13.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:13 smithi151 ceph-mon[28965]: pgmap v2187: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 6214/84474 objects degraded (7.356%); 4996/84474 objects misplaced (5.914%); 12 MiB/s, 230 objects/s recovering 2024-06-27T21:47:13.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:13 smithi190 ceph-mon[32247]: osdmap e894: 12 total, 12 up, 12 in 2024-06-27T21:47:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:13 smithi190 ceph-mon[32247]: pgmap v2187: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 6214/84474 objects degraded (7.356%); 4996/84474 objects misplaced (5.914%); 12 MiB/s, 230 objects/s recovering 2024-06-27T21:47:13.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:13.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:13 smithi192 ceph-mon[32956]: osdmap e894: 12 total, 12 up, 12 in 2024-06-27T21:47:13.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:13 smithi192 ceph-mon[32956]: pgmap v2187: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 6214/84474 objects degraded (7.356%); 4996/84474 objects misplaced (5.914%); 12 MiB/s, 230 objects/s recovering 2024-06-27T21:47:13.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:13.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:13.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:13.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:13.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:13.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:13.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:14.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:14.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:14.145 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:14.174 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:14.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:14.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:14.411 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:14.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:14.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:14.566 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:14.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:14.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:14 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:47:14.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:14.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:14 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:47:14.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:14 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:47:14.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:14.833 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:14.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:14.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:14.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:15.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:15.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:15.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:15.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:15.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:15.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:15.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:15.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:15.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:15.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:15.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:15.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:15 smithi151 ceph-mon[28965]: pgmap v2188: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 6207/78720 objects degraded (7.885%); 3997/78720 objects misplaced (5.077%); 16 MiB/s, 352 objects/s recovering 2024-06-27T21:47:15.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:15 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6207/78720 objects degraded (7.885%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:15.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:15 smithi151 ceph-mon[28965]: osdmap e895: 12 total, 12 up, 12 in 2024-06-27T21:47:15.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:15.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:15.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:15 smithi190 ceph-mon[32247]: pgmap v2188: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 6207/78720 objects degraded (7.885%); 3997/78720 objects misplaced (5.077%); 16 MiB/s, 352 objects/s recovering 2024-06-27T21:47:15.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:15 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6207/78720 objects degraded (7.885%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:15.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:15 smithi190 ceph-mon[32247]: osdmap e895: 12 total, 12 up, 12 in 2024-06-27T21:47:15.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:15 smithi192 ceph-mon[32956]: pgmap v2188: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 6207/78720 objects degraded (7.885%); 3997/78720 objects misplaced (5.077%); 16 MiB/s, 352 objects/s recovering 2024-06-27T21:47:15.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:15 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6207/78720 objects degraded (7.885%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:15.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:15 smithi192 ceph-mon[32956]: osdmap e895: 12 total, 12 up, 12 in 2024-06-27T21:47:15.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:15.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:15.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:15.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:15.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:16.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:16.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:16.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:16.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:16.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:16.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:16.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:16.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:16.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:16.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:16.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:16.752 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:16.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:16.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:16.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:16.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:17.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:17.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:17.137 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:17.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:17.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:17.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:17.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:17.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:17.529 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:17.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:17.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:17.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:17 smithi151 ceph-mon[28965]: pgmap v2190: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 6223/75135 objects degraded (8.282%); 4513/75135 objects misplaced (6.007%); 13 MiB/s, 261 objects/s recovering 2024-06-27T21:47:17.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:17.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:17 smithi190 ceph-mon[32247]: pgmap v2190: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 6223/75135 objects degraded (8.282%); 4513/75135 objects misplaced (6.007%); 13 MiB/s, 261 objects/s recovering 2024-06-27T21:47:17.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:17.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:17 smithi192 ceph-mon[32956]: pgmap v2190: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 6223/75135 objects degraded (8.282%); 4513/75135 objects misplaced (6.007%); 13 MiB/s, 261 objects/s recovering 2024-06-27T21:47:17.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:17.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:17.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:18.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:18.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:18.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:18.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:18.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:18.316 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:18.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:18.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:18.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:18.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:18.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:18.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 4 1 2024-06-27T21:47:18.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:18.725 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:18.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:18.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:18.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:18.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:18.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:19.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:19.126 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:19.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:19.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:19.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:19.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:19.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:19.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:19.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:19.601 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:19 smithi151 ceph-mon[28965]: pgmap v2191: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 6223/70395 objects degraded (8.840%); 4130/70395 objects misplaced (5.867%); 15 MiB/s, 272 objects/s recovering 2024-06-27T21:47:19.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:19.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:19.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:19.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:19 smithi190 ceph-mon[32247]: pgmap v2191: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 6223/70395 objects degraded (8.840%); 4130/70395 objects misplaced (5.867%); 15 MiB/s, 272 objects/s recovering 2024-06-27T21:47:19.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:19.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:19 smithi192 ceph-mon[32956]: pgmap v2191: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 6223/70395 objects degraded (8.840%); 4130/70395 objects misplaced (5.867%); 15 MiB/s, 272 objects/s recovering 2024-06-27T21:47:19.807 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:19.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:47:19.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:47:19.886 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:19.886 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:47:19.887 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:47:19.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:19.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:19.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:19.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:47:19.938 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:47:19.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:47:19.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:47:20.017 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:47:20.017 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:47:20.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:20.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:47:20.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:47:20.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:20.085 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:47:20.086 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:47:20.129 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:47:20.130 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:47:20.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:20.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:47:20.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:47:20.199 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:20.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:47:20.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:47:20.245 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:47:20.246 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:47:20.281 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:47:20.281 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:47:20.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:20.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:20.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:20.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:20.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:20.577 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:20.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:20.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2592602447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:47:20.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:20 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:47:20.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:20 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6223/70395 objects degraded (8.840%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:20.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:20.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2592602447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:47:20.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:20 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:47:20.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:20 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6223/70395 objects degraded (8.840%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:20.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2592602447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:47:20.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:20 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:47:20.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:20 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6223/70395 objects degraded (8.840%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:20.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:20.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:20.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:20.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:20.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:21.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:21.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:21.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:21.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:21.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:21.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:21.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:21.506 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:21.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:21.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:21.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:21 smithi151 ceph-mon[28965]: pgmap v2192: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 862 op/s; 6215/63378 objects degraded (9.806%); 3959/63378 objects misplaced (6.247%); 14 MiB/s, 343 objects/s recovering 2024-06-27T21:47:21.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:21.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:21 smithi190 ceph-mon[32247]: pgmap v2192: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 862 op/s; 6215/63378 objects degraded (9.806%); 3959/63378 objects misplaced (6.247%); 14 MiB/s, 343 objects/s recovering 2024-06-27T21:47:21.769 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:21.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:21 smithi192 ceph-mon[32956]: pgmap v2192: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 862 op/s; 6215/63378 objects degraded (9.806%); 3959/63378 objects misplaced (6.247%); 14 MiB/s, 343 objects/s recovering 2024-06-27T21:47:21.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:21.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:21.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:21.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:22.027 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:22.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:22.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:22.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:22.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:22.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:22.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:22.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:22.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:22.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:22.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:22.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:22.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:22.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:22.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:23.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:23.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:23.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:23.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:23.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:23.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:23.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:23.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:23.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:23.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:23.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:23.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:23.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:23 smithi190 ceph-mon[32247]: pgmap v2193: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 6215/61821 objects degraded (10.053%); 3551/61821 objects misplaced (5.744%); 14 MiB/s, 320 objects/s recovering 2024-06-27T21:47:23.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:23.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:23 smithi192 ceph-mon[32956]: pgmap v2193: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 6215/61821 objects degraded (10.053%); 3551/61821 objects misplaced (5.744%); 14 MiB/s, 320 objects/s recovering 2024-06-27T21:47:23.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:23.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:23.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:23.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:23.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:23 smithi151 ceph-mon[28965]: pgmap v2193: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 6215/61821 objects degraded (10.053%); 3551/61821 objects misplaced (5.744%); 14 MiB/s, 320 objects/s recovering 2024-06-27T21:47:24.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:24.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:24.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:24.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:24.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:24.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:24.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:24.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:24.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:24.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:24.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:24.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:24.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:24 smithi151 ceph-mon[28965]: osdmap e896: 12 total, 12 up, 12 in 2024-06-27T21:47:24.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:24 smithi190 ceph-mon[32247]: osdmap e896: 12 total, 12 up, 12 in 2024-06-27T21:47:24.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:24.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:24 smithi192 ceph-mon[32956]: osdmap e896: 12 total, 12 up, 12 in 2024-06-27T21:47:24.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:24.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:24.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:24.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:25.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:25.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:25.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:25.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:25.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:25.346 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:25.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:25.478 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:25.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:25.610 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:25.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.734 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:25.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:25.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:25 smithi190 ceph-mon[32247]: pgmap v2195: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s; 6214/53754 objects degraded (11.560%); 2400/53754 objects misplaced (4.465%); 10 MiB/s, 323 objects/s recovering 2024-06-27T21:47:25.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:25 smithi190 ceph-mon[32247]: osdmap e897: 12 total, 12 up, 12 in 2024-06-27T21:47:25.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:25 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 6214/53754 objects degraded (11.560%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:25.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:25 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:47:25.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:25 smithi192 ceph-mon[32956]: pgmap v2195: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s; 6214/53754 objects degraded (11.560%); 2400/53754 objects misplaced (4.465%); 10 MiB/s, 323 objects/s recovering 2024-06-27T21:47:25.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:25 smithi192 ceph-mon[32956]: osdmap e897: 12 total, 12 up, 12 in 2024-06-27T21:47:25.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:25 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 6214/53754 objects degraded (11.560%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:25.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:25 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:47:25.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:25.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:25.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:25 smithi151 ceph-mon[28965]: pgmap v2195: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 816 op/s; 6214/53754 objects degraded (11.560%); 2400/53754 objects misplaced (4.465%); 10 MiB/s, 323 objects/s recovering 2024-06-27T21:47:25.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:25 smithi151 ceph-mon[28965]: osdmap e897: 12 total, 12 up, 12 in 2024-06-27T21:47:25.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:25 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 6214/53754 objects degraded (11.560%), 4 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:25.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:25 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:47:25.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:25.990 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:25.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:26.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:26.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:26.268 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:26.290 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:26.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:26.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:26.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.538 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:26.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:26.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:26.662 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:26.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:26.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:26.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:26.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:26.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:27.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:27.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:27.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:27.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:27.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:27.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:27.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:27.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:27.488 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:27 smithi151 ceph-mon[28965]: pgmap v2197: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 5701/49308 objects degraded (11.562%); 2292/49308 objects misplaced (4.648%); 8.0 MiB/s, 424 objects/s recovering 2024-06-27T21:47:27.529 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:47:27.529 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:47:27.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:27.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:27.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:27.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:27.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:27 smithi190 ceph-mon[32247]: pgmap v2197: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 5701/49308 objects degraded (11.562%); 2292/49308 objects misplaced (4.648%); 8.0 MiB/s, 424 objects/s recovering 2024-06-27T21:47:27.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:27 smithi192 ceph-mon[32956]: pgmap v2197: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 5701/49308 objects degraded (11.562%); 2292/49308 objects misplaced (4.648%); 8.0 MiB/s, 424 objects/s recovering 2024-06-27T21:47:27.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:27.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:27.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:27.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:27.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:28.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:28.095 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:28.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:28.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:28.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:28.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:28.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:28.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:28.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:28.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:28.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:28.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:28.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:28.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:28.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:28.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:28.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:29.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:29.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:29.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:29.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:47:29.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:29.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:29.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:29.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:29.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:29.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:29.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:29 smithi151 ceph-mon[28965]: pgmap v2198: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 946 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 5701/45372 objects degraded (12.565%); 2292/45372 objects misplaced (5.052%); 6.6 MiB/s, 367 objects/s recovering 2024-06-27T21:47:29.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/444127882' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:47:29.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:29.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:29 smithi190 ceph-mon[32247]: pgmap v2198: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 946 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 5701/45372 objects degraded (12.565%); 2292/45372 objects misplaced (5.052%); 6.6 MiB/s, 367 objects/s recovering 2024-06-27T21:47:29.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/444127882' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:47:29.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:29 smithi192 ceph-mon[32956]: pgmap v2198: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 946 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 5701/45372 objects degraded (12.565%); 2292/45372 objects misplaced (5.052%); 6.6 MiB/s, 367 objects/s recovering 2024-06-27T21:47:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/444127882' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:47:29.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:29.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:29.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:29.947 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:47:29.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:30.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:30.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:30.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:30.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:30.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:30.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.425 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:30.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:30.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:30.554 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:30.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:30.684 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:30.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:47:30.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/444127882' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:47:30.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: osdmap e898: 12 total, 12 up, 12 in 2024-06-27T21:47:30.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:47:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:47:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5701/45372 objects degraded (12.565%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:30 smithi190 ceph-mon[32247]: osdmap e899: 12 total, 12 up, 12 in 2024-06-27T21:47:30.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:30.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:47:30.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/444127882' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:47:30.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: osdmap e898: 12 total, 12 up, 12 in 2024-06-27T21:47:30.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:47:30.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:47:30.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5701/45372 objects degraded (12.565%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:30.797 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:30 smithi192 ceph-mon[32956]: osdmap e899: 12 total, 12 up, 12 in 2024-06-27T21:47:30.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:30.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:30.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:47:30.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/444127882' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:47:30.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: osdmap e898: 12 total, 12 up, 12 in 2024-06-27T21:47:30.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:47:30.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:47:30.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5701/45372 objects degraded (12.565%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:30.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:30 smithi151 ceph-mon[28965]: osdmap e899: 12 total, 12 up, 12 in 2024-06-27T21:47:30.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:30.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:31.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:31.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:31.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:31.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:31.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:31.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:31.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.470 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:31.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:31.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:31.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:31.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:31.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:31.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:31.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:31.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:31.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:32.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:32.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:31 smithi151 ceph-mon[28965]: pgmap v2201: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 813 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 5695/38988 objects degraded (14.607%); 1288/38988 objects misplaced (3.304%); 4.6 MiB/s, 404 objects/s recovering 2024-06-27T21:47:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:31 smithi151 ceph-mon[28965]: 8.16 scrub starts 2024-06-27T21:47:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:31 smithi151 ceph-mon[28965]: 8.16 scrub ok 2024-06-27T21:47:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:32.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:32.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:31 smithi190 ceph-mon[32247]: pgmap v2201: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 813 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 5695/38988 objects degraded (14.607%); 1288/38988 objects misplaced (3.304%); 4.6 MiB/s, 404 objects/s recovering 2024-06-27T21:47:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:31 smithi190 ceph-mon[32247]: 8.16 scrub starts 2024-06-27T21:47:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:31 smithi190 ceph-mon[32247]: 8.16 scrub ok 2024-06-27T21:47:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:32.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:31 smithi192 ceph-mon[32956]: pgmap v2201: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 813 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 5695/38988 objects degraded (14.607%); 1288/38988 objects misplaced (3.304%); 4.6 MiB/s, 404 objects/s recovering 2024-06-27T21:47:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:31 smithi192 ceph-mon[32956]: 8.16 scrub starts 2024-06-27T21:47:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:31 smithi192 ceph-mon[32956]: 8.16 scrub ok 2024-06-27T21:47:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:32.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:32.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:32.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:32.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:32.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:32.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:47:32.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:32.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:32.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:32.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:32.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:32.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:33.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:33.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:33.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:33.186 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:32 smithi190 ceph-mon[32247]: 8.6 scrub starts 2024-06-27T21:47:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:32 smithi190 ceph-mon[32247]: osdmap e900: 12 total, 12 up, 12 in 2024-06-27T21:47:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:32 smithi190 ceph-mon[32247]: 8.6 scrub ok 2024-06-27T21:47:33.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:32 smithi190 ceph-mon[32247]: 8.b scrub starts 2024-06-27T21:47:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:33.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:32 smithi190 ceph-mon[32247]: 8.b scrub ok 2024-06-27T21:47:33.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:32 smithi192 ceph-mon[32956]: 8.6 scrub starts 2024-06-27T21:47:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:32 smithi192 ceph-mon[32956]: osdmap e900: 12 total, 12 up, 12 in 2024-06-27T21:47:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:32 smithi192 ceph-mon[32956]: 8.6 scrub ok 2024-06-27T21:47:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:32 smithi192 ceph-mon[32956]: 8.b scrub starts 2024-06-27T21:47:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:33.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:32 smithi192 ceph-mon[32956]: 8.b scrub ok 2024-06-27T21:47:33.315 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:33.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:32 smithi151 ceph-mon[28965]: 8.6 scrub starts 2024-06-27T21:47:33.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:32 smithi151 ceph-mon[28965]: osdmap e900: 12 total, 12 up, 12 in 2024-06-27T21:47:33.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:32 smithi151 ceph-mon[28965]: 8.6 scrub ok 2024-06-27T21:47:33.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:32 smithi151 ceph-mon[28965]: 8.b scrub starts 2024-06-27T21:47:33.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:33.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:32 smithi151 ceph-mon[28965]: 8.b scrub ok 2024-06-27T21:47:33.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:33.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:33.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:33.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:33.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:33.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:33.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:33.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:33.848 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:33.852 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:33.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:33.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:33.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:33.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:33 smithi151 ceph-mon[28965]: pgmap v2203: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 779 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 5695/37383 objects degraded (15.234%); 1288/37383 objects misplaced (3.445%); 3.5 MiB/s, 334 objects/s recovering 2024-06-27T21:47:33.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:33 smithi151 ceph-mon[28965]: osdmap e901: 12 total, 12 up, 12 in 2024-06-27T21:47:33.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:33 smithi151 ceph-mon[28965]: 8.a scrub starts 2024-06-27T21:47:33.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:33 smithi151 ceph-mon[28965]: 8.a scrub ok 2024-06-27T21:47:34.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:34.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:34.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:34.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:33 smithi190 ceph-mon[32247]: pgmap v2203: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 779 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 5695/37383 objects degraded (15.234%); 1288/37383 objects misplaced (3.445%); 3.5 MiB/s, 334 objects/s recovering 2024-06-27T21:47:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:33 smithi190 ceph-mon[32247]: osdmap e901: 12 total, 12 up, 12 in 2024-06-27T21:47:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:33 smithi190 ceph-mon[32247]: 8.a scrub starts 2024-06-27T21:47:34.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:33 smithi190 ceph-mon[32247]: 8.a scrub ok 2024-06-27T21:47:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:33 smithi192 ceph-mon[32956]: pgmap v2203: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 779 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 5695/37383 objects degraded (15.234%); 1288/37383 objects misplaced (3.445%); 3.5 MiB/s, 334 objects/s recovering 2024-06-27T21:47:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:33 smithi192 ceph-mon[32956]: osdmap e901: 12 total, 12 up, 12 in 2024-06-27T21:47:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:33 smithi192 ceph-mon[32956]: 8.a scrub starts 2024-06-27T21:47:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:33 smithi192 ceph-mon[32956]: 8.a scrub ok 2024-06-27T21:47:34.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:34.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:34.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:34.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:34.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:34.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:34.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:34.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:34.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:34.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:34.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-06-27T21:47:34.984 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:35.012 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:35.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:47:35.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:35.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:34 smithi151 ceph-mon[28965]: 8.14 scrub starts 2024-06-27T21:47:35.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1170068332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:47:35.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:34 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:47:35.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:34 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 5695/37383 objects degraded (15.234%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:35.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:34 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:47:35.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:34 smithi151 ceph-mon[28965]: osdmap e902: 12 total, 12 up, 12 in 2024-06-27T21:47:35.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:35.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:34 smithi190 ceph-mon[32247]: 8.14 scrub starts 2024-06-27T21:47:35.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1170068332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:47:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:34 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:47:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:34 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 5695/37383 objects degraded (15.234%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:34 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:47:35.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:34 smithi190 ceph-mon[32247]: osdmap e902: 12 total, 12 up, 12 in 2024-06-27T21:47:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:35.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:34 smithi192 ceph-mon[32956]: 8.14 scrub starts 2024-06-27T21:47:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1170068332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:47:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:34 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:47:35.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:34 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 5695/37383 objects degraded (15.234%), 3 pgs degraded (PG_DEGRADED) 2024-06-27T21:47:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:34 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:47:35.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:34 smithi192 ceph-mon[32956]: osdmap e902: 12 total, 12 up, 12 in 2024-06-27T21:47:35.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:35.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:35.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:35.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:35.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:35.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:35.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:35.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:35.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:35.890 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:35.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:35.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 5 1 2024-06-27T21:47:35.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:36.022 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:36.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:36.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: 8.14 scrub ok 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: pgmap v2205: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 650 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 5370/31179 objects degraded (17.223%); 1288/31179 objects misplaced (4.131%); 0 B/s, 233 objects/s recovering 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:36.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:35 smithi151 ceph-mon[28965]: osdmap e903: 12 total, 12 up, 12 in 2024-06-27T21:47:36.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: 8.14 scrub ok 2024-06-27T21:47:36.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: pgmap v2205: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 650 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 5370/31179 objects degraded (17.223%); 1288/31179 objects misplaced (4.131%); 0 B/s, 233 objects/s recovering 2024-06-27T21:47:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:47:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:47:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:36.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:35 smithi190 ceph-mon[32247]: osdmap e903: 12 total, 12 up, 12 in 2024-06-27T21:47:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:36.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: 8.14 scrub ok 2024-06-27T21:47:36.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: pgmap v2205: 57 pgs: 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 650 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 5370/31179 objects degraded (17.223%); 1288/31179 objects misplaced (4.131%); 0 B/s, 233 objects/s recovering 2024-06-27T21:47:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:47:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:47:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:36.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:35 smithi192 ceph-mon[32956]: osdmap e903: 12 total, 12 up, 12 in 2024-06-27T21:47:36.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:36.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:36.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:36.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:36.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:36.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:36.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:36.819 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:36.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:36.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:36.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:36.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:37.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:37.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:37.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:37.200 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:37.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:37.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:37.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:37.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:37.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:37.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:37.606 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:37.609 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:37.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:37.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:37.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:37.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:37.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:37.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:37.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:37.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:38.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:38.069 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:38.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: pgmap v2208: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 509 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3204/24417 objects degraded (13.122%); 1632/24417 objects misplaced (6.684%) 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 8.4 scrub starts 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 8.4 scrub ok 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 8.15 scrub starts 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 8.15 scrub ok 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: osdmap e904: 12 total, 12 up, 12 in 2024-06-27T21:47:38.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:37 smithi151 ceph-mon[28965]: 8.c scrub starts 2024-06-27T21:47:38.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: pgmap v2208: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 509 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3204/24417 objects degraded (13.122%); 1632/24417 objects misplaced (6.684%) 2024-06-27T21:47:38.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 8.4 scrub starts 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 8.4 scrub ok 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 8.15 scrub starts 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 8.15 scrub ok 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: osdmap e904: 12 total, 12 up, 12 in 2024-06-27T21:47:38.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:37 smithi190 ceph-mon[32247]: 8.c scrub starts 2024-06-27T21:47:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: pgmap v2208: 57 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 53 active+clean; 509 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3204/24417 objects degraded (13.122%); 1632/24417 objects misplaced (6.684%) 2024-06-27T21:47:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:47:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:47:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 8.4 scrub starts 2024-06-27T21:47:38.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 8.4 scrub ok 2024-06-27T21:47:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 8.15 scrub starts 2024-06-27T21:47:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 8.15 scrub ok 2024-06-27T21:47:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: osdmap e904: 12 total, 12 up, 12 in 2024-06-27T21:47:38.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:37 smithi192 ceph-mon[32956]: 8.c scrub starts 2024-06-27T21:47:38.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:38.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:38.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:38.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:38.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:38.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:38.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:38.720 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:38.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:38.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:38.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:38.972 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:38.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:39.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:39.098 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:39.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:47:39.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:47:39.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:39 smithi151 ceph-mon[28965]: 8.c scrub ok 2024-06-27T21:47:39.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:39 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:47:39.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:39 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:47:39.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:39 smithi151 ceph-mon[28965]: 8.18 scrub starts 2024-06-27T21:47:39.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:39 smithi151 ceph-mon[28965]: 8.18 scrub ok 2024-06-27T21:47:39.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:39 smithi151 ceph-mon[28965]: osdmap e905: 12 total, 12 up, 12 in 2024-06-27T21:47:39.169 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:47:39.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:47:39.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:39.201 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:47:39.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:47:39.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:39.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:47:39.238 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:47:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:39 smithi190 ceph-mon[32247]: 8.c scrub ok 2024-06-27T21:47:39.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:39 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:47:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:39 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:47:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:39 smithi190 ceph-mon[32247]: 8.18 scrub starts 2024-06-27T21:47:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:39 smithi190 ceph-mon[32247]: 8.18 scrub ok 2024-06-27T21:47:39.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:39 smithi190 ceph-mon[32247]: osdmap e905: 12 total, 12 up, 12 in 2024-06-27T21:47:39.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:47:39.272 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:47:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:39 smithi192 ceph-mon[32956]: 8.c scrub ok 2024-06-27T21:47:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:39 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:47:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:39 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:47:39.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:39 smithi192 ceph-mon[32956]: 8.18 scrub starts 2024-06-27T21:47:39.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:39 smithi192 ceph-mon[32956]: 8.18 scrub ok 2024-06-27T21:47:39.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:39 smithi192 ceph-mon[32956]: osdmap e905: 12 total, 12 up, 12 in 2024-06-27T21:47:39.301 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:47:39.301 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:47:39.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:47:39.332 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:47:39.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:39.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:39.364 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:47:39.364 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:47:39.405 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:47:39.405 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:47:39.446 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:47:39.447 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:47:39.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:39.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:47:39.478 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:47:39.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:39.509 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:47:39.510 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:47:39.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:39.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:39.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:39.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:39.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:39.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:39.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:39.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:40.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:40.112 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:40.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:40.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:40.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:40 smithi151 ceph-mon[28965]: pgmap v2211: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 429 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2747/20571 objects degraded (13.354%); 1632/20571 objects misplaced (7.933%) 2024-06-27T21:47:40.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1583881913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:47:40.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:40 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2747/20571 objects degraded (13.354%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:40 smithi190 ceph-mon[32247]: pgmap v2211: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 429 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2747/20571 objects degraded (13.354%); 1632/20571 objects misplaced (7.933%) 2024-06-27T21:47:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1583881913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:47:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:40 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2747/20571 objects degraded (13.354%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:40.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:40 smithi192 ceph-mon[32956]: pgmap v2211: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 429 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2747/20571 objects degraded (13.354%); 1632/20571 objects misplaced (7.933%) 2024-06-27T21:47:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1583881913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:47:40.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:40 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2747/20571 objects degraded (13.354%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:40.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:40.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:40.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:40.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:40.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:40.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:40.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:40.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:40.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:40.918 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:40.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:41.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:41.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:41.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:41.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:41 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:47:41.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:41 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:47:41.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:41 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:47:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:41 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:47:41.299 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:41.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:41.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:41 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:47:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:41 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:47:41.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:41.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:41.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:41.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:41.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:41.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:41.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:41.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:41.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:41.977 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:47:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:47:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27031.0:154663 8.a e905/797 8:54f2d60a:::benchmark_data_smithi192_167666_object75225:head v 905'9631, mlcod=905'9629)", 2024-06-27T21:47:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:47:39.650787+0000", 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0086433450000000002, 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086712049999999995, 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650787+0000", 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650787+0000", 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650788+0000", 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7400000000000002e-07 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:47:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650796+0000", 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3470000000000001e-06 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650833+0000", 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7097000000000001e-05 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650840+0000", 2024-06-27T21:47:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4749999999999996e-06 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650970+0000", 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012991200000000001 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.650989+0000", 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9111e-05 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:47:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27031.0:154664 8.a e905/797 8:543bdfce:::benchmark_data_smithi192_167666_object75226:head v 905'9632, mlcod=905'9629)", 2024-06-27T21:47:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:47:39.655012+0000", 2024-06-27T21:47:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0044185329999999997, 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044985750000000003, 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.655012+0000", 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.655012+0000", 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.655013+0000", 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0529999999999999e-06 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:47:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.655021+0000", 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6109999999999996e-06 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.655058+0000", 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7011999999999997e-05 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.655066+0000", 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4440000000000008e-06 2024-06-27T21:47:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.656290+0000", 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012241260000000001 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.656311+0000", 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0684999999999999e-05 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:154667 8.8 8:1428a89b:::benchmark_data_smithi192_167666_object75229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e905)", 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:47:39.657017+0000", 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0024139460000000001, 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025418929999999999, 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:47:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154667 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657017+0000", 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657017+0000", 2024-06-27T21:47:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657019+0000", 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729999999999999e-06 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657032+0000", 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.289e-05 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657049+0000", 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7558999999999999e-05 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657059+0000", 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5179999999999993e-06 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657120+0000", 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1470000000000006e-05 2024-06-27T21:47:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657245+0000", 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012529800000000001 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.657351+0000", 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105101 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27031.0:154636 8.13 8:cd0cf975:::benchmark_data_smithi192_167666_object75198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e905)", 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:47:39.620246+0000", 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.039184482, 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039372721999999999, 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27031", 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154636 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620246+0000", 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620246+0000", 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:47:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620248+0000", 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1739999999999999e-06 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620260+0000", 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1497e-05 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620281+0000", 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0605999999999999e-05 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620291+0000", 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0484e-05 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620367+0000", 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5975000000000004e-05 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620501+0000", 2024-06-27T21:47:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134475 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,11", 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.620612+0000", 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110173 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.623512+0000", 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029002139999999999 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:47:39.623595+0000", 2024-06-27T21:47:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3566000000000006e-05 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-06-27T21:47:41.997 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:47:42.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:42.073 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:42.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:42.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:42.263 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:42 smithi151 ceph-mon[28965]: pgmap v2212: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 54 active+clean; 316 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 2767/15147 objects degraded (18.268%); 1371/15147 objects misplaced (9.051%); 2.8 MiB/s, 323 objects/s recovering 2024-06-27T21:47:42.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:42 smithi190 ceph-mon[32247]: pgmap v2212: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 54 active+clean; 316 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 2767/15147 objects degraded (18.268%); 1371/15147 objects misplaced (9.051%); 2.8 MiB/s, 323 objects/s recovering 2024-06-27T21:47:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:42 smithi192 ceph-mon[32956]: pgmap v2212: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 54 active+clean; 316 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 2767/15147 objects degraded (18.268%); 1371/15147 objects misplaced (9.051%); 2.8 MiB/s, 323 objects/s recovering 2024-06-27T21:47:42.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:42.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:42.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:42.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:42.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:42.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:42.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:42.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:42.847 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:42.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:42.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:42.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:42.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:43.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:43.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:43.215 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:43.244 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:43.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:43.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:43.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:43.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:43.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:43.634 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:43.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:43.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:43.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:43.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:43.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:44.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:44.025 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:44.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:44.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:44.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:44 smithi190 ceph-mon[32247]: pgmap v2213: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 54 active+clean; 295 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 2767/14145 objects degraded (19.562%); 1371/14145 objects misplaced (9.692%); 2.4 MiB/s, 273 objects/s recovering 2024-06-27T21:47:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:44 smithi190 ceph-mon[32247]: 8.0 scrub starts 2024-06-27T21:47:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:44 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:47:44.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:44 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:47:44.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:44 smithi192 ceph-mon[32956]: pgmap v2213: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 54 active+clean; 295 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 2767/14145 objects degraded (19.562%); 1371/14145 objects misplaced (9.692%); 2.4 MiB/s, 273 objects/s recovering 2024-06-27T21:47:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:44 smithi192 ceph-mon[32956]: 8.0 scrub starts 2024-06-27T21:47:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:44 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:47:44.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:44 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:47:44.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:44.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:44.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:44 smithi151 ceph-mon[28965]: pgmap v2213: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 54 active+clean; 295 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 2767/14145 objects degraded (19.562%); 1371/14145 objects misplaced (9.692%); 2.4 MiB/s, 273 objects/s recovering 2024-06-27T21:47:44.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:44 smithi151 ceph-mon[28965]: 8.0 scrub starts 2024-06-27T21:47:44.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:44 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:47:44.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:44 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:47:44.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:44.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:44.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:44.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:44.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:44.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:44.792 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 79435 objects 2024-06-27T21:47:44.792 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :115.094 2024-06-27T21:47:44.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:44.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:44.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:44.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:45.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:45.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:45.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:45.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:45.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:45 smithi192 ceph-mon[32956]: 8.0 scrub ok 2024-06-27T21:47:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:45 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2767/14145 objects degraded (19.562%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:45.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:45.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:45 smithi151 ceph-mon[28965]: 8.0 scrub ok 2024-06-27T21:47:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:45 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2767/14145 objects degraded (19.562%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:45.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:45 smithi190 ceph-mon[32247]: 8.0 scrub ok 2024-06-27T21:47:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:45 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2767/14145 objects degraded (19.562%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:45.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:45.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:45.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:45.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:45.767 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:45.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:45.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:45.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:45.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:46.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:46.069 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-06-27T21:47:46.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-06-27T21:47:46.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:46.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:46.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:46.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:46 smithi192 ceph-mon[32956]: pgmap v2214: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 150 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 2767/7176 objects degraded (38.559%); 1371/7176 objects misplaced (19.105%); 1.9 MiB/s, 218 objects/s recovering 2024-06-27T21:47:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:46 smithi192 ceph-mon[32956]: 8.19 scrub starts 2024-06-27T21:47:46.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:46 smithi192 ceph-mon[32956]: 8.19 scrub ok 2024-06-27T21:47:46.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:46.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:46 smithi151 ceph-mon[28965]: pgmap v2214: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 150 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 2767/7176 objects degraded (38.559%); 1371/7176 objects misplaced (19.105%); 1.9 MiB/s, 218 objects/s recovering 2024-06-27T21:47:46.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:46 smithi151 ceph-mon[28965]: 8.19 scrub starts 2024-06-27T21:47:46.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:46 smithi151 ceph-mon[28965]: 8.19 scrub ok 2024-06-27T21:47:46.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:46 smithi190 ceph-mon[32247]: pgmap v2214: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 150 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 2767/7176 objects degraded (38.559%); 1371/7176 objects misplaced (19.105%); 1.9 MiB/s, 218 objects/s recovering 2024-06-27T21:47:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:46 smithi190 ceph-mon[32247]: 8.19 scrub starts 2024-06-27T21:47:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:46 smithi190 ceph-mon[32247]: 8.19 scrub ok 2024-06-27T21:47:46.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:46.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:46.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:46.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:46.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:46.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:46.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:46.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:47.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:47.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:47.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:47.184 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:47.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:47.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:47.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:47.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:47.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:47.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:47.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:47.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:47.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:47.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:47.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:47.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:47.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:47.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:48.047 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:48.068 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:48.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:48.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:48.270 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:48 smithi151 ceph-mon[28965]: pgmap v2215: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 27 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 825 op/s; 2842/1257 objects degraded (226.094%); 738/1257 objects misplaced (58.711%); 2.6 MiB/s, 255 objects/s recovering 2024-06-27T21:47:48.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:48.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:48 smithi192 ceph-mon[32956]: pgmap v2215: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 27 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 825 op/s; 2842/1257 objects degraded (226.094%); 738/1257 objects misplaced (58.711%); 2.6 MiB/s, 255 objects/s recovering 2024-06-27T21:47:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:48.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:48.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:48.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:48.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:48 smithi190 ceph-mon[32247]: pgmap v2215: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 27 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 825 op/s; 2842/1257 objects degraded (226.094%); 738/1257 objects misplaced (58.711%); 2.6 MiB/s, 255 objects/s recovering 2024-06-27T21:47:48.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:48.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:48.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.689 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:48.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:48.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:48.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:48.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:48.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:48.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:49.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:49.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:49.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:49.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:49.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:49.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:49.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:49.439 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:49.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:49.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:49.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:49.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:49.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:49.814 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:49.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:49.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:50.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:50.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:50.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:50.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:50.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:50 smithi151 ceph-mon[28965]: pgmap v2216: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 2.1 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 2842/84 objects degraded (3383.333%); 738/84 objects misplaced (878.571%); 2.4 MiB/s, 129 objects/s recovering 2024-06-27T21:47:50.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:50 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2842/84 objects degraded (3383.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:50.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:50 smithi151 ceph-mon[28965]: osdmap e906: 12 total, 12 up, 12 in 2024-06-27T21:47:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:50 smithi192 ceph-mon[32956]: pgmap v2216: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 2.1 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 2842/84 objects degraded (3383.333%); 738/84 objects misplaced (878.571%); 2.4 MiB/s, 129 objects/s recovering 2024-06-27T21:47:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:50 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2842/84 objects degraded (3383.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:50 smithi192 ceph-mon[32956]: osdmap e906: 12 total, 12 up, 12 in 2024-06-27T21:47:50.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.322 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:50.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:50.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:50.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:50.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:50 smithi190 ceph-mon[32247]: pgmap v2216: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 2.1 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 2842/84 objects degraded (3383.333%); 738/84 objects misplaced (878.571%); 2.4 MiB/s, 129 objects/s recovering 2024-06-27T21:47:50.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:50 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2842/84 objects degraded (3383.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:50.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:50 smithi190 ceph-mon[32247]: osdmap e906: 12 total, 12 up, 12 in 2024-06-27T21:47:50.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.574 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:50.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:50.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:50.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:50.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.821 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:50.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:50.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:50.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:50.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:51.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:51.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:51.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.192 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:51.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:51.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:51 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:47:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:51 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:47:51.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:51 smithi192 ceph-mon[32956]: osdmap e907: 12 total, 12 up, 12 in 2024-06-27T21:47:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:51.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:51.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:51.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:51 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:47:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:51 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:47:51.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:51 smithi151 ceph-mon[28965]: osdmap e907: 12 total, 12 up, 12 in 2024-06-27T21:47:51.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:51 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:47:51.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:51 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:47:51.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:51 smithi190 ceph-mon[32247]: osdmap e907: 12 total, 12 up, 12 in 2024-06-27T21:47:51.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:51.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:47:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.563 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:51.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:51.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:51.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:51.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:51.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:51.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:51.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:51.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:52.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:52.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:52.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:52.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:52 smithi192 ceph-mon[32956]: pgmap v2218: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 960 KiB/s, 172 objects/s recovering 2024-06-27T21:47:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:52 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:52 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:52 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:47:52.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:52 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:47:52.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:52.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:47:52.445 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:47:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:52 smithi151 ceph-mon[28965]: pgmap v2218: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 960 KiB/s, 172 objects/s recovering 2024-06-27T21:47:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:52 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:52 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:52 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:47:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:52 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:47:52.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:52 smithi190 ceph-mon[32247]: pgmap v2218: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 504 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 960 KiB/s, 172 objects/s recovering 2024-06-27T21:47:52.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:52 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:52.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:52 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:52.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:52 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:47:52.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:52 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:47:52.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:52.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:52.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:52.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:52.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:52.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:52.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:52.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:47:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:52.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:52.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:53.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:53.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:53.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:53.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:53.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:53.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:53.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:53.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:53.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:53.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:53.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:53.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:53.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:53.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:53.930 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:53.953 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:53.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:54.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:54.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:54.076 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:54.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:54.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:54 smithi192 ceph-mon[32956]: pgmap v2220: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 1.2 MiB/s, 215 objects/s recovering 2024-06-27T21:47:54.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:54.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:54.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:54.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:54 smithi151 ceph-mon[28965]: pgmap v2220: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 1.2 MiB/s, 215 objects/s recovering 2024-06-27T21:47:54.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:54.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:54 smithi190 ceph-mon[32247]: pgmap v2220: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 1.2 MiB/s, 215 objects/s recovering 2024-06-27T21:47:54.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:54.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:54.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:54.706 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:54.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:54.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:54.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:54.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:54.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:55.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:55.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:55.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:55 smithi151 ceph-mon[28965]: Health check update: Degraded data redundancy: 2842/6 objects degraded (47366.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:55.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:55.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:55.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:55 smithi192 ceph-mon[32956]: Health check update: Degraded data redundancy: 2842/6 objects degraded (47366.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:55.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:55.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:55.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:55.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:55.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:55 smithi190 ceph-mon[32247]: Health check update: Degraded data redundancy: 2842/6 objects degraded (47366.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-06-27T21:47:55.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:55.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:55.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:55.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:55.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:55.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:55.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:55.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:55.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:56.058 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:47:56.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:56.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:56.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:47:56.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:47:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:56 smithi192 ceph-mon[32956]: pgmap v2221: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 0 B/s, 134 objects/s recovering 2024-06-27T21:47:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:56 smithi192 ceph-mon[32956]: 8.8 scrub starts 2024-06-27T21:47:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:56 smithi192 ceph-mon[32956]: 8.8 scrub ok 2024-06-27T21:47:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:56 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:47:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:56 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:47:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1967807114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:56.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:56.334 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:56.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:56.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:56 smithi151 ceph-mon[28965]: pgmap v2221: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 0 B/s, 134 objects/s recovering 2024-06-27T21:47:56.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:56 smithi151 ceph-mon[28965]: 8.8 scrub starts 2024-06-27T21:47:56.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:56 smithi151 ceph-mon[28965]: 8.8 scrub ok 2024-06-27T21:47:56.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:56 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:47:56.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:56 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:47:56.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1967807114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:56.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:56.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:56 smithi190 ceph-mon[32247]: pgmap v2221: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 52 op/s; 2842/6 objects degraded (47366.667%); 3/6 objects misplaced (50.000%); 0 B/s, 134 objects/s recovering 2024-06-27T21:47:56.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:56 smithi190 ceph-mon[32247]: 8.8 scrub starts 2024-06-27T21:47:56.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:56 smithi190 ceph-mon[32247]: 8.8 scrub ok 2024-06-27T21:47:56.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:56 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:47:56.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:56 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:47:56.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1967807114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:47:56.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:56.585 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:56.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:56.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:56.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:56.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:56.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:56.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:56.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:57.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.086 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:57.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:57.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:57.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:47:57.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:47:57.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:47:57.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1967807114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:47:57.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: osdmap e908: 12 total, 12 up, 12 in 2024-06-27T21:47:57.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:47:57.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:57 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:47:57.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:47:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:47:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:47:57.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1967807114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:47:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: osdmap e908: 12 total, 12 up, 12 in 2024-06-27T21:47:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:47:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:57 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:47:57.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:47:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:57.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:47:57.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:57.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1967807114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: osdmap e908: 12 total, 12 up, 12 in 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:47:57.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:57 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:47:57.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:57.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:57.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:57.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:57.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:57.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:57.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:57.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:57.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:57.977 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:58.060 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-06-27T21:47:58.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:58.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:58.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:58.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:58 smithi192 ceph-mon[32956]: pgmap v2223: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2842/6 objects degraded (47366.667%) 2024-06-27T21:47:58.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/510414491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:47:58.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:58.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:58.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:47:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:58 smithi151 ceph-mon[28965]: pgmap v2223: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2842/6 objects degraded (47366.667%) 2024-06-27T21:47:58.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/510414491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:47:58.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:47:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:58.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:47:58.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:58 smithi190 ceph-mon[32247]: pgmap v2223: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2842/6 objects degraded (47366.667%) 2024-06-27T21:47:58.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/510414491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:47:58.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:47:58.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:47:58.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:58.728 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:58.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:58.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:58.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:47:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:58.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:58.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:47:59.066 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_6' removed 2024-06-27T21:47:59.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:59.130 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:59.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:47:59.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:47:59.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:47:59.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:47:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:47:59.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:59 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:47:59.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:59 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:47:59.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/510414491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:47:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:59 smithi151 ceph-mon[28965]: osdmap e909: 12 total, 12 up, 12 in 2024-06-27T21:47:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2623553630' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:47:59.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:47:59 smithi151 ceph-mon[28965]: pgmap v2225: 57 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2842/6 objects degraded (47366.667%) 2024-06-27T21:47:59.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:47:59.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:47:59.500 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:47:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:59 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:47:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:59 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:47:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/510414491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:47:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:59 smithi190 ceph-mon[32247]: osdmap e909: 12 total, 12 up, 12 in 2024-06-27T21:47:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2623553630' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:47:59.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:47:59 smithi190 ceph-mon[32247]: pgmap v2225: 57 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2842/6 objects degraded (47366.667%) 2024-06-27T21:47:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:59 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:47:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:59 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:47:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/510414491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:47:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:59 smithi192 ceph-mon[32956]: osdmap e909: 12 total, 12 up, 12 in 2024-06-27T21:47:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2623553630' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:47:59.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:47:59 smithi192 ceph-mon[32956]: pgmap v2225: 57 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2842/6 objects degraded (47366.667%) 2024-06-27T21:47:59.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:47:59.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:47:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:47:59.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:47:59.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:47:59.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:59.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:47:59.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:47:59.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:47:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:47:59.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:00.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:00.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:00.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:00.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:00.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:00.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:00.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2623553630' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: osdmap e910: 12 total, 12 up, 12 in 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:48:00.448 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:00 smithi192 ceph-mon[32956]: osdmap e911: 12 total, 12 up, 12 in 2024-06-27T21:48:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2623553630' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:48:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: osdmap e910: 12 total, 12 up, 12 in 2024-06-27T21:48:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:48:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:48:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:48:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:48:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:48:00.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:00 smithi151 ceph-mon[28965]: osdmap e911: 12 total, 12 up, 12 in 2024-06-27T21:48:00.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:00.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:00.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2623553630' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: osdmap e910: 12 total, 12 up, 12 in 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:48:00.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:00 smithi190 ceph-mon[32247]: osdmap e911: 12 total, 12 up, 12 in 2024-06-27T21:48:00.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:00.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:00.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:00.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:00.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:00.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:00.971 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:00.998 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:00.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:01.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:01.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:01.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:01.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:01.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:01.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:01.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:48:01.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:48:01.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:48:01.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:48:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:48:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:48:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: pgmap v2228: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:48:01.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:01 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:48:01.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:01.504 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:48:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:48:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:48:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:48:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:48:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:48:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: pgmap v2228: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:48:01.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:01 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:48:01.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:48:01.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:48:01.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:48:01.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:48:01.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:48:01.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:48:01.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: pgmap v2228: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:01.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:48:01.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:01 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:48:01.606 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:01.631 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:01.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:01.757 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:01.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:01.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:01.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:01.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:02.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:02.113 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-06-27T21:48:02.114 INFO:tasks.radosbench:Beginning radosbench... 2024-06-27T21:48:02.114 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-06-27T21:48:02.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-06-27T21:48:02.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:02.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:02.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:02.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:02.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:02.390 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:48:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:48:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:48:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:48:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2842/6 objects degraded (47366.667%), 1 pg degraded, 1 pg undersized) 2024-06-27T21:48:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:48:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:48:02.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:02 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:48:02.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2842/6 objects degraded (47366.667%), 1 pg degraded, 1 pg undersized) 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:48:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:02 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:48:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:02.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2842/6 objects degraded (47366.667%), 1 pg degraded, 1 pg undersized) 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:48:02.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:02 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:48:02.617 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:02.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:02.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:02.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:02.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:02.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:02.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:02.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:03.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:03.122 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:03.148 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:03.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:03.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:03.364 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:03 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:48:03.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:03 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:48:03.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:48:03.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:48:03.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:48:03.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:03 smithi151 ceph-mon[28965]: pgmap v2229: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:03.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:03.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:03.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:03.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:03 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:48:03.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:03 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:48:03.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:48:03.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:48:03.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:48:03.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:03 smithi190 ceph-mon[32247]: pgmap v2229: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:03.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:03.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:03 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:48:03.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:03 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:48:03.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:48:03.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:48:03.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:48:03.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:03 smithi192 ceph-mon[32956]: pgmap v2229: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:03.627 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:03.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:03.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:03.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:03.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:03.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 6 1 2024-06-27T21:48:03.910 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:03.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:04.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:04.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:04.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:04.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:04.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:04.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:04.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:04.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:04.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:04.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: osdmap e912: 12 total, 12 up, 12 in 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:48:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:04 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:48:04.603 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:48:04.603 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:48:04.603 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:04.603 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:04.603 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:48:04.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:48:04.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: osdmap e912: 12 total, 12 up, 12 in 2024-06-27T21:48:04.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:48:04.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:04 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:48:04.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:04.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:04.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:48:04.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:48:04.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:04.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:04.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:48:04.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:48:04.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: osdmap e912: 12 total, 12 up, 12 in 2024-06-27T21:48:04.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:48:04.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:04 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:48:04.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:04.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:04.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:04.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:04.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:04.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:48:04.992 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:48:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:48:05.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:48:05.025 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:05.036 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:48:05.036 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:48:05.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:05.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:48:05.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:48:05.081 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:48:05.082 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:48:05.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:48:05.104 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:48:05.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:48:05.127 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:48:05.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:05.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:05.177 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:05.178 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:05.246 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:48:05.247 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:48:05.269 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:48:05.269 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:48:05.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:05.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:48:05.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:48:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:05.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:05.314 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:48:05.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:48:05.337 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:48:05.337 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:48:05.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:05.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:05.525 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:05.551 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:05.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:05.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:05.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:05.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:05.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:05.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:05.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:06.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:06.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:06.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:06.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:05 smithi190 ceph-mon[32247]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:05 smithi190 ceph-mon[32247]: osdmap e913: 12 total, 12 up, 12 in 2024-06-27T21:48:06.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2423835794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:05 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:48:06.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:05 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:48:06.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:05 smithi192 ceph-mon[32956]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:05 smithi192 ceph-mon[32956]: osdmap e913: 12 total, 12 up, 12 in 2024-06-27T21:48:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2423835794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:05 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:48:06.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:05 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:48:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:06.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:06.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:05 smithi151 ceph-mon[28965]: pgmap v2231: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:06.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:05 smithi151 ceph-mon[28965]: osdmap e913: 12 total, 12 up, 12 in 2024-06-27T21:48:06.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2423835794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:06.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:05 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:48:06.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:05 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:48:06.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:06.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:06.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:06.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:06.655 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:06.656 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:06.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:06.673 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:06.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:06.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:48:06.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:06.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:06.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:06.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:06.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:07.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:07.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:07.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:07.197 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:07.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:06 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:48:07.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:06 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:48:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:06 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:48:07.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:06 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:48:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:06 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:48:07.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:06 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:48:07.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:07.323 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:07.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:07.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:07.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:07.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:07.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:07.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:07.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:07.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:07.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:07.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:07.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:08.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:08.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:08.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:48:08.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:08.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:07 smithi190 ceph-mon[32247]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s, 0 objects/s recovering 2024-06-27T21:48:08.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:07 smithi192 ceph-mon[32956]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s, 0 objects/s recovering 2024-06-27T21:48:08.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:08.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:08.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:08.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:07 smithi151 ceph-mon[28965]: pgmap v2233: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s, 0 objects/s recovering 2024-06-27T21:48:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:08.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:08.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:08.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:08.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:08.714 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:08.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:08.840 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:08.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:08.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:08.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:09.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:09.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:09.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:09.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:08 smithi151 ceph-mon[28965]: osdmap e914: 12 total, 12 up, 12 in 2024-06-27T21:48:09.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:09.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:08 smithi190 ceph-mon[32247]: osdmap e914: 12 total, 12 up, 12 in 2024-06-27T21:48:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:09.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:08 smithi192 ceph-mon[32956]: osdmap e914: 12 total, 12 up, 12 in 2024-06-27T21:48:09.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:09.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:09.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:09.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:09.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:09.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:09.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:09.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:09.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:09.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:09.855 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:09.951 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:48:09.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:09.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:09.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:10.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:10.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: pgmap v2235: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s, 0 objects/s recovering 2024-06-27T21:48:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:48:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:48:10.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1628148537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:48:10.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:10.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1628148537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:48:10.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:09 smithi151 ceph-mon[28965]: osdmap e915: 12 total, 12 up, 12 in 2024-06-27T21:48:10.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.235 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:10.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:10.243 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: pgmap v2235: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s, 0 objects/s recovering 2024-06-27T21:48:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:48:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:48:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1628148537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:48:10.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1628148537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:48:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:09 smithi190 ceph-mon[32247]: osdmap e915: 12 total, 12 up, 12 in 2024-06-27T21:48:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: pgmap v2235: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s, 0 objects/s recovering 2024-06-27T21:48:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:48:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:48:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1628148537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:48:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1628148537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:48:10.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:09 smithi192 ceph-mon[32956]: osdmap e915: 12 total, 12 up, 12 in 2024-06-27T21:48:10.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:10.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:10.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:10.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:10.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:10.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:10.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.742 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:10.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:10.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:10.867 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:10.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:10.973 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_7' created 2024-06-27T21:48:10.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:10.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:11.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:11.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:11.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:11.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:11.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2165330137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-27T21:48:11.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2165330137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-27T21:48:11.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2165330137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-06-27T21:48:11.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:11.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:11.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:11.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:11.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:11.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:11.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:11.757 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:11.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:11.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:11.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:11.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:12.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:12.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:12.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:12.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:12.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:11 smithi190 ceph-mon[32247]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2165330137' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-27T21:48:12.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:11 smithi190 ceph-mon[32247]: osdmap e916: 12 total, 12 up, 12 in 2024-06-27T21:48:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:11 smithi192 ceph-mon[32956]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:12.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2165330137' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-27T21:48:12.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:11 smithi192 ceph-mon[32956]: osdmap e916: 12 total, 12 up, 12 in 2024-06-27T21:48:12.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:11 smithi151 ceph-mon[28965]: pgmap v2237: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:12.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2165330137' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-06-27T21:48:12.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:11 smithi151 ceph-mon[28965]: osdmap e916: 12 total, 12 up, 12 in 2024-06-27T21:48:12.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:12.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:12.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.520 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:12.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:12.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:12.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:12.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:12.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:12.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:48:12.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:12.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:12.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:13.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:13.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:13.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:13.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:12 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:12 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:13.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:12 smithi190 ceph-mon[32247]: osdmap e917: 12 total, 12 up, 12 in 2024-06-27T21:48:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:13.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:12 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:12 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:13.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:12 smithi192 ceph-mon[32956]: osdmap e917: 12 total, 12 up, 12 in 2024-06-27T21:48:13.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:13.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:13.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:12 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:13.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:12 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:13.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:12 smithi151 ceph-mon[28965]: osdmap e917: 12 total, 12 up, 12 in 2024-06-27T21:48:13.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:13.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:13.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:13.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:13.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:13.772 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:13.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:13.899 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:13.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:14.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:14.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:14.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:14.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:14.157 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:13 smithi151 ceph-mon[28965]: pgmap v2240: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:14.157 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:13 smithi151 ceph-mon[28965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:48:14.157 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:13 smithi151 ceph-mon[28965]: osdmap e918: 12 total, 12 up, 12 in 2024-06-27T21:48:14.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:13 smithi190 ceph-mon[32247]: pgmap v2240: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:13 smithi190 ceph-mon[32247]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:48:14.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:13 smithi190 ceph-mon[32247]: osdmap e918: 12 total, 12 up, 12 in 2024-06-27T21:48:14.278 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:13 smithi192 ceph-mon[32956]: pgmap v2240: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:13 smithi192 ceph-mon[32956]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-27T21:48:14.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:13 smithi192 ceph-mon[32956]: osdmap e918: 12 total, 12 up, 12 in 2024-06-27T21:48:14.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:14.406 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:14.419 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:14.419 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:14.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:14.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:14.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:14.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:14.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:14.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:14.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:14.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:14.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:15.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:15.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:15.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:15.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:15.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:15.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:15.400 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:15.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:48:15.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:15.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:15.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:15.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:15.664 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:15.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:15.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:15.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:15.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:15.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:15.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:16.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:16.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:15 smithi151 ceph-mon[28965]: pgmap v2242: 47 pgs: 7 unknown, 40 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:16.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:15 smithi151 ceph-mon[28965]: osdmap e919: 12 total, 12 up, 12 in 2024-06-27T21:48:16.065 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:16.109 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:15 smithi190 ceph-mon[32247]: pgmap v2242: 47 pgs: 7 unknown, 40 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:16.109 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:15 smithi190 ceph-mon[32247]: osdmap e919: 12 total, 12 up, 12 in 2024-06-27T21:48:16.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:16.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:16.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:15 smithi192 ceph-mon[32956]: pgmap v2242: 47 pgs: 7 unknown, 40 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:16.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:15 smithi192 ceph-mon[32956]: osdmap e919: 12 total, 12 up, 12 in 2024-06-27T21:48:16.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:16.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:16.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:16.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:16.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:16.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:16.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:16.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-06-27T21:48:16.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:16.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:16.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:16.959 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-06-27T21:48:16.964 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:16.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:16.977 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:48:16.977 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:48:17.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:48:17.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:48:17.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2442820931' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:48:17.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2423181827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:17.032 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:48:17.032 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:48:17.053 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:48:17.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:48:17.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2442820931' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:48:17.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2423181827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:17.065 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:17.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:48:17.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:48:17.090 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:17.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:48:17.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:48:17.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:48:17.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:48:17.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:17.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2442820931' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:48:17.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2423181827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:17.301 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:48:17.301 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:48:17.301 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:48:17.301 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:48:17.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:17.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:17.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:17.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:17.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:17.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:17.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:17.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:17.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 9 1 2024-06-27T21:48:17.733 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:17.750 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:17.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:17.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:17.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:17.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:48:17.944 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:48:17.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:17.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:17.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:48:17.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:48:17.990 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:48:17.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:48:18.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:48:18.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:48:18.016 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:48:18.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:18.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:18.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:18.165 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:48:18.166 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:48:18.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:18.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:18.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:17 smithi151 ceph-mon[28965]: pgmap v2244: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:18.254 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:17 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2442820931' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:48:18.255 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:17 smithi151 ceph-mon[28965]: osdmap e920: 12 total, 12 up, 12 in 2024-06-27T21:48:18.255 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:17 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:18.255 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:17 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:17 smithi190 ceph-mon[32247]: pgmap v2244: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:17 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2442820931' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:48:18.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:17 smithi190 ceph-mon[32247]: osdmap e920: 12 total, 12 up, 12 in 2024-06-27T21:48:18.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:17 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:18.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:17 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:18.267 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:17 smithi192 ceph-mon[32956]: pgmap v2244: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:17 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2442820931' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:48:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:17 smithi192 ceph-mon[32956]: osdmap e920: 12 total, 12 up, 12 in 2024-06-27T21:48:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:17 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:18.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:17 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:18.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:18.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:18.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:18.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:18.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:18.581 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:18.683 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:18.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:18.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:18.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:18.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:18.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:18.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:19.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:19.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:19.182 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:19.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:18 smithi151 ceph-mon[28965]: osdmap e921: 12 total, 12 up, 12 in 2024-06-27T21:48:19.208 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:19.238 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:18 smithi190 ceph-mon[32247]: osdmap e921: 12 total, 12 up, 12 in 2024-06-27T21:48:19.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:18 smithi192 ceph-mon[32956]: osdmap e921: 12 total, 12 up, 12 in 2024-06-27T21:48:19.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:19.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:19.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:19.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:19.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:19.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:19.633 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:19.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:19.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:19.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:19.837 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:19.840 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:48:19.840 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:48:19.840 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:48:19.840 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:48:19.840 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:48:19.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:19.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:19.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:20.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:20.086 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:19 smithi151 ceph-mon[28965]: pgmap v2247: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:20.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:20.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:19 smithi190 ceph-mon[32247]: pgmap v2247: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:20.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:20.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:20.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:19 smithi192 ceph-mon[32956]: pgmap v2247: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:20.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:20.340 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:20.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:20.468 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:20.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:48:20.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:20.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:20.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:20.721 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:20.725 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:20.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:20.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:20.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:48:20.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:20.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:20.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:21.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:21.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:21.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:21.223 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:21.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:21.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:21.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:21.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:21.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:21.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:21.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:21.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:21.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:21.850 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:21.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:21.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:21.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:22.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:21 smithi190 ceph-mon[32247]: pgmap v2248: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:22.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:22.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:22.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:22.225 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:21 smithi192 ceph-mon[32956]: pgmap v2248: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:22.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:21 smithi151 ceph-mon[28965]: pgmap v2248: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:22.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:22.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:22.385 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:48:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:48:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:48:22.385 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:48:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e705,698 6:a6704fcb:::benchmark_data_smithi192_117994_object48223:head,540'9656)", 2024-06-27T21:48:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:08.659548+0000", 2024-06-27T21:48:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 553.72210111799996, 2024-06-27T21:48:22.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52699047600000004, 2024-06-27T21:48:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659548+0000", 2024-06-27T21:48:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659548+0000", 2024-06-27T21:48:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659549+0000", 2024-06-27T21:48:22.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.53e-06 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:08.659575+0000", 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5902e-05 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186447+0000", 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52687193300000001 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186452+0000", 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2399999999999998e-06 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186495+0000", 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.227e-05 2024-06-27T21:48:22.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186501+0000", 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8279999999999997e-06 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:09.186538+0000", 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6773000000000002e-05 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.392 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.16 e706,698 6:6ed1fc8a:::benchmark_data_smithi192_117994_object66041:head,607'10323)", 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 549.40213139900004, 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43442441300000001, 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.393 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979518+0000", 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0299999999999999e-07 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:12.979522+0000", 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0090000000000001e-06 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413778+0000", 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.434255693 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413784+0000", 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2709999999999998e-06 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413870+0000", 2024-06-27T21:48:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5923999999999999e-05 2024-06-27T21:48:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413876+0000", 2024-06-27T21:48:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2659999999999998e-06 2024-06-27T21:48:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:13.413942+0000", 2024-06-27T21:48:22.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5647000000000004e-05 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.16 e707,698 6:69a24610:::benchmark_data_smithi192_117994_object70316:head,625'10450)", 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:14.103552+0000", 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 548.27809677000005, 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48983110000000002, 2024-06-27T21:48:22.399 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103552+0000", 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103552+0000", 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103553+0000", 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.82e-07 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.103556+0000", 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5899999999999999e-06 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593246+0000", 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.489689924 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593256+0000", 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7499999999999998e-06 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593315+0000", 2024-06-27T21:48:22.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8947999999999999e-05 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593322+0000", 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8360000000000003e-06 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:14.593383+0000", 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1569999999999995e-05 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a2547159:::benchmark_data_smithi192_117994_object90208:head,669'11030)", 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:16.952936+0000", 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 545.428712851, 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48239369700000001, 2024-06-27T21:48:22.404 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952936+0000", 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952936+0000", 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952937+0000", 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2600000000000002e-07 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.952943+0000", 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2670000000000003e-06 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435072+0000", 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48212887100000001 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435085+0000", 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3072e-05 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435239+0000", 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000154067 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435253+0000", 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4606e-05 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.435330+0000", 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6287999999999999e-05 2024-06-27T21:48:22.408 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a6ffdd34:::benchmark_data_smithi192_117994_object90211:head,669'11031)", 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 545.426352318, 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48275176600000003, 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955297+0000", 2024-06-27T21:48:22.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4999999999999998e-07 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:16.955299+0000", 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3879999999999998e-06 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.437969+0000", 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.482669815 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.437973+0000", 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1570000000000002e-06 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.438010+0000", 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6628000000000002e-05 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.438015+0000", 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1630000000000004e-06 2024-06-27T21:48:22.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.438048+0000", 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3265000000000001e-05 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a6561f31:::benchmark_data_smithi192_117994_object91018:head,670'11058)", 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:17.493688+0000", 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 544.88796045300001, 2024-06-27T21:48:22.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36878778600000001, 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493688+0000", 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493688+0000", 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493689+0000", 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0699999999999997e-07 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.493696+0000", 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7719999999999997e-06 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862204+0000", 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.368508593 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862221+0000", 2024-06-27T21:48:22.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6082999999999998e-05 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862353+0000", 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000132053 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862372+0000", 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.929e-05 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.862476+0000", 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104388 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e709,698 6:a673efaf:::benchmark_data_smithi192_117994_object91022:head,670'11059)", 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:17.495674+0000", 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 544.88597488000005, 2024-06-27T21:48:22.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36941172900000002, 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495674+0000", 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495674+0000", 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495675+0000", 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999996e-07 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.495681+0000", 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3550000000000001e-06 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865007+0000", 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36932614899999999 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865012+0000", 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3469999999999998e-06 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865046+0000", 2024-06-27T21:48:22.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3961999999999998e-05 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865051+0000", 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.702e-06 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:17.865086+0000", 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4610000000000002e-05 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e710,698 6:a11fa313:::benchmark_data_smithi192_117994_object92325:head,673'11094)", 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 544.36408500799996, 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37082322099999998, 2024-06-27T21:48:22.424 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017564+0000", 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0500000000000004e-07 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.017567+0000", 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9809999999999999e-06 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388209+0000", 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37064207999999998 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388219+0000", 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9699999999999994e-06 2024-06-27T21:48:22.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388309+0000", 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0002999999999999e-05 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388317+0000", 2024-06-27T21:48:22.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2790000000000001e-06 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.388387+0000", 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0402999999999997e-05 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDelete(6.5 e710,698 6:a7bd4761:::benchmark_data_smithi192_117994_object92357:head,673'11095)", 2024-06-27T21:48:22.429 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 544.36186667499999, 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.372040078, 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019782+0000", 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6e-07 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.019785+0000", 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.419e-06 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391708+0000", 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.371923112 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391713+0000", 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2490000000000001e-06 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391781+0000", 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7336999999999996e-05 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391786+0000", 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0769999999999998e-06 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:39:18.391822+0000", 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6624e-05 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:59675 8.19 8.fc7e8c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:15.065138+0000", 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 187.31651089799999, 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83082461900000004, 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.434 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 59675 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065138+0000", 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065138+0000", 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065139+0000", 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065213+0000", 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3948000000000004e-05 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065228+0000", 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4748e-05 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065252+0000", 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3827000000000001e-05 2024-06-27T21:48:22.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065321+0000", 2024-06-27T21:48:22.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8992000000000006e-05 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065335+0000", 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3828e-05 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.895950+0000", 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83061501599999998 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.895963+0000", 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2866e-05 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.19 857/836 [PushOp(8:9d37a8bb:::benchmark_data_smithi192_167666_object16727:head, version: 810'1056, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:9d37a8bb:::benchmark_data_smithi192_167666_object16727:head@810'1056, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:02.906790+0000", 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 139.47485855400001, 2024-06-27T21:48:22.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23660510100000001, 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:02.906790+0000", 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:02.906790+0000", 2024-06-27T21:48:22.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:02.906791+0000", 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6199999999999997e-07 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:02.906891+0000", 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9715000000000003e-05 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.142212+0000", 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23532124300000001 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.142241+0000", 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9269e-05 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.142314+0000", 2024-06-27T21:48:22.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2249999999999994e-05 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.142323+0000", 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8679999999999994e-06 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.143395+0000", 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010719939999999999 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.445 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.d 857/838 [PushOp(8:bb5cad6c:::benchmark_data_smithi192_167666_object55735:head, version: 830'3540, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:bb5cad6c:::benchmark_data_smithi192_167666_object55735:head@830'3540, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:05.795373+0000", 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 136.58627584199999, 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30562369099999998, 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.446 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795373+0000", 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795373+0000", 2024-06-27T21:48:22.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795374+0000", 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8100000000000002e-07 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795430+0000", 2024-06-27T21:48:22.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5999999999999999e-05 2024-06-27T21:48:22.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098693+0000", 2024-06-27T21:48:22.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.303263055 2024-06-27T21:48:22.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098733+0000", 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0574999999999997e-05 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098837+0000", 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103335 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098851+0000", 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4251e-05 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.100997+0000", 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021457939999999999 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.16 867/839 [PushOp(8:6f6f41e4:::benchmark_data_smithi192_167666_object25554:head, version: 817'1579, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:6f6f41e4:::benchmark_data_smithi192_167666_object25554:head@817'1579, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:22.452 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:28.857788+0000", 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 113.523860513, 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31268284400000002, 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857788+0000", 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857788+0000", 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857789+0000", 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7400000000000003e-07 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857855+0000", 2024-06-27T21:48:22.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5889000000000006e-05 2024-06-27T21:48:22.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.168966+0000", 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31111138599999999 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.168994+0000", 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7463000000000001e-05 2024-06-27T21:48:22.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.169195+0000", 2024-06-27T21:48:22.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020071800000000001 2024-06-27T21:48:22.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.169215+0000", 2024-06-27T21:48:22.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0236000000000001e-05 2024-06-27T21:48:22.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.170471+0000", 2024-06-27T21:48:22.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012564779999999999 2024-06-27T21:48:22.459 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.459 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:100065 8.4 8:2399fd11:::benchmark_data_smithi192_167666_object20627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:29.129048+0000", 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 113.252600864, 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29965223299999999, 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.460 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:22.461 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:22.461 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:22.461 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:22.461 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 100065 2024-06-27T21:48:22.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.461 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129048+0000", 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129048+0000", 2024-06-27T21:48:22.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129051+0000", 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5940000000000001e-06 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129062+0000", 2024-06-27T21:48:22.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1556000000000001e-05 2024-06-27T21:48:22.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129101+0000", 2024-06-27T21:48:22.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8436000000000003e-05 2024-06-27T21:48:22.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129122+0000", 2024-06-27T21:48:22.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1066000000000001e-05 2024-06-27T21:48:22.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129245+0000", 2024-06-27T21:48:22.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012339 2024-06-27T21:48:22.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129373+0000", 2024-06-27T21:48:22.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128217 2024-06-27T21:48:22.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,10", 2024-06-27T21:48:22.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129486+0000", 2024-06-27T21:48:22.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011226399999999999 2024-06-27T21:48:22.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:22.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.131155+0000", 2024-06-27T21:48:22.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016692510000000001 2024-06-27T21:48:22.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:22.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.131180+0000", 2024-06-27T21:48:22.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4938000000000001e-05 2024-06-27T21:48:22.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:22.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.428630+0000", 2024-06-27T21:48:22.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29745006400000001 2024-06-27T21:48:22.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:22.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.428668+0000", 2024-06-27T21:48:22.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7716000000000003e-05 2024-06-27T21:48:22.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.428700+0000", 2024-06-27T21:48:22.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2740999999999997e-05 2024-06-27T21:48:22.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.473 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.474 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(8.4 e879,873 8:221570d9:::benchmark_data_smithi192_167666_object17847:head,865'4997)", 2024-06-27T21:48:22.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:48:22.474 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 87.686142985999993, 2024-06-27T21:48:22.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.2588125, 2024-06-27T21:48:22.475 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.475 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.475 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:48:22.475 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:48:22.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4700000000000001e-07 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695511+0000", 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5020000000000003e-06 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954076+0000", 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25856520100000002 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954083+0000", 2024-06-27T21:48:22.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1980000000000004e-06 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954136+0000", 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2806999999999998e-05 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954318+0000", 2024-06-27T21:48:22.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018224499999999999 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:129217 8.4 8:273ad32a:::benchmark_data_smithi192_167666_object49779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.244361+0000", 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 76.137287946000001, 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34760285299999999, 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.480 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129217 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244361+0000", 2024-06-27T21:48:22.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244361+0000", 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244362+0000", 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0920000000000001e-06 2024-06-27T21:48:22.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244367+0000", 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4689999999999999e-06 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244378+0000", 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1639e-05 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244386+0000", 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3799999999999996e-06 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244433+0000", 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7658999999999999e-05 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244517+0000", 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3786e-05 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 10,11", 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244588+0000", 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1005000000000002e-05 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:22.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246288+0000", 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001699823 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246855+0000", 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00056720100000000003 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:22.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591902+0000", 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34504665400000001 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591935+0000", 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3105000000000002e-05 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591964+0000", 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.904e-05 2024-06-27T21:48:22.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:129217 8.4 e889/881 ondisk, result = 0)", 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.246606+0000", 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 76.135042622, 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34536166299999999, 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.488 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246606+0000", 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246606+0000", 2024-06-27T21:48:22.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246607+0000", 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1600000000000001e-07 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246610+0000", 2024-06-27T21:48:22.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3739999999999998e-06 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591854+0000", 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34524346299999997 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591861+0000", 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.002e-06 2024-06-27T21:48:22.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591894+0000", 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2817000000000001e-05 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591898+0000", 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3510000000000004e-06 2024-06-27T21:48:22.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591968+0000", 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.004e-05 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:129225 8.16 e889/876 8:689e3fb0:::benchmark_data_smithi192_167666_object49787:head v 889'5801, mlcod=889'5798)", 2024-06-27T21:48:22.493 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.253857+0000", 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 76.127791497999993, 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.455897161, 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253857+0000", 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253857+0000", 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253858+0000", 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.51e-07 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253863+0000", 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9799999999999998e-06 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708079+0000", 2024-06-27T21:48:22.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.45421639400000002 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708086+0000", 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6290000000000004e-06 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708130+0000", 2024-06-27T21:48:22.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4048000000000002e-05 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708145+0000", 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.521e-05 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.709719+0000", 2024-06-27T21:48:22.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001573577 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.709755+0000", 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5772000000000002e-05 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.499 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:138829 8.11 e895/846 8:8e75fe10:::benchmark_data_smithi192_167666_object59391:head v 895'6216, mlcod=895'6213)", 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:19.559264+0000", 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 62.822384515000003, 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25218020499999999, 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:22.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559264+0000", 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559264+0000", 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559265+0000", 2024-06-27T21:48:22.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8800000000000002e-07 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559271+0000", 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8329999999999999e-06 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809754+0000", 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.250483067 2024-06-27T21:48:22.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809761+0000", 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4489999999999996e-06 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809818+0000", 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7824000000000001e-05 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809847+0000", 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8424000000000001e-05 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.811416+0000", 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00156879 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:22.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.811445+0000", 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9030000000000002e-05 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:22.505 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:48:22.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:22.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:22.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:22.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:22.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:22.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:22.850 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:22.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:22.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:22.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:22.989 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-06-27T21:48:22.989 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:48:23.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:23.032 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:23.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:23.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:23.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:48:23.258 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:22 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:22 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:22 smithi190 ceph-mon[32247]: osdmap e922: 12 total, 12 up, 12 in 2024-06-27T21:48:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3323689112' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:48:23.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/880347385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:48:23.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:22 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:22 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:22 smithi192 ceph-mon[32956]: osdmap e922: 12 total, 12 up, 12 in 2024-06-27T21:48:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3323689112' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:48:23.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/880347385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:48:23.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:22 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:23.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:22 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:23.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:22 smithi151 ceph-mon[28965]: osdmap e922: 12 total, 12 up, 12 in 2024-06-27T21:48:23.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3323689112' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-06-27T21:48:23.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/880347385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:48:23.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:23.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:23.457 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:23.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:23.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:23.634 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:23.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:23.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:23.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:23.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:23.911 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:23.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:24.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:24.038 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:24.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:23 smithi151 ceph-mon[28965]: pgmap v2250: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:24.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:23 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:24.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3323689112' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:48:24.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/880347385' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:48:24.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:23 smithi151 ceph-mon[28965]: osdmap e923: 12 total, 12 up, 12 in 2024-06-27T21:48:24.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3652899640' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:48:24.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:24.165 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:24.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:23 smithi190 ceph-mon[32247]: pgmap v2250: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:24.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:23 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:24.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3323689112' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:48:24.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/880347385' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:48:24.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:23 smithi190 ceph-mon[32247]: osdmap e923: 12 total, 12 up, 12 in 2024-06-27T21:48:24.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3652899640' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:48:24.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:24.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:23 smithi192 ceph-mon[32956]: pgmap v2250: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:23 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3323689112' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-06-27T21:48:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/880347385' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:48:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:23 smithi192 ceph-mon[32956]: osdmap e923: 12 total, 12 up, 12 in 2024-06-27T21:48:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3652899640' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:48:24.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:24.410 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-06-27T21:48:24.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:24.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:24.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:24.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:24.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:24.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:24.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:24.820 DEBUG:teuthology.orchestra.run.smithi192:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-06-27T21:48:24.823 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-06-27T21:48:24.823 INFO:teuthology.orchestra.run:waiting for 3000 2024-06-27T21:48:24.886 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:hints = 1 2024-06-27T21:48:24.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-06-27T21:48:24.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object prefix: benchmark_data_smithi192_179902 2024-06-27T21:48:24.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:48:24.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 0 0 0 0 0 0 - 0 2024-06-27T21:48:24.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:24.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:24.921 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:25.022 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:25.027 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:48:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:48:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:48:25.028 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:48:25.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:25.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:25.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:25.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 10 1 2024-06-27T21:48:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3652899640' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:48:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:25 smithi190 ceph-mon[32247]: osdmap e924: 12 total, 12 up, 12 in 2024-06-27T21:48:25.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:25 smithi190 ceph-mon[32247]: osdmap e925: 12 total, 12 up, 12 in 2024-06-27T21:48:25.270 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3652899640' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:48:25.270 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:25 smithi151 ceph-mon[28965]: osdmap e924: 12 total, 12 up, 12 in 2024-06-27T21:48:25.270 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:25 smithi151 ceph-mon[28965]: osdmap e925: 12 total, 12 up, 12 in 2024-06-27T21:48:25.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3652899640' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:48:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:25 smithi192 ceph-mon[32956]: osdmap e924: 12 total, 12 up, 12 in 2024-06-27T21:48:25.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:25 smithi192 ceph-mon[32956]: osdmap e925: 12 total, 12 up, 12 in 2024-06-27T21:48:25.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:25.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:25.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:25.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:25.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:25.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:25.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:25.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:48:25.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:25.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:25.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 1 16 1674 1658 103.619 103.625 0.0339315 0.00957281 2024-06-27T21:48:25.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:25.936 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:25.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:25.997 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:26.002 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:26 smithi190 ceph-mon[32247]: pgmap v2253: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:26.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:26 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:48:26.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:26 smithi190 ceph-mon[32247]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:48:26.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:26.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:26 smithi151 ceph-mon[28965]: pgmap v2253: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:26.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:26 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:48:26.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:26 smithi151 ceph-mon[28965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:48:26.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:26.022 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:26.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:26.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:26.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:26.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:26.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:26.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:26 smithi192 ceph-mon[32956]: pgmap v2253: 47 pgs: 47 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:48:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:26 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:48:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:26 smithi192 ceph-mon[32956]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-27T21:48:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:26.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:26.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:26.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:26.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:26.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:26.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:26.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:26.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:26.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:26.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:26.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 2 16 3437 3421 106.899 110.188 0.00292914 0.0093402 2024-06-27T21:48:26.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:26.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:26.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:27.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:27.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:27.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:27.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:27.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:27 smithi190 ceph-mon[32247]: osdmap e926: 12 total, 12 up, 12 in 2024-06-27T21:48:27.266 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:27 smithi151 ceph-mon[28965]: osdmap e926: 12 total, 12 up, 12 in 2024-06-27T21:48:27.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:27 smithi192 ceph-mon[32956]: osdmap e926: 12 total, 12 up, 12 in 2024-06-27T21:48:27.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:27.351 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:27.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:27.476 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:27.516 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:27.578 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:27.599 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:27.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:27.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:27.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:48:27.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:27.849 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:48:27.849 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:48:27.850 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:48:27.850 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:48:27.850 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:48:27.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:27.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:27.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 3 16 5271 5255 109.47 114.625 0.0033726 0.00909655 2024-06-27T21:48:27.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:28.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:28.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:28.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:28.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:48:28.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:48:28.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:28.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:48:28.241 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:48:28.241 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:28 smithi151 ceph-mon[28965]: pgmap v2256: 47 pgs: 47 active+clean; 9.9 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 37 op/s 2024-06-27T21:48:28.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:28 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:28.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:28 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:28.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:28 smithi151 ceph-mon[28965]: osdmap e927: 12 total, 12 up, 12 in 2024-06-27T21:48:28.242 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2112289310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:28.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:28 smithi190 ceph-mon[32247]: pgmap v2256: 47 pgs: 47 active+clean; 9.9 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 37 op/s 2024-06-27T21:48:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:28 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:28 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:28 smithi190 ceph-mon[32247]: osdmap e927: 12 total, 12 up, 12 in 2024-06-27T21:48:28.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2112289310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:28.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:48:28.268 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:48:28.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:48:28.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:48:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:28 smithi192 ceph-mon[32956]: pgmap v2256: 47 pgs: 47 active+clean; 9.9 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s wr, 37 op/s 2024-06-27T21:48:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:28 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:28 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:28 smithi192 ceph-mon[32956]: osdmap e927: 12 total, 12 up, 12 in 2024-06-27T21:48:28.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2112289310' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:28.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:48:28.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:48:28.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:48:28.336 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:48:28.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:48:28.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:48:28.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:28.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:48:28.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:48:28.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:28.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:48:28.403 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:48:28.424 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:48:28.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:48:28.451 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:48:28.452 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:48:28.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:48:28.478 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:48:28.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:28.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:28.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:48:28.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.640 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:28.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:28.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:28.768 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:28.820 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:28.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:28.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 4 16 6608 6592 102.992 83.5625 0.00820087 0.00968264 2024-06-27T21:48:28.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:28.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:28.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:29.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:29.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:29.158 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:29.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:29.286 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:29.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:29.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:29.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:29.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:29.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:29.661 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:29.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:29.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:29.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:29.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 5 15 8273 8258 103.217 104.125 0.0166958 0.00966669 2024-06-27T21:48:29.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:29.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:29.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:29.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:30.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:30.045 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:30.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:30.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:30.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:30 smithi151 ceph-mon[28965]: pgmap v2258: 47 pgs: 47 active+clean; 148 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 543 op/s 2024-06-27T21:48:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:30 smithi190 ceph-mon[32247]: pgmap v2258: 47 pgs: 47 active+clean; 148 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 543 op/s 2024-06-27T21:48:30.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:30.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:30 smithi192 ceph-mon[32956]: pgmap v2258: 47 pgs: 47 active+clean; 148 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 543 op/s 2024-06-27T21:48:30.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:30.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:30.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:30.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:30.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:30.543 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:30.544 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:30.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:30.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:30.681 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:30.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:30.813 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:30.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 6 15 10023 10008 104.242 109.375 0.00785757 0.00958129 2024-06-27T21:48:30.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:30.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:30.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:31.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:31.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:31.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:31.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:31.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:31.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:31.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:48:31.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:31.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:31.548 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:31.576 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:31.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:31.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:48:31.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:31.736 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:31.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:31.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:31.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 7 16 11756 11740 104.814 108.25 0.00307082 0.00953044 2024-06-27T21:48:31.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:31.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:31.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:32.021 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-06-27T21:48:32.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:32.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:32.117 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:32 smithi190 ceph-mon[32247]: pgmap v2259: 47 pgs: 47 active+clean; 357 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-06-27T21:48:32.117 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:32 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:48:32.118 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3665813336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:48:32.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:32.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:32 smithi192 ceph-mon[32956]: pgmap v2259: 47 pgs: 47 active+clean; 357 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-06-27T21:48:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:32 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:48:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3665813336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:48:32.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:32.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:32.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:32 smithi151 ceph-mon[28965]: pgmap v2259: 47 pgs: 47 active+clean; 357 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-06-27T21:48:32.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:32 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:48:32.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3665813336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:48:32.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:32.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:32.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:32.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:32.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:32.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:32.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:32.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:32.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 8 16 12780 12764 99.7114 64 0.00252299 0.0100193 2024-06-27T21:48:32.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:32.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:32.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:33.032 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:48:33.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:33.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:33.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:33.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:33.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3665813336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:48:33.251 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:33 smithi190 ceph-mon[32247]: osdmap e928: 12 total, 12 up, 12 in 2024-06-27T21:48:33.251 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3758864216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:48:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3665813336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:48:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:33 smithi192 ceph-mon[32956]: osdmap e928: 12 total, 12 up, 12 in 2024-06-27T21:48:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3758864216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:48:33.315 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:33.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:33.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3665813336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:48:33.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:33 smithi151 ceph-mon[28965]: osdmap e928: 12 total, 12 up, 12 in 2024-06-27T21:48:33.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3758864216' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:48:33.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 2 1 2024-06-27T21:48:33.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:33.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:33.494 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:33.495 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:33.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:33.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:33.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:33.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:33.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:33.845 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:33.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 9 16 13971 13955 96.9025 74.4375 0.0078439 0.0103022 2024-06-27T21:48:33.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:33.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:33.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:34.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:34.091 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:34.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:34.115 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:34.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:34 smithi190 ceph-mon[32247]: pgmap v2261: 47 pgs: 47 active+clean; 421 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:48:34.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:34 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:34.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3758864216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:48:34.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:34 smithi190 ceph-mon[32247]: osdmap e929: 12 total, 12 up, 12 in 2024-06-27T21:48:34.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:34 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:48:34.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:34 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:48:34.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:34.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:48:34.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:34 smithi192 ceph-mon[32956]: pgmap v2261: 47 pgs: 47 active+clean; 421 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:48:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:34 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3758864216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:48:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:34 smithi192 ceph-mon[32956]: osdmap e929: 12 total, 12 up, 12 in 2024-06-27T21:48:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:34 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:48:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:34 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:48:34.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:34.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:34.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:34 smithi151 ceph-mon[28965]: pgmap v2261: 47 pgs: 47 active+clean; 421 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:48:34.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:34 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:34.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3758864216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:48:34.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:34 smithi151 ceph-mon[28965]: osdmap e929: 12 total, 12 up, 12 in 2024-06-27T21:48:34.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:34 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:48:34.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:34 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:48:34.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.493 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:34.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:34.555 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:34.594 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:34.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:34.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:34.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:34.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:34.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 10 16 15592 15576 97.3428 101.312 0.0213108 0.010257 2024-06-27T21:48:34.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:34.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:34.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:35.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:35.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:35.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:35.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:35.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:35.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:35.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:35.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:35.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:35.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:35.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:35.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:35.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:35.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:35.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:35.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 11 16 17182 17166 97.5269 99.375 0.0179911 0.0102436 2024-06-27T21:48:35.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:36.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:36.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:48:36.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:36 smithi151 ceph-mon[28965]: pgmap v2263: 47 pgs: 47 active+clean; 689 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-06-27T21:48:36.106 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:36 smithi190 ceph-mon[32247]: pgmap v2263: 47 pgs: 47 active+clean; 689 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-06-27T21:48:36.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:36.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:36.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:36.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:36 smithi192 ceph-mon[32956]: pgmap v2263: 47 pgs: 47 active+clean; 689 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.48k op/s 2024-06-27T21:48:36.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:36.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:36.436 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:36.436 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:36.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:36.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:36.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:36.656 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:36.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:36.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:36.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:36.887 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 12 16 18688 18672 97.2427 94.125 0.0342369 0.0102718 2024-06-27T21:48:36.909 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:36.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:37.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:37.035 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:37.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:48:37.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:37.160 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:37.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:37 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:48:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:37 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:48:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:37.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:37 smithi151 ceph-mon[28965]: osdmap e930: 12 total, 12 up, 12 in 2024-06-27T21:48:37.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:37 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:48:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:37 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:48:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:37.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:37 smithi190 ceph-mon[32247]: osdmap e930: 12 total, 12 up, 12 in 2024-06-27T21:48:37.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:37.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:37.418 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:37.420 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:48:37.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:37.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:37 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:48:37.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:37 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:48:37.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:37.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:37.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:37 smithi192 ceph-mon[32956]: osdmap e930: 12 total, 12 up, 12 in 2024-06-27T21:48:37.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:37.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:37.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:37.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:37.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:37.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 13 16 20353 20337 97.7667 104.062 0.00373077 0.0102177 2024-06-27T21:48:37.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:37.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:37.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:38.016 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:38.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:38.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:38.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:38 smithi190 ceph-mon[32247]: pgmap v2265: 47 pgs: 47 active+clean; 887 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:48:38.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:38 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:48:38.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:38 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:48:38.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:38 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:48:38.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:38 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:48:38.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:38.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:38.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:38 smithi151 ceph-mon[28965]: pgmap v2265: 47 pgs: 47 active+clean; 887 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:48:38.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:38 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:48:38.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:38 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:48:38.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:38 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:48:38.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:38 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:48:38.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:38.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:38.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:38.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:38.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:38 smithi192 ceph-mon[32956]: pgmap v2265: 47 pgs: 47 active+clean; 887 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:48:38.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:38 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:48:38.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:38 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:48:38.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:38 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:48:38.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:38 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:48:38.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:38.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:38.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:38.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:38.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:38.881 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:38.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:38.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 14 16 22343 22327 99.6666 124.375 0.00776869 0.0100237 2024-06-27T21:48:38.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:39.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:39.054 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-06-27T21:48:39.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:39.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:39.180 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3360490593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:48:39.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:48:39.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:48:39.239 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:39.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:48:39.250 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:48:39.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:39.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:48:39.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:48:39.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:48:39.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:48:39.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3360490593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:48:39.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:48:39.334 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:48:39.360 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:48:39.361 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:48:39.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:39.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:48:39.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:48:39.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:39.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:48:39.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:48:39.443 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:48:39.444 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:48:39.460 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:48:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:48:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:48:39.460 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.d 836/821 [PushOp(8:b1be6d51:::benchmark_data_smithi192_167666_object19431:head, version: 812'1214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:b1be6d51:::benchmark_data_smithi192_167666_object19431:head@812'1214, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:16.435593+0000", 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.02205052400001, 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35761547199999999, 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.461 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435593+0000", 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435593+0000", 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435594+0000", 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9500000000000002e-07 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435617+0000", 2024-06-27T21:48:39.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2955999999999998e-05 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.791922+0000", 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35630468100000001 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.791935+0000", 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3341e-05 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.792038+0000", 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010332600000000001 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.792054+0000", 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.571e-05 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.793209+0000", 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011548629999999999 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.465 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:79966 8.15 e852/844 8:aa0046c5:::benchmark_data_smithi192_167666_object528:head v 852'4206, mlcod=852'4203)", 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:50.771108+0000", 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 168.68653603199999, 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23469005200000001, 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:50.771108+0000", 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:50.771108+0000", 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:50.771109+0000", 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.083e-06 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:50.771120+0000", 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0635999999999999e-05 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.004619+0000", 2024-06-27T21:48:39.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23349925899999999 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.004624+0000", 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8910000000000003e-06 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.004680+0000", 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6379999999999999e-05 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.004697+0000", 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6390999999999999e-05 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.005747+0000", 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001050165 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.005798+0000", 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1246999999999999e-05 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:80108 8.15 e853/844 8:acb44bb1:::benchmark_data_smithi192_167666_object670:head v 853'4211, mlcod=853'4208)", 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:51.208345+0000", 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 168.24929909299999, 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24365014600000001, 2024-06-27T21:48:39.471 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208345+0000", 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208345+0000", 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208346+0000", 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208355+0000", 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4450000000000002e-06 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450867+0000", 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24251208199999999 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450874+0000", 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6030000000000001e-06 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450916+0000", 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2466999999999998e-05 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450931+0000", 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4611e-05 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.451965+0000", 2024-06-27T21:48:39.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001034594 2024-06-27T21:48:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.451995+0000", 2024-06-27T21:48:39.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9484000000000001e-05 2024-06-27T21:48:39.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:48:39.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:48:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.478 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:81055 8.15 e855/844 8:ab25a5cb:::benchmark_data_smithi192_167666_object1617:head v 855'4241, mlcod=855'4238)", 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:53.977158+0000", 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 165.48048585500001, 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.369013596, 2024-06-27T21:48:39.479 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977158+0000", 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977158+0000", 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977159+0000", 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.173e-06 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977166+0000", 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0979999999999996e-06 2024-06-27T21:48:39.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344918+0000", 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36775122500000001 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344922+0000", 2024-06-27T21:48:39.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.622e-06 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344967+0000", 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4387e-05 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344982+0000", 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5218000000000001e-05 2024-06-27T21:48:39.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.346142+0000", 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011605179999999999 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.346172+0000", 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9354999999999999e-05 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:81708 8.10 8:0a177ce8:::benchmark_data_smithi192_167666_object2270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 163.65040433999999, 2024-06-27T21:48:39.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26758338500000001, 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 81708 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1800000000000005e-07 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807244+0000", 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.771e-06 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807253+0000", 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1749999999999994e-06 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807258+0000", 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1059999999999999e-06 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807288+0000", 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9498000000000001e-05 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807366+0000", 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8258999999999997e-05 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:48:39.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807434+0000", 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7651999999999995e-05 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808283+0000", 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00084945300000000001 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808559+0000", 2024-06-27T21:48:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027518600000000002 2024-06-27T21:48:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074743+0000", 2024-06-27T21:48:39.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.266184639 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074777+0000", 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3547000000000003e-05 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074823+0000", 2024-06-27T21:48:39.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6281e-05 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:81708 8.10 e855/854 ondisk, result = 0)", 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 163.649313224, 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26651063000000003, 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.494 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7700000000000004e-07 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808335+0000", 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4189999999999998e-06 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074678+0000", 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26634306400000002 2024-06-27T21:48:39.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074684+0000", 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4980000000000003e-06 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074735+0000", 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0504000000000003e-05 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074739+0000", 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6859999999999997e-06 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074841+0000", 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101882 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.498 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:81708 8.10 8:0a177ce8:::benchmark_data_smithi192_167666_object2270:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 163.64439183900001, 2024-06-27T21:48:39.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26153279400000001, 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for ondisk", 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 81708 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.500 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0899999999999997e-07 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813254+0000", 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.539e-06 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813259+0000", 2024-06-27T21:48:39.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4829999999999999e-06 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813263+0000", 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1379999999999999e-06 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813357+0000", 2024-06-27T21:48:39.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4815000000000006e-05 2024-06-27T21:48:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for ondisk", 2024-06-27T21:48:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813391+0000", 2024-06-27T21:48:39.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4125000000000003e-05 2024-06-27T21:48:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074785+0000", 2024-06-27T21:48:39.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26139338499999998 2024-06-27T21:48:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:81976 8.10 8:0e64fe29:::benchmark_data_smithi192_167666_object2538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:56.390372+0000", 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 163.06727172399999, 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24159777099999999, 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.507 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 81976 2024-06-27T21:48:39.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390372+0000", 2024-06-27T21:48:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390372+0000", 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390374+0000", 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.316e-06 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390378+0000", 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8760000000000004e-06 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390390+0000", 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1409999999999999e-05 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390396+0000", 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6789999999999999e-06 2024-06-27T21:48:39.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390446+0000", 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9728000000000002e-05 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390528+0000", 2024-06-27T21:48:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2253e-05 2024-06-27T21:48:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:48:39.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.390599+0000", 2024-06-27T21:48:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0073000000000005e-05 2024-06-27T21:48:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.391702+0000", 2024-06-27T21:48:39.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001103368 2024-06-27T21:48:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.392142+0000", 2024-06-27T21:48:39.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044003800000000002 2024-06-27T21:48:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631893+0000", 2024-06-27T21:48:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.239751363 2024-06-27T21:48:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631939+0000", 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6171999999999998e-05 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631970+0000", 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0494999999999999e-05 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:81976 8.10 e855/854 ondisk, result = 0)", 2024-06-27T21:48:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:56.391959+0000", 2024-06-27T21:48:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 163.06568509799999, 2024-06-27T21:48:39.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24001497899999999, 2024-06-27T21:48:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.391959+0000", 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.391959+0000", 2024-06-27T21:48:39.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.391960+0000", 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2699999999999998e-07 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.391963+0000", 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3610000000000001e-06 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631816+0000", 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.239852551 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631822+0000", 2024-06-27T21:48:39.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3860000000000004e-06 2024-06-27T21:48:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631880+0000", 2024-06-27T21:48:39.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7753000000000001e-05 2024-06-27T21:48:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631887+0000", 2024-06-27T21:48:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8560000000000001e-06 2024-06-27T21:48:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.631974+0000", 2024-06-27T21:48:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7145000000000007e-05 2024-06-27T21:48:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.12 856/836 [PushOp(8:4b894608:::benchmark_data_smithi192_167666_object27253:head, version: 818'1716, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4b894608:::benchmark_data_smithi192_167666_object27253:head@818'1716, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 161.98755925200001, 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46211215900000002, 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.528 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:48:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:48:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:48:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8100000000000002e-07 2024-06-27T21:48:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470131+0000", 2024-06-27T21:48:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5881999999999999e-05 2024-06-27T21:48:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931108+0000", 2024-06-27T21:48:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46097701899999999 2024-06-27T21:48:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931116+0000", 2024-06-27T21:48:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6079999999999998e-06 2024-06-27T21:48:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931191+0000", 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5159000000000003e-05 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931205+0000", 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.384e-05 2024-06-27T21:48:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.932197+0000", 2024-06-27T21:48:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00099186999999999995 2024-06-27T21:48:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(8.16 e857/839 cost 1049576)", 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:58.249383+0000", 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 161.20826054700001, 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.737237174, 2024-06-27T21:48:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249383+0000", 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249383+0000", 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249384+0000", 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249393+0000", 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6130000000000004e-06 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986132+0000", 2024-06-27T21:48:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73673969100000003 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986138+0000", 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986185+0000", 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6563999999999997e-05 2024-06-27T21:48:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986621+0000", 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043603799999999997 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.17 857/836 [PushOp(8:ec39b37b:::benchmark_data_smithi192_167666_object21643:head, version: 814'1359, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ec39b37b:::benchmark_data_smithi192_167666_object21643:head@814'1359, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:59.975389+0000", 2024-06-27T21:48:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 159.48225481200001, 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30655363200000002, 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975389+0000", 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975389+0000", 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975391+0000", 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.477e-06 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975450+0000", 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9800999999999999e-05 2024-06-27T21:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280774+0000", 2024-06-27T21:48:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30532361099999999 2024-06-27T21:48:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:39.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3360490593' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:48:39.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:48:39.547 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280792+0000", 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8151e-05 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280899+0000", 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106951 2024-06-27T21:48:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280912+0000", 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3278000000000001e-05 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.281943+0000", 2024-06-27T21:48:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010303630000000001 2024-06-27T21:48:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.17 857/836 [PushOp(8:e964d975:::benchmark_data_smithi192_167666_object23698:head, version: 817'1491, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e964d975:::benchmark_data_smithi192_167666_object23698:head@817'1491, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:01.507697+0000", 2024-06-27T21:48:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 157.94994706, 2024-06-27T21:48:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38872538699999998, 2024-06-27T21:48:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507697+0000", 2024-06-27T21:48:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507697+0000", 2024-06-27T21:48:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507698+0000", 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.446e-06 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507760+0000", 2024-06-27T21:48:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2121999999999999e-05 2024-06-27T21:48:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895295+0000", 2024-06-27T21:48:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38753470499999998 2024-06-27T21:48:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895310+0000", 2024-06-27T21:48:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5237e-05 2024-06-27T21:48:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895392+0000", 2024-06-27T21:48:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1422999999999996e-05 2024-06-27T21:48:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895400+0000", 2024-06-27T21:48:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4780000000000008e-06 2024-06-27T21:48:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.896422+0000", 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001021976 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.17 857/836 [PushOp(8:ed779c3f:::benchmark_data_smithi192_167666_object27105:head, version: 818'1718, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ed779c3f:::benchmark_data_smithi192_167666_object27105:head@818'1718, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:03.393984+0000", 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 156.06366000200001, 2024-06-27T21:48:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52261519899999997, 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.393984+0000", 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.393984+0000", 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.393985+0000", 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2200000000000003e-07 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.394077+0000", 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2316000000000002e-05 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915130+0000", 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52105278899999996 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915146+0000", 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6432000000000001e-05 2024-06-27T21:48:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915249+0000", 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010308500000000001 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915267+0000", 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8003e-05 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.916599+0000", 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013318519999999999 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:83642 8.d e858/838 8:bd2afb08:::benchmark_data_smithi192_167666_object4204:head v 858'5247, mlcod=831'3586)", 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:06.532557+0000", 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 152.92508694599999, 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50356170499999997, 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532557+0000", 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532557+0000", 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532558+0000", 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.78e-07 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532569+0000", 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1034e-05 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034199+0000", 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50163044199999995 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034206+0000", 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2110000000000003e-06 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034255+0000", 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8973e-05 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034275+0000", 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0055999999999999e-05 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.036072+0000", 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001797416 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.036119+0000", 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6694999999999999e-05 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:94715 8.10 8:0b89cc6a:::benchmark_data_smithi192_167666_object15277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:23.482765+0000", 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 135.974878964, 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27123174100000003, 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 94715 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482765+0000", 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482765+0000", 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482766+0000", 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1790000000000001e-06 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482774+0000", 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5020000000000003e-06 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482787+0000", 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3838e-05 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482795+0000", 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1489999999999997e-06 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482860+0000", 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5247999999999996e-05 2024-06-27T21:48:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482988+0000", 2024-06-27T21:48:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012832699999999999 2024-06-27T21:48:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.483102+0000", 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011404900000000001 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.484543+0000", 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00144103 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.484860+0000", 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031666700000000002 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753930+0000", 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26906978100000001 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753963+0000", 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3531000000000001e-05 2024-06-27T21:48:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753997+0000", 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3439999999999998e-05 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:100117 8.17 8:eacbdbc0:::benchmark_data_smithi192_167666_object20679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e868)", 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:29.525411+0000", 2024-06-27T21:48:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 129.93223276399999, 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33249853000000001, 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 100117 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525411+0000", 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525411+0000", 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525412+0000", 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2600000000000002e-07 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525416+0000", 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8689999999999997e-06 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525427+0000", 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1401e-05 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525441+0000", 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4335000000000001e-05 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525592+0000", 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015089500000000001 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525698+0000", 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010518699999999999 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,8", 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525802+0000", 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104598 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.528214+0000", 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024122800000000002 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.529086+0000", 2024-06-27T21:48:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087144999999999996 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857818+0000", 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.328732207 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857880+0000", 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2353000000000004e-05 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857910+0000", 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9229000000000001e-05 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:100117 8.17 e868/867 ondisk, result = 0)", 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:29.574463+0000", 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 129.883180458, 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.283453173, 2024-06-27T21:48:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574463+0000", 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574463+0000", 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574464+0000", 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.06e-07 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574467+0000", 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3969999999999998e-06 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857759+0000", 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.283291505 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857764+0000", 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5310000000000003e-06 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857802+0000", 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8501999999999998e-05 2024-06-27T21:48:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857811+0000", 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1090000000000004e-06 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857917+0000", 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105523 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:103778 8.12 8:4addb60a:::benchmark_data_smithi192_167666_object24340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e872)", 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:33.717225+0000", 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 125.74041930999999, 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.261469805, 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:48:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 103778 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717225+0000", 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717225+0000", 2024-06-27T21:48:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717226+0000", 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2109999999999999e-06 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717230+0000", 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000003e-06 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717242+0000", 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2055999999999999e-05 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717248+0000", 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6829999999999996e-06 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717306+0000", 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7979000000000002e-05 2024-06-27T21:48:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717438+0000", 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013185200000000001 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,10", 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717562+0000", 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123843 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:48:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719019+0000", 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001457161 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719447+0000", 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042779800000000001 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978505+0000", 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25905842699999998 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978593+0000", 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7960999999999994e-05 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978694+0000", 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010153399999999999 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:103778 8.12 e872/864 ondisk, result = 0)", 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:33.719231+0000", 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 125.738412782, 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25947442999999998, 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719231+0000", 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719231+0000", 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719232+0000", 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2900000000000002e-07 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719236+0000", 2024-06-27T21:48:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5130000000000001e-06 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978373+0000", 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25913739299999999 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978386+0000", 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2619999999999999e-05 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978475+0000", 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9701000000000001e-05 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978490+0000", 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4399000000000001e-05 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978706+0000", 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000215875 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:48:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:48:39.600 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:48:39.600 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:48:39.600 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:48:39.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:39.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:39.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:39.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:39.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:39.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:39.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 15 16 24012 23996 99.9758 104.312 0.0028701 0.00999005 2024-06-27T21:48:39.982 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:40.013 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:40.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:40.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:40.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:40.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:40.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:40.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:40 smithi151 ceph-mon[28965]: pgmap v2266: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-06-27T21:48:40.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3360490593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:48:40.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:40 smithi151 ceph-mon[28965]: osdmap e931: 12 total, 12 up, 12 in 2024-06-27T21:48:40.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:40 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2131642409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:40.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:40 smithi190 ceph-mon[32247]: pgmap v2266: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-06-27T21:48:40.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3360490593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:48:40.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:40 smithi190 ceph-mon[32247]: osdmap e931: 12 total, 12 up, 12 in 2024-06-27T21:48:40.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:40 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2131642409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:40.393 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:40.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 8 1 2024-06-27T21:48:40.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:48:40.464 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:40 smithi192 ceph-mon[32956]: pgmap v2266: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-06-27T21:48:40.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3360490593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:48:40.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:40 smithi192 ceph-mon[32956]: osdmap e931: 12 total, 12 up, 12 in 2024-06-27T21:48:40.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:40 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2131642409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:40.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:40.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:40.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:40.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:40.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:40.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:40.804 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:48:40.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:40.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 16 16 25526 25510 99.6409 94.625 0.00925022 0.0100256 2024-06-27T21:48:40.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:40.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:41.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:41.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:41.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:41.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:41.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:41.288 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:41 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:48:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:41 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:48:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:41 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:48:41.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:41 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:48:41.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:41.421 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:41 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:48:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:41 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:48:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:41 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:48:41.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:41 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:48:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:41 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:48:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:41 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:48:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:41 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:48:41.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:41 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:48:41.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:41.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:41.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:41.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:41.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:41.805 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:41.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 17 16 26899 26883 98.8271 85.8125 0.0177147 0.0100487 2024-06-27T21:48:41.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:41.935 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:41.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:42.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:42.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:42.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:42.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:42.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:42.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:42.397 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:42 smithi151 ceph-mon[28965]: pgmap v2268: 47 pgs: 47 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-06-27T21:48:42.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:42.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:42.452 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:42 smithi192 ceph-mon[32956]: pgmap v2268: 47 pgs: 47 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-06-27T21:48:42.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:42 smithi190 ceph-mon[32247]: pgmap v2268: 47 pgs: 47 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.64k op/s 2024-06-27T21:48:42.547 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:42.580 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:42.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:42.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:42.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:42.838 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:42.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 18 16 28763 28747 99.8084 116.5 0.0077943 0.0100114 2024-06-27T21:48:42.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:42.963 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:42.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:43.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:43.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:43.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:43 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:48:43.142 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:43 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:48:43.182 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:48:43.182 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:48:43.182 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:48:43.182 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:48:43.188 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:43.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:43.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:43 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:48:43.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:43 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:48:43.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:43.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:43.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:43.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:43.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:43 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:48:43.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:43 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:48:43.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:43.604 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:43.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:43.737 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:43.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:43.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:43.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 19 16 30569 30553 100.496 112.875 0.00985775 0.00994301 2024-06-27T21:48:43.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:43.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:48:43.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:43.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:44.069 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:48:44.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:44.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:44.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:44.262 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:44 smithi192 ceph-mon[32956]: pgmap v2269: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-27T21:48:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:44 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:48:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:44 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:48:44.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2592422771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:48:44.302 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:48:44.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:44 smithi151 ceph-mon[28965]: pgmap v2269: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-27T21:48:44.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:44 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:48:44.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:44 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:48:44.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2592422771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:48:44.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:44.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:44.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:44 smithi190 ceph-mon[32247]: pgmap v2269: 47 pgs: 47 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-06-27T21:48:44.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:44 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:48:44.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:44 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:48:44.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2592422771' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:48:44.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:44.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:44.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:44.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:44.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:44.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:44.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:48:44.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:48:44.888141+0000 min lat: 0.0010169 max lat: 0.260908 avg lat: 0.0099634 2024-06-27T21:48:44.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:48:44.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 20 16 32113 32097 100.295 96.5 0.0187521 0.0099634 2024-06-27T21:48:44.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:44.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:45.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:45.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:45.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:45.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:45 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:48:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:45 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:48:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2592422771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:48:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:45 smithi151 ceph-mon[28965]: osdmap e932: 12 total, 12 up, 12 in 2024-06-27T21:48:45.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:45 smithi151 ceph-mon[28965]: osdmap e933: 12 total, 12 up, 12 in 2024-06-27T21:48:45.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:45.287 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:45.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:45.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:45.499 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:45 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:48:45.499 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:45 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:48:45.499 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2592422771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:48:45.499 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:45 smithi192 ceph-mon[32956]: osdmap e932: 12 total, 12 up, 12 in 2024-06-27T21:48:45.499 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:45 smithi192 ceph-mon[32956]: osdmap e933: 12 total, 12 up, 12 in 2024-06-27T21:48:45.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:45 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:48:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:45 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:48:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2592422771' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:48:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:45 smithi190 ceph-mon[32247]: osdmap e932: 12 total, 12 up, 12 in 2024-06-27T21:48:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:45 smithi190 ceph-mon[32247]: osdmap e933: 12 total, 12 up, 12 in 2024-06-27T21:48:45.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:45.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:45.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:45.700 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:45.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:45.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:45.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:45.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 21 16 33270 33254 98.9626 72.3125 0.00322921 0.0100964 2024-06-27T21:48:45.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:45.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:46.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:46.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:46.086 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-06-27T21:48:46.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:46.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:46.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:46.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:46.330 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:48:46.330 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:48:46.330 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:48:46.330 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:48:46.330 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:48:46.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:46.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:46.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:46 smithi151 ceph-mon[28965]: pgmap v2271: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-27T21:48:46.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1974340681' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:48:46.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:46 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:48:46.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:46 smithi190 ceph-mon[32247]: pgmap v2271: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-27T21:48:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1974340681' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:48:46.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:46 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:48:46.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:46.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:46 smithi192 ceph-mon[32956]: pgmap v2271: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-06-27T21:48:46.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1974340681' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:48:46.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:46 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:48:46.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:46.674 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:46.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:46.808 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:46.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 22 16 34483 34467 97.9101 75.8125 0.0286052 0.0102069 2024-06-27T21:48:46.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:46.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:46.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:47.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:47.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:47.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:47.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:47 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:48:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:47 smithi151 ceph-mon[28965]: osdmap e934: 12 total, 12 up, 12 in 2024-06-27T21:48:47.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:47 smithi151 ceph-mon[28965]: pgmap v2274: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-27T21:48:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:47 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:48:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:47 smithi190 ceph-mon[32247]: osdmap e934: 12 total, 12 up, 12 in 2024-06-27T21:48:47.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:47 smithi190 ceph-mon[32247]: pgmap v2274: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-27T21:48:47.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:47.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:47.398 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:47 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:48:47.398 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:47 smithi192 ceph-mon[32956]: osdmap e934: 12 total, 12 up, 12 in 2024-06-27T21:48:47.398 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:47 smithi192 ceph-mon[32956]: pgmap v2274: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-06-27T21:48:47.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:47.440 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:48:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:47.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:47.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:47.567 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:47.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:48:47.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:47.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:47.731 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:48:47.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:47.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 1 1 2024-06-27T21:48:47.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:47.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 23 16 35689 35673 96.93 75.375 0.0118459 0.0103104 2024-06-27T21:48:47.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:47.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:47.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:48.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:48.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:48.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:48.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:48.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:48.339 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:48.340 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.348 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:48.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:48.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:48.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:48.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3487550198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:48.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:48.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:48.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:48.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:48.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3487550198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:48.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3487550198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:48.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:48.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:48.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 24 15 36941 36926 96.154 78.3125 0.00378658 0.01039 2024-06-27T21:48:48.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:48.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:48.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:49.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:49.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:49.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:48:49.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:48:49.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:48:49.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:48:49.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:48:49.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:48:49.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:49.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:48:49.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:48:49.236 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:49.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:48:49.238 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:48:49.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:48:49.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:48:49.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:48:49.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:48:49.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:48:49.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:48:49.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:49.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:48:49.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:48:49.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:49.379 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:48:49.379 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:48:49.410 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:48:49.410 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:48:49.410 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:49 smithi151 ceph-mon[28965]: pgmap v2275: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-06-27T21:48:49.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:48:49.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:48:49.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:49.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:49 smithi192 ceph-mon[32956]: pgmap v2275: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-06-27T21:48:49.552 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:49.552 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:49.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:49.623 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:49.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:49.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:49.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:49 smithi190 ceph-mon[32247]: pgmap v2275: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 MiB/s wr, 1.89k op/s 2024-06-27T21:48:49.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:49.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:49.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:49.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 25 16 38119 38103 95.2501 73.5625 0.0200704 0.0104912 2024-06-27T21:48:49.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:50.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:50.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:50.124 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:50.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:50.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:48:50.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:50.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:50.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:50.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:50.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:50.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:50.629 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:48:50.648 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:50.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:50.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:50.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:50.888 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 26 16 39597 39581 95.1394 92.375 0.00339218 0.0105044 2024-06-27T21:48:50.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:50.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:51.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:51.039 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:50 smithi192 ceph-mon[32956]: osdmap e935: 12 total, 12 up, 12 in 2024-06-27T21:48:51.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:51.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:50 smithi151 ceph-mon[28965]: osdmap e935: 12 total, 12 up, 12 in 2024-06-27T21:48:51.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:51.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:51.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:50 smithi190 ceph-mon[32247]: osdmap e935: 12 total, 12 up, 12 in 2024-06-27T21:48:51.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:51.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:51.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:51.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:51.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:51.570 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:51.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:51.702 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:51.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:51.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:51.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 27 16 41129 41113 95.1616 95.75 0.00594381 0.0104621 2024-06-27T21:48:51.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:51.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:51.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:52.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:52.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:52.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:51 smithi151 ceph-mon[28965]: pgmap v2277: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-27T21:48:52.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:48:52.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:48:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:52.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:52.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:51 smithi190 ceph-mon[32247]: pgmap v2277: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-27T21:48:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:48:52.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:48:52.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:51 smithi192 ceph-mon[32956]: pgmap v2277: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-06-27T21:48:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:48:52.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:48:52.323 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:52.347 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:52.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:52.480 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:52.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:52.614 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:52.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:52.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:52.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:52.859 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:52.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 28 16 42387 42371 94.5707 78.625 0.177427 0.0105668 2024-06-27T21:48:52.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:52.980 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:48:52.980 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-06-27T21:48:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:52.993 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:52.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:53.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:53.128 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:53.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:53.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:53.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:52 smithi190 ceph-mon[32247]: osdmap e936: 12 total, 12 up, 12 in 2024-06-27T21:48:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1359509711' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:48:53.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3909448446' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:48:53.266 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:52 smithi192 ceph-mon[32956]: osdmap e936: 12 total, 12 up, 12 in 2024-06-27T21:48:53.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1359509711' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:48:53.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3909448446' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:48:53.328 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:52 smithi151 ceph-mon[28965]: osdmap e936: 12 total, 12 up, 12 in 2024-06-27T21:48:53.328 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1359509711' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:48:53.328 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3909448446' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:48:53.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:53.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:53.421 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:53.421 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:53.496 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:53.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:53.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:48:53.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:53.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:53.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:53.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:53.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:53.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 29 16 43465 43449 93.6327 67.375 0.0635286 0.0106381 2024-06-27T21:48:53.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:53.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:54.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:54.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:54.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:48:54.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:54.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: pgmap v2279: 47 pgs: 47 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: 1.1a deep-scrub starts 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: 1.1a deep-scrub ok 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1359509711' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3909448446' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:48:54.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:53 smithi151 ceph-mon[28965]: osdmap e937: 12 total, 12 up, 12 in 2024-06-27T21:48:54.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:54.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: pgmap v2279: 47 pgs: 47 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:48:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: 1.1a deep-scrub starts 2024-06-27T21:48:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: 1.1a deep-scrub ok 2024-06-27T21:48:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1359509711' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:48:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3909448446' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:48:54.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:53 smithi190 ceph-mon[32247]: osdmap e937: 12 total, 12 up, 12 in 2024-06-27T21:48:54.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:54.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: pgmap v2279: 47 pgs: 47 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:48:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: 1.1a deep-scrub starts 2024-06-27T21:48:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: 1.1a deep-scrub ok 2024-06-27T21:48:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:48:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1359509711' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:48:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3909448446' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:48:54.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:53 smithi192 ceph-mon[32956]: osdmap e937: 12 total, 12 up, 12 in 2024-06-27T21:48:54.391 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:54.420 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:54.459 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:48:54.522 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:54.550 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:54.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 7 1 2024-06-27T21:48:54.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:54.675 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:54.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:54.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:54.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:54.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 30 16 44742 44726 93.1718 79.8125 0.00567627 0.0107246 2024-06-27T21:48:54.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:54.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:54.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:55.030 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:55.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:55.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:55.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:55.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:55.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:55.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:55.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:55.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:55.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:55.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:55.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:55.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:48:55.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:55.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:48:55.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 31 16 46380 46364 93.4684 102.375 0.00260988 0.0106926 2024-06-27T21:48:55.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:55.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:48:55.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:55.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:56.003 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:48:56.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:56.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:56.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:56.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:56.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:55 smithi190 ceph-mon[32247]: pgmap v2281: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:48:56.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:55 smithi151 ceph-mon[28965]: pgmap v2281: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:48:56.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:55 smithi192 ceph-mon[32956]: pgmap v2281: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:48:56.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:56.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:56.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:56.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:56.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:56.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:56.610 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:48:56.610 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:48:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:56.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:56.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:56.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:56.874 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:56.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:56.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 32 16 47734 47718 93.1918 84.625 0.00602193 0.0107255 2024-06-27T21:48:56.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:48:56.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:48:56.922 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:48:56.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:48:56.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:48:56.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:48:56.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:56.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:48:56.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:48:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:57.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:57.009 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:48:57.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:48:57.031 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:48:57.031 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:48:57.053 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:48:57.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:48:57.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:48:57.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:48:57.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:57.105 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:48:57.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:48:57.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2699170812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:57.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:57.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:48:57.127 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:48:57.149 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:48:57.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:48:57.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:48:57.170 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:48:57.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:48:57.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:48:57.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2699170812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:57.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2699170812' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:48:57.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:57.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:57.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:48:57.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:48:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:57.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:48:57.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:57.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:57.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:57.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:57.862 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:48:57.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:57.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 33 16 48995 48979 92.7559 78.8125 0.00387227 0.0107683 2024-06-27T21:48:57.905 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:57.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:58.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:58.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:58.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:58.155 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:58.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:48:58.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:48:57 smithi190 ceph-mon[32247]: pgmap v2282: 47 pgs: 47 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:48:58.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:48:58.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:48:57 smithi192 ceph-mon[32956]: pgmap v2282: 47 pgs: 47 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:48:58.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:58.407 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:58.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:48:57 smithi151 ceph-mon[28965]: pgmap v2282: 47 pgs: 47 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-06-27T21:48:58.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:58.534 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:58.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:58.660 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:58.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:48:58.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:48:58.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 34 16 50005 49989 91.8842 63.125 0.00238726 0.0108745 2024-06-27T21:48:58.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:48:58.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:58.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:48:59.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:59.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:59.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:48:59.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:48:59.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:48:59.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:48:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:48:59.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:48:59.428 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:48:59.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:59.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:48:59.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:48:59.685 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:48:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:48:59.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:48:59.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:48:59.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 35 16 51466 51450 91.8676 91.3125 0.00259487 0.0108733 2024-06-27T21:48:59.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:48:59.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:48:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:48:59.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:00.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:00.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:00.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:00.203 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:00 smithi190 ceph-mon[32247]: pgmap v2283: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:49:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:00 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:00 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:00.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:00 smithi190 ceph-mon[32247]: osdmap e938: 12 total, 12 up, 12 in 2024-06-27T21:49:00.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:00 smithi190 ceph-mon[32247]: osdmap e939: 12 total, 12 up, 12 in 2024-06-27T21:49:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:00 smithi192 ceph-mon[32956]: pgmap v2283: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:49:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:00 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:00 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:00 smithi192 ceph-mon[32956]: osdmap e938: 12 total, 12 up, 12 in 2024-06-27T21:49:00.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:00 smithi192 ceph-mon[32956]: osdmap e939: 12 total, 12 up, 12 in 2024-06-27T21:49:00.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:00.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:00.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:00.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:00 smithi151 ceph-mon[28965]: pgmap v2283: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-06-27T21:49:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:00 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:00 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:00 smithi151 ceph-mon[28965]: osdmap e938: 12 total, 12 up, 12 in 2024-06-27T21:49:00.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:00 smithi151 ceph-mon[28965]: osdmap e939: 12 total, 12 up, 12 in 2024-06-27T21:49:00.466 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:00.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:00.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:00.649 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:00.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:00.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:00.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:00.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:00.889 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 36 16 52872 52856 91.7565 87.875 0.0314994 0.0108901 2024-06-27T21:49:00.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:00.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:00.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:01.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:01.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:01.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:01.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:01.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:01.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:01.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:01.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:01.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:01.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:01.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:01.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:01.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:01.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:01.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:01.907 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 37 16 54335 54319 91.7036 91.4375 0.0134759 0.0108884 2024-06-27T21:49:01.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:02.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:02.094 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:02 smithi192 ceph-mon[32956]: pgmap v2286: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:49:02.094 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:02.094 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:02.094 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:02 smithi192 ceph-mon[32956]: osdmap e940: 12 total, 12 up, 12 in 2024-06-27T21:49:02.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGRecoveryDeleteReply(6.5 e707,698 6:a62b003a:::benchmark_data_smithi192_117994_object75931:head,643'10604)", 2024-06-27T21:49:02.134 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:14.231833+0000", 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 587.89949842999999, 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.88407170000000002, 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231833+0000", 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231833+0000", 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231834+0000", 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8800000000000002e-07 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:14.231837+0000", 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.63e-06 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115556+0000", 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.883718958 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115566+0000", 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0165e-05 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115625+0000", 2024-06-27T21:49:02.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8303000000000001e-05 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:15.115905+0000", 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028005599999999999 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7073 7.a 7:5854dda9:::benchmark_data_smithi192_150129_object7072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.138 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.702641+0000", 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.42869068300001, 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0929825260000001, 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7073 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702641+0000", 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702641+0000", 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702642+0000", 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0410000000000001e-06 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702663+0000", 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1299999999999999e-05 2024-06-27T21:49:02.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702677+0000", 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3947999999999999e-05 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702684+0000", 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.861e-06 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702737+0000", 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2877999999999998e-05 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.702943+0000", 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020602399999999999 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.703106+0000", 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016296600000000001 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.704211+0000", 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001105396 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.706633+0000", 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002422113 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795514+0000", 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.088880246 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795562+0000", 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8405000000000002e-05 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795623+0000", 2024-06-27T21:49:02.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1347999999999996e-05 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7081 7.a 7:589100a8:::benchmark_data_smithi192_150129_object7080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.705118+0000", 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.42621356799998, 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0906754319999998, 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.146 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7081 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705118+0000", 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705118+0000", 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705120+0000", 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7069999999999999e-06 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705149+0000", 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9462e-05 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705166+0000", 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.666e-05 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705174+0000", 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7179999999999996e-06 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705226+0000", 2024-06-27T21:49:02.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1969000000000003e-05 2024-06-27T21:49:02.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705359+0000", 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000133857 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.705538+0000", 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017908900000000001 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.706085+0000", 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054701999999999995 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.714773+0000", 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0086879260000000003 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795682+0000", 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0809086040000002 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795746+0000", 2024-06-27T21:49:02.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4462999999999996e-05 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.795793+0000", 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6956999999999998e-05 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7113 7.a 7:518e8ad9:::benchmark_data_smithi192_150129_object7112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.722268+0000", 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.40906322399996, 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0741637069999999, 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7113 2024-06-27T21:49:02.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722268+0000", 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722268+0000", 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722269+0000", 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.68e-07 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722291+0000", 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2226999999999999e-05 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722299+0000", 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0129999999999993e-06 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722305+0000", 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5999999999999997e-06 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722358+0000", 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2775999999999998e-05 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722499+0000", 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000140968 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.722636+0000", 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000137942 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.723239+0000", 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060242800000000001 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.724713+0000", 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001473741 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796327+0000", 2024-06-27T21:49:02.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.071613948 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796387+0000", 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0509e-05 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796432+0000", 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4987000000000001e-05 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.161 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7144 7.a 7:58d52035:::benchmark_data_smithi192_150129_object7143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.734600+0000", 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.39673110000001, 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0619978899999998, 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7144 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734600+0000", 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734600+0000", 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734602+0000", 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.846e-06 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734627+0000", 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4635999999999999e-05 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734650+0000", 2024-06-27T21:49:02.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3322000000000001e-05 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734659+0000", 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5879999999999998e-06 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734720+0000", 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1525999999999993e-05 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.734853+0000", 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013227900000000001 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.735030+0000", 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017699600000000001 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.735604+0000", 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00057416300000000004 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.736022+0000", 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000417933 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796489+0000", 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.060467429 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796553+0000", 2024-06-27T21:49:02.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3582000000000002e-05 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796598+0000", 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5590000000000002e-05 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7161 7.a 7:5c50f347:::benchmark_data_smithi192_150129_object7160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.38474082300002, 2024-06-27T21:49:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0504606029999999, 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7161 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746591+0000", 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4099999999999998e-07 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746608+0000", 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6625000000000001e-05 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746618+0000", 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0011e-05 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746623+0000", 2024-06-27T21:49:02.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3360000000000002e-06 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746663+0000", 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0018000000000002e-05 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746745+0000", 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1840000000000002e-05 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.746828+0000", 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2491999999999996e-05 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.747302+0000", 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047436100000000002 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.747576+0000", 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027408000000000001 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.796916+0000", 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0493401699999998 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797001+0000", 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5148000000000005e-05 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797051+0000", 2024-06-27T21:49:02.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9880999999999999e-05 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7163 7.a 7:551d8d3c:::benchmark_data_smithi192_150129_object7162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.751481+0000", 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.37985061500001, 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.045761766, 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7163 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751481+0000", 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751481+0000", 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751482+0000", 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.201e-06 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751512+0000", 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9422000000000001e-05 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751525+0000", 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3512999999999999e-05 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751533+0000", 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0190000000000006e-06 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751584+0000", 2024-06-27T21:49:02.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0909999999999999e-05 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751667+0000", 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3344999999999996e-05 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.751758+0000", 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0295000000000002e-05 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.752302+0000", 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054424699999999998 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.752958+0000", 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00065651900000000005 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797146+0000", 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0441872939999999 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797204+0000", 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8359000000000001e-05 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.797243+0000", 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8642e-05 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7170 7.a 7:5e96af1b:::benchmark_data_smithi192_150129_object7169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.754627+0000", 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.37670486499997, 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0436859030000001, 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.182 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7170 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754627+0000", 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754627+0000", 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754628+0000", 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.16e-07 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754647+0000", 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9052999999999999e-05 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754657+0000", 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0349999999999999e-05 2024-06-27T21:49:02.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754662+0000", 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.074e-06 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754696+0000", 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4245000000000002e-05 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754770+0000", 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3411999999999994e-05 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.754850+0000", 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0724999999999997e-05 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.755252+0000", 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000401717 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.755609+0000", 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035673299999999997 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798066+0000", 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0424571569999999 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798271+0000", 2024-06-27T21:49:02.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00020472700000000001 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798313+0000", 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1894000000000002e-05 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7173 7.a 7:5e8032bc:::benchmark_data_smithi192_150129_object7172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.758350+0000", 2024-06-27T21:49:02.188 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.37298149599997, 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0401055800000001, 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7173 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758350+0000", 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758350+0000", 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758351+0000", 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8400000000000003e-07 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758369+0000", 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7813999999999999e-05 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758379+0000", 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0657e-05 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758384+0000", 2024-06-27T21:49:02.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9599999999999999e-06 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758431+0000", 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6353000000000002e-05 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758548+0000", 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011780700000000001 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.758687+0000", 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000139004 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.759099+0000", 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041104999999999999 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.759403+0000", 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030414899999999998 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798361+0000", 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0389582499999999 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798414+0000", 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2806999999999998e-05 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798456+0000", 2024-06-27T21:49:02.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1944999999999999e-05 2024-06-27T21:49:02.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.195 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:02.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:02.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7194 7.a 7:54efe092:::benchmark_data_smithi192_150129_object7193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-06-27T21:49:02.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.804471+0000", 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.32686035899997, 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9944490020000001, 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7194 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804471+0000", 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804471+0000", 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804472+0000", 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4000000000000001e-07 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804489+0000", 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6928000000000001e-05 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804498+0000", 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3740000000000004e-06 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804503+0000", 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5460000000000002e-06 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804540+0000", 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7283999999999997e-05 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804619+0000", 2024-06-27T21:49:02.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8466000000000004e-05 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.804704+0000", 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5797999999999994e-05 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.805270+0000", 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056605399999999997 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.805570+0000", 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030003099999999997 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798830+0000", 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9932598270000002 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798883+0000", 2024-06-27T21:49:02.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2920999999999997e-05 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798920+0000", 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7033000000000003e-05 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7203 7.a 7:54cc3b12:::benchmark_data_smithi192_150129_object7202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.932232+0000", 2024-06-27T21:49:02.203 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.19909921800001, 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.866820186, 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7203 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932232+0000", 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932232+0000", 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932234+0000", 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.167e-06 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932254+0000", 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0332000000000001e-05 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932266+0000", 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1739000000000001e-05 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932272+0000", 2024-06-27T21:49:02.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1979999999999998e-06 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932317+0000", 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4972000000000002e-05 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932414+0000", 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6875999999999999e-05 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.932500+0000", 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6761000000000005e-05 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.933047+0000", 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00054685900000000002 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.933322+0000", 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027448799999999998 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.798967+0000", 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8656448829999999 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799020+0000", 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.363e-05 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799053+0000", 2024-06-27T21:49:02.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2280999999999998e-05 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7206 7.a 7:5fcfa5ab:::benchmark_data_smithi192_150129_object7205:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:50.968510+0000", 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.16282118300001, 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8313247289999999, 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.210 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7206 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968510+0000", 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968510+0000", 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968512+0000", 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.669e-06 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968551+0000", 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8559999999999997e-05 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968562+0000", 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0910999999999999e-05 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968567+0000", 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2039999999999996e-06 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968603+0000", 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6644999999999999e-05 2024-06-27T21:49:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968704+0000", 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100657 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.968799+0000", 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4724999999999999e-05 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.969300+0000", 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00050102600000000001 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:50.969718+0000", 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041863499999999999 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799711+0000", 2024-06-27T21:49:02.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8299927789999999 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799791+0000", 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0276000000000002e-05 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799835+0000", 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3642e-05 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.216 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7212 7.a 7:59657ab7:::benchmark_data_smithi192_150129_object7211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.090909+0000", 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 551.04042250500004, 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7091218459999999, 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.217 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7212 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090909+0000", 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090909+0000", 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090910+0000", 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2600000000000001e-07 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090929+0000", 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9236e-05 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090940+0000", 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0614999999999999e-05 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090947+0000", 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9330000000000004e-06 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.090998+0000", 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1405e-05 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.091083+0000", 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5008999999999999e-05 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.091170+0000", 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6833e-05 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.091624+0000", 2024-06-27T21:49:02.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045440399999999999 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.092085+0000", 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00046061799999999999 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799883+0000", 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7077979029999999 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.799991+0000", 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000108422 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800031+0000", 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9542000000000002e-05 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.223 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7223 7.a 7:59acbc88:::benchmark_data_smithi192_150129_object7222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.227352+0000", 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 550.90397991500004, 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5728116769999998, 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7223 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227352+0000", 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227352+0000", 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227353+0000", 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4020000000000001e-06 2024-06-27T21:49:02.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227373+0000", 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0409000000000001e-05 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227385+0000", 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1406999999999999e-05 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227392+0000", 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1350000000000003e-06 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227440+0000", 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7726999999999999e-05 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227529+0000", 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8769000000000003e-05 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.227615+0000", 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6230000000000006e-05 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.228120+0000", 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00050544700000000001 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.228405+0000", 2024-06-27T21:49:02.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028437900000000002 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800072+0000", 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.571667862 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800126+0000", 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3245000000000003e-05 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800163+0000", 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7664999999999999e-05 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7236 7.a 7:5bae2729:::benchmark_data_smithi192_150129_object7235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e729)", 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.314672+0000", 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 550.81665940300002, 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.485620113, 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.230 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7236 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314672+0000", 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314672+0000", 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314673+0000", 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9199999999999999e-07 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314692+0000", 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9357000000000001e-05 2024-06-27T21:49:02.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314703+0000", 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0854e-05 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314709+0000", 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3990000000000003e-06 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314760+0000", 2024-06-27T21:49:02.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0980000000000003e-05 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314848+0000", 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8475999999999998e-05 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.314935+0000", 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6965999999999994e-05 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.315411+0000", 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047577600000000002 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.316018+0000", 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060737699999999998 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800204+0000", 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.484185246 2024-06-27T21:49:02.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800257+0000", 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3349e-05 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800292+0000", 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5441000000000002e-05 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7275 7.a 7:56cae157:::benchmark_data_smithi192_150129_object7274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e730)", 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:51.900012+0000", 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 550.23131971500004, 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9010095310000001, 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7275 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.237 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900012+0000", 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900012+0000", 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900013+0000", 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6000000000000003e-07 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900056+0000", 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3597999999999998e-05 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900067+0000", 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1022000000000001e-05 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900073+0000", 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2599999999999996e-06 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900113+0000", 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0071999999999999e-05 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900200+0000", 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7625000000000002e-05 2024-06-27T21:49:02.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900284+0000", 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3646000000000005e-05 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.900850+0000", 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056577099999999996 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:51.901121+0000", 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027103400000000002 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800754+0000", 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8996332389999999 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.800954+0000", 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019964599999999999 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801021+0000", 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7858e-05 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:7276 7.a 7:5d7fcb60:::benchmark_data_smithi192_150129_object7275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e730)", 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:39:52.017209+0000", 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 550.11412303400004, 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7839677919999999, 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.243 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7276 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017209+0000", 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017209+0000", 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017210+0000", 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.421e-06 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017254+0000", 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3893999999999997e-05 2024-06-27T21:49:02.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017267+0000", 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3006999999999999e-05 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017275+0000", 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0320000000000003e-06 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017331+0000", 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5615999999999997e-05 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017446+0000", 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115633 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,11", 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.017547+0000", 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101346 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.018283+0000", 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00073507599999999998 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:52.018792+0000", 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000509075 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801070+0000", 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7822779560000002 2024-06-27T21:49:02.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801123+0000", 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3492999999999998e-05 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:39:53.801176+0000", 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3242999999999998e-05 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(8.8 857/839 [PushReplyOp(8:1346eb01:::benchmark_data_smithi192_167666_object31587:head)])", 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:58.522540+0000", 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 183.60879134999999, 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78787486799999995, 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.522540+0000", 2024-06-27T21:49:02.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.522540+0000", 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.522541+0000", 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-07 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.522545+0000", 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7610000000000001e-06 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.309156+0000", 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.78661161999999996 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.309173+0000", 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6609999999999999e-05 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.309283+0000", 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000109935 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.310415+0000", 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011323609999999999 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.254 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPush(8.13 857/839 [PushOp(8:c9002e11:::benchmark_data_smithi192_167666_object57884:head, version: 832'3594, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c9002e11:::benchmark_data_smithi192_167666_object57884:head@832'3594, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:49:02.254 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:59.693349+0000", 2024-06-27T21:49:02.254 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 182.43798243800001, 2024-06-27T21:49:02.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.857180952, 2024-06-27T21:49:02.254 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.254 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.693349+0000", 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.693349+0000", 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.693350+0000", 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.693420+0000", 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0596e-05 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:00.549384+0000", 2024-06-27T21:49:02.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.85596420200000001 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:00.549394+0000", 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:00.549477+0000", 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2819999999999996e-05 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:00.549490+0000", 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3455000000000001e-05 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:00.550530+0000", 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001039646 2024-06-27T21:49:02.258 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:131996 8.15 8:a8017a7b:::benchmark_data_smithi192_167666_object52558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e892)", 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:47:10.866596+0000", 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 111.264735849, 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30328159700000001, 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:02.259 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131996 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866596+0000", 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866596+0000", 2024-06-27T21:49:02.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866597+0000", 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.302e-06 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866602+0000", 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7389999999999999e-06 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:02.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866613+0000", 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0847e-05 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866619+0000", 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9750000000000004e-06 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866669+0000", 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0059999999999998e-05 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:02.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866774+0000", 2024-06-27T21:49:02.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105457 2024-06-27T21:49:02.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:49:02.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.866879+0000", 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010442600000000001 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.868665+0000", 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001786957 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:10.868785+0000", 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119129 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:11.169810+0000", 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.30102522799999998 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:11.169848+0000", 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7727e-05 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:02.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:47:11.169877+0000", 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9750000000000001e-05 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:02.268 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:49:02.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:02 smithi190 ceph-mon[32247]: pgmap v2286: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:49:02.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:02.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:02.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:02 smithi190 ceph-mon[32247]: osdmap e940: 12 total, 12 up, 12 in 2024-06-27T21:49:02.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:02.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:02.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:02 smithi151 ceph-mon[28965]: pgmap v2286: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-06-27T21:49:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:02.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:02 smithi151 ceph-mon[28965]: osdmap e940: 12 total, 12 up, 12 in 2024-06-27T21:49:02.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:02.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:02.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:02.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:49:02.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:02.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:02.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:02.846 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:02.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:02.907 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 38 16 55622 55606 91.4071 80.4375 0.0246133 0.010934 2024-06-27T21:49:02.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:02.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:02.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:02.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:03.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:03 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:49:03.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:03.107 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:03.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:03.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:03.233 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:03 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:49:03.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:03 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:49:03.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:03.359 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:03.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:03.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:03.519 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:03.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:03.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:03.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:03.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:03.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:03.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:03.907 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 39 16 56745 56729 90.8631 70.1875 0.0125057 0.010997 2024-06-27T21:49:03.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:49:03.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:04.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:04.015 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:49:04.018 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-06-27T21:49:04.104 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:04.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:04.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:04.278 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:04.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:04 smithi192 ceph-mon[32956]: pgmap v2288: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-27T21:49:04.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4061924007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:49:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:49:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:04 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3893244497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:04.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:04 smithi192 ceph-mon[32956]: from='client.27281 172.21.15.151:0/3893244497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:04.377 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:49:04.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.414 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:04.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:04 smithi151 ceph-mon[28965]: pgmap v2288: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-27T21:49:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4061924007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:49:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:49:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:04 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3893244497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:04.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:04 smithi151 ceph-mon[28965]: from='client.27281 172.21.15.151:0/3893244497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:04 smithi190 ceph-mon[32247]: pgmap v2288: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-06-27T21:49:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4061924007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:49:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:49:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:04 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3893244497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:04 smithi190 ceph-mon[32247]: from='client.27281 172.21.15.151:0/3893244497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:04.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:04.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:04.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:04.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:04.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:04.794 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:04.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:04.907 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:49:04.907628+0000 min lat: 0.0010169 max lat: 0.278399 avg lat: 0.0110671 2024-06-27T21:49:04.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:49:04.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 40 16 57843 57827 90.3073 68.625 0.00300356 0.0110671 2024-06-27T21:49:04.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:04.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:05.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:05.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:05.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:05.178 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:05 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:05 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:05 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:05 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:49:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:05 smithi151 ceph-mon[28965]: from='client.27281 172.21.15.151:0/3893244497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:05.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:05 smithi151 ceph-mon[28965]: osdmap e941: 12 total, 12 up, 12 in 2024-06-27T21:49:05.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:05.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:05 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:05 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:05 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:05 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:49:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:05 smithi192 ceph-mon[32956]: from='client.27281 172.21.15.151:0/3893244497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:05.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:05 smithi192 ceph-mon[32956]: osdmap e941: 12 total, 12 up, 12 in 2024-06-27T21:49:05.301 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:05.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:05.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:05.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:05 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:05.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:05 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:05.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:05 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:05.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:05 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:49:05.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:05 smithi190 ceph-mon[32247]: from='client.27281 172.21.15.151:0/3893244497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:05.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:05 smithi190 ceph-mon[32247]: osdmap e941: 12 total, 12 up, 12 in 2024-06-27T21:49:05.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:05.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:05.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:05.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:05.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:05.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:05.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:05.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 41 15 58673 58658 89.3717 51.9375 0.00963439 0.0111825 2024-06-27T21:49:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:05.937 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:05.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:06.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.061 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:06.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:06.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:06.169 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:06 smithi192 ceph-mon[32956]: pgmap v2290: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:49:06.194 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:06.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:06 smithi151 ceph-mon[28965]: pgmap v2290: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:49:06.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:06.320 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:06.333 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:49:06.333 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:49:06.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.333 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:59035 9.9 9:9258a34d:::benchmark_data_smithi192_179902_object59034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:49:06.333 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.323313+0000", 2024-06-27T21:49:06.333 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0075670870000000001, 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0075985319999999999, 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59035 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323313+0000", 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323313+0000", 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323315+0000", 2024-06-27T21:49:06.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323342+0000", 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7199000000000002e-05 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323360+0000", 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7405e-05 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323368+0000", 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2800000000000003e-06 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323438+0000", 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9445999999999999e-05 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323580+0000", 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000142416 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,11", 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.323726+0000", 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000145685 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.325929+0000", 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022029950000000001 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:06.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329101+0000", 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031727550000000002 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:59039 9.6 9:6f5c80de:::benchmark_data_smithi192_179902_object59038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.326603+0000", 2024-06-27T21:49:06.339 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0042772820000000003, 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043934530000000003, 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59039 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326603+0000", 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326603+0000", 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326604+0000", 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3260000000000001e-06 2024-06-27T21:49:06.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326626+0000", 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2212e-05 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326644+0000", 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.77e-05 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326653+0000", 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5660000000000007e-06 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326703+0000", 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9832e-05 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:06.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326908+0000", 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000205534 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.327057+0000", 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000149276 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.344 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:59040 9.6 9:60d1455d:::benchmark_data_smithi192_179902_object59039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.326877+0000", 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0040036689999999996, 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041815890000000003, 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59040 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326877+0000", 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326877+0000", 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326877+0000", 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5899999999999996e-07 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326898+0000", 2024-06-27T21:49:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0543e-05 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326907+0000", 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3200000000000006e-06 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.326913+0000", 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3550000000000001e-06 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328946+0000", 2024-06-27T21:49:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020327280000000001 2024-06-27T21:49:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329111+0000", 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000164666 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329269+0000", 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015779100000000001 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:59041 9.e e941/916 9:734095f9:::benchmark_data_smithi192_179902_object59040:head v 941'3743, mlcod=941'3741)", 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.328492+0000", 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0023877550000000001, 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026213439999999998, 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328492+0000", 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328492+0000", 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328493+0000", 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0500000000000002e-07 2024-06-27T21:49:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328584+0000", 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0912e-05 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328604+0000", 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9888e-05 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328611+0000", 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9840000000000004e-06 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328931+0000", 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031949000000000002 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.328948+0000", 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7268000000000001e-05 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.27262.0:59039 9.6 e941/916)", 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.329011+0000", 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.001869117, 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021470539999999998, 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329011+0000", 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329011+0000", 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329012+0000", 2024-06-27T21:49:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7400000000000002e-07 2024-06-27T21:49:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.329016+0000", 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2760000000000002e-06 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330849+0000", 2024-06-27T21:49:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001832581 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330853+0000", 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.758e-06 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.27262.0:59017 9.6 e941/916)", 2024-06-27T21:49:06.362 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.330903+0000", 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9999766, 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00032404500000000002, 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330903+0000", 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330903+0000", 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330904+0000", 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330906+0000", 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9010000000000001e-06 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330915+0000", 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8070000000000006e-06 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.330918+0000", 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5499999999999999e-06 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.27262.0:59039 9.6 e941/916)", 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.331114+0000", 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9997663, 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000190022, 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:49:06.366 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.331114+0000", 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.331114+0000", 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.331114+0000", 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7599999999999998e-07 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.331116+0000", 2024-06-27T21:49:06.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.331124+0000", 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8390000000000007e-06 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.331127+0000", 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8150000000000002e-06 2024-06-27T21:49:06.369 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:59017 9.6 9:6fda9f20:::benchmark_data_smithi192_179902_object59016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:06.307652+0000", 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.023228134000000001, 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023694025000000001, 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:06.370 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59017 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307652+0000", 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307652+0000", 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307653+0000", 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4199999999999996e-07 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307675+0000", 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2305000000000001e-05 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307683+0000", 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4069999999999993e-06 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307689+0000", 2024-06-27T21:49:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7699999999999998e-06 2024-06-27T21:49:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:06.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307748+0000", 2024-06-27T21:49:06.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8888000000000003e-05 2024-06-27T21:49:06.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.307924+0000", 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017586899999999999 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,5", 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.308058+0000", 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000134408 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.310595+0000", 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002537021 2024-06-27T21:49:06.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:06.325903+0000", 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015307777999999999 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 8 2024-06-27T21:49:06.379 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:49:06.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:06 smithi190 ceph-mon[32247]: pgmap v2290: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-06-27T21:49:06.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:06.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:06.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:06.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:06.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:06.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:06.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:06.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:06.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 42 16 59685 59669 88.7484 63.1875 0.00620563 0.0112626 2024-06-27T21:49:06.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:06.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:06.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:07.066 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:07.074 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:49:07.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:49:07.093 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:07.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:49:07.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:49:07.123 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:49:07.123 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:49:07.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:49:07.147 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:49:07.174 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:49:07.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:49:07.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:07.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:49:07.203 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:49:07.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:07.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:49:07.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:49:07.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:49:07.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:49:07.280 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:49:07.280 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:49:07.287 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:49:07.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:49:07.306 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:49:07.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:07.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:49:07.332 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:49:07.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:07.356 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:49:07.356 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:49:07.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:07.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:07.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:07.602 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:07.620 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:49:07.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:07.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:07.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:07.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:07.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 43 16 60779 60763 88.2746 68.375 0.00420185 0.0113223 2024-06-27T21:49:07.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:07.986 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:07.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:08.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:08.116 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:08.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:08.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:08.250 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:08.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:49:08.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:08 smithi192 ceph-mon[32956]: pgmap v2291: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-27T21:49:08.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/414955018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:08.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:08 smithi151 ceph-mon[28965]: pgmap v2291: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-27T21:49:08.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/414955018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:08.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:08.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:08.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:08.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:08.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:08 smithi190 ceph-mon[32247]: pgmap v2291: 47 pgs: 47 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-06-27T21:49:08.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/414955018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:08.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 11 1 2024-06-27T21:49:08.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:08.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:08.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:08.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:08.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:08.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:08.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:08.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 44 16 62100 62084 88.1447 82.5625 0.0155512 0.0113304 2024-06-27T21:49:08.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:09.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:09.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:09.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:09.160 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:09.186 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:09.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:09.278 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:09.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:09.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:09.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.538 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:09.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:09.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:09.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:09.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:09.796 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:09.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:09.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 45 16 63290 63274 87.8387 74.375 0.0271506 0.0113775 2024-06-27T21:49:09.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:09.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:10.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:10.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:10.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:10.177 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:10.279 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:10 smithi192 ceph-mon[32956]: pgmap v2292: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:49:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:10.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:10 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:10.311 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:10.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:10 smithi151 ceph-mon[28965]: pgmap v2292: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:49:10.349 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:10.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:10 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:10.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:10.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:10 smithi190 ceph-mon[32247]: pgmap v2292: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-06-27T21:49:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:10 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:10.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:10.571 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:10.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:10.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:10.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:10.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:10.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 46 16 64653 64637 87.781 85.1875 0.00359712 0.0113868 2024-06-27T21:49:10.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:10.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:10.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:11.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:11.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:11.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:49:11.092 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:11.092 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:11.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:11.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:11.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:11.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:11.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:11.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:11.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:11.601 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:11.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:11.736 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:11.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:11.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:11.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:49:11.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 47 16 66063 66047 87.7881 88.125 0.00843143 0.0113772 2024-06-27T21:49:11.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:11.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:11.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:12.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:12.123 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:12.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:12.228 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:49:12.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:12.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:12 smithi192 ceph-mon[32956]: pgmap v2293: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:49:12.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:12.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:12 smithi151 ceph-mon[28965]: pgmap v2293: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:49:12.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:12.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:12.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:12 smithi190 ceph-mon[32247]: pgmap v2293: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-06-27T21:49:12.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:12.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:12.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:12.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:12.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:12.903 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:12.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:12.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 48 16 67394 67378 87.6922 83.1875 0.0149933 0.0113981 2024-06-27T21:49:13.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:13.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:13.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:13.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:13.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: osdmap e942: 12 total, 12 up, 12 in 2024-06-27T21:49:13.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:49:13.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:49:13.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:13.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/600286351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:49:13.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:13.166 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:13.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:13.278 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:49:13.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:13.305 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:13.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:13.427 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: osdmap e942: 12 total, 12 up, 12 in 2024-06-27T21:49:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:49:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:49:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:13.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/600286351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:49:13.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:13.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:13.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:13.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: osdmap e942: 12 total, 12 up, 12 in 2024-06-27T21:49:13.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:49:13.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:49:13.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:49:13.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/600286351' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:49:13.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:13.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:13.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:13.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:13.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:13.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 49 15 68714 68699 87.5874 82.5625 0.007945 0.0114111 2024-06-27T21:49:13.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:13.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:13.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:14.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:14.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:14.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:14.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:14.234 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: pgmap v2295: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s 2024-06-27T21:49:14.234 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:14.234 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/600286351' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:49:14.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: osdmap e943: 12 total, 12 up, 12 in 2024-06-27T21:49:14.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2920223136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:49:14.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:49:14.235 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:14 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:49:14.261 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: pgmap v2295: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s 2024-06-27T21:49:14.262 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:14.262 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/600286351' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:49:14.262 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: osdmap e943: 12 total, 12 up, 12 in 2024-06-27T21:49:14.262 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2920223136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:49:14.262 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:49:14.262 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:14 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:49:14.279 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-06-27T21:49:14.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:14.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:14.423 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:14.423 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:14.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:14.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:14.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: pgmap v2295: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s 2024-06-27T21:49:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/600286351' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:49:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: osdmap e943: 12 total, 12 up, 12 in 2024-06-27T21:49:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2920223136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:49:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:49:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:14 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:49:14.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:14.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:14.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:14.725 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:14.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:14.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:14.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 50 16 69915 69899 87.3356 75 0.00744524 0.0114443 2024-06-27T21:49:14.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:14.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:14.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:15.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:15.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:15.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:15.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:49:15.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:15.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.370 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:15.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: 9.8 scrub starts 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2920223136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: osdmap e944: 12 total, 12 up, 12 in 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:49:15.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:15 smithi151 ceph-mon[28965]: osdmap e945: 12 total, 12 up, 12 in 2024-06-27T21:49:15.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:15.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: 9.8 scrub starts 2024-06-27T21:49:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2920223136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:49:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: osdmap e944: 12 total, 12 up, 12 in 2024-06-27T21:49:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:49:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:49:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:15 smithi190 ceph-mon[32247]: osdmap e945: 12 total, 12 up, 12 in 2024-06-27T21:49:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: 9.8 scrub starts 2024-06-27T21:49:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2920223136' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:49:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: osdmap e944: 12 total, 12 up, 12 in 2024-06-27T21:49:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:49:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:49:15.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:15 smithi192 ceph-mon[32956]: osdmap e945: 12 total, 12 up, 12 in 2024-06-27T21:49:15.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:15.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:15.630 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:49:15.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:15.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:15.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:15.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:15.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:15.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 51 16 71048 71032 87.0116 70.8125 0.00774903 0.0114864 2024-06-27T21:49:15.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:16.005 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:16.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:49:16.055 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:49:16.055 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:49:16.084 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:49:16.084 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:49:16.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:16.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:49:16.115 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:49:16.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:16.144 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:49:16.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:49:16.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:49:16.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:49:16.195 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:49:16.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:49:16.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:49:16.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:49:16.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:49:16.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:49:16.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:16.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:49:16.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:49:16.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:16.284 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:49:16.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:49:16.307 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:49:16.308 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:49:16.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:16 smithi151 ceph-mon[28965]: pgmap v2298: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-06-27T21:49:16.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:16 smithi151 ceph-mon[28965]: 9.8 scrub ok 2024-06-27T21:49:16.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:16 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:49:16.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:16 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:49:16.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:16 smithi151 ceph-mon[28965]: 9.f scrub starts 2024-06-27T21:49:16.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2981865894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:16.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:49:16.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:49:16.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:16.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:16.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:16 smithi190 ceph-mon[32247]: pgmap v2298: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-06-27T21:49:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:16 smithi190 ceph-mon[32247]: 9.8 scrub ok 2024-06-27T21:49:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:16 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:49:16.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:16 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:49:16.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:16 smithi190 ceph-mon[32247]: 9.f scrub starts 2024-06-27T21:49:16.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2981865894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:16.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:16 smithi192 ceph-mon[32956]: pgmap v2298: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s 2024-06-27T21:49:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:16 smithi192 ceph-mon[32956]: 9.8 scrub ok 2024-06-27T21:49:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:16 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:49:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:16 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:49:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:16 smithi192 ceph-mon[32956]: 9.f scrub starts 2024-06-27T21:49:16.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2981865894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:16.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:16.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:16.667 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-27T21:49:16.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:49:16.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:16.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:16.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:16.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:16.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:16.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 52 16 71963 71947 86.4382 57.1875 0.00972824 0.0115624 2024-06-27T21:49:16.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:17.028 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:17.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:17.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:17.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:17.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:17.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:17.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:17.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:17 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:49:17.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:17 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:49:17.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:17 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:49:17.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:17 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:49:17.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:17 smithi151 ceph-mon[28965]: 9.f scrub ok 2024-06-27T21:49:17.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:17.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:17 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:49:17.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:17 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:49:17.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:17 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:49:17.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:17 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:49:17.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:17 smithi190 ceph-mon[32247]: 9.f scrub ok 2024-06-27T21:49:17.442 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:17.482 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:17.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:17.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:17.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:17 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:49:17.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:17 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:49:17.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:17 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:49:17.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:17 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:49:17.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:17 smithi192 ceph-mon[32956]: 9.f scrub ok 2024-06-27T21:49:17.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:17.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:17.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:17.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:17.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:17.908 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 53 16 72975 72959 86.0007 63.25 0.0186029 0.0116224 2024-06-27T21:49:17.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:17.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:18.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:18.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:18.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:18.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:18.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:18.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:18.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:18.427 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:18 smithi192 ceph-mon[32956]: pgmap v2300: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-06-27T21:49:18.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:18 smithi151 ceph-mon[28965]: pgmap v2300: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-06-27T21:49:18.449 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.450 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:18.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:18 smithi190 ceph-mon[32247]: pgmap v2300: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.53k op/s 2024-06-27T21:49:18.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:18.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:18.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:18.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:18.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:18.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:18.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:18.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 54 16 74219 74203 85.8479 77.75 0.00458482 0.0116404 2024-06-27T21:49:18.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:18.960 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:49:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:49:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:74233 9.5 9:a92eb0eb:::benchmark_data_smithi192_179902_object74232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:18.920944+0000", 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0071859660000000002, 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072150840000000001, 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 74233 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.920944+0000", 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.920944+0000", 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.920946+0000", 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6980000000000001e-06 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.920973+0000", 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6495e-05 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.920996+0000", 2024-06-27T21:49:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3651000000000001e-05 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.921006+0000", 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4490000000000005e-06 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.921072+0000", 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5962000000000003e-05 2024-06-27T21:49:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.921220+0000", 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000148455 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.921391+0000", 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017095700000000001 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.924114+0000", 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027230779999999999 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.928046+0000", 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039318629999999999 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:74185 9.5 9:a9f638ff:::benchmark_data_smithi192_179902_object74184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:18.876133+0000", 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.051996993999999998, 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.052106736000000001, 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 74185 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876133+0000", 2024-06-27T21:49:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876133+0000", 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876135+0000", 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3939999999999999e-06 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876158+0000", 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3266999999999999e-05 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876171+0000", 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3147e-05 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876179+0000", 2024-06-27T21:49:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4590000000000001e-06 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876245+0000", 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6723000000000005e-05 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876382+0000", 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013679400000000001 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:49:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.876486+0000", 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103733 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.878551+0000", 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020656009999999998 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.879353+0000", 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080145800000000001 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:74239 9.0 9:0a6ce884:::benchmark_data_smithi192_179902_object74238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:18.927757+0000", 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00037320400000000003, 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056042899999999998, 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 74239 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927757+0000", 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927757+0000", 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927759+0000", 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.485e-06 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927781+0000", 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2651e-05 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927796+0000", 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5095e-05 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927806+0000", 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3780000000000007e-06 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.927867+0000", 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0934000000000003e-05 2024-06-27T21:49:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.928030+0000", 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016378000000000001 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.928127+0000", 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6576000000000005e-05 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:74221 9.5 9:ab342c03:::benchmark_data_smithi192_179902_object74220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:18.910296+0000", 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017833934999999999, 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018084445000000001, 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 74221 2024-06-27T21:49:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910296+0000", 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910296+0000", 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910297+0000", 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.71e-07 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910316+0000", 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8984e-05 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910327+0000", 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.046e-05 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910332+0000", 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7490000000000004e-06 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910383+0000", 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0952000000000003e-05 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910511+0000", 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012798599999999999 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,7", 2024-06-27T21:49:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.910659+0000", 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014719200000000001 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.913480+0000", 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002821048 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.913518+0000", 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8482000000000002e-05 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:74225 9.0 9:04c1362a:::benchmark_data_smithi192_179902_object74224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:18.912131+0000", 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015999742000000001, 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016320036999999999, 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:49:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 74225 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912131+0000", 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912131+0000", 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912132+0000", 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.257e-06 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:49:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912158+0000", 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6157e-05 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912173+0000", 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4970999999999999e-05 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912181+0000", 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4940000000000003e-06 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912273+0000", 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1594000000000004e-05 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912410+0000", 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000136837 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.912568+0000", 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015830900000000001 2024-06-27T21:49:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.915237+0000", 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026683150000000001 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:18.915271+0000", 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4462999999999999e-05 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:49:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:49:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 5 2024-06-27T21:49:18.990 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:49:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:18.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:18.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:19.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:19.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:19.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:19.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:19.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:19 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:49:19.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:19 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:49:19.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:19.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:19.415 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:19 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:49:19.415 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:19 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:49:19.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:49:19.473 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:19.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:19.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:19 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:49:19.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:19 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:49:19.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:19.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:19.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:19.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:19.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:49:19.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:19.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:19.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:19.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 55 16 75248 75232 85.4563 64.3125 0.0342511 0.0116799 2024-06-27T21:49:19.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:20.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:20.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:20.146 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:49:20.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:20.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:20.320 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:49:20.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:20 smithi151 ceph-mon[28965]: pgmap v2301: 47 pgs: 47 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-06-27T21:49:20.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:20 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:49:20.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:20 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:49:20.323 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:49:20.370 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:20.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:20.442 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:20 smithi192 ceph-mon[32956]: pgmap v2301: 47 pgs: 47 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-06-27T21:49:20.442 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:20 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:49:20.442 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:20 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:49:20.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:20 smithi190 ceph-mon[32247]: pgmap v2301: 47 pgs: 47 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-06-27T21:49:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:20 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:49:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:20 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:49:20.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:20.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:20.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:20.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:20.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:20.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:20.906 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:20.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:20.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 56 16 76207 76191 85.0007 59.9375 0.00315125 0.0117585 2024-06-27T21:49:21.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:21.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:21.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:21.162 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:21 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:49:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:21 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:49:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:21 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:49:21.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:21 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:49:21.264 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:21.268 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:21.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:21.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:21.430 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:21.481 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:21 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:49:21.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:21 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:49:21.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:21 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:49:21.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:21 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:49:21.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:21 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:49:21.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:21 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:49:21.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:21 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:49:21.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:21 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:49:21.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:21.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:21.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:21.695 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:21.792 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2301,"stamp":"2024-06-27T21:49:18.356898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4659741216,"num_objects":71095,"num_object_clones":0,"num_object_copies":213285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71093,"num_write_kb":4549952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73945,"ondisk_log_size":73945,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":22533084,"kb_used_data":13471276,"kb_used_omap":110,"kb_used_meta":9059793,"kb_avail":1102408740,"statfs":{"total":1151940427776,"available":1128866549760,"internally_reserved":0,"allocated":13794586624,"data_stored":13788642073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9277229015},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":594870272,"num_objects":9077,"num_object_clones":0,"num_object_copies":27231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":580928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001075"},"pg_stats":[{"pgid":"9.3","version":"945'4335","reported_seq":4402,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:15.388053+0000","last_change":"2024-06-27T21:48:11.986786+0000","last_active":"2024-06-27T21:49:15.388053+0000","last_peered":"2024-06-27T21:49:15.388053+0000","last_clean":"2024-06-27T21:49:15.388053+0000","last_became_active":"2024-06-27T21:48:11.986220+0000","last_became_peered":"2024-06-27T21:48:11.986220+0000","last_unstale":"2024-06-27T21:49:15.388053+0000","last_undegraded":"2024-06-27T21:49:15.388053+0000","last_fullsized":"2024-06-27T21:49:15.388053+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.515921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4335,"num_write_kb":277440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":168,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.287107+0000","last_change":"2024-06-27T21:48:37.072258+0000","last_active":"2024-06-27T21:49:14.287107+0000","last_peered":"2024-06-27T21:49:14.287107+0000","last_clean":"2024-06-27T21:49:14.287107+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:49:14.287107+0000","last_undegraded":"2024-06-27T21:49:14.287107+0000","last_fullsized":"2024-06-27T21:49:14.287107+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:37.072188+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:56.646935+0000","last_clean_scrub_stamp":"2024-06-27T21:48:37.072188+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:52.196369+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"945'4720","reported_seq":4786,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:18.146778+0000","last_change":"2024-06-27T21:48:12.310120+0000","last_active":"2024-06-27T21:49:18.146778+0000","last_peered":"2024-06-27T21:49:18.146778+0000","last_clean":"2024-06-27T21:49:18.146778+0000","last_became_active":"2024-06-27T21:48:12.309522+0000","last_became_peered":"2024-06-27T21:48:12.309522+0000","last_unstale":"2024-06-27T21:49:18.146778+0000","last_undegraded":"2024-06-27T21:49:18.146778+0000","last_fullsized":"2024-06-27T21:49:18.146778+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:35.020638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309329920,"num_objects":4720,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":302080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":135,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.973723+0000","last_change":"2024-06-27T21:48:39.725238+0000","last_active":"2024-06-27T21:49:14.973723+0000","last_peered":"2024-06-27T21:49:14.973723+0000","last_clean":"2024-06-27T21:49:14.973723+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:49:14.973723+0000","last_undegraded":"2024-06-27T21:49:14.973723+0000","last_fullsized":"2024-06-27T21:49:14.973723+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:39.725111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:39.725111+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:42.976869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"944'4343","reported_seq":4409,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.872912+0000","last_change":"2024-06-27T21:48:11.990938+0000","last_active":"2024-06-27T21:49:14.872912+0000","last_peered":"2024-06-27T21:49:14.872912+0000","last_clean":"2024-06-27T21:49:14.872912+0000","last_became_active":"2024-06-27T21:48:11.990332+0000","last_became_peered":"2024-06-27T21:48:11.990332+0000","last_unstale":"2024-06-27T21:49:14.872912+0000","last_undegraded":"2024-06-27T21:49:14.872912+0000","last_fullsized":"2024-06-27T21:49:14.872912+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284622848,"num_objects":4343,"num_object_clones":0,"num_object_copies":13029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":277952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":271,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.284865+0000","last_change":"2024-06-27T21:48:35.993226+0000","last_active":"2024-06-27T21:49:13.284865+0000","last_peered":"2024-06-27T21:49:13.284865+0000","last_clean":"2024-06-27T21:49:13.284865+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:49:13.284865+0000","last_undegraded":"2024-06-27T21:49:13.284865+0000","last_fullsized":"2024-06-27T21:49:13.284865+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:35.993117+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:35.993117+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:57.566564+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"944'4435","reported_seq":4500,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.852730+0000","last_change":"2024-06-27T21:48:12.310003+0000","last_active":"2024-06-27T21:49:14.852730+0000","last_peered":"2024-06-27T21:49:14.852730+0000","last_clean":"2024-06-27T21:49:14.852730+0000","last_became_active":"2024-06-27T21:48:12.309579+0000","last_became_peered":"2024-06-27T21:48:12.309579+0000","last_unstale":"2024-06-27T21:49:14.852730+0000","last_undegraded":"2024-06-27T21:49:14.852730+0000","last_fullsized":"2024-06-27T21:49:14.852730+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:40.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290652160,"num_objects":4435,"num_object_clones":0,"num_object_copies":13305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":283840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1041,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.280233+0000","last_change":"2024-06-27T21:48:08.767858+0000","last_active":"2024-06-27T21:49:13.280233+0000","last_peered":"2024-06-27T21:49:13.280233+0000","last_clean":"2024-06-27T21:49:13.280233+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:49:13.280233+0000","last_undegraded":"2024-06-27T21:49:13.280233+0000","last_fullsized":"2024-06-27T21:49:13.280233+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:08.767758+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:48:08.767758+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:20.221675+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"944'4314","reported_seq":4380,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.850710+0000","last_change":"2024-06-27T21:48:11.990014+0000","last_active":"2024-06-27T21:49:14.850710+0000","last_peered":"2024-06-27T21:49:14.850710+0000","last_clean":"2024-06-27T21:49:14.850710+0000","last_became_active":"2024-06-27T21:48:11.989652+0000","last_became_peered":"2024-06-27T21:48:11.989652+0000","last_unstale":"2024-06-27T21:49:14.850710+0000","last_undegraded":"2024-06-27T21:49:14.850710+0000","last_fullsized":"2024-06-27T21:49:14.850710+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4314,"log_dups_size":0,"ondisk_log_size":4314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:24.552031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282722304,"num_objects":4314,"num_object_clones":0,"num_object_copies":12942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4314,"num_write_kb":276096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1654,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.973445+0000","last_change":"2024-06-27T21:47:34.859171+0000","last_active":"2024-06-27T21:49:14.973445+0000","last_peered":"2024-06-27T21:49:14.973445+0000","last_clean":"2024-06-27T21:49:14.973445+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:49:14.973445+0000","last_undegraded":"2024-06-27T21:49:14.973445+0000","last_fullsized":"2024-06-27T21:49:14.973445+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:34.859102+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:47:34.859102+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:16.602474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"945'4435","reported_seq":4537,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:15.352421+0000","last_change":"2024-06-27T21:49:14.968114+0000","last_active":"2024-06-27T21:49:15.352421+0000","last_peered":"2024-06-27T21:49:15.352421+0000","last_clean":"2024-06-27T21:49:15.352421+0000","last_became_active":"2024-06-27T21:48:25.035404+0000","last_became_peered":"2024-06-27T21:48:25.035404+0000","last_unstale":"2024-06-27T21:49:15.352421+0000","last_undegraded":"2024-06-27T21:49:15.352421+0000","last_fullsized":"2024-06-27T21:49:15.352421+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'4404","last_scrub_stamp":"2024-06-27T21:49:14.967854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:14.967854+0000","objects_scrubbed":4367,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:19.717925+0000","scrub_duration":1019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290652160,"num_objects":4435,"num_object_clones":0,"num_object_copies":13305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":283840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":637,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.971894+0000","last_change":"2024-06-27T21:48:43.028417+0000","last_active":"2024-06-27T21:49:14.971894+0000","last_peered":"2024-06-27T21:49:14.971894+0000","last_clean":"2024-06-27T21:49:14.971894+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:49:14.971894+0000","last_undegraded":"2024-06-27T21:49:14.971894+0000","last_fullsized":"2024-06-27T21:49:14.971894+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:43.028305+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:48:43.028305+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:45.889033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.b","version":"945'4594","reported_seq":4661,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:18.185690+0000","last_change":"2024-06-27T21:48:11.989882+0000","last_active":"2024-06-27T21:49:18.185690+0000","last_peered":"2024-06-27T21:49:18.185690+0000","last_clean":"2024-06-27T21:49:18.185690+0000","last_became_active":"2024-06-27T21:48:11.989420+0000","last_became_peered":"2024-06-27T21:48:11.989420+0000","last_unstale":"2024-06-27T21:49:18.185690+0000","last_undegraded":"2024-06-27T21:49:18.185690+0000","last_fullsized":"2024-06-27T21:49:18.185690+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.290203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301072384,"num_objects":4594,"num_object_clones":0,"num_object_copies":13782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":294016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":1029,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.279913+0000","last_change":"2024-06-27T21:48:18.178132+0000","last_active":"2024-06-27T21:49:13.279913+0000","last_peered":"2024-06-27T21:49:13.279913+0000","last_clean":"2024-06-27T21:49:13.279913+0000","last_became_active":"2024-06-27T21:48:18.177749+0000","last_became_peered":"2024-06-27T21:48:18.177749+0000","last_unstale":"2024-06-27T21:49:13.279913+0000","last_undegraded":"2024-06-27T21:49:13.279913+0000","last_fullsized":"2024-06-27T21:49:13.279913+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:42.888645+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_clean_scrub_stamp":"2024-06-27T21:47:42.888645+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"945'4514","reported_seq":4582,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:17.752076+0000","last_change":"2024-06-27T21:48:12.311450+0000","last_active":"2024-06-27T21:49:17.752076+0000","last_peered":"2024-06-27T21:49:17.752076+0000","last_clean":"2024-06-27T21:49:17.752076+0000","last_became_active":"2024-06-27T21:48:12.311072+0000","last_became_peered":"2024-06-27T21:48:12.311072+0000","last_unstale":"2024-06-27T21:49:17.752076+0000","last_undegraded":"2024-06-27T21:49:17.752076+0000","last_fullsized":"2024-06-27T21:49:17.752076+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:34.768612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295829504,"num_objects":4514,"num_object_clones":0,"num_object_copies":13542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4514,"num_write_kb":288896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":264,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.283855+0000","last_change":"2024-06-27T21:48:00.010050+0000","last_active":"2024-06-27T21:49:13.283855+0000","last_peered":"2024-06-27T21:49:13.283855+0000","last_clean":"2024-06-27T21:49:13.283855+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:49:13.283855+0000","last_undegraded":"2024-06-27T21:49:13.283855+0000","last_fullsized":"2024-06-27T21:49:13.283855+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.009982+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.009982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:28.111931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"944'4312","reported_seq":4378,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.826450+0000","last_change":"2024-06-27T21:48:11.998116+0000","last_active":"2024-06-27T21:49:14.826450+0000","last_peered":"2024-06-27T21:49:14.826450+0000","last_clean":"2024-06-27T21:49:14.826450+0000","last_became_active":"2024-06-27T21:48:11.997839+0000","last_became_peered":"2024-06-27T21:48:11.997839+0000","last_unstale":"2024-06-27T21:49:14.826450+0000","last_undegraded":"2024-06-27T21:49:14.826450+0000","last_fullsized":"2024-06-27T21:49:14.826450+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4312,"log_dups_size":0,"ondisk_log_size":4312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:37.072712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282591232,"num_objects":4312,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":275968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":274,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:16.258246+0000","last_change":"2024-06-27T21:49:16.258267+0000","last_active":"2024-06-27T21:49:16.258246+0000","last_peered":"2024-06-27T21:49:16.258246+0000","last_clean":"2024-06-27T21:49:16.258246+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:49:16.258246+0000","last_undegraded":"2024-06-27T21:49:16.258246+0000","last_fullsized":"2024-06-27T21:49:16.258246+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:16.258197+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:49:16.258197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:37.926239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"945'4545","reported_seq":4671,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:17.741090+0000","last_change":"2024-06-27T21:49:16.636144+0000","last_active":"2024-06-27T21:49:17.741090+0000","last_peered":"2024-06-27T21:49:17.741090+0000","last_clean":"2024-06-27T21:49:17.741090+0000","last_became_active":"2024-06-27T21:48:24.958699+0000","last_became_peered":"2024-06-27T21:48:24.958699+0000","last_unstale":"2024-06-27T21:49:17.741090+0000","last_undegraded":"2024-06-27T21:49:17.741090+0000","last_fullsized":"2024-06-27T21:49:17.741090+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'4466","last_scrub_stamp":"2024-06-27T21:49:16.635894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:16.635894+0000","objects_scrubbed":4435,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:41.281934+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297861120,"num_objects":4545,"num_object_clones":0,"num_object_copies":13635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4545,"num_write_kb":290880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1367,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.284503+0000","last_change":"2024-06-27T21:48:18.178256+0000","last_active":"2024-06-27T21:49:14.284503+0000","last_peered":"2024-06-27T21:49:14.284503+0000","last_clean":"2024-06-27T21:49:14.284503+0000","last_became_active":"2024-06-27T21:48:18.177860+0000","last_became_peered":"2024-06-27T21:48:18.177860+0000","last_unstale":"2024-06-27T21:49:14.284503+0000","last_undegraded":"2024-06-27T21:49:14.284503+0000","last_fullsized":"2024-06-27T21:49:14.284503+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:56.354808+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:47:56.354808+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:19.483984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"943'4358","reported_seq":4420,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.862524+0000","last_change":"2024-06-27T21:48:12.309880+0000","last_active":"2024-06-27T21:49:13.862524+0000","last_peered":"2024-06-27T21:49:13.862524+0000","last_clean":"2024-06-27T21:49:13.862524+0000","last_became_active":"2024-06-27T21:48:12.309446+0000","last_became_peered":"2024-06-27T21:48:12.309446+0000","last_unstale":"2024-06-27T21:49:13.862524+0000","last_undegraded":"2024-06-27T21:49:13.862524+0000","last_fullsized":"2024-06-27T21:49:13.862524+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:30.856501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285605888,"num_objects":4358,"num_object_clones":0,"num_object_copies":13074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4358,"num_write_kb":278912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1440,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.285915+0000","last_change":"2024-06-27T21:48:00.289561+0000","last_active":"2024-06-27T21:49:14.285915+0000","last_peered":"2024-06-27T21:49:14.285915+0000","last_clean":"2024-06-27T21:49:14.285915+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:49:14.285915+0000","last_undegraded":"2024-06-27T21:49:14.285915+0000","last_fullsized":"2024-06-27T21:49:14.285915+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.289470+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.289470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:28.638908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"943'4363","reported_seq":4425,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.831175+0000","last_change":"2024-06-27T21:48:12.310346+0000","last_active":"2024-06-27T21:49:13.831175+0000","last_peered":"2024-06-27T21:49:13.831175+0000","last_clean":"2024-06-27T21:49:13.831175+0000","last_became_active":"2024-06-27T21:48:12.309793+0000","last_became_peered":"2024-06-27T21:48:12.309793+0000","last_unstale":"2024-06-27T21:49:13.831175+0000","last_undegraded":"2024-06-27T21:49:13.831175+0000","last_fullsized":"2024-06-27T21:49:13.831175+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:36.073340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285933568,"num_objects":4363,"num_object_clones":0,"num_object_copies":13089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4363,"num_write_kb":279232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1461,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.284370+0000","last_change":"2024-06-27T21:48:18.178298+0000","last_active":"2024-06-27T21:49:14.284370+0000","last_peered":"2024-06-27T21:49:14.284370+0000","last_clean":"2024-06-27T21:49:14.284370+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:49:14.284370+0000","last_undegraded":"2024-06-27T21:49:14.284370+0000","last_fullsized":"2024-06-27T21:49:14.284370+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:55.400894+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:47:55.400894+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:21.330907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"945'4512","reported_seq":4579,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:17.438744+0000","last_change":"2024-06-27T21:48:11.986615+0000","last_active":"2024-06-27T21:49:17.438744+0000","last_peered":"2024-06-27T21:49:17.438744+0000","last_clean":"2024-06-27T21:49:17.438744+0000","last_became_active":"2024-06-27T21:48:11.986349+0000","last_became_peered":"2024-06-27T21:48:11.986349+0000","last_unstale":"2024-06-27T21:49:17.438744+0000","last_undegraded":"2024-06-27T21:49:17.438744+0000","last_fullsized":"2024-06-27T21:49:17.438744+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:22.977189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295698432,"num_objects":4512,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4512,"num_write_kb":288768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":162,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.286983+0000","last_change":"2024-06-27T21:48:01.046827+0000","last_active":"2024-06-27T21:49:14.286983+0000","last_peered":"2024-06-27T21:49:14.286983+0000","last_clean":"2024-06-27T21:49:14.286983+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:49:14.286983+0000","last_undegraded":"2024-06-27T21:49:14.286983+0000","last_fullsized":"2024-06-27T21:49:14.286983+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:01.046728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:01.046728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:22.168436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"945'4467","reported_seq":4535,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:17.410308+0000","last_change":"2024-06-27T21:48:12.309660+0000","last_active":"2024-06-27T21:49:17.410308+0000","last_peered":"2024-06-27T21:49:17.410308+0000","last_clean":"2024-06-27T21:49:17.410308+0000","last_became_active":"2024-06-27T21:48:12.309342+0000","last_became_peered":"2024-06-27T21:48:12.309342+0000","last_unstale":"2024-06-27T21:49:17.410308+0000","last_undegraded":"2024-06-27T21:49:17.410308+0000","last_fullsized":"2024-06-27T21:49:17.410308+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:20.858395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292749312,"num_objects":4467,"num_object_clones":0,"num_object_copies":13401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":285888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":65,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.971853+0000","last_change":"2024-06-27T21:48:37.006935+0000","last_active":"2024-06-27T21:49:14.971853+0000","last_peered":"2024-06-27T21:49:14.971853+0000","last_clean":"2024-06-27T21:49:14.971853+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:49:14.971853+0000","last_undegraded":"2024-06-27T21:49:14.971853+0000","last_fullsized":"2024-06-27T21:49:14.971853+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:37.006836+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:48:37.006836+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:59.789286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"945'4470","reported_seq":4538,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:17.427681+0000","last_change":"2024-06-27T21:48:12.198669+0000","last_active":"2024-06-27T21:49:17.427681+0000","last_peered":"2024-06-27T21:49:17.427681+0000","last_clean":"2024-06-27T21:49:17.427681+0000","last_became_active":"2024-06-27T21:48:12.198416+0000","last_became_peered":"2024-06-27T21:48:12.198416+0000","last_unstale":"2024-06-27T21:49:17.427681+0000","last_undegraded":"2024-06-27T21:49:17.427681+0000","last_fullsized":"2024-06-27T21:49:17.427681+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:22.499845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292945920,"num_objects":4470,"num_object_clones":0,"num_object_copies":13410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4470,"num_write_kb":286080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":272,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.283778+0000","last_change":"2024-06-27T21:48:03.112579+0000","last_active":"2024-06-27T21:49:13.283778+0000","last_peered":"2024-06-27T21:49:13.283778+0000","last_clean":"2024-06-27T21:49:13.283778+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:49:13.283778+0000","last_undegraded":"2024-06-27T21:49:13.283778+0000","last_fullsized":"2024-06-27T21:49:13.283778+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:03.112494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:03.112494+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:24.935386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"944'4376","reported_seq":4441,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.850837+0000","last_change":"2024-06-27T21:48:12.309489+0000","last_active":"2024-06-27T21:49:14.850837+0000","last_peered":"2024-06-27T21:49:14.850837+0000","last_clean":"2024-06-27T21:49:14.850837+0000","last_became_active":"2024-06-27T21:48:12.309089+0000","last_became_peered":"2024-06-27T21:48:12.309089+0000","last_unstale":"2024-06-27T21:49:14.850837+0000","last_undegraded":"2024-06-27T21:49:14.850837+0000","last_fullsized":"2024-06-27T21:49:14.850837+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:31.892235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286785536,"num_objects":4376,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":280064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,6],"acting":[11,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1348,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.280635+0000","last_change":"2024-06-27T21:48:00.530707+0000","last_active":"2024-06-27T21:49:14.280635+0000","last_peered":"2024-06-27T21:49:14.280635+0000","last_clean":"2024-06-27T21:49:14.280635+0000","last_became_active":"2024-06-27T21:47:59.079888+0000","last_became_peered":"2024-06-27T21:47:59.079888+0000","last_unstale":"2024-06-27T21:49:14.280635+0000","last_undegraded":"2024-06-27T21:49:14.280635+0000","last_fullsized":"2024-06-27T21:49:14.280635+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.530618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.530618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:21.819984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":89,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.974646+0000","last_change":"2024-06-27T21:48:03.757178+0000","last_active":"2024-06-27T21:49:14.974646+0000","last_peered":"2024-06-27T21:49:14.974646+0000","last_clean":"2024-06-27T21:49:14.974646+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:49:14.974646+0000","last_undegraded":"2024-06-27T21:49:14.974646+0000","last_fullsized":"2024-06-27T21:49:14.974646+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:03.757083+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:48:03.757083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:18.904595+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1056,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:15.211103+0000","last_change":"2024-06-27T21:49:15.211124+0000","last_active":"2024-06-27T21:49:15.211103+0000","last_peered":"2024-06-27T21:49:15.211103+0000","last_clean":"2024-06-27T21:49:15.211103+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:49:15.211103+0000","last_undegraded":"2024-06-27T21:49:15.211103+0000","last_fullsized":"2024-06-27T21:49:15.211103+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:15.211050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:15.211050+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:21.569767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1083,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.973332+0000","last_change":"2024-06-27T21:48:43.000826+0000","last_active":"2024-06-27T21:49:14.973332+0000","last_peered":"2024-06-27T21:49:14.973332+0000","last_clean":"2024-06-27T21:49:14.973332+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:49:14.973332+0000","last_undegraded":"2024-06-27T21:49:14.973332+0000","last_fullsized":"2024-06-27T21:49:14.973332+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:43.000735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:43.000735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:07.945537+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":162,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.286801+0000","last_change":"2024-06-27T21:48:33.250275+0000","last_active":"2024-06-27T21:49:14.286801+0000","last_peered":"2024-06-27T21:49:14.286801+0000","last_clean":"2024-06-27T21:49:14.286801+0000","last_became_active":"2024-06-27T21:46:34.530850+0000","last_became_peered":"2024-06-27T21:46:34.530850+0000","last_unstale":"2024-06-27T21:49:14.286801+0000","last_undegraded":"2024-06-27T21:49:14.286801+0000","last_fullsized":"2024-06-27T21:49:14.286801+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:33.250197+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:48:33.250197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:43.368122+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1045,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.280705+0000","last_change":"2024-06-27T21:47:59.741785+0000","last_active":"2024-06-27T21:49:14.280705+0000","last_peered":"2024-06-27T21:49:14.280705+0000","last_clean":"2024-06-27T21:49:14.280705+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:49:14.280705+0000","last_undegraded":"2024-06-27T21:49:14.280705+0000","last_fullsized":"2024-06-27T21:49:14.280705+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:59.741634+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:47:59.741634+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:15.671298+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":271,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.284060+0000","last_change":"2024-06-27T21:48:06.087520+0000","last_active":"2024-06-27T21:49:13.284060+0000","last_peered":"2024-06-27T21:49:13.284060+0000","last_clean":"2024-06-27T21:49:13.284060+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:49:13.284060+0000","last_undegraded":"2024-06-27T21:49:13.284060+0000","last_fullsized":"2024-06-27T21:49:13.284060+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:06.087437+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:06.087437+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:28.217034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":299,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.554366+0000","last_change":"2024-06-27T21:49:13.554401+0000","last_active":"2024-06-27T21:49:13.554366+0000","last_peered":"2024-06-27T21:49:13.554366+0000","last_clean":"2024-06-27T21:49:13.554366+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:49:13.554366+0000","last_undegraded":"2024-06-27T21:49:13.554366+0000","last_fullsized":"2024-06-27T21:49:13.554366+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:13.554287+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:49:13.554287+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:40.613036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":711,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.973984+0000","last_change":"2024-06-27T21:47:59.947241+0000","last_active":"2024-06-27T21:49:14.973984+0000","last_peered":"2024-06-27T21:49:14.973984+0000","last_clean":"2024-06-27T21:49:14.973984+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:49:14.973984+0000","last_undegraded":"2024-06-27T21:49:14.973984+0000","last_fullsized":"2024-06-27T21:49:14.973984+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:59.947125+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:47:59.947125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:17.824709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":93,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.974576+0000","last_change":"2024-06-27T21:47:59.802279+0000","last_active":"2024-06-27T21:49:14.974576+0000","last_peered":"2024-06-27T21:49:14.974576+0000","last_clean":"2024-06-27T21:49:14.974576+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:49:14.974576+0000","last_undegraded":"2024-06-27T21:49:14.974576+0000","last_fullsized":"2024-06-27T21:49:14.974576+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:59.802203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:47:59.802203+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:18.139232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":895,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.284342+0000","last_change":"2024-06-27T21:47:51.396070+0000","last_active":"2024-06-27T21:49:14.284342+0000","last_peered":"2024-06-27T21:49:14.284342+0000","last_clean":"2024-06-27T21:49:14.284342+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:49:14.284342+0000","last_undegraded":"2024-06-27T21:49:14.284342+0000","last_fullsized":"2024-06-27T21:49:14.284342+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:51.395994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:47:51.395994+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:17.400948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":996,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-06-27T21:49:13.279639+0000","last_change":"2024-06-27T21:48:52.674891+0000","last_active":"2024-06-27T21:49:13.279639+0000","last_peered":"2024-06-27T21:49:13.279639+0000","last_clean":"2024-06-27T21:49:13.279639+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:49:13.279639+0000","last_undegraded":"2024-06-27T21:49:13.279639+0000","last_fullsized":"2024-06-27T21:49:13.279639+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_clean_scrub_stamp":"2024-06-27T21:48:52.674762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:09.638022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":693,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.973864+0000","last_change":"2024-06-27T21:48:24.958797+0000","last_active":"2024-06-27T21:49:14.973864+0000","last_peered":"2024-06-27T21:49:14.973864+0000","last_clean":"2024-06-27T21:49:14.973864+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:49:14.973864+0000","last_undegraded":"2024-06-27T21:49:14.973864+0000","last_fullsized":"2024-06-27T21:49:14.973864+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:04.976398+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:48:04.976398+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:30.531216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":711,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.973928+0000","last_change":"2024-06-27T21:48:02.964528+0000","last_active":"2024-06-27T21:49:14.973928+0000","last_peered":"2024-06-27T21:49:14.973928+0000","last_clean":"2024-06-27T21:49:14.973928+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:49:14.973928+0000","last_undegraded":"2024-06-27T21:49:14.973928+0000","last_fullsized":"2024-06-27T21:49:14.973928+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:02.964429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:02.964429+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:17.824747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":162,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.287035+0000","last_change":"2024-06-27T21:48:02.060711+0000","last_active":"2024-06-27T21:49:14.287035+0000","last_peered":"2024-06-27T21:49:14.287035+0000","last_clean":"2024-06-27T21:49:14.287035+0000","last_became_active":"2024-06-27T21:46:34.528582+0000","last_became_peered":"2024-06-27T21:46:34.528582+0000","last_unstale":"2024-06-27T21:49:14.287035+0000","last_undegraded":"2024-06-27T21:49:14.287035+0000","last_fullsized":"2024-06-27T21:49:14.287035+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:02.060563+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:02.060563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:25.296698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":162,"reported_epoch":944,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.286994+0000","last_change":"2024-06-27T21:48:40.018430+0000","last_active":"2024-06-27T21:49:14.286994+0000","last_peered":"2024-06-27T21:49:14.286994+0000","last_clean":"2024-06-27T21:49:14.286994+0000","last_became_active":"2024-06-27T21:46:34.528359+0000","last_became_peered":"2024-06-27T21:46:34.528359+0000","last_unstale":"2024-06-27T21:49:14.286994+0000","last_undegraded":"2024-06-27T21:49:14.286994+0000","last_fullsized":"2024-06-27T21:49:14.286994+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:40.018304+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:48:40.018304+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:05.786297+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4659150848,"num_objects":71093,"num_object_clones":0,"num_object_copies":213279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71093,"num_write_kb":4549952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13873184768,"data_stored":13873184768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71093,"ondisk_log_size":71093,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253345,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2118604,"kb_used_data":1104148,"kb_used_omap":1,"kb_used_meta":1014078,"kb_avail":91626548,"statfs":{"total":95995035648,"available":93825585152,"internally_reserved":0,"allocated":1130647552,"data_stored":1130116941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1038416325},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514981,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2314092,"kb_used_data":1390060,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":91431060,"statfs":{"total":95995035648,"available":93625405440,"internally_reserved":0,"allocated":1423421440,"data_stored":1422947385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":946010560},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743913,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2389236,"kb_used_data":1668076,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91355916,"statfs":{"total":95995035648,"available":93548457984,"internally_reserved":0,"allocated":1708109824,"data_stored":1707635769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":738458048},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972846,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173880,"kb_used_data":592944,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92571272,"statfs":{"total":95995035648,"available":94792982528,"internally_reserved":0,"allocated":607174656,"data_stored":606719967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267181,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2170068,"kb_used_data":1362460,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":91575084,"statfs":{"total":95995035648,"available":93772886016,"internally_reserved":0,"allocated":1395159040,"data_stored":1394624877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":826603978},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528823,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271804,"kb_used_data":558704,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92473348,"statfs":{"total":95995035648,"available":94692708352,"internally_reserved":0,"allocated":572112896,"data_stored":571658207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757757,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3244800,"kb_used_data":2618040,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":90500352,"statfs":{"total":95995035648,"available":92672360448,"internally_reserved":0,"allocated":2680872960,"data_stored":2680414719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161462,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099460,"kb_used_data":547004,"kb_used_omap":92,"kb_used_meta":552355,"kb_avail":92645692,"statfs":{"total":95995035648,"available":94869188608,"internally_reserved":0,"allocated":560132096,"data_stored":559542200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":565611618},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182917,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2046456,"kb_used_data":1423856,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91698696,"statfs":{"total":95995035648,"available":93899464704,"internally_reserved":0,"allocated":1458028544,"data_stored":1457573855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":637532615},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411849,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809052,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936100,"statfs":{"total":95995035648,"available":95166566400,"internally_reserved":0,"allocated":1667072,"data_stored":1132909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575376,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3014420,"kb_used_data":2202836,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":90730732,"statfs":{"total":95995035648,"available":92908269568,"internally_reserved":0,"allocated":2255704064,"data_stored":2255173453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881212,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92863940,"statfs":{"total":95995035648,"available":95092674560,"internally_reserved":0,"allocated":1556480,"data_stored":1101791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2262892544,"data_stored":2262892544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1467809792,"data_stored":1467809792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":564920320,"data_stored":564920320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2687303680,"data_stored":2687303680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":575602688,"data_stored":575602688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1408172032,"data_stored":1408172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":610467840,"data_stored":610467840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1720057856,"data_stored":1720057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1433534464,"data_stored":1433534464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1142423552,"data_stored":1142423552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:49:21.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:49:21.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:21.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:21.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 57 16 77439 77423 84.8603 77 0.00484443 0.0117783 2024-06-27T21:49:21.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:21.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:21.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:22.094 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:22.116 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:49:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:49:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:49:22.116 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:49:22.116 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:49:22.127 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:22.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:22.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:22.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:22 smithi151 ceph-mon[28965]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:22 smithi151 ceph-mon[28965]: pgmap v2302: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-27T21:49:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:22 smithi151 ceph-mon[28965]: 9.2 scrub starts 2024-06-27T21:49:22.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:22.385 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:22.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:22.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:22 smithi190 ceph-mon[32247]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:22 smithi190 ceph-mon[32247]: pgmap v2302: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-27T21:49:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:22 smithi190 ceph-mon[32247]: 9.2 scrub starts 2024-06-27T21:49:22.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:22 smithi192 ceph-mon[32956]: from='client.17799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:22.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:22 smithi192 ceph-mon[32956]: pgmap v2302: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-06-27T21:49:22.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:22 smithi192 ceph-mon[32956]: 9.2 scrub starts 2024-06-27T21:49:22.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:22.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:22.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:22.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:22.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:22.897 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:22.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:22.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 58 16 78627 78611 84.6773 74.25 0.00520581 0.0118037 2024-06-27T21:49:22.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:23.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:23.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:23.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:23.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:23 smithi151 ceph-mon[28965]: 9.2 scrub ok 2024-06-27T21:49:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:23 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:49:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:23 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:49:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:23 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:49:23.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:23 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:49:23.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:23.285 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:23.388 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:23.415 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:23 smithi190 ceph-mon[32247]: 9.2 scrub ok 2024-06-27T21:49:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:23 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:49:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:23 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:49:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:23 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:49:23.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:23 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:49:23.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:23.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:23.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:23 smithi192 ceph-mon[32956]: 9.2 scrub ok 2024-06-27T21:49:23.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:23 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:49:23.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:23 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:49:23.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:23 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:49:23.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:23 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:49:23.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:49:23.640 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:23.665 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:23.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:23.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:23.880 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:49:23.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:23.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 59 16 79376 79360 84.0357 46.8125 0.0136238 0.0118812 2024-06-27T21:49:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:23.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:23.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:24.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:24.050 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:24.093 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:49:24.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:24 smithi151 ceph-mon[28965]: pgmap v2303: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:49:24.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:24 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3819240336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:24.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:24 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:24.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:24 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:49:24.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:24 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:49:24.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:24.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:49:24.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:49:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:24.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:24.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:49:24.207 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:49:24.233 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:49:24.233 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:49:24.261 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:49:24.261 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:49:24.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:24.289 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:49:24.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:49:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:24 smithi192 ceph-mon[32956]: pgmap v2303: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:49:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:24 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3819240336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:24 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:24 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:49:24.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:24 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:49:24.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:24.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:49:24.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:49:24.345 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:49:24.346 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:49:24.357 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:24.375 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:49:24.375 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:49:24.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:24.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:24 smithi190 ceph-mon[32247]: pgmap v2303: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-06-27T21:49:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:24 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3819240336' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:24 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:24 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:49:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:24 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:49:24.533 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:24.556 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:24.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:49:24.605 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:49:24.636 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:49:24.636 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:49:24.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:24.663 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:49:24.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:49:24.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:24.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:49:24.692 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:49:24.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:24.816 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:24.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:49:24.909230+0000 min lat: 0.0010169 max lat: 0.321704 avg lat: 0.0119415 2024-06-27T21:49:24.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:49:24.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 60 16 80396 80380 83.6976 63.75 0.00466832 0.0119415 2024-06-27T21:49:24.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:24.943 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:24.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:25.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:25.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:25.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: osdmap e946: 12 total, 12 up, 12 in 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/202242947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: pgmap v2305: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:49:25.162 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:25 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:49:25.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:25.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:25.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:25.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:25.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:25.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: osdmap e946: 12 total, 12 up, 12 in 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/202242947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: pgmap v2305: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:49:25.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:25 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:49:25.450 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:25.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: osdmap e946: 12 total, 12 up, 12 in 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/202242947' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: pgmap v2305: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:49:25.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:25 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:49:25.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:25.579 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:25.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:25.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:25.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:25.839 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:25.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 61 16 81158 81142 83.1064 47.625 0.0411031 0.0120216 2024-06-27T21:49:25.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:25.970 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:25.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:26.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:26.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:26.134 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:26.134 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:26.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:26.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:26.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:26 smithi151 ceph-mon[28965]: 9.1 deep-scrub starts 2024-06-27T21:49:26.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:26 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:49:26.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:26 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:49:26.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:26.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:26.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:26.472 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:26 smithi192 ceph-mon[32956]: 9.1 deep-scrub starts 2024-06-27T21:49:26.472 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:26 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:49:26.472 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:26 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:49:26.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:26.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:26 smithi190 ceph-mon[32247]: 9.1 deep-scrub starts 2024-06-27T21:49:26.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:26 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:49:26.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:26 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:49:26.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:26.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:26.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:26.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:26.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:26.871 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:26.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:49:26.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 62 16 82104 82088 82.7196 59.125 0.00405581 0.012083 2024-06-27T21:49:26.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:49:26.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:26.997 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:26.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:27.099 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:27.125 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:27.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:27.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:27 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:49:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:27 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:49:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:27 smithi190 ceph-mon[32247]: pgmap v2306: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:49:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:27 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:49:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:27 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:49:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:27 smithi192 ceph-mon[32956]: pgmap v2306: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:49:27.353 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:27.361 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:49:27.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:27.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:27 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:49:27.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:27 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:49:27.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:27 smithi151 ceph-mon[28965]: pgmap v2306: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:49:27.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:27.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:27.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:27.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:27.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:49:27.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:27.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:27.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:27.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 63 15 83113 83098 82.4086 63.125 0.00937782 0.012129 2024-06-27T21:49:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:27.938 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:27.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:28.039 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:28.066 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:28.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:28.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:28.298 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:28.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:28.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:28.452 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:28.457 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:28.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:28.574 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:28 smithi151 ceph-mon[28965]: 9.c scrub starts 2024-06-27T21:49:28.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:28.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:28.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:28 smithi190 ceph-mon[32247]: 9.c scrub starts 2024-06-27T21:49:28.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:28.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:28 smithi192 ceph-mon[32956]: 9.c scrub starts 2024-06-27T21:49:28.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:28.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:28.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 64 16 84239 84223 82.2195 70.3125 0.0158789 0.0121557 2024-06-27T21:49:28.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:28.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:28.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:29.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:29.099 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:29.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:29.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:29.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:29.346 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:29.346 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:29.357 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:29.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:29 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:49:29.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:29 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:49:29.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:29 smithi151 ceph-mon[28965]: pgmap v2307: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:49:29.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:29 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:49:29.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:29 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:49:29.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:29 smithi151 ceph-mon[28965]: 9.c scrub ok 2024-06-27T21:49:29.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:29.484 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:29.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:29.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:29.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:29.642 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:29.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:29 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:49:29.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:29 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:49:29.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:29 smithi190 ceph-mon[32247]: pgmap v2307: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:49:29.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:29 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:49:29.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:29 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:49:29.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:29 smithi190 ceph-mon[32247]: 9.c scrub ok 2024-06-27T21:49:29.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:29.749 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:29.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:29 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:49:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:29 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:49:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:29 smithi192 ceph-mon[32956]: pgmap v2307: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-06-27T21:49:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:29 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:49:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:29 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:49:29.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:29 smithi192 ceph-mon[32956]: 9.c scrub ok 2024-06-27T21:49:29.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:29.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:29.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 65 16 85068 85052 81.7518 51.8125 0.0324035 0.0122255 2024-06-27T21:49:29.932 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:49:29.933 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:49:29.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:30.006 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:30.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:49:30.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:30.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:30.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:30.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:30.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:30.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:30.410 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:49:30.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:30.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:30.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:30.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:30.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:30.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:30.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:30.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 66 16 86015 85999 81.4099 59.1875 0.02554 0.0122758 2024-06-27T21:49:30.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:30.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:31.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:31.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:31.055 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2307,"stamp":"2024-06-27T21:49:28.358984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5337121312,"num_objects":81431,"num_object_clones":0,"num_object_copies":244293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81429,"num_write_kb":5211456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84785,"ondisk_log_size":84785,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":24593168,"kb_used_data":15487832,"kb_used_omap":110,"kb_used_meta":9104209,"kb_avail":1100348656,"statfs":{"total":1151940427776,"available":1126757023744,"internally_reserved":0,"allocated":15859539968,"data_stored":15854538100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9322710999},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":677380096,"num_objects":10336,"num_object_clones":0,"num_object_copies":31008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10336,"num_write_kb":661504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002086"},"pg_stats":[{"pgid":"9.3","version":"946'4971","reported_seq":5040,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:25.426677+0000","last_change":"2024-06-27T21:48:11.986786+0000","last_active":"2024-06-27T21:49:25.426677+0000","last_peered":"2024-06-27T21:49:25.426677+0000","last_clean":"2024-06-27T21:49:25.426677+0000","last_became_active":"2024-06-27T21:48:11.986220+0000","last_became_peered":"2024-06-27T21:48:11.986220+0000","last_unstale":"2024-06-27T21:49:25.426677+0000","last_undegraded":"2024-06-27T21:49:25.426677+0000","last_fullsized":"2024-06-27T21:49:25.426677+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.515921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325779456,"num_objects":4971,"num_object_clones":0,"num_object_copies":14913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4971,"num_write_kb":318144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":172,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.093653+0000","last_change":"2024-06-27T21:48:37.072258+0000","last_active":"2024-06-27T21:49:24.093653+0000","last_peered":"2024-06-27T21:49:24.093653+0000","last_clean":"2024-06-27T21:49:24.093653+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:49:24.093653+0000","last_undegraded":"2024-06-27T21:49:24.093653+0000","last_fullsized":"2024-06-27T21:49:24.093653+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:37.072188+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:56.646935+0000","last_clean_scrub_stamp":"2024-06-27T21:48:37.072188+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:52.196369+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"946'5319","reported_seq":5388,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:28.158545+0000","last_change":"2024-06-27T21:48:12.310120+0000","last_active":"2024-06-27T21:49:28.158545+0000","last_peered":"2024-06-27T21:49:28.158545+0000","last_clean":"2024-06-27T21:49:28.158545+0000","last_became_active":"2024-06-27T21:48:12.309522+0000","last_became_peered":"2024-06-27T21:48:12.309522+0000","last_unstale":"2024-06-27T21:49:28.158545+0000","last_undegraded":"2024-06-27T21:49:28.158545+0000","last_fullsized":"2024-06-27T21:49:28.158545+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5319,"log_dups_size":0,"ondisk_log_size":5319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:35.020638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348585984,"num_objects":5319,"num_object_clones":0,"num_object_copies":15957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":340416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":138,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.092116+0000","last_change":"2024-06-27T21:48:39.725238+0000","last_active":"2024-06-27T21:49:24.092116+0000","last_peered":"2024-06-27T21:49:24.092116+0000","last_clean":"2024-06-27T21:49:24.092116+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:49:24.092116+0000","last_undegraded":"2024-06-27T21:49:24.092116+0000","last_fullsized":"2024-06-27T21:49:24.092116+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:39.725111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:39.725111+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:42.976869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"946'5022","reported_seq":5092,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.853094+0000","last_change":"2024-06-27T21:48:11.990938+0000","last_active":"2024-06-27T21:49:24.853094+0000","last_peered":"2024-06-27T21:49:24.853094+0000","last_clean":"2024-06-27T21:49:24.853094+0000","last_became_active":"2024-06-27T21:48:11.990332+0000","last_became_peered":"2024-06-27T21:48:11.990332+0000","last_unstale":"2024-06-27T21:49:24.853094+0000","last_undegraded":"2024-06-27T21:49:24.853094+0000","last_fullsized":"2024-06-27T21:49:24.853094+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329121792,"num_objects":5022,"num_object_clones":0,"num_object_copies":15066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5022,"num_write_kb":321408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":276,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.974956+0000","last_change":"2024-06-27T21:48:35.993226+0000","last_active":"2024-06-27T21:49:14.974956+0000","last_peered":"2024-06-27T21:49:14.974956+0000","last_clean":"2024-06-27T21:49:14.974956+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:49:14.974956+0000","last_undegraded":"2024-06-27T21:49:14.974956+0000","last_fullsized":"2024-06-27T21:49:14.974956+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:35.993117+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:35.993117+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:57.566564+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"946'5074","reported_seq":5143,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.844572+0000","last_change":"2024-06-27T21:48:12.310003+0000","last_active":"2024-06-27T21:49:24.844572+0000","last_peered":"2024-06-27T21:49:24.844572+0000","last_clean":"2024-06-27T21:49:24.844572+0000","last_became_active":"2024-06-27T21:48:12.309579+0000","last_became_peered":"2024-06-27T21:48:12.309579+0000","last_unstale":"2024-06-27T21:49:24.844572+0000","last_undegraded":"2024-06-27T21:49:24.844572+0000","last_fullsized":"2024-06-27T21:49:24.844572+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:40.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332529664,"num_objects":5074,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5074,"num_write_kb":324736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1053,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:20.921399+0000","last_change":"2024-06-27T21:49:20.921441+0000","last_active":"2024-06-27T21:49:20.921399+0000","last_peered":"2024-06-27T21:49:20.921399+0000","last_clean":"2024-06-27T21:49:20.921399+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:49:20.921399+0000","last_undegraded":"2024-06-27T21:49:20.921399+0000","last_fullsized":"2024-06-27T21:49:20.921399+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:20.921338+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:49:20.921338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:38.530191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"946'4968","reported_seq":5037,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.854101+0000","last_change":"2024-06-27T21:48:11.990014+0000","last_active":"2024-06-27T21:49:24.854101+0000","last_peered":"2024-06-27T21:49:24.854101+0000","last_clean":"2024-06-27T21:49:24.854101+0000","last_became_active":"2024-06-27T21:48:11.989652+0000","last_became_peered":"2024-06-27T21:48:11.989652+0000","last_unstale":"2024-06-27T21:49:24.854101+0000","last_undegraded":"2024-06-27T21:49:24.854101+0000","last_fullsized":"2024-06-27T21:49:24.854101+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325582848,"num_objects":4968,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":317952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1664,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.092869+0000","last_change":"2024-06-27T21:49:19.557403+0000","last_active":"2024-06-27T21:49:24.092869+0000","last_peered":"2024-06-27T21:49:24.092869+0000","last_clean":"2024-06-27T21:49:24.092869+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:49:24.092869+0000","last_undegraded":"2024-06-27T21:49:24.092869+0000","last_fullsized":"2024-06-27T21:49:24.092869+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:19.557294+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:49:19.557294+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:25.339126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"946'5088","reported_seq":5192,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:25.420632+0000","last_change":"2024-06-27T21:49:14.968114+0000","last_active":"2024-06-27T21:49:25.420632+0000","last_peered":"2024-06-27T21:49:25.420632+0000","last_clean":"2024-06-27T21:49:25.420632+0000","last_became_active":"2024-06-27T21:48:25.035404+0000","last_became_peered":"2024-06-27T21:48:25.035404+0000","last_unstale":"2024-06-27T21:49:25.420632+0000","last_undegraded":"2024-06-27T21:49:25.420632+0000","last_fullsized":"2024-06-27T21:49:25.420632+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'4404","last_scrub_stamp":"2024-06-27T21:49:14.967854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:14.967854+0000","objects_scrubbed":4367,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:19.717925+0000","scrub_duration":1019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333447168,"num_objects":5088,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5088,"num_write_kb":325632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":639,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.095400+0000","last_change":"2024-06-27T21:48:43.028417+0000","last_active":"2024-06-27T21:49:24.095400+0000","last_peered":"2024-06-27T21:49:24.095400+0000","last_clean":"2024-06-27T21:49:24.095400+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:49:24.095400+0000","last_undegraded":"2024-06-27T21:49:24.095400+0000","last_fullsized":"2024-06-27T21:49:24.095400+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:43.028305+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:48:43.028305+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:45.889033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.b","version":"946'5236","reported_seq":5305,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:28.155273+0000","last_change":"2024-06-27T21:48:11.989882+0000","last_active":"2024-06-27T21:49:28.155273+0000","last_peered":"2024-06-27T21:49:28.155273+0000","last_clean":"2024-06-27T21:49:28.155273+0000","last_became_active":"2024-06-27T21:48:11.989420+0000","last_became_peered":"2024-06-27T21:48:11.989420+0000","last_unstale":"2024-06-27T21:49:28.155273+0000","last_undegraded":"2024-06-27T21:49:28.155273+0000","last_fullsized":"2024-06-27T21:49:28.155273+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.290203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":335104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":1042,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.968237+0000","last_change":"2024-06-27T21:49:14.931890+0000","last_active":"2024-06-27T21:49:14.968237+0000","last_peered":"2024-06-27T21:49:14.968237+0000","last_clean":"2024-06-27T21:49:14.968237+0000","last_became_active":"2024-06-27T21:48:18.177749+0000","last_became_peered":"2024-06-27T21:48:18.177749+0000","last_unstale":"2024-06-27T21:49:14.968237+0000","last_undegraded":"2024-06-27T21:49:14.968237+0000","last_fullsized":"2024-06-27T21:49:14.968237+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:14.931780+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_clean_scrub_stamp":"2024-06-27T21:49:14.931780+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:23.596370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"946'5139","reported_seq":5209,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:27.754097+0000","last_change":"2024-06-27T21:48:12.311450+0000","last_active":"2024-06-27T21:49:27.754097+0000","last_peered":"2024-06-27T21:49:27.754097+0000","last_clean":"2024-06-27T21:49:27.754097+0000","last_became_active":"2024-06-27T21:48:12.311072+0000","last_became_peered":"2024-06-27T21:48:12.311072+0000","last_unstale":"2024-06-27T21:49:27.754097+0000","last_undegraded":"2024-06-27T21:49:27.754097+0000","last_fullsized":"2024-06-27T21:49:27.754097+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:34.768612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5139,"num_write_kb":328896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":268,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.976286+0000","last_change":"2024-06-27T21:48:00.010050+0000","last_active":"2024-06-27T21:49:14.976286+0000","last_peered":"2024-06-27T21:49:14.976286+0000","last_clean":"2024-06-27T21:49:14.976286+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:49:14.976286+0000","last_undegraded":"2024-06-27T21:49:14.976286+0000","last_fullsized":"2024-06-27T21:49:14.976286+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.009982+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.009982+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:28.111931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"946'5003","reported_seq":5072,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.825357+0000","last_change":"2024-06-27T21:48:11.998116+0000","last_active":"2024-06-27T21:49:24.825357+0000","last_peered":"2024-06-27T21:49:24.825357+0000","last_clean":"2024-06-27T21:49:24.825357+0000","last_became_active":"2024-06-27T21:48:11.997839+0000","last_became_peered":"2024-06-27T21:48:11.997839+0000","last_unstale":"2024-06-27T21:49:24.825357+0000","last_undegraded":"2024-06-27T21:49:24.825357+0000","last_fullsized":"2024-06-27T21:49:24.825357+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:37.072712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327876608,"num_objects":5003,"num_object_clones":0,"num_object_copies":15009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5003,"num_write_kb":320192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":276,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.389017+0000","last_change":"2024-06-27T21:49:16.258267+0000","last_active":"2024-06-27T21:49:24.389017+0000","last_peered":"2024-06-27T21:49:24.389017+0000","last_clean":"2024-06-27T21:49:24.389017+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:49:24.389017+0000","last_undegraded":"2024-06-27T21:49:24.389017+0000","last_fullsized":"2024-06-27T21:49:24.389017+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:16.258197+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:49:16.258197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:37.926239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"946'5199","reported_seq":5326,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:27.798938+0000","last_change":"2024-06-27T21:49:16.636144+0000","last_active":"2024-06-27T21:49:27.798938+0000","last_peered":"2024-06-27T21:49:27.798938+0000","last_clean":"2024-06-27T21:49:27.798938+0000","last_became_active":"2024-06-27T21:48:24.958699+0000","last_became_peered":"2024-06-27T21:48:24.958699+0000","last_unstale":"2024-06-27T21:49:27.798938+0000","last_undegraded":"2024-06-27T21:49:27.798938+0000","last_fullsized":"2024-06-27T21:49:27.798938+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'4466","last_scrub_stamp":"2024-06-27T21:49:16.635894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:16.635894+0000","objects_scrubbed":4435,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:41.281934+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340721664,"num_objects":5199,"num_object_clones":0,"num_object_copies":15597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5199,"num_write_kb":332736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1371,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.094938+0000","last_change":"2024-06-27T21:48:18.178256+0000","last_active":"2024-06-27T21:49:24.094938+0000","last_peered":"2024-06-27T21:49:24.094938+0000","last_clean":"2024-06-27T21:49:24.094938+0000","last_became_active":"2024-06-27T21:48:18.177860+0000","last_became_peered":"2024-06-27T21:48:18.177860+0000","last_unstale":"2024-06-27T21:49:24.094938+0000","last_undegraded":"2024-06-27T21:49:24.094938+0000","last_fullsized":"2024-06-27T21:49:24.094938+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:56.354808+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:47:56.354808+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"945'5001","reported_seq":5068,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:23.763329+0000","last_change":"2024-06-27T21:48:12.309880+0000","last_active":"2024-06-27T21:49:23.763329+0000","last_peered":"2024-06-27T21:49:23.763329+0000","last_clean":"2024-06-27T21:49:23.763329+0000","last_became_active":"2024-06-27T21:48:12.309446+0000","last_became_peered":"2024-06-27T21:48:12.309446+0000","last_unstale":"2024-06-27T21:49:23.763329+0000","last_undegraded":"2024-06-27T21:49:23.763329+0000","last_fullsized":"2024-06-27T21:49:23.763329+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:30.856501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327745536,"num_objects":5001,"num_object_clones":0,"num_object_copies":15003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":320064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1444,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.094789+0000","last_change":"2024-06-27T21:48:00.289561+0000","last_active":"2024-06-27T21:49:24.094789+0000","last_peered":"2024-06-27T21:49:24.094789+0000","last_clean":"2024-06-27T21:49:24.094789+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:49:24.094789+0000","last_undegraded":"2024-06-27T21:49:24.094789+0000","last_fullsized":"2024-06-27T21:49:24.094789+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.289470+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.289470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:28.638908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"945'5046","reported_seq":5112,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:23.780535+0000","last_change":"2024-06-27T21:48:12.310346+0000","last_active":"2024-06-27T21:49:23.780535+0000","last_peered":"2024-06-27T21:49:23.780535+0000","last_clean":"2024-06-27T21:49:23.780535+0000","last_became_active":"2024-06-27T21:48:12.309793+0000","last_became_peered":"2024-06-27T21:48:12.309793+0000","last_unstale":"2024-06-27T21:49:23.780535+0000","last_undegraded":"2024-06-27T21:49:23.780535+0000","last_fullsized":"2024-06-27T21:49:23.780535+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:36.073340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5046,"num_write_kb":322944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1465,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.094617+0000","last_change":"2024-06-27T21:48:18.178298+0000","last_active":"2024-06-27T21:49:24.094617+0000","last_peered":"2024-06-27T21:49:24.094617+0000","last_clean":"2024-06-27T21:49:24.094617+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:49:24.094617+0000","last_undegraded":"2024-06-27T21:49:24.094617+0000","last_fullsized":"2024-06-27T21:49:24.094617+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:47:55.400894+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:47:55.400894+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"946'5112","reported_seq":5182,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:27.457674+0000","last_change":"2024-06-27T21:48:11.986615+0000","last_active":"2024-06-27T21:49:27.457674+0000","last_peered":"2024-06-27T21:49:27.457674+0000","last_clean":"2024-06-27T21:49:27.457674+0000","last_became_active":"2024-06-27T21:48:11.986349+0000","last_became_peered":"2024-06-27T21:48:11.986349+0000","last_unstale":"2024-06-27T21:49:27.457674+0000","last_undegraded":"2024-06-27T21:49:27.457674+0000","last_fullsized":"2024-06-27T21:49:27.457674+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5112,"log_dups_size":0,"ondisk_log_size":5112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335020032,"num_objects":5112,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":327168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":174,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.093728+0000","last_change":"2024-06-27T21:49:22.189855+0000","last_active":"2024-06-27T21:49:24.093728+0000","last_peered":"2024-06-27T21:49:24.093728+0000","last_clean":"2024-06-27T21:49:24.093728+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:49:24.093728+0000","last_undegraded":"2024-06-27T21:49:24.093728+0000","last_fullsized":"2024-06-27T21:49:24.093728+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:22.189748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:22.189748+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:32.782620+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"946'5075","reported_seq":5199,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:27.436512+0000","last_change":"2024-06-27T21:49:21.857234+0000","last_active":"2024-06-27T21:49:27.436512+0000","last_peered":"2024-06-27T21:49:27.436512+0000","last_clean":"2024-06-27T21:49:27.436512+0000","last_became_active":"2024-06-27T21:48:12.309342+0000","last_became_peered":"2024-06-27T21:48:12.309342+0000","last_unstale":"2024-06-27T21:49:27.436512+0000","last_undegraded":"2024-06-27T21:49:27.436512+0000","last_fullsized":"2024-06-27T21:49:27.436512+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'4766","last_scrub_stamp":"2024-06-27T21:49:21.857113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:21.857113+0000","objects_scrubbed":4732,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:30.845701+0000","scrub_duration":986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332595200,"num_objects":5075,"num_object_clones":0,"num_object_copies":15225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5075,"num_write_kb":324800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":67,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.099990+0000","last_change":"2024-06-27T21:48:37.006935+0000","last_active":"2024-06-27T21:49:24.099990+0000","last_peered":"2024-06-27T21:49:24.099990+0000","last_clean":"2024-06-27T21:49:24.099990+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:49:24.099990+0000","last_undegraded":"2024-06-27T21:49:24.099990+0000","last_fullsized":"2024-06-27T21:49:24.099990+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:37.006836+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:48:37.006836+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:59.789286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"946'5634","reported_seq":5744,"reported_epoch":946,"state":"active+clean+scrubbing+deep","last_fresh":"2024-06-27T21:49:27.462785+0000","last_change":"2024-06-27T21:49:24.818222+0000","last_active":"2024-06-27T21:49:27.462785+0000","last_peered":"2024-06-27T21:49:27.462785+0000","last_clean":"2024-06-27T21:49:27.462785+0000","last_became_active":"2024-06-27T21:48:12.198416+0000","last_became_peered":"2024-06-27T21:48:12.198416+0000","last_unstale":"2024-06-27T21:49:27.462785+0000","last_undegraded":"2024-06-27T21:49:27.462785+0000","last_fullsized":"2024-06-27T21:49:27.462785+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":899,"log_size":5634,"log_dups_size":0,"ondisk_log_size":5634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336199680,"num_objects":5130,"num_object_clones":0,"num_object_copies":15390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":328320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":276,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.974920+0000","last_change":"2024-06-27T21:48:03.112579+0000","last_active":"2024-06-27T21:49:14.974920+0000","last_peered":"2024-06-27T21:49:14.974920+0000","last_clean":"2024-06-27T21:49:14.974920+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:49:14.974920+0000","last_undegraded":"2024-06-27T21:49:14.974920+0000","last_fullsized":"2024-06-27T21:49:14.974920+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:03.112494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:03.112494+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:24.935386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"946'5046","reported_seq":5116,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.887134+0000","last_change":"2024-06-27T21:48:12.309489+0000","last_active":"2024-06-27T21:49:24.887134+0000","last_peered":"2024-06-27T21:49:24.887134+0000","last_clean":"2024-06-27T21:49:24.887134+0000","last_became_active":"2024-06-27T21:48:12.309089+0000","last_became_peered":"2024-06-27T21:48:12.309089+0000","last_unstale":"2024-06-27T21:49:24.887134+0000","last_undegraded":"2024-06-27T21:49:24.887134+0000","last_fullsized":"2024-06-27T21:49:24.887134+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:31.892235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5046,"num_write_kb":322944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,6],"acting":[11,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1352,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.389300+0000","last_change":"2024-06-27T21:48:00.530707+0000","last_active":"2024-06-27T21:49:24.389300+0000","last_peered":"2024-06-27T21:49:24.389300+0000","last_clean":"2024-06-27T21:49:24.389300+0000","last_became_active":"2024-06-27T21:47:59.079888+0000","last_became_peered":"2024-06-27T21:47:59.079888+0000","last_unstale":"2024-06-27T21:49:24.389300+0000","last_undegraded":"2024-06-27T21:49:24.389300+0000","last_fullsized":"2024-06-27T21:49:24.389300+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.530618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.530618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":99,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.092708+0000","last_change":"2024-06-27T21:49:23.589601+0000","last_active":"2024-06-27T21:49:24.092708+0000","last_peered":"2024-06-27T21:49:24.092708+0000","last_clean":"2024-06-27T21:49:24.092708+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:49:24.092708+0000","last_undegraded":"2024-06-27T21:49:24.092708+0000","last_fullsized":"2024-06-27T21:49:24.092708+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:23.589530+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:49:23.589530+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:47.854335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1058,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.388935+0000","last_change":"2024-06-27T21:49:15.211124+0000","last_active":"2024-06-27T21:49:24.388935+0000","last_peered":"2024-06-27T21:49:24.388935+0000","last_clean":"2024-06-27T21:49:24.388935+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:49:24.388935+0000","last_undegraded":"2024-06-27T21:49:24.388935+0000","last_fullsized":"2024-06-27T21:49:24.388935+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:15.211050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:15.211050+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:21.569767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1085,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.092888+0000","last_change":"2024-06-27T21:48:43.000826+0000","last_active":"2024-06-27T21:49:24.092888+0000","last_peered":"2024-06-27T21:49:24.092888+0000","last_clean":"2024-06-27T21:49:24.092888+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:49:24.092888+0000","last_undegraded":"2024-06-27T21:49:24.092888+0000","last_fullsized":"2024-06-27T21:49:24.092888+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:43.000735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:43.000735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:07.945537+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":166,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.093555+0000","last_change":"2024-06-27T21:48:33.250275+0000","last_active":"2024-06-27T21:49:24.093555+0000","last_peered":"2024-06-27T21:49:24.093555+0000","last_clean":"2024-06-27T21:49:24.093555+0000","last_became_active":"2024-06-27T21:46:34.530850+0000","last_became_peered":"2024-06-27T21:46:34.530850+0000","last_unstale":"2024-06-27T21:49:24.093555+0000","last_undegraded":"2024-06-27T21:49:24.093555+0000","last_fullsized":"2024-06-27T21:49:24.093555+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:33.250197+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:48:33.250197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:43.368122+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1057,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.389184+0000","last_change":"2024-06-27T21:49:15.693241+0000","last_active":"2024-06-27T21:49:24.389184+0000","last_peered":"2024-06-27T21:49:24.389184+0000","last_clean":"2024-06-27T21:49:24.389184+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:49:24.389184+0000","last_undegraded":"2024-06-27T21:49:24.389184+0000","last_fullsized":"2024-06-27T21:49:24.389184+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:15.693106+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:49:15.693106+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:20.863365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":275,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.974823+0000","last_change":"2024-06-27T21:48:06.087520+0000","last_active":"2024-06-27T21:49:14.974823+0000","last_peered":"2024-06-27T21:49:14.974823+0000","last_clean":"2024-06-27T21:49:14.974823+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:49:14.974823+0000","last_undegraded":"2024-06-27T21:49:14.974823+0000","last_fullsized":"2024-06-27T21:49:14.974823+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:06.087437+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:06.087437+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:28.217034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":303,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.974880+0000","last_change":"2024-06-27T21:49:13.554401+0000","last_active":"2024-06-27T21:49:14.974880+0000","last_peered":"2024-06-27T21:49:14.974880+0000","last_clean":"2024-06-27T21:49:14.974880+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:49:14.974880+0000","last_undegraded":"2024-06-27T21:49:14.974880+0000","last_fullsized":"2024-06-27T21:49:14.974880+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:13.554287+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:49:13.554287+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:40.613036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":721,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.106113+0000","last_change":"2024-06-27T21:49:20.087934+0000","last_active":"2024-06-27T21:49:24.106113+0000","last_peered":"2024-06-27T21:49:24.106113+0000","last_clean":"2024-06-27T21:49:24.106113+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:49:24.106113+0000","last_undegraded":"2024-06-27T21:49:24.106113+0000","last_fullsized":"2024-06-27T21:49:24.106113+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:20.087806+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:49:20.087806+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:41.382744+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":103,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.092667+0000","last_change":"2024-06-27T21:49:22.643869+0000","last_active":"2024-06-27T21:49:24.092667+0000","last_peered":"2024-06-27T21:49:24.092667+0000","last_clean":"2024-06-27T21:49:24.092667+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:49:24.092667+0000","last_undegraded":"2024-06-27T21:49:24.092667+0000","last_fullsized":"2024-06-27T21:49:24.092667+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:22.643782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:22.643782+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:42.743060+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":907,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.094351+0000","last_change":"2024-06-27T21:49:18.385781+0000","last_active":"2024-06-27T21:49:24.094351+0000","last_peered":"2024-06-27T21:49:24.094351+0000","last_clean":"2024-06-27T21:49:24.094351+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:49:24.094351+0000","last_undegraded":"2024-06-27T21:49:24.094351+0000","last_fullsized":"2024-06-27T21:49:24.094351+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:18.385670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:18.385670+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:21.967265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1000,"reported_epoch":945,"state":"active+clean","last_fresh":"2024-06-27T21:49:14.968035+0000","last_change":"2024-06-27T21:48:52.674891+0000","last_active":"2024-06-27T21:49:14.968035+0000","last_peered":"2024-06-27T21:49:14.968035+0000","last_clean":"2024-06-27T21:49:14.968035+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:49:14.968035+0000","last_undegraded":"2024-06-27T21:49:14.968035+0000","last_fullsized":"2024-06-27T21:49:14.968035+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_clean_scrub_stamp":"2024-06-27T21:48:52.674762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:09.638022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":696,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.105883+0000","last_change":"2024-06-27T21:48:24.958797+0000","last_active":"2024-06-27T21:49:24.105883+0000","last_peered":"2024-06-27T21:49:24.105883+0000","last_clean":"2024-06-27T21:49:24.105883+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:49:24.105883+0000","last_undegraded":"2024-06-27T21:49:24.105883+0000","last_fullsized":"2024-06-27T21:49:24.105883+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:04.976398+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:48:04.976398+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:30.531216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":722,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.106094+0000","last_change":"2024-06-27T21:49:23.841442+0000","last_active":"2024-06-27T21:49:24.106094+0000","last_peered":"2024-06-27T21:49:24.106094+0000","last_clean":"2024-06-27T21:49:24.106094+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:49:24.106094+0000","last_undegraded":"2024-06-27T21:49:24.106094+0000","last_fullsized":"2024-06-27T21:49:24.106094+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:23.841370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:23.841370+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:29.522864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":166,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.093652+0000","last_change":"2024-06-27T21:48:02.060711+0000","last_active":"2024-06-27T21:49:24.093652+0000","last_peered":"2024-06-27T21:49:24.093652+0000","last_clean":"2024-06-27T21:49:24.093652+0000","last_became_active":"2024-06-27T21:46:34.528582+0000","last_became_peered":"2024-06-27T21:46:34.528582+0000","last_unstale":"2024-06-27T21:49:24.093652+0000","last_undegraded":"2024-06-27T21:49:24.093652+0000","last_fullsized":"2024-06-27T21:49:24.093652+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:02.060563+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:02.060563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:25.296698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":166,"reported_epoch":946,"state":"active+clean","last_fresh":"2024-06-27T21:49:24.093528+0000","last_change":"2024-06-27T21:48:40.018430+0000","last_active":"2024-06-27T21:49:24.093528+0000","last_peered":"2024-06-27T21:49:24.093528+0000","last_clean":"2024-06-27T21:49:24.093528+0000","last_became_active":"2024-06-27T21:46:34.528359+0000","last_became_peered":"2024-06-27T21:46:34.528359+0000","last_unstale":"2024-06-27T21:49:24.093528+0000","last_undegraded":"2024-06-27T21:49:24.093528+0000","last_fullsized":"2024-06-27T21:49:24.093528+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:40.018304+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:48:40.018304+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:05.786297+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5336530944,"num_objects":81429,"num_object_clones":0,"num_object_copies":244287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81429,"num_write_kb":5211456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15920988160,"data_stored":15920988160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81933,"ondisk_log_size":81933,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253347,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2286728,"kb_used_data":1276616,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":91458424,"statfs":{"total":95995035648,"available":93653426176,"internally_reserved":0,"allocated":1307254784,"data_stored":1306858227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1034156485},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514983,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2522440,"kb_used_data":1602568,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":91222712,"statfs":{"total":95995035648,"available":93412057088,"internally_reserved":0,"allocated":1641029632,"data_stored":1640633075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":941750720},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743915,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2657992,"kb_used_data":1918600,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91087160,"statfs":{"total":95995035648,"available":93273251840,"internally_reserved":0,"allocated":1964646400,"data_stored":1964249843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":756939200},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972848,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241488,"kb_used_data":673032,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92503664,"statfs":{"total":95995035648,"available":94723751936,"internally_reserved":0,"allocated":689184768,"data_stored":688788211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267183,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2385280,"kb_used_data":1582200,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91359872,"statfs":{"total":95995035648,"available":93552508928,"internally_reserved":0,"allocated":1620172800,"data_stored":1619714559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":822344138},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528825,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1361288,"kb_used_data":644104,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92383864,"statfs":{"total":95995035648,"available":94601076736,"internally_reserved":0,"allocated":659562496,"data_stored":659165939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757759,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3617432,"kb_used_data":2982352,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":90127720,"statfs":{"total":95995035648,"available":92290785280,"internally_reserved":0,"allocated":3053928448,"data_stored":3053528339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161464,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192184,"kb_used_data":635568,"kb_used_omap":92,"kb_used_meta":556515,"kb_avail":92552968,"statfs":{"total":95995035648,"available":94774239232,"internally_reserved":0,"allocated":650821632,"data_stored":650366943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":569871458},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182919,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258256,"kb_used_data":1623176,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91486896,"statfs":{"total":95995035648,"available":93682581504,"internally_reserved":0,"allocated":1662132224,"data_stored":1661735667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":650312135},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411851,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808640,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936512,"statfs":{"total":95995035648,"available":95166988288,"internally_reserved":0,"allocated":1433600,"data_stored":975359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575378,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3380472,"kb_used_data":2546864,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":90364680,"statfs":{"total":95995035648,"available":92533432320,"internally_reserved":0,"allocated":2607988736,"data_stored":2607534047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":853604805},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880968,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864184,"statfs":{"total":95995035648,"available":95092924416,"internally_reserved":0,"allocated":1384448,"data_stored":987891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2610364416,"data_stored":2610364416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1672085504,"data_stored":1672085504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":654049280,"data_stored":654049280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3063021568,"data_stored":3063021568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":661848064,"data_stored":661848064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1627717632,"data_stored":1627717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":691798016,"data_stored":691798016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1972568064,"data_stored":1972568064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1647968256,"data_stored":1647968256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1319567360,"data_stored":1319567360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:49:31.058 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-27T21:49:31.058 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:49:31.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:49:31.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:31.171 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:31.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:30 smithi190 ceph-mon[32247]: from='client.27373 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:31.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:30 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:49:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:30 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:49:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:30 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:49:31.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:30 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:49:31.268 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:30 smithi151 ceph-mon[28965]: from='client.27373 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:31.268 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:30 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:49:31.268 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:30 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:49:31.268 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:30 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:49:31.268 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:30 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:49:31.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:31.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:30 smithi192 ceph-mon[32956]: from='client.27373 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:31.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:30 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:49:31.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:30 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:49:31.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:30 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:49:31.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:30 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:49:31.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:31.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:31.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:31.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:31.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:31.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:31.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:31.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:31.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:31.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 67 16 86943 86927 81.0606 58 0.0136071 0.0123303 2024-06-27T21:49:31.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:31.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:31.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:32.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:32.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: pgmap v2308: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 999 op/s 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: 9.e scrub starts 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:32.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:32.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:31 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:32.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:32.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:32.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: pgmap v2308: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 999 op/s 2024-06-27T21:49:32.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:49:32.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:49:32.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: 9.e scrub starts 2024-06-27T21:49:32.249 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:32.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:32.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:32.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:31 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:32.290 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: pgmap v2308: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 999 op/s 2024-06-27T21:49:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:49:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:49:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: 9.e scrub starts 2024-06-27T21:49:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:32.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:31 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:32.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:32.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:32.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:32.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:32.507 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:32.507 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:32.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:32.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:32.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:32.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:32.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:32.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:32.878 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:32.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:32.909 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 68 16 88004 87988 80.8437 66.3125 0.0594245 0.0123625 2024-06-27T21:49:32.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:32.996 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:49:33.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:33.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:33.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:32 smithi190 ceph-mon[32247]: osdmap e947: 12 total, 12 up, 12 in 2024-06-27T21:49:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1528084398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:49:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:32 smithi190 ceph-mon[32247]: 9.e scrub ok 2024-06-27T21:49:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:32 smithi190 ceph-mon[32247]: 9.7 scrub starts 2024-06-27T21:49:33.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:49:33.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:33.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:33.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:33.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:33.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:32 smithi192 ceph-mon[32956]: osdmap e947: 12 total, 12 up, 12 in 2024-06-27T21:49:33.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1528084398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:49:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:32 smithi192 ceph-mon[32956]: 9.e scrub ok 2024-06-27T21:49:33.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:32 smithi192 ceph-mon[32956]: 9.7 scrub starts 2024-06-27T21:49:33.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:33.395 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:32 smithi151 ceph-mon[28965]: osdmap e947: 12 total, 12 up, 12 in 2024-06-27T21:49:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1528084398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:49:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:32 smithi151 ceph-mon[28965]: 9.e scrub ok 2024-06-27T21:49:33.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:32 smithi151 ceph-mon[28965]: 9.7 scrub starts 2024-06-27T21:49:33.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:33.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:33.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:33.627 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:49:33.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:33.758 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:33.784 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:33.886 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:33.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 69 15 89026 89011 80.5984 63.9375 0.0121425 0.0124001 2024-06-27T21:49:33.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:33.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:34.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:33 smithi190 ceph-mon[32247]: pgmap v2310: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 828 op/s 2024-06-27T21:49:34.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:33 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:34.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1528084398' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:49:34.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:33 smithi190 ceph-mon[32247]: osdmap e948: 12 total, 12 up, 12 in 2024-06-27T21:49:34.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:34.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:34.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:34.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:33 smithi192 ceph-mon[32956]: pgmap v2310: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 828 op/s 2024-06-27T21:49:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:33 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1528084398' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:49:34.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:33 smithi192 ceph-mon[32956]: osdmap e948: 12 total, 12 up, 12 in 2024-06-27T21:49:34.307 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:34.331 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:34.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:33 smithi151 ceph-mon[28965]: pgmap v2310: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 828 op/s 2024-06-27T21:49:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:33 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1528084398' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:49:34.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:33 smithi151 ceph-mon[28965]: osdmap e948: 12 total, 12 up, 12 in 2024-06-27T21:49:34.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:34.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:34.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:34.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:34.713 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:34.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:34.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:34.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 70 16 89924 89908 80.2479 56.0625 0.0124649 0.0124548 2024-06-27T21:49:34.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:34.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:34.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:35.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:35.108 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:35 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:35 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:35 smithi190 ceph-mon[32247]: osdmap e949: 12 total, 12 up, 12 in 2024-06-27T21:49:35.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:35.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:35 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:35 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:35 smithi192 ceph-mon[32956]: osdmap e949: 12 total, 12 up, 12 in 2024-06-27T21:49:35.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.362 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:35.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:35.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:35 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:35.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:35 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:35.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:35 smithi151 ceph-mon[28965]: osdmap e949: 12 total, 12 up, 12 in 2024-06-27T21:49:35.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:35.541 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:35.541 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:35.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:35.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:35.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:35.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:35.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:35.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:35.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:35.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 71 16 90788 90772 79.8783 54 0.00676242 0.012514 2024-06-27T21:49:35.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:36.001 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:36.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:36.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:36.212 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:36 smithi190 ceph-mon[32247]: pgmap v2313: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-06-27T21:49:36.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:36.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-06-27T21:49:36.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:36.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:36 smithi151 ceph-mon[28965]: pgmap v2313: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-06-27T21:49:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:36 smithi192 ceph-mon[32956]: pgmap v2313: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-06-27T21:49:36.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:36.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:36.488 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:36.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:36.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:36.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:36.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:36.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:36.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:36.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:36.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:36.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:36.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 72 16 91670 91654 79.5345 55.125 0.00989546 0.0125673 2024-06-27T21:49:37.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:37.032 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:37.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:37.163 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:37.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:37.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:37.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:37.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:37.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:37.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:37.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:37.680 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:37.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:37.807 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:37.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:37.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 73 16 92690 92674 79.3183 63.75 0.0166342 0.0126016 2024-06-27T21:49:37.934 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:37.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:38.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:38.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:38.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:38.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:38.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:38 smithi151 ceph-mon[28965]: pgmap v2314: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 905 op/s 2024-06-27T21:49:38.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:38 smithi190 ceph-mon[32247]: pgmap v2314: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 905 op/s 2024-06-27T21:49:38.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:38 smithi192 ceph-mon[32956]: pgmap v2314: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 905 op/s 2024-06-27T21:49:38.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:38.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:38.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:38.451 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:38.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:38.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:38.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:38.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:38.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:38.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:38.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 74 16 93655 93639 79.0614 60.3125 0.00683773 0.0126419 2024-06-27T21:49:38.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:38.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:38.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:38.999 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:39.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:39.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:39.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:39.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:49:39.282 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:49:39.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:39.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:39.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:39.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:39.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.630 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:39.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:39.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:39.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:39.858 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:39.885 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:39.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:39.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 75 16 94699 94683 78.8772 65.25 0.00440258 0.0126708 2024-06-27T21:49:39.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:40.014 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:40.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:40.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:40.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:49:40.212 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:49:40.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:49:40.237 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:49:40.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:40.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:49:40.264 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:49:40.264 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:49:40.264 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:40 smithi151 ceph-mon[28965]: pgmap v2315: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:49:40.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:40 smithi190 ceph-mon[32247]: pgmap v2315: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:49:40.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:40.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:49:40.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:49:40.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:40 smithi192 ceph-mon[32956]: pgmap v2315: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-06-27T21:49:40.316 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:49:40.316 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:49:40.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:49:40.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:49:40.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:40.378 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:49:40.378 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:49:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:40.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:40.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:49:40.407 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:49:40.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:49:40.434 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:49:40.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:49:40.461 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:49:40.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:49:40.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:49:40.498 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:40.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:49:40.514 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:49:40.524 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:40.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.654 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:40.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:40.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:40.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:40.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:40.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 76 16 95511 95495 78.5072 50.75 0.00444521 0.0127292 2024-06-27T21:49:40.915 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:40.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:41.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:41.046 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:41.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:41.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:41 smithi151 ceph-mon[28965]: from='client.17823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:41.160 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3745986465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:41.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:41.278 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:41.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:41 smithi192 ceph-mon[32956]: from='client.17823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:41.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3745986465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:41.306 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:41.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:41.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:41.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:41 smithi190 ceph-mon[32247]: from='client.17823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:49:41.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3745986465' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:41.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:41.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:41.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:41.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:41.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.822 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:41.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:41.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 77 16 96485 96469 78.2782 60.875 0.0453446 0.0127678 2024-06-27T21:49:41.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:41.940 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2315,"stamp":"2024-06-27T21:49:38.360977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5964038688,"num_objects":90997,"num_object_clones":0,"num_object_copies":272991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90995,"num_write_kb":5823680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95925,"ondisk_log_size":95925,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":26433552,"kb_used_data":17304824,"kb_used_omap":110,"kb_used_meta":9126417,"kb_avail":1098508272,"statfs":{"total":1151940427776,"available":1124872470528,"internally_reserved":0,"allocated":17720139776,"data_stored":17716792828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9345451991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":454295552,"num_objects":6932,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.381326"},"pg_stats":[{"pgid":"9.3","version":"949'5555","reported_seq":5630,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:35.437060+0000","last_change":"2024-06-27T21:48:11.986786+0000","last_active":"2024-06-27T21:49:35.437060+0000","last_peered":"2024-06-27T21:49:35.437060+0000","last_clean":"2024-06-27T21:49:35.437060+0000","last_became_active":"2024-06-27T21:48:11.986220+0000","last_became_peered":"2024-06-27T21:48:11.986220+0000","last_unstale":"2024-06-27T21:49:35.437060+0000","last_undegraded":"2024-06-27T21:49:35.437060+0000","last_fullsized":"2024-06-27T21:49:35.437060+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.515921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364052480,"num_objects":5555,"num_object_clones":0,"num_object_copies":16665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5555,"num_write_kb":355520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":178,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.006477+0000","last_change":"2024-06-27T21:48:37.072258+0000","last_active":"2024-06-27T21:49:34.006477+0000","last_peered":"2024-06-27T21:49:34.006477+0000","last_clean":"2024-06-27T21:49:34.006477+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:49:34.006477+0000","last_undegraded":"2024-06-27T21:49:34.006477+0000","last_fullsized":"2024-06-27T21:49:34.006477+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:37.072188+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:56.646935+0000","last_clean_scrub_stamp":"2024-06-27T21:48:37.072188+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:52.196369+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"949'5925","reported_seq":5999,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:38.196347+0000","last_change":"2024-06-27T21:48:12.310120+0000","last_active":"2024-06-27T21:49:38.196347+0000","last_peered":"2024-06-27T21:49:38.196347+0000","last_clean":"2024-06-27T21:49:38.196347+0000","last_became_active":"2024-06-27T21:48:12.309522+0000","last_became_peered":"2024-06-27T21:48:12.309522+0000","last_unstale":"2024-06-27T21:49:38.196347+0000","last_undegraded":"2024-06-27T21:49:38.196347+0000","last_fullsized":"2024-06-27T21:49:38.196347+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5925,"log_dups_size":0,"ondisk_log_size":5925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:42.785802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388300800,"num_objects":5925,"num_object_clones":0,"num_object_copies":17775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5925,"num_write_kb":379200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":144,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.999445+0000","last_change":"2024-06-27T21:48:39.725238+0000","last_active":"2024-06-27T21:49:33.999445+0000","last_peered":"2024-06-27T21:49:33.999445+0000","last_clean":"2024-06-27T21:49:33.999445+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:49:33.999445+0000","last_undegraded":"2024-06-27T21:49:33.999445+0000","last_fullsized":"2024-06-27T21:49:33.999445+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:39.725111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:39.725111+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:42.976869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"949'5646","reported_seq":5722,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.875236+0000","last_change":"2024-06-27T21:48:11.990938+0000","last_active":"2024-06-27T21:49:34.875236+0000","last_peered":"2024-06-27T21:49:34.875236+0000","last_clean":"2024-06-27T21:49:34.875236+0000","last_became_active":"2024-06-27T21:48:11.990332+0000","last_became_peered":"2024-06-27T21:48:11.990332+0000","last_unstale":"2024-06-27T21:49:34.875236+0000","last_undegraded":"2024-06-27T21:49:34.875236+0000","last_fullsized":"2024-06-27T21:49:34.875236+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.131391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370016256,"num_objects":5646,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":361344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":282,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.004294+0000","last_change":"2024-06-27T21:48:35.993226+0000","last_active":"2024-06-27T21:49:33.004294+0000","last_peered":"2024-06-27T21:49:33.004294+0000","last_clean":"2024-06-27T21:49:33.004294+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:49:33.004294+0000","last_undegraded":"2024-06-27T21:49:33.004294+0000","last_fullsized":"2024-06-27T21:49:33.004294+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:35.993117+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:48:35.993117+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:57.566564+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"949'5652","reported_seq":5728,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.821489+0000","last_change":"2024-06-27T21:48:12.310003+0000","last_active":"2024-06-27T21:49:34.821489+0000","last_peered":"2024-06-27T21:49:34.821489+0000","last_clean":"2024-06-27T21:49:34.821489+0000","last_became_active":"2024-06-27T21:48:12.309579+0000","last_became_peered":"2024-06-27T21:48:12.309579+0000","last_unstale":"2024-06-27T21:49:34.821489+0000","last_undegraded":"2024-06-27T21:49:34.821489+0000","last_fullsized":"2024-06-27T21:49:34.821489+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5652,"log_dups_size":0,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:40.576125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":361728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1059,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:32.998447+0000","last_change":"2024-06-27T21:49:20.921441+0000","last_active":"2024-06-27T21:49:32.998447+0000","last_peered":"2024-06-27T21:49:32.998447+0000","last_clean":"2024-06-27T21:49:32.998447+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:49:32.998447+0000","last_undegraded":"2024-06-27T21:49:32.998447+0000","last_fullsized":"2024-06-27T21:49:32.998447+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:20.921338+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:49:20.921338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:38.530191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"949'5560","reported_seq":5654,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.793466+0000","last_change":"2024-06-27T21:49:33.010934+0000","last_active":"2024-06-27T21:49:34.793466+0000","last_peered":"2024-06-27T21:49:34.793466+0000","last_clean":"2024-06-27T21:49:34.793466+0000","last_became_active":"2024-06-27T21:48:11.989652+0000","last_became_peered":"2024-06-27T21:48:11.989652+0000","last_unstale":"2024-06-27T21:49:34.793466+0000","last_undegraded":"2024-06-27T21:49:34.793466+0000","last_fullsized":"2024-06-27T21:49:34.793466+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":1449,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.005590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364380160,"num_objects":5560,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":355840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1670,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.002149+0000","last_change":"2024-06-27T21:49:19.557403+0000","last_active":"2024-06-27T21:49:34.002149+0000","last_peered":"2024-06-27T21:49:34.002149+0000","last_clean":"2024-06-27T21:49:34.002149+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:49:34.002149+0000","last_undegraded":"2024-06-27T21:49:34.002149+0000","last_fullsized":"2024-06-27T21:49:34.002149+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:19.557294+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:49:19.557294+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:25.339126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"949'5681","reported_seq":5791,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:35.426826+0000","last_change":"2024-06-27T21:49:14.968114+0000","last_active":"2024-06-27T21:49:35.426826+0000","last_peered":"2024-06-27T21:49:35.426826+0000","last_clean":"2024-06-27T21:49:35.426826+0000","last_became_active":"2024-06-27T21:48:25.035404+0000","last_became_peered":"2024-06-27T21:48:25.035404+0000","last_unstale":"2024-06-27T21:49:35.426826+0000","last_undegraded":"2024-06-27T21:49:35.426826+0000","last_fullsized":"2024-06-27T21:49:35.426826+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'4404","last_scrub_stamp":"2024-06-27T21:49:14.967854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:14.967854+0000","objects_scrubbed":4367,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:19.717925+0000","scrub_duration":1019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372310016,"num_objects":5681,"num_object_clones":0,"num_object_copies":17043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5681,"num_write_kb":363584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":645,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.000821+0000","last_change":"2024-06-27T21:48:43.028417+0000","last_active":"2024-06-27T21:49:34.000821+0000","last_peered":"2024-06-27T21:49:34.000821+0000","last_clean":"2024-06-27T21:49:34.000821+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:49:34.000821+0000","last_undegraded":"2024-06-27T21:49:34.000821+0000","last_fullsized":"2024-06-27T21:49:34.000821+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:43.028305+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:48:43.028305+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:45.889033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.b","version":"949'5789","reported_seq":5862,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:38.174133+0000","last_change":"2024-06-27T21:48:11.989882+0000","last_active":"2024-06-27T21:49:38.174133+0000","last_peered":"2024-06-27T21:49:38.174133+0000","last_clean":"2024-06-27T21:49:38.174133+0000","last_became_active":"2024-06-27T21:48:11.989420+0000","last_became_peered":"2024-06-27T21:48:11.989420+0000","last_unstale":"2024-06-27T21:49:38.174133+0000","last_undegraded":"2024-06-27T21:49:38.174133+0000","last_fullsized":"2024-06-27T21:49:38.174133+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.290203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379387904,"num_objects":5789,"num_object_clones":0,"num_object_copies":17367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5789,"num_write_kb":370496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":1048,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:32.998052+0000","last_change":"2024-06-27T21:49:14.931890+0000","last_active":"2024-06-27T21:49:32.998052+0000","last_peered":"2024-06-27T21:49:32.998052+0000","last_clean":"2024-06-27T21:49:32.998052+0000","last_became_active":"2024-06-27T21:48:18.177749+0000","last_became_peered":"2024-06-27T21:48:18.177749+0000","last_unstale":"2024-06-27T21:49:32.998052+0000","last_undegraded":"2024-06-27T21:49:32.998052+0000","last_fullsized":"2024-06-27T21:49:32.998052+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:14.931780+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:45.673000+0000","last_clean_scrub_stamp":"2024-06-27T21:49:14.931780+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:23.596370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"949'5744","reported_seq":5820,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:37.766579+0000","last_change":"2024-06-27T21:48:12.311450+0000","last_active":"2024-06-27T21:49:37.766579+0000","last_peered":"2024-06-27T21:49:37.766579+0000","last_clean":"2024-06-27T21:49:37.766579+0000","last_became_active":"2024-06-27T21:48:12.311072+0000","last_became_peered":"2024-06-27T21:48:12.311072+0000","last_unstale":"2024-06-27T21:49:37.766579+0000","last_undegraded":"2024-06-27T21:49:37.766579+0000","last_fullsized":"2024-06-27T21:49:37.766579+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376438784,"num_objects":5744,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5744,"num_write_kb":367616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":282,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.000456+0000","last_change":"2024-06-27T21:49:28.234089+0000","last_active":"2024-06-27T21:49:33.000456+0000","last_peered":"2024-06-27T21:49:33.000456+0000","last_clean":"2024-06-27T21:49:33.000456+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:49:33.000456+0000","last_undegraded":"2024-06-27T21:49:33.000456+0000","last_fullsized":"2024-06-27T21:49:33.000456+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:28.233996+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:49:28.233996+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:45.666240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"949'5580","reported_seq":5656,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.822940+0000","last_change":"2024-06-27T21:48:11.998116+0000","last_active":"2024-06-27T21:49:34.822940+0000","last_peered":"2024-06-27T21:49:34.822940+0000","last_clean":"2024-06-27T21:49:34.822940+0000","last_became_active":"2024-06-27T21:48:11.997839+0000","last_became_peered":"2024-06-27T21:48:11.997839+0000","last_unstale":"2024-06-27T21:49:34.822940+0000","last_undegraded":"2024-06-27T21:49:34.822940+0000","last_fullsized":"2024-06-27T21:49:34.822940+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5580,"log_dups_size":0,"ondisk_log_size":5580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:37.072712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365690880,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5580,"num_write_kb":357120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":282,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.996878+0000","last_change":"2024-06-27T21:49:16.258267+0000","last_active":"2024-06-27T21:49:33.996878+0000","last_peered":"2024-06-27T21:49:33.996878+0000","last_clean":"2024-06-27T21:49:33.996878+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:49:33.996878+0000","last_undegraded":"2024-06-27T21:49:33.996878+0000","last_fullsized":"2024-06-27T21:49:33.996878+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:16.258197+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:49:16.258197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:37.926239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"949'5824","reported_seq":5958,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:37.785407+0000","last_change":"2024-06-27T21:49:16.636144+0000","last_active":"2024-06-27T21:49:37.785407+0000","last_peered":"2024-06-27T21:49:37.785407+0000","last_clean":"2024-06-27T21:49:37.785407+0000","last_became_active":"2024-06-27T21:48:24.958699+0000","last_became_peered":"2024-06-27T21:48:24.958699+0000","last_unstale":"2024-06-27T21:49:37.785407+0000","last_undegraded":"2024-06-27T21:49:37.785407+0000","last_fullsized":"2024-06-27T21:49:37.785407+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'4466","last_scrub_stamp":"2024-06-27T21:49:16.635894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:16.635894+0000","objects_scrubbed":4435,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:41.281934+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381681664,"num_objects":5824,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5824,"num_write_kb":372736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1385,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.997460+0000","last_change":"2024-06-27T21:49:25.518932+0000","last_active":"2024-06-27T21:49:33.997460+0000","last_peered":"2024-06-27T21:49:33.997460+0000","last_clean":"2024-06-27T21:49:33.997460+0000","last_became_active":"2024-06-27T21:48:18.177860+0000","last_became_peered":"2024-06-27T21:48:18.177860+0000","last_unstale":"2024-06-27T21:49:33.997460+0000","last_undegraded":"2024-06-27T21:49:33.997460+0000","last_fullsized":"2024-06-27T21:49:33.997460+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:25.518679+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:49:25.518679+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:43.714160+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"948'5576","reported_seq":5711,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.870000+0000","last_change":"2024-06-27T21:49:32.153599+0000","last_active":"2024-06-27T21:49:33.870000+0000","last_peered":"2024-06-27T21:49:33.870000+0000","last_clean":"2024-06-27T21:49:33.870000+0000","last_became_active":"2024-06-27T21:48:12.309446+0000","last_became_peered":"2024-06-27T21:48:12.309446+0000","last_unstale":"2024-06-27T21:49:33.870000+0000","last_undegraded":"2024-06-27T21:49:33.870000+0000","last_fullsized":"2024-06-27T21:49:33.870000+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"947'5477","last_scrub_stamp":"2024-06-27T21:49:32.153278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:32.153278+0000","objects_scrubbed":5448,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:53.377863+0000","scrub_duration":997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365428736,"num_objects":5576,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5576,"num_write_kb":356864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1451,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.997438+0000","last_change":"2024-06-27T21:48:00.289561+0000","last_active":"2024-06-27T21:49:33.997438+0000","last_peered":"2024-06-27T21:49:33.997438+0000","last_clean":"2024-06-27T21:49:33.997438+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:49:33.997438+0000","last_undegraded":"2024-06-27T21:49:33.997438+0000","last_fullsized":"2024-06-27T21:49:33.997438+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:00.289470+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:48:00.289470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:38.508646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"948'5640","reported_seq":5713,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.848059+0000","last_change":"2024-06-27T21:48:12.310346+0000","last_active":"2024-06-27T21:49:33.848059+0000","last_peered":"2024-06-27T21:49:33.848059+0000","last_clean":"2024-06-27T21:49:33.848059+0000","last_became_active":"2024-06-27T21:48:12.309793+0000","last_became_peered":"2024-06-27T21:48:12.309793+0000","last_unstale":"2024-06-27T21:49:33.848059+0000","last_undegraded":"2024-06-27T21:49:33.848059+0000","last_fullsized":"2024-06-27T21:49:33.848059+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:36.073340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369623040,"num_objects":5640,"num_object_clones":0,"num_object_copies":16920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5640,"num_write_kb":360960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1479,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.997290+0000","last_change":"2024-06-27T21:49:28.536302+0000","last_active":"2024-06-27T21:49:33.997290+0000","last_peered":"2024-06-27T21:49:33.997290+0000","last_clean":"2024-06-27T21:49:33.997290+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:49:33.997290+0000","last_undegraded":"2024-06-27T21:49:33.997290+0000","last_fullsized":"2024-06-27T21:49:33.997290+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:28.536174+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:49:28.536174+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:53.809194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"949'5753","reported_seq":5888,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:37.445347+0000","last_change":"2024-06-27T21:49:28.997386+0000","last_active":"2024-06-27T21:49:37.445347+0000","last_peered":"2024-06-27T21:49:37.445347+0000","last_clean":"2024-06-27T21:49:37.445347+0000","last_became_active":"2024-06-27T21:48:11.986349+0000","last_became_peered":"2024-06-27T21:48:11.986349+0000","last_unstale":"2024-06-27T21:49:37.445347+0000","last_undegraded":"2024-06-27T21:49:37.445347+0000","last_fullsized":"2024-06-27T21:49:37.445347+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"946'5230","last_scrub_stamp":"2024-06-27T21:49:28.997268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:28.997268+0000","objects_scrubbed":5187,"log_size":5753,"log_dups_size":0,"ondisk_log_size":5753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:42.565115+0000","scrub_duration":1131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377028608,"num_objects":5753,"num_object_clones":0,"num_object_copies":17259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":368192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":180,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.006507+0000","last_change":"2024-06-27T21:49:22.189855+0000","last_active":"2024-06-27T21:49:34.006507+0000","last_peered":"2024-06-27T21:49:34.006507+0000","last_clean":"2024-06-27T21:49:34.006507+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:49:34.006507+0000","last_undegraded":"2024-06-27T21:49:34.006507+0000","last_fullsized":"2024-06-27T21:49:34.006507+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:22.189748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:22.189748+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:32.782620+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"949'5699","reported_seq":5830,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:37.481072+0000","last_change":"2024-06-27T21:49:21.857234+0000","last_active":"2024-06-27T21:49:37.481072+0000","last_peered":"2024-06-27T21:49:37.481072+0000","last_clean":"2024-06-27T21:49:37.481072+0000","last_became_active":"2024-06-27T21:48:12.309342+0000","last_became_peered":"2024-06-27T21:48:12.309342+0000","last_unstale":"2024-06-27T21:49:37.481072+0000","last_undegraded":"2024-06-27T21:49:37.481072+0000","last_fullsized":"2024-06-27T21:49:37.481072+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'4766","last_scrub_stamp":"2024-06-27T21:49:21.857113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:49:21.857113+0000","objects_scrubbed":4732,"log_size":5699,"log_dups_size":0,"ondisk_log_size":5699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:30.845701+0000","scrub_duration":986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373489664,"num_objects":5699,"num_object_clones":0,"num_object_copies":17097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5699,"num_write_kb":364736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":73,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.000843+0000","last_change":"2024-06-27T21:48:37.006935+0000","last_active":"2024-06-27T21:49:34.000843+0000","last_peered":"2024-06-27T21:49:34.000843+0000","last_clean":"2024-06-27T21:49:34.000843+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:49:34.000843+0000","last_undegraded":"2024-06-27T21:49:34.000843+0000","last_fullsized":"2024-06-27T21:49:34.000843+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:37.006836+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:48:37.006836+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:59.789286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"949'7835","reported_seq":8050,"reported_epoch":949,"state":"active+clean+scrubbing+deep","last_fresh":"2024-06-27T21:49:37.500387+0000","last_change":"2024-06-27T21:49:24.818222+0000","last_active":"2024-06-27T21:49:37.500387+0000","last_peered":"2024-06-27T21:49:37.500387+0000","last_clean":"2024-06-27T21:49:37.500387+0000","last_became_active":"2024-06-27T21:48:12.198416+0000","last_became_peered":"2024-06-27T21:48:12.198416+0000","last_unstale":"2024-06-27T21:49:37.500387+0000","last_undegraded":"2024-06-27T21:49:37.500387+0000","last_fullsized":"2024-06-27T21:49:37.500387+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":3503,"log_size":7835,"log_dups_size":0,"ondisk_log_size":7835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377290752,"num_objects":5757,"num_object_clones":0,"num_object_copies":17271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5757,"num_write_kb":368448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":290,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.000336+0000","last_change":"2024-06-27T21:49:26.212697+0000","last_active":"2024-06-27T21:49:33.000336+0000","last_peered":"2024-06-27T21:49:33.000336+0000","last_clean":"2024-06-27T21:49:33.000336+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:49:33.000336+0000","last_undegraded":"2024-06-27T21:49:33.000336+0000","last_fullsized":"2024-06-27T21:49:33.000336+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:26.212587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:26.212587+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:42.632708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"949'5614","reported_seq":5690,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.748881+0000","last_change":"2024-06-27T21:48:12.309489+0000","last_active":"2024-06-27T21:49:34.748881+0000","last_peered":"2024-06-27T21:49:34.748881+0000","last_clean":"2024-06-27T21:49:34.748881+0000","last_became_active":"2024-06-27T21:48:12.309089+0000","last_became_peered":"2024-06-27T21:48:12.309089+0000","last_unstale":"2024-06-27T21:49:34.748881+0000","last_undegraded":"2024-06-27T21:49:34.748881+0000","last_fullsized":"2024-06-27T21:49:34.748881+0000","mapping_epoch":916,"log_start":"0'0","ondisk_log_start":"0'0","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:48:10.967611+0000","objects_scrubbed":0,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367919104,"num_objects":5614,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,6],"acting":[11,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1366,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.007698+0000","last_change":"2024-06-27T21:49:24.772444+0000","last_active":"2024-06-27T21:49:34.007698+0000","last_peered":"2024-06-27T21:49:34.007698+0000","last_clean":"2024-06-27T21:49:34.007698+0000","last_became_active":"2024-06-27T21:47:59.079888+0000","last_became_peered":"2024-06-27T21:47:59.079888+0000","last_unstale":"2024-06-27T21:49:34.007698+0000","last_undegraded":"2024-06-27T21:49:34.007698+0000","last_fullsized":"2024-06-27T21:49:34.007698+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:24.772368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:24.772368+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:46.566404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":105,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.001956+0000","last_change":"2024-06-27T21:49:23.589601+0000","last_active":"2024-06-27T21:49:34.001956+0000","last_peered":"2024-06-27T21:49:34.001956+0000","last_clean":"2024-06-27T21:49:34.001956+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:49:34.001956+0000","last_undegraded":"2024-06-27T21:49:34.001956+0000","last_fullsized":"2024-06-27T21:49:34.001956+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:23.589530+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:49:23.589530+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:47.854335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1064,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.996777+0000","last_change":"2024-06-27T21:49:15.211124+0000","last_active":"2024-06-27T21:49:33.996777+0000","last_peered":"2024-06-27T21:49:33.996777+0000","last_clean":"2024-06-27T21:49:33.996777+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:49:33.996777+0000","last_undegraded":"2024-06-27T21:49:33.996777+0000","last_fullsized":"2024-06-27T21:49:33.996777+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:15.211050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:15.211050+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:21.569767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1091,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.002276+0000","last_change":"2024-06-27T21:48:43.000826+0000","last_active":"2024-06-27T21:49:34.002276+0000","last_peered":"2024-06-27T21:49:34.002276+0000","last_clean":"2024-06-27T21:49:34.002276+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:49:34.002276+0000","last_undegraded":"2024-06-27T21:49:34.002276+0000","last_fullsized":"2024-06-27T21:49:34.002276+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:43.000735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:48:43.000735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:07.945537+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":172,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.006589+0000","last_change":"2024-06-27T21:48:33.250275+0000","last_active":"2024-06-27T21:49:34.006589+0000","last_peered":"2024-06-27T21:49:34.006589+0000","last_clean":"2024-06-27T21:49:34.006589+0000","last_became_active":"2024-06-27T21:46:34.530850+0000","last_became_peered":"2024-06-27T21:46:34.530850+0000","last_unstale":"2024-06-27T21:49:34.006589+0000","last_undegraded":"2024-06-27T21:49:34.006589+0000","last_fullsized":"2024-06-27T21:49:34.006589+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:33.250197+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:16:15.304450+0000","last_clean_scrub_stamp":"2024-06-27T21:48:33.250197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:49:43.368122+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1063,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.007731+0000","last_change":"2024-06-27T21:49:15.693241+0000","last_active":"2024-06-27T21:49:34.007731+0000","last_peered":"2024-06-27T21:49:34.007731+0000","last_clean":"2024-06-27T21:49:34.007731+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:49:34.007731+0000","last_undegraded":"2024-06-27T21:49:34.007731+0000","last_fullsized":"2024-06-27T21:49:34.007731+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:15.693106+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:49:15.693106+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:20.863365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":289,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.000596+0000","last_change":"2024-06-27T21:49:30.217877+0000","last_active":"2024-06-27T21:49:33.000596+0000","last_peered":"2024-06-27T21:49:33.000596+0000","last_clean":"2024-06-27T21:49:33.000596+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:49:33.000596+0000","last_undegraded":"2024-06-27T21:49:33.000596+0000","last_fullsized":"2024-06-27T21:49:33.000596+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:30.217745+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:49:30.217745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:53.233678+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":310,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.000845+0000","last_change":"2024-06-27T21:49:13.554401+0000","last_active":"2024-06-27T21:49:33.000845+0000","last_peered":"2024-06-27T21:49:33.000845+0000","last_clean":"2024-06-27T21:49:33.000845+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:49:33.000845+0000","last_undegraded":"2024-06-27T21:49:33.000845+0000","last_fullsized":"2024-06-27T21:49:33.000845+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:13.554287+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:49:13.554287+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:40.613036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":727,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.003091+0000","last_change":"2024-06-27T21:49:20.087934+0000","last_active":"2024-06-27T21:49:34.003091+0000","last_peered":"2024-06-27T21:49:34.003091+0000","last_clean":"2024-06-27T21:49:34.003091+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:49:34.003091+0000","last_undegraded":"2024-06-27T21:49:34.003091+0000","last_fullsized":"2024-06-27T21:49:34.003091+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:20.087806+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:49:20.087806+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:41.382744+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":109,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.001918+0000","last_change":"2024-06-27T21:49:22.643869+0000","last_active":"2024-06-27T21:49:34.001918+0000","last_peered":"2024-06-27T21:49:34.001918+0000","last_clean":"2024-06-27T21:49:34.001918+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:49:34.001918+0000","last_undegraded":"2024-06-27T21:49:34.001918+0000","last_fullsized":"2024-06-27T21:49:34.001918+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:22.643782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:22.643782+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:42.743060+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":913,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:33.997335+0000","last_change":"2024-06-27T21:49:18.385781+0000","last_active":"2024-06-27T21:49:33.997335+0000","last_peered":"2024-06-27T21:49:33.997335+0000","last_clean":"2024-06-27T21:49:33.997335+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:49:33.997335+0000","last_undegraded":"2024-06-27T21:49:33.997335+0000","last_fullsized":"2024-06-27T21:49:33.997335+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:18.385670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:18.385670+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:21.967265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1006,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-06-27T21:49:32.997865+0000","last_change":"2024-06-27T21:48:52.674891+0000","last_active":"2024-06-27T21:49:32.997865+0000","last_peered":"2024-06-27T21:49:32.997865+0000","last_clean":"2024-06-27T21:49:32.997865+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:49:32.997865+0000","last_undegraded":"2024-06-27T21:49:32.997865+0000","last_fullsized":"2024-06-27T21:49:32.997865+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_clean_scrub_stamp":"2024-06-27T21:48:52.674762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:09.638022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":710,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.002826+0000","last_change":"2024-06-27T21:49:30.832075+0000","last_active":"2024-06-27T21:49:34.002826+0000","last_peered":"2024-06-27T21:49:34.002826+0000","last_clean":"2024-06-27T21:49:34.002826+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:49:34.002826+0000","last_undegraded":"2024-06-27T21:49:34.002826+0000","last_fullsized":"2024-06-27T21:49:34.002826+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:30.831947+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:49:30.831947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:48.423510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":728,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.003084+0000","last_change":"2024-06-27T21:49:23.841442+0000","last_active":"2024-06-27T21:49:34.003084+0000","last_peered":"2024-06-27T21:49:34.003084+0000","last_clean":"2024-06-27T21:49:34.003084+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:49:34.003084+0000","last_undegraded":"2024-06-27T21:49:34.003084+0000","last_fullsized":"2024-06-27T21:49:34.003084+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:23.841370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:49:23.841370+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:29.522864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":180,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.006477+0000","last_change":"2024-06-27T21:49:30.185699+0000","last_active":"2024-06-27T21:49:34.006477+0000","last_peered":"2024-06-27T21:49:34.006477+0000","last_clean":"2024-06-27T21:49:34.006477+0000","last_became_active":"2024-06-27T21:46:34.528582+0000","last_became_peered":"2024-06-27T21:46:34.528582+0000","last_unstale":"2024-06-27T21:49:34.006477+0000","last_undegraded":"2024-06-27T21:49:34.006477+0000","last_fullsized":"2024-06-27T21:49:34.006477+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:49:30.185572+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:49:30.185572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:39.273719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":172,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-06-27T21:49:34.006567+0000","last_change":"2024-06-27T21:48:40.018430+0000","last_active":"2024-06-27T21:49:34.006567+0000","last_peered":"2024-06-27T21:49:34.006567+0000","last_clean":"2024-06-27T21:49:34.006567+0000","last_became_active":"2024-06-27T21:46:34.528359+0000","last_became_peered":"2024-06-27T21:46:34.528359+0000","last_unstale":"2024-06-27T21:49:34.006567+0000","last_undegraded":"2024-06-27T21:49:34.006567+0000","last_fullsized":"2024-06-27T21:49:34.006567+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:48:40.018304+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:48:40.018304+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:50:05.786297+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5963448320,"num_objects":90995,"num_object_clones":0,"num_object_copies":272985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90995,"num_write_kb":5823680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17799118848,"data_stored":17799118848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93073,"ondisk_log_size":93073,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253349,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2433992,"kb_used_data":1432184,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":91311160,"statfs":{"total":95995035648,"available":93502627840,"internally_reserved":0,"allocated":1466556416,"data_stored":1466278345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1025636805},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514985,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2705216,"kb_used_data":1789304,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":91039936,"statfs":{"total":95995035648,"available":93224894464,"internally_reserved":0,"allocated":1832247296,"data_stored":1831969225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":937490880},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743917,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2891720,"kb_used_data":2148152,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":90853432,"statfs":{"total":95995035648,"available":93033914368,"internally_reserved":0,"allocated":2199707648,"data_stored":2199429577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":761199040},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972850,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1337920,"kb_used_data":747256,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92407232,"statfs":{"total":95995035648,"available":94625005568,"internally_reserved":0,"allocated":765190144,"data_stored":764912073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267185,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2584000,"kb_used_data":1768256,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91161152,"statfs":{"total":95995035648,"available":93349019648,"internally_reserved":0,"allocated":1810694144,"data_stored":1810412661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528827,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1432008,"kb_used_data":718968,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92313144,"statfs":{"total":95995035648,"available":94528659456,"internally_reserved":0,"allocated":736223232,"data_stored":735945161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757761,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3960584,"kb_used_data":3321344,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":89784568,"statfs":{"total":95995035648,"available":91939397632,"internally_reserved":0,"allocated":3401056256,"data_stored":3400774633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":654571973},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161466,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1270960,"kb_used_data":709816,"kb_used_omap":92,"kb_used_meta":560675,"kb_avail":92474192,"statfs":{"total":95995035648,"available":94693572608,"internally_reserved":0,"allocated":726851584,"data_stored":726573653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":574131298},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182921,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2441152,"kb_used_data":1818552,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91304000,"statfs":{"total":95995035648,"available":93495296000,"internally_reserved":0,"allocated":1862197248,"data_stored":1861919177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":637532615},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808320,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936832,"statfs":{"total":95995035648,"available":95167315968,"internally_reserved":0,"allocated":917504,"data_stored":636021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575380,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3686848,"kb_used_data":2849080,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":90058304,"statfs":{"total":95995035648,"available":92219703296,"internally_reserved":0,"allocated":2917457920,"data_stored":2917179989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":857864645},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880832,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864320,"statfs":{"total":95995035648,"available":95093063680,"internally_reserved":0,"allocated":1040384,"data_stored":762313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2923888640,"data_stored":2923888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1874591744,"data_stored":1874591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":730136576,"data_stored":730136576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3414032384,"data_stored":3414032384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":738852864,"data_stored":738852864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1824391168,"data_stored":1824391168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":767688704,"data_stored":767688704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2212626432,"data_stored":2212626432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1838350336,"data_stored":1838350336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1474560000,"data_stored":1474560000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:49:41.944 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:41.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:42.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:42.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:42.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:42 smithi151 ceph-mon[28965]: pgmap v2316: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:49:42.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:42 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:42 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:42.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:42 smithi151 ceph-mon[28965]: osdmap e950: 12 total, 12 up, 12 in 2024-06-27T21:49:42.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:42.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:42 smithi192 ceph-mon[32956]: pgmap v2316: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:49:42.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:42 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:42 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:42.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:42 smithi192 ceph-mon[32956]: osdmap e950: 12 total, 12 up, 12 in 2024-06-27T21:49:42.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:42.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:42.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:42.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:42.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:42 smithi190 ceph-mon[32247]: pgmap v2316: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-06-27T21:49:42.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:42 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:42.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:42 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:42.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:42 smithi190 ceph-mon[32247]: osdmap e950: 12 total, 12 up, 12 in 2024-06-27T21:49:42.529 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:49:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:49:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:49:42.529 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:49:42.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:42.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:42.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:42.739 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:42.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:42.872 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:42.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:42.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 78 16 97229 97213 77.8708 46.5 0.0142428 0.0128333 2024-06-27T21:49:42.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:42.999 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:43.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:43.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:43.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:43.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:43.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:43.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-06-27T21:49:43.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:43.400 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:43.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:43.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:49:43.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:43.619 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.647 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:43.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:43.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:43.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:43.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:43.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:43.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:43.910 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 79 16 98135 98119 77.6019 56.625 0.0107448 0.0128784 2024-06-27T21:49:44.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:44.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:44.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:44.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:44.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:44 smithi151 ceph-mon[28965]: pgmap v2318: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 819 op/s 2024-06-27T21:49:44.255 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:44.265 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:44.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:44.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:44 smithi192 ceph-mon[32956]: pgmap v2318: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 819 op/s 2024-06-27T21:49:44.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.416 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:44.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:44 smithi190 ceph-mon[32247]: pgmap v2318: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 819 op/s 2024-06-27T21:49:44.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:44.547 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:44.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:44.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:44.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:44.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:44.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:44.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:2024-06-27T21:49:44.911045+0000 min lat: 0.0010169 max lat: 0.321704 avg lat: 0.0129222 2024-06-27T21:49:44.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-06-27T21:49:44.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 80 16 99054 99038 77.3498 57.4375 0.00698456 0.0129222 2024-06-27T21:49:44.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:44.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:45.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:45.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:45.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=345453879,config=2353167246,health=551204407,kv=1983919777}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4040311372}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=156582587}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3548616673}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733297477}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132737050}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542718088}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3947966068}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2478361987}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=726190333}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517575067}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2763171313}) 2024-06-27T21:49:45.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590816692}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2743927474}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784378140}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073686239}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979144634}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926238158}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317525775}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229289854}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1428330730}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2096934358}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1475660644}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177375028}) 2024-06-27T21:49:45.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437277774}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=654170650}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575933542}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331707489}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3763028219}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3337814621}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=432727748}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634441261}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2023181911}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2424716763}) 2024-06-27T21:49:45.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962100358}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3647476233}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262703113}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1792939778}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3223857268}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=669355464}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3741783219}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3883586536}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65685481}) 2024-06-27T21:49:45.068 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1543102156}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642139022}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695057256}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613063505}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4130512270}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747104976}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460146921}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443882096}) 2024-06-27T21:49:45.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735246988}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2988039449}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4226350077}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312279932}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238212219}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970747910}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1745277535}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971999660}) 2024-06-27T21:49:45.070 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=715013531}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321362990}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059385658}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726399422}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314944262}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321724324}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=66223345}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3133609678}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878972944}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2133893713}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442130630}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784183433}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025933570}) 2024-06-27T21:49:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1493676776}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2915921048}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500595421}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4186847310}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20618180}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851902628}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349045502}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3979913797}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615893147}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267488341}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2778919093}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812356362}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3222051861}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=66,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10} crc {logm=3079202619,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=3349880920}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1014230307}) 2024-06-27T21:49:45.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=219971853}) 2024-06-27T21:49:45.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=91,monmap=5,osd_metadata=4} crc {mon_config_key=633041497,monmap=3596043196,osd_metadata=2465043430}) 2024-06-27T21:49:45.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=8,osd_pg_creating=1,osdmap=91} crc {osd_metadata=4052796637,osd_pg_creating=576692271,osdmap=1301829846}) 2024-06-27T21:49:45.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=2696935399}) 2024-06-27T21:49:45.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:45.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:45.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=345453879,config=2353167246,health=551204407,kv=1983919777}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4040311372}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=156582587}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3548616673}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733297477}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132737050}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542718088}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3947966068}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2478361987}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=726190333}) 2024-06-27T21:49:45.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517575067}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2763171313}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590816692}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2743927474}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784378140}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073686239}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979144634}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926238158}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317525775}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229289854}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1428330730}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2096934358}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1475660644}) 2024-06-27T21:49:45.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177375028}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437277774}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=654170650}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575933542}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331707489}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3763028219}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3337814621}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=432727748}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634441261}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2023181911}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2424716763}) 2024-06-27T21:49:45.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962100358}) 2024-06-27T21:49:45.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3647476233}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262703113}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1792939778}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3223857268}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=669355464}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3741783219}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3883586536}) 2024-06-27T21:49:45.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65685481}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1543102156}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642139022}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695057256}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613063505}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4130512270}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747104976}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460146921}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443882096}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735246988}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2988039449}) 2024-06-27T21:49:45.301 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4226350077}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312279932}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238212219}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970747910}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1745277535}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971999660}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=715013531}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321362990}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059385658}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726399422}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314944262}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321724324}) 2024-06-27T21:49:45.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=66223345}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3133609678}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878972944}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2133893713}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442130630}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784183433}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025933570}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1493676776}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2915921048}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500595421}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4186847310}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20618180}) 2024-06-27T21:49:45.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851902628}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349045502}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3979913797}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615893147}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267488341}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2778919093}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812356362}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3222051861}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=66,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10} crc {logm=3079202619,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=3349880920}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1014230307}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=219971853}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=91,monmap=5,osd_metadata=4} crc {mon_config_key=633041497,monmap=3596043196,osd_metadata=2465043430}) 2024-06-27T21:49:45.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=8,osd_pg_creating=1,osdmap=91} crc {osd_metadata=4052796637,osd_pg_creating=576692271,osdmap=1301829846}) 2024-06-27T21:49:45.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=2696935399}) 2024-06-27T21:49:45.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:45.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:45.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=59} crc {auth=345453879,config=2353167246,health=551204407,kv=1983919777}) 2024-06-27T21:49:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4040311372}) 2024-06-27T21:49:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=156582587}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3548616673}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733297477}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132737050}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542718088}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3947966068}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2478361987}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=726190333}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517575067}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2763171313}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590816692}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2743927474}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=784378140}) 2024-06-27T21:49:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073686239}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2979144634}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2926238158}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317525775}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229289854}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1428330730}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2096934358}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1475660644}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1177375028}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437277774}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=654170650}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2575933542}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3331707489}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3763028219}) 2024-06-27T21:49:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3337814621}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=432727748}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634441261}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2023181911}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2424716763}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962100358}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3647476233}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3262703113}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1792939778}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3223857268}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=669355464}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3741783219}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3883586536}) 2024-06-27T21:49:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65685481}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1543102156}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642139022}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3695057256}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613063505}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4130512270}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747104976}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2460146921}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443882096}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735246988}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2988039449}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4226350077}) 2024-06-27T21:49:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312279932}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238212219}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970747910}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1745277535}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=971999660}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=715013531}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321362990}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059385658}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726399422}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1314944262}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1321724324}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=66223345}) 2024-06-27T21:49:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3133609678}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878972944}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2133893713}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1442130630}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2784183433}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025933570}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1493676776}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2915921048}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500595421}) 2024-06-27T21:49:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4186847310}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20618180}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851902628}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=349045502}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3979913797}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615893147}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=267488341}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2778919093}) 2024-06-27T21:49:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812356362}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3222051861}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=66,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10} crc {logm=3079202619,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=3349880920}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1014230307}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=219971853}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=91,monmap=5,osd_metadata=4} crc {mon_config_key=633041497,monmap=3596043196,osd_metadata=2465043430}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=8,osd_pg_creating=1,osdmap=91} crc {osd_metadata=4052796637,osd_pg_creating=576692271,osdmap=1301829846}) 2024-06-27T21:49:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=24} crc {osdmap=2696935399}) 2024-06-27T21:49:45.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:45.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:45.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:45.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:45.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:45.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:45.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 81 16 99969 99953 77.1009 57.1875 0.00649629 0.0129634 2024-06-27T21:49:45.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:45.954 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:45.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:46.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:46.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:46.183 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:46.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:46 smithi151 ceph-mon[28965]: pgmap v2319: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 941 op/s 2024-06-27T21:49:46.209 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:46.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:46 smithi192 ceph-mon[32956]: pgmap v2319: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 941 op/s 2024-06-27T21:49:46.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:46.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:46.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:46.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:46 smithi190 ceph-mon[32247]: pgmap v2319: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 941 op/s 2024-06-27T21:49:46.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:46.592 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:46.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:46.718 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:46.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:46.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:46.843 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:46.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 82 16 100844 100828 76.8276 54.6875 0.0130837 0.0130109 2024-06-27T21:49:46.943 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 1, 7, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:49:46.943 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:49:46.943 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-06-27T21:49:46.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-06-27T21:49:46.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:46.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:46.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:47.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:47.105 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:47.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:47.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:47.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:47.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:47.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:47.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:47.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:47.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:47.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:47.762 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:47.864 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:47.893 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:47.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:47.900 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:47.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 83 16 101927 101911 76.7174 67.6875 0.00479139 0.0130297 2024-06-27T21:49:47.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:48.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:48.087 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:48 smithi151 ceph-mon[28965]: pgmap v2320: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s 2024-06-27T21:49:48.087 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:48 smithi151 ceph-mon[28965]: 9.1 deep-scrub ok 2024-06-27T21:49:48.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:48.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:48.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:48.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:48.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:48 smithi192 ceph-mon[32956]: pgmap v2320: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s 2024-06-27T21:49:48.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:48 smithi192 ceph-mon[32956]: 9.1 deep-scrub ok 2024-06-27T21:49:48.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:48.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:48.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:48.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:48 smithi190 ceph-mon[32247]: pgmap v2320: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 926 op/s 2024-06-27T21:49:48.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:48 smithi190 ceph-mon[32247]: 9.1 deep-scrub ok 2024-06-27T21:49:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.540 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:48.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:48.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:48.669 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:48.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:48.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:48.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:48.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 84 16 102943 102927 76.56 63.5 0.00860245 0.0130552 2024-06-27T21:49:48.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:48.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:49.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:49.038 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:49:49.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:49.170 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:49.195 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:49.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1911538205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:49.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:49 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:49.297 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:49.326 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:49.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:49.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1911538205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:49.445 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:49 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:49.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1911538205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:49 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:49:49.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:49.584 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:49.601 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:49:49.602 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:49:49.602 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:49:49.602 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:49:49.602 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:49:49.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:49.711 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:49.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:49.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:49.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 85 16 103919 103903 76.3769 61 0.0418477 0.0130871 2024-06-27T21:49:49.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:49.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:49.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:50.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:50.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:50.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:50.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:50 smithi151 ceph-mon[28965]: pgmap v2321: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s 2024-06-27T21:49:50.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:50 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:50 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:50 smithi151 ceph-mon[28965]: osdmap e951: 12 total, 12 up, 12 in 2024-06-27T21:49:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:49:50.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:49:50.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:50.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:50 smithi192 ceph-mon[32956]: pgmap v2321: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s 2024-06-27T21:49:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:50 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:50 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:50 smithi192 ceph-mon[32956]: osdmap e951: 12 total, 12 up, 12 in 2024-06-27T21:49:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:49:50.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:49:50.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:50.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:50.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:50.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:50 smithi190 ceph-mon[32247]: pgmap v2321: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s 2024-06-27T21:49:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:50 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:49:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:50 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:49:50.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:50 smithi190 ceph-mon[32247]: osdmap e951: 12 total, 12 up, 12 in 2024-06-27T21:49:50.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:49:50.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:49:50.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:50.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:50.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:50.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:50.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:50.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:50.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:50.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 86 16 105125 105109 76.3652 75.375 0.0194211 0.0130902 2024-06-27T21:49:50.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:51.006 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:51.012 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:51.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:51.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:51.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:51 smithi151 ceph-mon[28965]: osdmap e952: 12 total, 12 up, 12 in 2024-06-27T21:49:51.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:51.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:51.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:51 smithi192 ceph-mon[32956]: osdmap e952: 12 total, 12 up, 12 in 2024-06-27T21:49:51.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:51.397 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:51.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-06-27T21:49:51.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:51.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:51 smithi190 ceph-mon[32247]: osdmap e952: 12 total, 12 up, 12 in 2024-06-27T21:49:51.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:51.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:51.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:51.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:51.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:51.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:51.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:51.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:51.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 87 16 106168 106152 76.2367 65.1875 0.00402746 0.0131112 2024-06-27T21:49:52.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:49:52.001 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:49:52.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:52.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:49:52.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:49:52.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:52.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:49:52.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:49:52.081 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:49:52.081 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:49:52.110 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:49:52.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:49:52.136 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:49:52.137 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:49:52.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:52.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:52.169 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:49:52.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:49:52.202 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:49:52.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:49:52.230 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:49:52.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:49:52.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:52 smithi151 ceph-mon[28965]: pgmap v2324: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-27T21:49:52.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4021959755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:52.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:49:52.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:49:52.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:52.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:52 smithi192 ceph-mon[32956]: pgmap v2324: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-27T21:49:52.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4021959755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:52.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:52.299 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:49:52.299 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:49:52.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:49:52.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:49:52.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:52.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:52.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:52 smithi190 ceph-mon[32247]: pgmap v2324: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.08k op/s 2024-06-27T21:49:52.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4021959755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:49:52.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:52.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:52.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:52.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:52.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:52.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:52.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 88 16 107284 107268 76.163 69.75 0.00731362 0.0131249 2024-06-27T21:49:52.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:52.948 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:52.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:52.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:49:53.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:53.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:53.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:53.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:53.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:53.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:53.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:53.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:53.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:53.600 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:53.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:53.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:53.791 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:53.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:53.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:53.911 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 89 16 108447 108431 76.1239 72.6875 0.00493411 0.0131309 2024-06-27T21:49:53.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:53.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:53.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:54.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:54.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:54.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:54 smithi151 ceph-mon[28965]: pgmap v2325: 47 pgs: 47 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 832 op/s 2024-06-27T21:49:54.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:54.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:54.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:54.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:54.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:54.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:54.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:54 smithi190 ceph-mon[32247]: pgmap v2325: 47 pgs: 47 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 832 op/s 2024-06-27T21:49:54.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:54 smithi192 ceph-mon[32956]: pgmap v2325: 47 pgs: 47 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 832 op/s 2024-06-27T21:49:54.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:54.617 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:54.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:54.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:54.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:54.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:54.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:54.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout: 90 7 109447 109440 75.9786 63.0625 0.00990954 0.0131541 2024-06-27T21:49:54.981 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:55.002 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:55.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:55.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:55.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:55.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:55.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:55.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:55.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.501 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:55.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:55.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:49:55.603 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:55.624 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:55.676 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:55.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:55.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:55.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:55.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:55.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total time run: 90.7007 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Total writes made: 109447 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Write size: 65536 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Object size: 65536 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Bandwidth (MB/sec): 75.4177 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Bandwidth: 18.3733 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max bandwidth (MB/sec): 124.375 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min bandwidth (MB/sec): 46.5 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average IOPS: 1206 2024-06-27T21:49:55.912 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev IOPS: 293.973 2024-06-27T21:49:55.913 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max IOPS: 1990 2024-06-27T21:49:55.913 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min IOPS: 744 2024-06-27T21:49:55.913 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Average Latency(s): 0.0131848 2024-06-27T21:49:55.913 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Stddev Latency(s): 0.0184999 2024-06-27T21:49:55.913 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Max latency(s): 0.715998 2024-06-27T21:49:55.913 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Min latency(s): 0.0010169 2024-06-27T21:49:55.921 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Cleaning up (deleting benchmark objects) 2024-06-27T21:49:55.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:49:56.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:49:56.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:56.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:56.229 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.257 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:56.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:56.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:56.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:56.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:56 smithi151 ceph-mon[28965]: pgmap v2326: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 972 op/s 2024-06-27T21:49:56.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:49:56.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-06-27T21:49:56.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:49:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:56 smithi190 ceph-mon[32247]: pgmap v2326: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 972 op/s 2024-06-27T21:49:56.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:56 smithi192 ceph-mon[32956]: pgmap v2326: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 972 op/s 2024-06-27T21:49:56.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.635 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:56.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:56.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:56.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:56.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:56.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:56.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:56.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:57.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:57.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:57.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:49:57.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:57.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:57.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:57.400 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:57.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:57.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:57.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:57.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:57.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:57.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:57.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:49:57.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:57.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:49:58.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:58.041 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:58.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:49:58 smithi151 ceph-mon[28965]: pgmap v2327: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:49:58.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:58.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:58.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:58.300 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:58.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.433 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:58.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:49:58 smithi190 ceph-mon[32247]: pgmap v2327: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:49:58.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:49:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:49:58 smithi192 ceph-mon[32956]: pgmap v2327: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-06-27T21:49:58.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:49:58.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:58.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:58.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:49:58.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:49:58.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:49:58.956 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:49:58.956 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:49:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:58.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:58.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:49:59.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:49:59.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:49:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:49:59.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:59.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:59.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:59.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:49:59.324 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:49:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:59.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:49:59.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:59.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:49:59.597 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:49:59.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:49:59.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:49:59.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:49:59.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:49:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:49:59.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:49:59.976 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:49:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:49:59.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:00.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:00.104 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:00 smithi151 ceph-mon[28965]: pgmap v2328: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 1.01k op/s 2024-06-27T21:50:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:00 smithi151 ceph-mon[28965]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-06-27T21:50:00.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:00 smithi151 ceph-mon[28965]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-06-27T21:50:00.205 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:00.237 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:00.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:00.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:00.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:00.497 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:00 smithi190 ceph-mon[32247]: pgmap v2328: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 1.01k op/s 2024-06-27T21:50:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:00 smithi190 ceph-mon[32247]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-06-27T21:50:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:00 smithi190 ceph-mon[32247]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-06-27T21:50:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:00 smithi192 ceph-mon[32956]: pgmap v2328: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 1.01k op/s 2024-06-27T21:50:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:00 smithi192 ceph-mon[32956]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-06-27T21:50:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:00 smithi192 ceph-mon[32956]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-06-27T21:50:00.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:00.626 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:00.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:00.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:00.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:00.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:00.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:00.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:01.015 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:01.116 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:01.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:01.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.277 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:01.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:01.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:01.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:01.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:01 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:01.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:01 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:01.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:01 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:01.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:01 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:01.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:01 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:01 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:01.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:01.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:01.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:01.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:01.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:01.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:01.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:01.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:01.973 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:02.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-06-27T21:50:02.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:02.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:02.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:02.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:02.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:02.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:02 smithi151 ceph-mon[28965]: pgmap v2329: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 B/s rd, 43 MiB/s wr, 946 op/s 2024-06-27T21:50:02.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:02 smithi151 ceph-mon[28965]: osdmap e953: 12 total, 12 up, 12 in 2024-06-27T21:50:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:02.452 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.482 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:02.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:02.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:02 smithi192 ceph-mon[32956]: pgmap v2329: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 B/s rd, 43 MiB/s wr, 946 op/s 2024-06-27T21:50:02.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:02 smithi192 ceph-mon[32956]: osdmap e953: 12 total, 12 up, 12 in 2024-06-27T21:50:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:02 smithi190 ceph-mon[32247]: pgmap v2329: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 B/s rd, 43 MiB/s wr, 946 op/s 2024-06-27T21:50:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:02 smithi190 ceph-mon[32247]: osdmap e953: 12 total, 12 up, 12 in 2024-06-27T21:50:02.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:02.615 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:02.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:02.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:02.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:02.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:02.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:02.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:03.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:03.092 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:50:03.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:03.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:03.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:03.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.398 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:03.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:03.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:03 smithi151 ceph-mon[28965]: osdmap e954: 12 total, 12 up, 12 in 2024-06-27T21:50:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3277732416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:03.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:03 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:03.499 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:03 smithi190 ceph-mon[32247]: osdmap e954: 12 total, 12 up, 12 in 2024-06-27T21:50:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3277732416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:03 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.531 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:03.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:03 smithi192 ceph-mon[32956]: osdmap e954: 12 total, 12 up, 12 in 2024-06-27T21:50:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3277732416' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:03 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:03.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:03.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:03.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:03.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:03.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:03.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:03.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:04.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:04.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:04.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:04.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:04.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:04 smithi151 ceph-mon[28965]: pgmap v2332: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 27 MiB/s wr, 840 op/s 2024-06-27T21:50:04.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:04 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:04.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:04 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:50:04.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:04 smithi151 ceph-mon[28965]: osdmap e955: 12 total, 12 up, 12 in 2024-06-27T21:50:04.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:04 smithi151 ceph-mon[28965]: 9.d scrub starts 2024-06-27T21:50:04.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.309 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:04.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:04.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:04.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:04 smithi190 ceph-mon[32247]: pgmap v2332: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 27 MiB/s wr, 840 op/s 2024-06-27T21:50:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:04 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:04 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:50:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:04 smithi190 ceph-mon[32247]: osdmap e955: 12 total, 12 up, 12 in 2024-06-27T21:50:04.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:04 smithi190 ceph-mon[32247]: 9.d scrub starts 2024-06-27T21:50:04.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:04 smithi192 ceph-mon[32956]: pgmap v2332: 47 pgs: 47 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 27 MiB/s wr, 840 op/s 2024-06-27T21:50:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:04 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:04 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:50:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:04 smithi192 ceph-mon[32956]: osdmap e955: 12 total, 12 up, 12 in 2024-06-27T21:50:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:04 smithi192 ceph-mon[32956]: 9.d scrub starts 2024-06-27T21:50:04.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:04.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:04.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:04.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:04.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:04.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:04.949 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:04.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:05.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:05.074 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:05.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:05.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:05.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:05.324 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:05.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:05.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:05.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:05.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:05.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-06-27T21:50:05.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:05.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:05.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:05.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:05.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:05.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:50:05.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:05.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:06.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:06.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:06.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:06 smithi151 ceph-mon[28965]: pgmap v2334: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.3 MiB/s wr, 930 op/s 2024-06-27T21:50:06.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:06 smithi151 ceph-mon[28965]: 9.d scrub ok 2024-06-27T21:50:06.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:06.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:06.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:06.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:06.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:06.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:06.486 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:06.486 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:06.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:06.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:06 smithi190 ceph-mon[32247]: pgmap v2334: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.3 MiB/s wr, 930 op/s 2024-06-27T21:50:06.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:06 smithi190 ceph-mon[32247]: 9.d scrub ok 2024-06-27T21:50:06.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:50:06.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:50:06.517 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:06.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:06 smithi192 ceph-mon[32956]: pgmap v2334: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.3 MiB/s wr, 930 op/s 2024-06-27T21:50:06.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:06 smithi192 ceph-mon[32956]: 9.d scrub ok 2024-06-27T21:50:06.556 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:50:06.556 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:50:06.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:50:06.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:50:06.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:06.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:50:06.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:50:06.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:06.642 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:50:06.642 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:50:06.676 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:50:06.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:50:06.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:50:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:50:06.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:50:06.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:50:06.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:06.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:50:06.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:50:06.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:06.800 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:50:06.801 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:50:06.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:50:06.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:50:06.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:50:06.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:50:06.871 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:06.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:06.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:07.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:07.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:07.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:07.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:07.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:07.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:07 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:50:07.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:07 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:50:07.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:07 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/129346675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:07.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:07.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:07.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:07.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:07 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:50:07.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:07 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:50:07.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:07 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/129346675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:07.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:07.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:07.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:07 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:50:07.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:07 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:50:07.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:07 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/129346675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:07.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:07.670 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:07.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:07.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:07.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:07.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:07.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:08.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:08.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:08.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:08.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:08.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:08 smithi151 ceph-mon[28965]: pgmap v2335: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:50:08.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:08 smithi151 ceph-mon[28965]: 9.3 scrub starts 2024-06-27T21:50:08.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:08.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:08.408 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.441 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:08.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:08 smithi190 ceph-mon[32247]: pgmap v2335: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:50:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:08 smithi190 ceph-mon[32247]: 9.3 scrub starts 2024-06-27T21:50:08.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:08 smithi192 ceph-mon[32956]: pgmap v2335: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-06-27T21:50:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:08 smithi192 ceph-mon[32956]: 9.3 scrub starts 2024-06-27T21:50:08.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:08.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:08.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:08.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:08.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:08.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:08.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:08.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:09.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:09.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:09.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:09.216 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:09.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:09.350 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 9.4 scrub starts 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 9.3 scrub ok 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 9.0 scrub starts 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: pgmap v2336: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 9.a scrub starts 2024-06-27T21:50:09.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:09 smithi151 ceph-mon[28965]: 9.7 scrub starts 2024-06-27T21:50:09.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:09.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 9.4 scrub starts 2024-06-27T21:50:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:50:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:50:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 9.3 scrub ok 2024-06-27T21:50:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 9.0 scrub starts 2024-06-27T21:50:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: pgmap v2336: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-06-27T21:50:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 9.a scrub starts 2024-06-27T21:50:09.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:09 smithi190 ceph-mon[32247]: 9.7 scrub starts 2024-06-27T21:50:09.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 9.4 scrub starts 2024-06-27T21:50:09.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:50:09.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:50:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 9.3 scrub ok 2024-06-27T21:50:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 9.0 scrub starts 2024-06-27T21:50:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: pgmap v2336: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-06-27T21:50:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 9.a scrub starts 2024-06-27T21:50:09.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:09 smithi192 ceph-mon[32956]: 9.7 scrub starts 2024-06-27T21:50:09.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:09.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:09.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:09.741 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:09.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:09.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:09.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:09.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:10.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:10.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:10.141 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:10.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:10.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:10.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:10.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:10 smithi151 ceph-mon[28965]: 9.4 scrub ok 2024-06-27T21:50:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:10 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:50:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:10 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:50:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:10 smithi151 ceph-mon[28965]: 9.a scrub ok 2024-06-27T21:50:10.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:10 smithi190 ceph-mon[32247]: 9.4 scrub ok 2024-06-27T21:50:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:10 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:50:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:10 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:50:10.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:10 smithi190 ceph-mon[32247]: 9.a scrub ok 2024-06-27T21:50:10.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:10 smithi192 ceph-mon[32956]: 9.4 scrub ok 2024-06-27T21:50:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:10 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:50:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:10 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:50:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:10 smithi192 ceph-mon[32956]: 9.a scrub ok 2024-06-27T21:50:10.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:10.658 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:10.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:10.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:10.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:10.913 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:10.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:11.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:11.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:11.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:11.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:11.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:11.294 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:11.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:11.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:11.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: 9.0 scrub ok 2024-06-27T21:50:11.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: 9.7 scrub ok 2024-06-27T21:50:11.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:50:11.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:50:11.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: pgmap v2337: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-06-27T21:50:11.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:50:11.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:11 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:50:11.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: 9.0 scrub ok 2024-06-27T21:50:11.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: 9.7 scrub ok 2024-06-27T21:50:11.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:50:11.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:50:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: pgmap v2337: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-06-27T21:50:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:50:11.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:11 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:50:11.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:11.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: 9.0 scrub ok 2024-06-27T21:50:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: 9.7 scrub ok 2024-06-27T21:50:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:50:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:50:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: pgmap v2337: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-06-27T21:50:11.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:50:11.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:11 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:50:11.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:11.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:11.678 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:11.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:11.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:11.911 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:11.940 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:11.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:12.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:12.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:12.171 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:12.184 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:50:12.184 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:50:12.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:125508 9.f e955/924 9:fca8b03b:::benchmark_data_smithi192_179902_object16058:head v 955'7875, mlcod=955'7873)", 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:50:11.903874+0000", 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0043186850000000001, 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043442680000000001, 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903874+0000", 2024-06-27T21:50:12.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903874+0000", 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903875+0000", 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.54e-07 2024-06-27T21:50:12.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903881+0000", 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8080000000000001e-06 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903929+0000", 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7936000000000003e-05 2024-06-27T21:50:12.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903936+0000", 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3939999999999998e-06 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903999+0000", 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3161000000000001e-05 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:50:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.904017+0000", 2024-06-27T21:50:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8223000000000002e-05 2024-06-27T21:50:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:50:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.190 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:125506 9.2 e955/916 9:414ffa41:::benchmark_data_smithi192_179902_object16056:head v 955'7808, mlcod=955'7804)", 2024-06-27T21:50:12.190 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:50:11.903903+0000", 2024-06-27T21:50:12.190 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0042899740000000002, 2024-06-27T21:50:12.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00436791, 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:50:12.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903903+0000", 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903903+0000", 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903903+0000", 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1699999999999999e-07 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:50:12.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903907+0000", 2024-06-27T21:50:12.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7960000000000002e-06 2024-06-27T21:50:12.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:50:12.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903974+0000", 2024-06-27T21:50:12.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6863e-05 2024-06-27T21:50:12.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:50:12.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.903985+0000", 2024-06-27T21:50:12.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0954e-05 2024-06-27T21:50:12.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:50:12.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.904029+0000", 2024-06-27T21:50:12.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4005999999999998e-05 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.904043+0000", 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3861e-05 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:125509 9.a e955/916 9:581905de:::benchmark_data_smithi192_179902_object16059:head v 955'7734, mlcod=955'7732)", 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:50:11.906049+0000", 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0021436789999999999, 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022663779999999999, 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906049+0000", 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906049+0000", 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906081+0000", 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1720999999999997e-05 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906087+0000", 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3999999999999997e-06 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906174+0000", 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6879999999999995e-05 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:50:12.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906181+0000", 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9879999999999998e-06 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906226+0000", 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4715000000000001e-05 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.906245+0000", 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9071000000000001e-05 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:50:12.201 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:125512 9.4 e955/916 9:2df77d1a:::benchmark_data_smithi192_179902_object16062:head v 955'7946, mlcod=955'7943)", 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:50:11.907798+0000", 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.000394662, 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055937300000000005, 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:50:12.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907798+0000", 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907798+0000", 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907799+0000", 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7199999999999998e-07 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907805+0000", 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8710000000000002e-06 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907839+0000", 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4275e-05 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907846+0000", 2024-06-27T21:50:12.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7100000000000001e-06 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907902+0000", 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6679999999999999e-05 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907931+0000", 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8252e-05 2024-06-27T21:50:12.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:125511 9.0 e955/916 9:0bee1b14:::benchmark_data_smithi192_179902_object16061:head v 955'7864, mlcod=955'7862)", 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:50:11.907821+0000", 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00037176100000000002, 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058021199999999998, 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:50:12.206 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907821+0000", 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907821+0000", 2024-06-27T21:50:12.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907822+0000", 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.111e-06 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907829+0000", 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4330000000000001e-06 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907864+0000", 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5565000000000003e-05 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907871+0000", 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7479999999999996e-06 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907917+0000", 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5899999999999998e-05 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:50:11.907960+0000", 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3689000000000001e-05 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:50:12.210 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:50:12.211 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 5 2024-06-27T21:50:12.211 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:50:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.212 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:12.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:12.313 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:12.340 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:12.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:12.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:12 smithi151 ceph-mon[28965]: 9.b scrub starts 2024-06-27T21:50:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:12.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:12.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:12 smithi190 ceph-mon[32247]: 9.b scrub starts 2024-06-27T21:50:12.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:12 smithi192 ceph-mon[32956]: 9.b scrub starts 2024-06-27T21:50:12.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:12.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:12.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:12.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:12.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:50:12.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:12.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:12.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:12.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:12.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:13.079 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:13.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:13.127 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:13 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:50:13.127 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:13 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:50:13.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:13.251 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:13.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:13 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:13 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:50:13.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:13 smithi151 ceph-mon[28965]: pgmap v2338: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1k op/s 2024-06-27T21:50:13.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:13 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:50:13.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:13.509 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:13 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:50:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:13 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:50:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:13 smithi190 ceph-mon[32247]: pgmap v2338: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1k op/s 2024-06-27T21:50:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:13 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:50:13.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:13 smithi192 ceph-mon[32956]: pgmap v2338: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1k op/s 2024-06-27T21:50:13.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:13 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:50:13.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:13.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:13.738 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:13.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:13.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:13.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:13.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:13.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-06-27T21:50:13.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:14.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:14.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:14.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:14.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:14.289 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:14.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:14.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:14.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: 9.b scrub ok 2024-06-27T21:50:14.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:14.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:14.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: osdmap e956: 12 total, 12 up, 12 in 2024-06-27T21:50:14.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: 9.9 scrub starts 2024-06-27T21:50:14.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:14.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:14 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:14.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: 9.b scrub ok 2024-06-27T21:50:14.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:14.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:14.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: osdmap e956: 12 total, 12 up, 12 in 2024-06-27T21:50:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: 9.9 scrub starts 2024-06-27T21:50:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:14.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:14 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:14.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:14.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: 9.b scrub ok 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: osdmap e956: 12 total, 12 up, 12 in 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: 9.9 scrub starts 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:14.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:14 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:14.642 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:14.674 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:14.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.804 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:14.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:14.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:14.939 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:14.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:15.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:15.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:15.168 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:15.201 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:15.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-06-27T21:50:15.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:15.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:15.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: 9.5 scrub starts 2024-06-27T21:50:15.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:50:15.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:50:15.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:15.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:50:15.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:50:15.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:15 smithi151 ceph-mon[28965]: pgmap v2340: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-06-27T21:50:15.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:15.464 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: 9.5 scrub starts 2024-06-27T21:50:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:50:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:50:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:50:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:50:15.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:15 smithi190 ceph-mon[32247]: pgmap v2340: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: 9.5 scrub starts 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:50:15.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:15 smithi192 ceph-mon[32956]: pgmap v2340: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-06-27T21:50:15.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:15.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:15.634 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:15.686 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:15.693 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:15.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:15.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:15.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:15.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:15.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:15.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:15.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:15.981 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:50:16.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:16.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:16.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:16.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:16.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:16.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:16.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:16.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:16.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:16.608 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:16.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:16.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:16.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: 9.9 scrub ok 2024-06-27T21:50:16.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3200072357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:50:16.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:50:16.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:50:16.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3200072357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:50:16.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:16 smithi151 ceph-mon[28965]: osdmap e957: 12 total, 12 up, 12 in 2024-06-27T21:50:16.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:16.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: 9.9 scrub ok 2024-06-27T21:50:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3200072357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:50:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:50:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:50:16.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3200072357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:50:16.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:16 smithi190 ceph-mon[32247]: osdmap e957: 12 total, 12 up, 12 in 2024-06-27T21:50:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: 9.9 scrub ok 2024-06-27T21:50:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3200072357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:50:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:50:16.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:50:16.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3200072357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:50:16.796 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:16 smithi192 ceph-mon[32956]: osdmap e957: 12 total, 12 up, 12 in 2024-06-27T21:50:16.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:16.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:16.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:16.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:16.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:17.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:17.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:17.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:17.233 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:17.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:17.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:17.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:17.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:17 smithi190 ceph-mon[32247]: 9.5 scrub ok 2024-06-27T21:50:17.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:17 smithi190 ceph-mon[32247]: pgmap v2342: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:50:17.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:17 smithi190 ceph-mon[32247]: 9.6 deep-scrub starts 2024-06-27T21:50:17.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:17.613 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:17 smithi151 ceph-mon[28965]: 9.5 scrub ok 2024-06-27T21:50:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:17 smithi151 ceph-mon[28965]: pgmap v2342: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:50:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:17 smithi151 ceph-mon[28965]: 9.6 deep-scrub starts 2024-06-27T21:50:17.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:17.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:17 smithi192 ceph-mon[32956]: 9.5 scrub ok 2024-06-27T21:50:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:17 smithi192 ceph-mon[32956]: pgmap v2342: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 869 op/s 2024-06-27T21:50:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:17 smithi192 ceph-mon[32956]: 9.6 deep-scrub starts 2024-06-27T21:50:17.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:17.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:17.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:17.988 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:17.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:18.090 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:18.114 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:18.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:18.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:18.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:18.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:18.490 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:18.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:18.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:18.718 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:18.744 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:18.846 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:18.875 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:18.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:18.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:19.003 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:19.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:19.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:19.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:19.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:19.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:19.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:19.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:19.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:19.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:19.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:19.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:19 smithi151 ceph-mon[28965]: pgmap v2343: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-06-27T21:50:19.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:19.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:19 smithi190 ceph-mon[32247]: pgmap v2343: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-06-27T21:50:19.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:19.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:19 smithi192 ceph-mon[32956]: pgmap v2343: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-06-27T21:50:19.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:50:19.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:19.907 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:19.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:20.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:20.034 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:20.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:20.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:20.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.284 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:20.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:20.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:20.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:20.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:20.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:20.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:20.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:20.762 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.790 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:20.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:20.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:20.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:20.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:21.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:21.146 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:21.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:21.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:21.314 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:21.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:21.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:21.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:21.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:21.672 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:21.698 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:21.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:21.826 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:21.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:21.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:21.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:22.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:22.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:21 smithi151 ceph-mon[28965]: pgmap v2344: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-06-27T21:50:22.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:22.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:22.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:22.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:21 smithi190 ceph-mon[32247]: pgmap v2344: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-06-27T21:50:22.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:21 smithi192 ceph-mon[32956]: pgmap v2344: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-06-27T21:50:22.330 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:50:22.330 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:50:22.330 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:50:22.330 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:50:22.330 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:50:22.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:22.361 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:22.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:22.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:22.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:22.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:22.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:22.745 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:22.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:22.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:22.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:23.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:23.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:23.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:23.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:22 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:50:23.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:22 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:50:23.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:22 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:50:23.193 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:22 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:50:23.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:23.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-06-27T21:50:23.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:22 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:50:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:22 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:50:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:22 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:50:23.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:22 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:50:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:22 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:50:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:22 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:50:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:22 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:50:23.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:22 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:50:23.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:23.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:23.480 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.507 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:23.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:23.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:23.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:23.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:23.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:23.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:23.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:23.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:23.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:23.988 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:23.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:24.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:24.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-06-27T21:50:24.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:24.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:24.243 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:23 smithi151 ceph-mon[28965]: pgmap v2345: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-06-27T21:50:24.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:24.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:23 smithi190 ceph-mon[32247]: pgmap v2345: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-06-27T21:50:24.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:23 smithi192 ceph-mon[32956]: pgmap v2345: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-06-27T21:50:24.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:24.411 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:24.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:24.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:24.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:24.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:24.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:50:24.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:50:24.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:24.782 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:50:24.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:50:24.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:24.814 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:50:24.814 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:50:24.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:50:24.845 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:50:24.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:50:24.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:50:24.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:24.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:50:24.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:50:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:24.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:24.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:24.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:50:24.938 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:50:24.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:50:24.970 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:50:24.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:50:24.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:50:25.019 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:50:25.019 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:50:25.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:25.043 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:50:25.043 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:50:25.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:25 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:50:25.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:25 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:50:25.044 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:25 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2310168408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:25.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:25.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:50:25.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:50:25.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:25.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:25 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:50:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:25 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:50:25.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:25 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2310168408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:25.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:25 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:50:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:25 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:50:25.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:25 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2310168408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:25.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:25.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.431 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:25.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:25.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:25.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:25.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:25.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:25.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:25.811 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:25.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:25.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:25.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:26.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:26.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:26.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:26.164 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:26.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:26 smithi190 ceph-mon[32247]: pgmap v2346: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 799 op/s 2024-06-27T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:26 smithi190 ceph-mon[32247]: 1.3 deep-scrub starts 2024-06-27T21:50:26.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:26 smithi190 ceph-mon[32247]: 1.3 deep-scrub ok 2024-06-27T21:50:26.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:26 smithi190 ceph-mon[32247]: osdmap e958: 12 total, 12 up, 12 in 2024-06-27T21:50:26.275 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:26.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:26.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:26 smithi151 ceph-mon[28965]: pgmap v2346: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 799 op/s 2024-06-27T21:50:26.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:26 smithi151 ceph-mon[28965]: 1.3 deep-scrub starts 2024-06-27T21:50:26.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:26 smithi151 ceph-mon[28965]: 1.3 deep-scrub ok 2024-06-27T21:50:26.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:26 smithi151 ceph-mon[28965]: osdmap e958: 12 total, 12 up, 12 in 2024-06-27T21:50:26.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:26 smithi192 ceph-mon[32956]: pgmap v2346: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 799 op/s 2024-06-27T21:50:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:26 smithi192 ceph-mon[32956]: 1.3 deep-scrub starts 2024-06-27T21:50:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:26 smithi192 ceph-mon[32956]: 1.3 deep-scrub ok 2024-06-27T21:50:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:26 smithi192 ceph-mon[32956]: osdmap e958: 12 total, 12 up, 12 in 2024-06-27T21:50:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:26.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:26.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:26.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:26.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:26.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:26.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:26.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:26.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:26.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:26.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:26.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:27.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:27.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:27.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:27.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:27 smithi151 ceph-mon[28965]: 1.13 deep-scrub starts 2024-06-27T21:50:27.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:27 smithi151 ceph-mon[28965]: 1.13 deep-scrub ok 2024-06-27T21:50:27.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:27 smithi151 ceph-mon[28965]: 1.f deep-scrub starts 2024-06-27T21:50:27.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:27 smithi151 ceph-mon[28965]: 1.f deep-scrub ok 2024-06-27T21:50:27.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:27 smithi190 ceph-mon[32247]: 1.13 deep-scrub starts 2024-06-27T21:50:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:27 smithi190 ceph-mon[32247]: 1.13 deep-scrub ok 2024-06-27T21:50:27.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:27 smithi190 ceph-mon[32247]: 1.f deep-scrub starts 2024-06-27T21:50:27.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:27 smithi190 ceph-mon[32247]: 1.f deep-scrub ok 2024-06-27T21:50:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:27 smithi192 ceph-mon[32956]: 1.13 deep-scrub starts 2024-06-27T21:50:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:27 smithi192 ceph-mon[32956]: 1.13 deep-scrub ok 2024-06-27T21:50:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:27 smithi192 ceph-mon[32956]: 1.f deep-scrub starts 2024-06-27T21:50:27.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:27 smithi192 ceph-mon[32956]: 1.f deep-scrub ok 2024-06-27T21:50:27.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:27.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:27.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:27.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:27.564 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:27.590 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:27.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:27.713 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:27.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:27.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:27.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:27.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:27.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:27.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:28.068 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.092 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:28.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:28.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:28 smithi151 ceph-mon[28965]: pgmap v2348: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s 2024-06-27T21:50:28.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:28 smithi151 ceph-mon[28965]: 1.b deep-scrub starts 2024-06-27T21:50:28.096 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:28 smithi151 ceph-mon[28965]: 1.b deep-scrub ok 2024-06-27T21:50:28.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:50:28.193 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:28.214 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:28 smithi190 ceph-mon[32247]: pgmap v2348: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s 2024-06-27T21:50:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:28 smithi190 ceph-mon[32247]: 1.b deep-scrub starts 2024-06-27T21:50:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:28 smithi190 ceph-mon[32247]: 1.b deep-scrub ok 2024-06-27T21:50:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:28 smithi192 ceph-mon[32956]: pgmap v2348: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 758 op/s 2024-06-27T21:50:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:28 smithi192 ceph-mon[32956]: 1.b deep-scrub starts 2024-06-27T21:50:28.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:28 smithi192 ceph-mon[32956]: 1.b deep-scrub ok 2024-06-27T21:50:28.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:28.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:28.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:28.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:28.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:28.598 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:28.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:28.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:28.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:28.856 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:28.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:28.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:28.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:29.012 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:50:29.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:29.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:29 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:50:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:29 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:50:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3873827760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:50:29.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:29.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:29 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:50:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:29 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:50:29.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3873827760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:50:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:29 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:50:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:29 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:50:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3873827760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:50:29.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:29.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:29.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:29.511 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:29.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:29.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:29.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:29.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:29.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:29.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:29.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:29.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-06-27T21:50:29.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:30.020 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:30.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:30.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:30.171 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:30 smithi151 ceph-mon[28965]: pgmap v2349: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s 2024-06-27T21:50:30.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:30 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:30.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3873827760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:50:30.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:30 smithi151 ceph-mon[28965]: osdmap e959: 12 total, 12 up, 12 in 2024-06-27T21:50:30.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:30 smithi190 ceph-mon[32247]: pgmap v2349: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s 2024-06-27T21:50:30.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:30 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3873827760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:50:30.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:30 smithi190 ceph-mon[32247]: osdmap e959: 12 total, 12 up, 12 in 2024-06-27T21:50:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.278 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:30.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:30 smithi192 ceph-mon[32956]: pgmap v2349: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s 2024-06-27T21:50:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:30 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3873827760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:50:30.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:30 smithi192 ceph-mon[32956]: osdmap e959: 12 total, 12 up, 12 in 2024-06-27T21:50:30.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:30.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:30.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:30.522 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:30.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:30.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:30.662 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:30.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:30.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:30.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:30.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:30.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:31.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:31.037 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:31.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:31.169 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:31.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:31.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:31.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:31.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:31.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:31.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:31.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:31.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:31.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:31.809 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:31.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:31.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:31.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:32.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:32.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:32.162 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:32.191 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:32.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:32 smithi151 ceph-mon[28965]: pgmap v2351: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s 2024-06-27T21:50:32.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:32 smithi192 ceph-mon[32956]: pgmap v2351: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s 2024-06-27T21:50:32.320 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:32.422 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:32.453 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:32 smithi190 ceph-mon[32247]: pgmap v2351: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s 2024-06-27T21:50:32.536 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:50:32.536 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:50:32.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:32.582 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:32.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.707 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:32.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:32.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:32.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:32.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:32.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:32.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:33.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:33.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:33.202 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:33.227 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:33.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.355 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:33.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:33.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:33.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:33.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:33.612 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:33.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:33.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:33.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:33.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-06-27T21:50:33.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:33.912 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:33.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:33.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:33.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:34.076 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:34 smithi151 ceph-mon[28965]: pgmap v2352: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 617 op/s 2024-06-27T21:50:34.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:34.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:34.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:34.245 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:34 smithi192 ceph-mon[32956]: pgmap v2352: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 617 op/s 2024-06-27T21:50:34.348 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:34.378 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:34.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:34.483 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:50:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:50:34.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:34.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:34 smithi190 ceph-mon[32247]: pgmap v2352: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 617 op/s 2024-06-27T21:50:34.517 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:50:34.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:50:34.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:50:34.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:50:34.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:50:34.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:50:34.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:50:34.596 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:50:34.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:34.624 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:50:34.624 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:50:34.638 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:34.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:50:34.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:50:34.675 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:50:34.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:50:34.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:50:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:50:34.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:50:34.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:50:34.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:34.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:50:34.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:50:34.765 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:34.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:50:34.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:50:34.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:34.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:34.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:34.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:50:35.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:35.023 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:35.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:35.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:35.201 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3802359884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:35.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:35.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:35.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3802359884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:35.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:35.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:35.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:35.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3802359884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:35.533 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:35.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:35.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:35.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:35.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:35.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:35.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:35.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:35.974 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:36.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:36.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:36.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:36.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:36.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:36.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:36 smithi151 ceph-mon[28965]: pgmap v2353: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s 2024-06-27T21:50:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:36 smithi192 ceph-mon[32956]: pgmap v2353: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s 2024-06-27T21:50:36.297 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:36.399 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:36.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:36 smithi190 ceph-mon[32247]: pgmap v2353: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s 2024-06-27T21:50:36.422 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:36.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:36.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:36.657 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:36.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:36.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:36.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:36.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:36.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:36.945 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:36.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:37.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:37.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:37.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:37.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:37.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:37.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:37.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:37.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:37.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:37.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:37.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:37.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:37.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:37.859 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:37.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:37.983 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:37.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:38.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:38.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:38.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:38.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:38 smithi192 ceph-mon[32956]: pgmap v2354: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s 2024-06-27T21:50:38.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:38.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:38.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:38 smithi151 ceph-mon[28965]: pgmap v2354: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s 2024-06-27T21:50:38.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:38 smithi190 ceph-mon[32247]: pgmap v2354: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s 2024-06-27T21:50:38.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:38.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:38.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:38.627 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:38.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:50:38.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:38.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:38.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:38.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:38.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:38.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:39.009 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:39.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:39.132 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:50:39.132 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:50:39.132 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-06-27T21:50:39.133 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:50:39.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:39.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:39.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:39 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:39 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:39 smithi192 ceph-mon[32956]: osdmap e960: 12 total, 12 up, 12 in 2024-06-27T21:50:39.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:39.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:39 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:39.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:39 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:39.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:39 smithi151 ceph-mon[28965]: osdmap e960: 12 total, 12 up, 12 in 2024-06-27T21:50:39.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:39.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:39 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:39 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:39 smithi190 ceph-mon[32247]: osdmap e960: 12 total, 12 up, 12 in 2024-06-27T21:50:39.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:39.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:39.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:39.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:39.781 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.782 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:39.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:39.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:39.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:40.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:40.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:40.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:40.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:40.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-06-27T21:50:40.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:40.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:40 smithi192 ceph-mon[32956]: pgmap v2356: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-06-27T21:50:40.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:40.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:40.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:40 smithi151 ceph-mon[28965]: pgmap v2356: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-06-27T21:50:40.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:40.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:40.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:40 smithi190 ceph-mon[32247]: pgmap v2356: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s 2024-06-27T21:50:40.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:40.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:40.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:40.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:40.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:40.830 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:40.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:40.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:40.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:41.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:41.081 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:41.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:41.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:41.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:41.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:41.341 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:41.432 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:41.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:41.469 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:41.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-06-27T21:50:41.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:41.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:41.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:41.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:41.831 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.862 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:41.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:41.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:41.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:41.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:42.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:42.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:42.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:42.239 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:42.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:42 smithi151 ceph-mon[28965]: pgmap v2357: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s 2024-06-27T21:50:42.266 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:42.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:42 smithi192 ceph-mon[32956]: pgmap v2357: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s 2024-06-27T21:50:42.341 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:42.366 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:42.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:42.494 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:42.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:42 smithi190 ceph-mon[32247]: pgmap v2357: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s 2024-06-27T21:50:42.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:42.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:42.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:42.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:42.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:42.880 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:42.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:42.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:43.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:43.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:43.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:43.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:43.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:43.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:43.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:43.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:43.525 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:43.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:43.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:43.754 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:43.777 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:43.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:43.902 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:43.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:44.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:44.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:44.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:44.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:44.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:44 smithi151 ceph-mon[28965]: pgmap v2358: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-06-27T21:50:44.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:44.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:44.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:44 smithi192 ceph-mon[32956]: pgmap v2358: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-06-27T21:50:44.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:44.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:44 smithi190 ceph-mon[32247]: pgmap v2358: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-06-27T21:50:44.521 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:44.544 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:44.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:44.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:44.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:44.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:44.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:44.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:44.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:45.032 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:45.049 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:50:45.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:45.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:45.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:45.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:45.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:45.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/235915251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:50:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.319 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:45.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:45.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:45.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/235915251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:50:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/235915251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:50:45.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.583 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:45.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:45.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:45.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:45.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:45.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:45.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:45.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:46.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:46.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:46.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:46.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:46.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:46.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:46.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:46 smithi151 ceph-mon[28965]: pgmap v2359: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s 2024-06-27T21:50:46.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:46 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:46.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/235915251' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:50:46.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:46 smithi151 ceph-mon[28965]: osdmap e961: 12 total, 12 up, 12 in 2024-06-27T21:50:46.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:46.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:46 smithi190 ceph-mon[32247]: pgmap v2359: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s 2024-06-27T21:50:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:46 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/235915251' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:50:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:46 smithi190 ceph-mon[32247]: osdmap e961: 12 total, 12 up, 12 in 2024-06-27T21:50:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:46 smithi192 ceph-mon[32956]: pgmap v2359: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s 2024-06-27T21:50:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:46 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/235915251' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:50:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:46 smithi192 ceph-mon[32956]: osdmap e961: 12 total, 12 up, 12 in 2024-06-27T21:50:46.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:46.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:46.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:46.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:46.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:46.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:46.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:46.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:47.011 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:47.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:47.138 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:47.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:47.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:47.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:47.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:47.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:47.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:47.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:47.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:47.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:47.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:47.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:47.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:47.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:47.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:47.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:50:47.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:50:47.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:48.008 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:50:48.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:50:48.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:48.037 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:50:48.037 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:50:48.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:50:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:50:48.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:50:48.094 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:50:48.123 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:50:48.123 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:50:48.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:48.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:48.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:50:48.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:50:48.181 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:50:48.181 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:50:48.207 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:50:48.207 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:50:48.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:48.281 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:48.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:48 smithi151 ceph-mon[28965]: pgmap v2361: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-06-27T21:50:48.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/56171588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:48.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:48.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:48.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:48 smithi190 ceph-mon[32247]: pgmap v2361: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-06-27T21:50:48.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/56171588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:48.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:48.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:48 smithi192 ceph-mon[32956]: pgmap v2361: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s 2024-06-27T21:50:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/56171588' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:50:48.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:48.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:48.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:48.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:48.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:50:48.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:48.919 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:50:48.919 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:50:48.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:48.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:49.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:49.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:49.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:49.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:49 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:49 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:49.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:49.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:49.410 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:49.436 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:49 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:49.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:49 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:49.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:49 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:49 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:49.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:49.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:49.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:49.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:49.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:49.863 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:49.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:49.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:49.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:50.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:50.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:50.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:50.213 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:50.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:50 smithi151 ceph-mon[28965]: pgmap v2362: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s 2024-06-27T21:50:50.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:50 smithi151 ceph-mon[28965]: osdmap e962: 12 total, 12 up, 12 in 2024-06-27T21:50:50.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:50.304 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:50.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:50.344 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:50.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-06-27T21:50:50.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:50.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:50.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:50 smithi190 ceph-mon[32247]: pgmap v2362: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s 2024-06-27T21:50:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:50 smithi190 ceph-mon[32247]: osdmap e962: 12 total, 12 up, 12 in 2024-06-27T21:50:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:50 smithi192 ceph-mon[32956]: pgmap v2362: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 4.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s 2024-06-27T21:50:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:50 smithi192 ceph-mon[32956]: osdmap e962: 12 total, 12 up, 12 in 2024-06-27T21:50:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:50.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:50.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:50.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:50.730 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:50.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:50.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:50.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:50.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:50.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:51.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:51.115 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:51.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:51 smithi151 ceph-mon[28965]: osdmap e963: 12 total, 12 up, 12 in 2024-06-27T21:50:51.217 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:51.246 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:51.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:51.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:51.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:51.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:51.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:51 smithi190 ceph-mon[32247]: osdmap e963: 12 total, 12 up, 12 in 2024-06-27T21:50:51.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:51 smithi192 ceph-mon[32956]: osdmap e963: 12 total, 12 up, 12 in 2024-06-27T21:50:51.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:51.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:51.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:51.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:51.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:50:51.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:51.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:51.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:51.990 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:52.016 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:52.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:52.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:52 smithi151 ceph-mon[28965]: pgmap v2365: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-27T21:50:52.143 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:52.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:52.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:52.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:52.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:52.398 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:52.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:52.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:52 smithi190 ceph-mon[32247]: pgmap v2365: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-27T21:50:52.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:52 smithi192 ceph-mon[32956]: pgmap v2365: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-27T21:50:52.622 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:50:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.651 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:52.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:50:52.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:52.780 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:52.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:52.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:52.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:53.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:53.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:53.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:53.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:53.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:53.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:53.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:53.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:53.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.565 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:53.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:53.666 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:53.690 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:53.792 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:53.817 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:53.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:53.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:53.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:53.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:54.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:54.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:54.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:54 smithi151 ceph-mon[28965]: pgmap v2366: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s 2024-06-27T21:50:54.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:54.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:54.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:54.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:54.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:54 smithi190 ceph-mon[32247]: pgmap v2366: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s 2024-06-27T21:50:54.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:54 smithi192 ceph-mon[32956]: pgmap v2366: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s 2024-06-27T21:50:54.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:54.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:54.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:54.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-06-27T21:50:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:54.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:54.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:54.852 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:54.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:54.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:54.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:55.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:55.121 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:55.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:55.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:55.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:55.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:55.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:55 smithi151 ceph-mon[28965]: osdmap e964: 12 total, 12 up, 12 in 2024-06-27T21:50:55.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:55 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:50:55.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:55 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:50:55.349 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:55.376 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:55.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:55.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:55.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:55.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:55.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:55 smithi190 ceph-mon[32247]: osdmap e964: 12 total, 12 up, 12 in 2024-06-27T21:50:55.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:55 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:50:55.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:55 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:50:55.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:55.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:55.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:55 smithi192 ceph-mon[32956]: osdmap e964: 12 total, 12 up, 12 in 2024-06-27T21:50:55.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:55 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:50:55.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:55 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:50:55.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:55.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:55.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.768 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:55.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:55.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:55.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:55.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:56.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:56.028 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:56.099 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:50:56.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:56.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:56.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:56.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:56.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:56 smithi151 ceph-mon[28965]: pgmap v2368: 47 pgs: 47 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s 2024-06-27T21:50:56.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:56 smithi151 ceph-mon[28965]: osdmap e965: 12 total, 12 up, 12 in 2024-06-27T21:50:56.291 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/822400041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:56.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:56.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:56 smithi190 ceph-mon[32247]: pgmap v2368: 47 pgs: 47 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s 2024-06-27T21:50:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:56 smithi190 ceph-mon[32247]: osdmap e965: 12 total, 12 up, 12 in 2024-06-27T21:50:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/822400041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:56.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:56.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:56.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:56 smithi192 ceph-mon[32956]: pgmap v2368: 47 pgs: 47 active+clean; 3.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s 2024-06-27T21:50:56.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:56 smithi192 ceph-mon[32956]: osdmap e965: 12 total, 12 up, 12 in 2024-06-27T21:50:56.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/822400041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:50:56.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:56.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:56.772 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:50:56.793 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:50:56.822 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:50:56.823 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:50:56.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:56.923 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:56.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:57.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:57.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:57.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:57.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:57.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:57.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:57.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:50:57.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:50:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:57 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/822400041' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:50:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:57 smithi151 ceph-mon[28965]: osdmap e966: 12 total, 12 up, 12 in 2024-06-27T21:50:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:57 smithi151 ceph-mon[28965]: pgmap v2371: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:50:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:57 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:50:57.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:57 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:50:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:57 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/822400041' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:50:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:57 smithi190 ceph-mon[32247]: osdmap e966: 12 total, 12 up, 12 in 2024-06-27T21:50:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:57 smithi190 ceph-mon[32247]: pgmap v2371: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:50:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:57 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:50:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:57 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:50:57.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:57 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:50:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/822400041' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:50:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:57 smithi192 ceph-mon[32956]: osdmap e966: 12 total, 12 up, 12 in 2024-06-27T21:50:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:57 smithi192 ceph-mon[32956]: pgmap v2371: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 765 op/s 2024-06-27T21:50:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:57 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:50:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:57 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:50:57.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:57.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:50:57.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:50:57.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:57.823 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:57.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:57.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:57.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:57.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-06-27T21:50:58.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:50:58.095 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:58.095 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:58.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:50:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:58 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:50:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:58 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:50:58.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:58 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:50:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:58 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:50:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:58 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:50:58.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:58 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:50:58.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:58.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:58.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:58.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:58.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:50:58.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:50:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:58 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:50:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:58 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:50:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:58 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:50:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:58 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:50:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:58 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:50:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:58 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:50:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:58 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:50:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:58 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:50:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:58 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:50:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:58 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:50:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:58 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:50:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:58 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:50:58.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:58.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:58.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:58.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:58.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:58.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:58.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:58.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:58.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:59.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:50:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.110 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:50:59.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:50:59.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:50:59.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:50:59.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:59.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:59.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:59 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:50:59.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:59 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:50:59.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:59 smithi151 ceph-mon[28965]: pgmap v2372: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s 2024-06-27T21:50:59.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:50:59 smithi151 ceph-mon[28965]: 9.6 scrub starts 2024-06-27T21:50:59.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:50:59.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:50:59.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:59 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:50:59.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:59 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:50:59.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:59 smithi190 ceph-mon[32247]: pgmap v2372: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s 2024-06-27T21:50:59.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:50:59 smithi190 ceph-mon[32247]: 9.6 scrub starts 2024-06-27T21:50:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:59 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:50:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:59 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:50:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:59 smithi192 ceph-mon[32956]: pgmap v2372: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 984 op/s 2024-06-27T21:50:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:50:59 smithi192 ceph-mon[32956]: 9.6 scrub starts 2024-06-27T21:50:59.546 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:50:59.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:50:59.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:50:59.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:50:59.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:50:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:50:59.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:50:59.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:50:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:50:59.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:50:59.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:00.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:00.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:00.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:00.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:51:00.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:00.265 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:00.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:00.370 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:00 smithi151 ceph-mon[28965]: 9.8 scrub starts 2024-06-27T21:51:00.370 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:00 smithi151 ceph-mon[28965]: 9.6 scrub ok 2024-06-27T21:51:00.396 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:00.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:00 smithi190 ceph-mon[32247]: 9.8 scrub starts 2024-06-27T21:51:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:00 smithi190 ceph-mon[32247]: 9.6 scrub ok 2024-06-27T21:51:00.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:00 smithi192 ceph-mon[32956]: 9.8 scrub starts 2024-06-27T21:51:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:00 smithi192 ceph-mon[32956]: 9.6 scrub ok 2024-06-27T21:51:00.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:00.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:00.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:00.789 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:00.892 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:00.920 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:00.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:01.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:01.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:01.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:01.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:01.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:01.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:01.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:01.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:01.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: 9.2 scrub starts 2024-06-27T21:51:01.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: 9.2 scrub ok 2024-06-27T21:51:01.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: 9.8 scrub ok 2024-06-27T21:51:01.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: 9.c scrub starts 2024-06-27T21:51:01.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: pgmap v2373: 47 pgs: 47 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 884 op/s 2024-06-27T21:51:01.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:51:01.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:01 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:51:01.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: 9.2 scrub starts 2024-06-27T21:51:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: 9.2 scrub ok 2024-06-27T21:51:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: 9.8 scrub ok 2024-06-27T21:51:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: 9.c scrub starts 2024-06-27T21:51:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: pgmap v2373: 47 pgs: 47 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 884 op/s 2024-06-27T21:51:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:51:01.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:01 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:51:01.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: 9.2 scrub starts 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: 9.2 scrub ok 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: 9.8 scrub ok 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: 9.c scrub starts 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: pgmap v2373: 47 pgs: 47 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 884 op/s 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:51:01.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:01 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:51:01.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:01.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:01.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:01.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:51:01.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:51:01.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:51:01.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:51:01.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:51:01.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:51:01.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:01.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:51:01.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:51:01.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:01.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:51:01.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:51:01.875 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:51:01.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:51:01.900 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:51:01.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:51:01.922 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:51:01.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:51:01.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:01.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:51:01.946 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:51:01.955 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:01.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:01.976 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:51:01.976 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:51:02.003 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:51:02.003 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:51:02.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:51:02.033 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:51:02.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:02.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:02.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: 9.c scrub ok 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3705430673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:02.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:51:02.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:02 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:51:02.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:02.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:02.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:02.422 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:02.445 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: 9.c scrub ok 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3705430673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:51:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:02 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:51:02.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:02.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: 9.c scrub ok 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3705430673' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:51:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:02 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:51:02.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:02.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:02.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:02.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:02.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:02.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:02.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:02.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:03.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27262.0:164500 9.2 9:4cba217f:::benchmark_data_smithi192_179902_object55050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967)", 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:51:03.001380+0000", 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0060781910000000001, 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061039739999999999, 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:51:03.009 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27262", 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164500 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001380+0000", 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001380+0000", 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001381+0000", 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.13e-07 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001384+0000", 2024-06-27T21:51:03.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5109999999999999e-06 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001392+0000", 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8140000000000001e-06 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001396+0000", 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2849999999999996e-06 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001450+0000", 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3248000000000002e-05 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001557+0000", 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107419 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-06-27T21:51:03.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001664+0000", 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107157 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006167+0000", 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045030590000000002 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006254+0000", 2024-06-27T21:51:03.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7015999999999995e-05 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:164501 9.7 e967/916 9:e76a1f1a:::benchmark_data_smithi192_179902_object55051:head v 967'10183, mlcod=967'10180)", 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:51:03.001779+0000", 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0056788409999999996, 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057770479999999999, 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:51:03.015 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001779+0000", 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001779+0000", 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001780+0000", 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.23e-07 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001786+0000", 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9750000000000004e-06 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:51:03.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001804+0000", 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7486000000000002e-05 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001811+0000", 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9589999999999998e-06 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001870+0000", 2024-06-27T21:51:03.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.914e-05 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.001888+0000", 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8698999999999998e-05 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27262.0:164504 9.7 e967/916 9:e8507998:::benchmark_data_smithi192_179902_object55054:head v 967'10184, mlcod=967'10180)", 2024-06-27T21:51:03.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:51:03.006349+0000", 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.001109608, 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001253779, 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006349+0000", 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006349+0000", 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006349+0000", 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2699999999999999e-07 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006354+0000", 2024-06-27T21:51:03.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8890000000000001e-06 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006368+0000", 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3844e-05 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006374+0000", 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7309999999999998e-06 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006450+0000", 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6536e-05 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006469+0000", 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8199999999999999e-05 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.023 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27262.0:164505 9.1 9:8792f649:::benchmark_data_smithi192_179902_object55055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e967)", 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-06-27T21:51:03.006638+0000", 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00082005999999999995, 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010096009999999999, 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27262", 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 164505 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-06-27T21:51:03.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006638+0000", 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006638+0000", 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006639+0000", 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8800000000000001e-07 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006643+0000", 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7330000000000001e-06 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006651+0000", 2024-06-27T21:51:03.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9249999999999995e-06 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006656+0000", 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7999999999999998e-06 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006690+0000", 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4054000000000003e-05 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006768+0000", 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8907999999999997e-05 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-06-27T21:51:03.006837+0000", 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8503999999999993e-05 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:51:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:51:03.029 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-06-27T21:51:03.029 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:51:03.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:03.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:03.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:03 smithi151 ceph-mon[28965]: osdmap e967: 12 total, 12 up, 12 in 2024-06-27T21:51:03.200 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:03 smithi151 ceph-mon[28965]: pgmap v2375: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s 2024-06-27T21:51:03.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:03.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:03.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:03.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:03.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:03 smithi190 ceph-mon[32247]: osdmap e967: 12 total, 12 up, 12 in 2024-06-27T21:51:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:03 smithi190 ceph-mon[32247]: pgmap v2375: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s 2024-06-27T21:51:03.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:03 smithi192 ceph-mon[32956]: osdmap e967: 12 total, 12 up, 12 in 2024-06-27T21:51:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:03 smithi192 ceph-mon[32956]: pgmap v2375: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s 2024-06-27T21:51:03.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:03.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:03.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:03.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:03.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:03.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:03.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:03.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:04.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:04.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:04.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:04.164 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:04.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:04.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:04.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:51:04.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:51:04.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 9.1 scrub starts 2024-06-27T21:51:04.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:51:04.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:51:04.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:51:04.308 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:04 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:51:04.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:04.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.435 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:04.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 9.1 scrub starts 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:51:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:04 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:51:04.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:51:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:51:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 9.1 scrub starts 2024-06-27T21:51:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:51:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:51:04.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:51:04.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:04 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:51:04.561 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:04.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:04.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:04.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:04.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:04.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:04.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:04.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:04.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:05.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:05.077 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:05.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:05.205 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:05.298 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:05 smithi151 ceph-mon[28965]: 9.1 scrub ok 2024-06-27T21:51:05.298 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:05 smithi151 ceph-mon[28965]: pgmap v2376: 47 pgs: 47 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s 2024-06-27T21:51:05.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:05.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:05.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:05.460 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:05.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:05 smithi190 ceph-mon[32247]: 9.1 scrub ok 2024-06-27T21:51:05.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:05 smithi190 ceph-mon[32247]: pgmap v2376: 47 pgs: 47 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s 2024-06-27T21:51:05.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-06-27T21:51:05.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:05 smithi192 ceph-mon[32956]: 9.1 scrub ok 2024-06-27T21:51:05.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:05 smithi192 ceph-mon[32956]: pgmap v2376: 47 pgs: 47 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 806 op/s 2024-06-27T21:51:05.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:05.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:05.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:05.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:05.818 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:05.844 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:05.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:05.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:05.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:06.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:06.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:06.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:06.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:06.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:06.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:06.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:06 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:51:06.381 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:06 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:51:06.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:06 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:51:06.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:06 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:51:06.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-06-27T21:51:06.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:06.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:06.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:06 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:51:06.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:06 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:51:06.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:06.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:06.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:06.731 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:06.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:06.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:06.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:06.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:07.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:07.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:07.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:51:07.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:07.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:07.254 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:07.280 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:07.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:07 smithi151 ceph-mon[28965]: pgmap v2377: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s 2024-06-27T21:51:07.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:07.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:07.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:07.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:07 smithi190 ceph-mon[32247]: pgmap v2377: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s 2024-06-27T21:51:07.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:07.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:07 smithi192 ceph-mon[32956]: pgmap v2377: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s 2024-06-27T21:51:07.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:07.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:07.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:07.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:07.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:07.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:07.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:08.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:08.032 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:08.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:08.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:08.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:08.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:08.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:08.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:08 smithi151 ceph-mon[28965]: osdmap e968: 12 total, 12 up, 12 in 2024-06-27T21:51:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:08 smithi151 ceph-mon[28965]: 9.f scrub starts 2024-06-27T21:51:08.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:08 smithi190 ceph-mon[32247]: osdmap e968: 12 total, 12 up, 12 in 2024-06-27T21:51:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:08 smithi190 ceph-mon[32247]: 9.f scrub starts 2024-06-27T21:51:08.528 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:08 smithi192 ceph-mon[32956]: osdmap e968: 12 total, 12 up, 12 in 2024-06-27T21:51:08.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:08 smithi192 ceph-mon[32956]: 9.f scrub starts 2024-06-27T21:51:08.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:08.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:08.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:08.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:08.819 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.849 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:08.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:08.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:08.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:08.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:09.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:09.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:09.166 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:51:09.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:09.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:09.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:09.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:09.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1537640643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:09.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:09 smithi151 ceph-mon[28965]: pgmap v2379: 47 pgs: 47 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s 2024-06-27T21:51:09.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:09 smithi151 ceph-mon[28965]: 9.f scrub ok 2024-06-27T21:51:09.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:09.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1537640643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:09 smithi190 ceph-mon[32247]: pgmap v2379: 47 pgs: 47 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s 2024-06-27T21:51:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:09 smithi190 ceph-mon[32247]: 9.f scrub ok 2024-06-27T21:51:09.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1537640643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:09.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:09 smithi192 ceph-mon[32956]: pgmap v2379: 47 pgs: 47 active+clean; 3.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s 2024-06-27T21:51:09.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:09 smithi192 ceph-mon[32956]: 9.f scrub ok 2024-06-27T21:51:09.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:09.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:09.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.747 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:09.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:09.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:09.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:09.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:09.977 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:10.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:10.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:10.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:10.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:10.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:10.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:10.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:51:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:51:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1537640643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: osdmap e969: 12 total, 12 up, 12 in 2024-06-27T21:51:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:51:10.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:51:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:51:10.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:10 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:51:10.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1537640643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: osdmap e969: 12 total, 12 up, 12 in 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:51:10.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:10 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:51:10.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1537640643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: osdmap e969: 12 total, 12 up, 12 in 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:51:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:51:10.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:10 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:51:10.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:10.649 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:10.751 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:10.774 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:10.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:10.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:10.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:11.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:11.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:11.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:11.159 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:11.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:11 smithi151 ceph-mon[28965]: pgmap v2381: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:51:11.220 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:51:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-06-27T21:51:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-06-27T21:51:11.220 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-06-27T21:51:11.220 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:51:11.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:11.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:11.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:11.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:11.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:11 smithi190 ceph-mon[32247]: pgmap v2381: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:51:11.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:11.543 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:11 smithi192 ceph-mon[32956]: pgmap v2381: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-06-27T21:51:11.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:11.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:11.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:11.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:11.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:11.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:11.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:11.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-06-27T21:51:12.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:12.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:12.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:12.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:12.308 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:12.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:12.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:12.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:12.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:12.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:12.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:51:12.681 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:12.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.806 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:12.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:12.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:12.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:12.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:13.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:13.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:13.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:13.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:13.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:13.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:13.417 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:13.442 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:13.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:13.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:13.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:13 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:51:13.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:13 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:51:13.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:13 smithi151 ceph-mon[28965]: pgmap v2382: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s 2024-06-27T21:51:13.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:13.693 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:13.699 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:13.745 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:13.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:13 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:51:13.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:13 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:51:13.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:13 smithi190 ceph-mon[32247]: pgmap v2382: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s 2024-06-27T21:51:13.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:13 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:51:13.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:13 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:51:13.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:13 smithi192 ceph-mon[32956]: pgmap v2382: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s 2024-06-27T21:51:13.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:13.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:13.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:13.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:13.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:14.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:14.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:14.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:14.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:14.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:14.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:14.428 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:14.457 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:14.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:14.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:14.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:14.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:14.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:14.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:14.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:14.981 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:14.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:15.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:15.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:15.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:15.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:15.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:15.364 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:15.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:15.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:15.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:15.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:15.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:15 smithi151 ceph-mon[28965]: pgmap v2383: 47 pgs: 47 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-06-27T21:51:15.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:15 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:51:15.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:15 smithi151 ceph-mon[28965]: 9.0 scrub starts 2024-06-27T21:51:15.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:15.759 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:15.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:15 smithi190 ceph-mon[32247]: pgmap v2383: 47 pgs: 47 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-06-27T21:51:15.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:15 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:51:15.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:15 smithi190 ceph-mon[32247]: 9.0 scrub starts 2024-06-27T21:51:15.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:15 smithi192 ceph-mon[32956]: pgmap v2383: 47 pgs: 47 active+clean; 2.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s 2024-06-27T21:51:15.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:15 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:51:15.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:15 smithi192 ceph-mon[32956]: 9.0 scrub starts 2024-06-27T21:51:15.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:15.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:15.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:15.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:16.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:16.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:16.147 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:16.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:16.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:16.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:16.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:16.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:16.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:16.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:16.652 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:16.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:16 smithi151 ceph-mon[28965]: 9.0 scrub ok 2024-06-27T21:51:16.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:16 smithi190 ceph-mon[32247]: 9.0 scrub ok 2024-06-27T21:51:16.778 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:16.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:16.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:16 smithi192 ceph-mon[32956]: 9.0 scrub ok 2024-06-27T21:51:16.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:16.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:16.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:17.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.034 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:17.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:17.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:17.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:17.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:17.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:17.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:17.425 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:17.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:17.557 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:17.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:17.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:17 smithi151 ceph-mon[28965]: pgmap v2384: 47 pgs: 47 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s 2024-06-27T21:51:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:51:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:51:17.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:51:17.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:17 smithi190 ceph-mon[32247]: pgmap v2384: 47 pgs: 47 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s 2024-06-27T21:51:17.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:51:17.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:51:17.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:51:17.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:17 smithi192 ceph-mon[32956]: pgmap v2384: 47 pgs: 47 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s 2024-06-27T21:51:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:51:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:51:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:51:17.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:17.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:17.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:17.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:18.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:18.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:51:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:51:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:51:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:51:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:51:18.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:51:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:51:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:51:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:51:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:51:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:51:18.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:51:18.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:18.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:51:18.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:51:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:18.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:18.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:51:18.214 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:51:18.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:51:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:51:18.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:51:18.274 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:51:18.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.335 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:18.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:18.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:51:18.426 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:51:18.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:18.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:51:18.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:51:18.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:18.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:18.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:18.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:18 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:18.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:18 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:18.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3079065091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:18.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:18 smithi151 ceph-mon[28965]: osdmap e970: 12 total, 12 up, 12 in 2024-06-27T21:51:18.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:18.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:18.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:18 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:18.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:18 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:18.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3079065091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:18.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:18 smithi190 ceph-mon[32247]: osdmap e970: 12 total, 12 up, 12 in 2024-06-27T21:51:18.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:18 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:18.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:18 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:18.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3079065091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:18.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:18 smithi192 ceph-mon[32956]: osdmap e970: 12 total, 12 up, 12 in 2024-06-27T21:51:18.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:18.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:18.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:18.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:18.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:19.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:19.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:19.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:19.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:19.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:19.368 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:19.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:19.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:19.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:19.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:19 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:51:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:19 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:51:19.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:19 smithi151 ceph-mon[28965]: pgmap v2386: 47 pgs: 47 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 910 op/s 2024-06-27T21:51:19.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:19.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:19.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:19 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:51:19.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:19 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:51:19.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:19 smithi190 ceph-mon[32247]: pgmap v2386: 47 pgs: 47 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 910 op/s 2024-06-27T21:51:19.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:19 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:51:19.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:19 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:51:19.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:19 smithi192 ceph-mon[32956]: pgmap v2386: 47 pgs: 47 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 910 op/s 2024-06-27T21:51:19.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:19.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:19.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:19.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:20.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:20.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:20.144 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:20.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:20.271 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:20.373 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:20.399 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:20.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-06-27T21:51:20.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:20.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:20.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:20.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:20.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:20 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:51:20.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:20 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:51:20.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:20.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:20 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:51:20.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:20 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:51:20.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:20.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:20 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:51:20.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:20 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:51:20.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:20.917 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:20.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:21.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:21.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:21.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.167 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:21.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:21.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:21.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:21.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:21.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:21.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:21.558 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:21.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:51:21.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:21.687 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:21 smithi151 ceph-mon[28965]: 9.e scrub starts 2024-06-27T21:51:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:21 smithi151 ceph-mon[28965]: pgmap v2387: 47 pgs: 47 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s 2024-06-27T21:51:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:21 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/408741303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:51:21.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:21 smithi190 ceph-mon[32247]: 9.e scrub starts 2024-06-27T21:51:21.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:21 smithi190 ceph-mon[32247]: pgmap v2387: 47 pgs: 47 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s 2024-06-27T21:51:21.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:21 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/408741303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:51:21.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:21 smithi192 ceph-mon[32956]: 9.e scrub starts 2024-06-27T21:51:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:21 smithi192 ceph-mon[32956]: pgmap v2387: 47 pgs: 47 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s 2024-06-27T21:51:21.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:21 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/408741303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:51:21.810 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:21.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:21.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:21.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:21.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:22.002 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:51:22.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:22.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:22.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:22.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:22.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:22.332 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-06-27T21:51:22.332 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:51:22.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:22.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:22.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:22.583 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:22.605 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:22.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:22 smithi151 ceph-mon[28965]: 9.e scrub ok 2024-06-27T21:51:22.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:22 smithi151 ceph-mon[28965]: 9.d deep-scrub starts 2024-06-27T21:51:22.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:22 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:22.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/408741303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:51:22.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:22 smithi151 ceph-mon[28965]: osdmap e971: 12 total, 12 up, 12 in 2024-06-27T21:51:22.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:22.735 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:22.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:22 smithi190 ceph-mon[32247]: 9.e scrub ok 2024-06-27T21:51:22.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:22 smithi190 ceph-mon[32247]: 9.d deep-scrub starts 2024-06-27T21:51:22.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:22 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:22.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/408741303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:51:22.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:22 smithi190 ceph-mon[32247]: osdmap e971: 12 total, 12 up, 12 in 2024-06-27T21:51:22.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:22 smithi192 ceph-mon[32956]: 9.e scrub ok 2024-06-27T21:51:22.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:22 smithi192 ceph-mon[32956]: 9.d deep-scrub starts 2024-06-27T21:51:22.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:22 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:22.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/408741303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:51:22.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:22 smithi192 ceph-mon[32956]: osdmap e971: 12 total, 12 up, 12 in 2024-06-27T21:51:22.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:22.866 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:22.968 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:22.995 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:22.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:23.096 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:23.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-06-27T21:51:23.118 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:23.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:23.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:23.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:23.364 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:23.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:23.436 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:23.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:23.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:23.616 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:23.641 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:23.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:23.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:23.833 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:23.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:23.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:23.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:23.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:24.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:24.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:24 smithi151 ceph-mon[28965]: pgmap v2389: 47 pgs: 47 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s 2024-06-27T21:51:24.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:24.151 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:24.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:24.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:24 smithi190 ceph-mon[32247]: pgmap v2389: 47 pgs: 47 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s 2024-06-27T21:51:24.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:24.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:24 smithi192 ceph-mon[32956]: pgmap v2389: 47 pgs: 47 active+clean; 2.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 736 op/s 2024-06-27T21:51:24.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:24.409 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:24.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:24.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:24.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:24.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:24.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:24.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:24.896 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:24.919 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:24.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:25.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:25.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:25.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:25.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:25.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:25.298 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:25.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:25.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:25.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:25.470 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:51:25.470 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:51:25.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:25.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:25.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:25.676 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:25.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:25.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:25.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:51:25.905 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:25.928 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:25.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:26.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:26.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:26.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:26 smithi151 ceph-mon[28965]: pgmap v2390: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s 2024-06-27T21:51:26.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:26.162 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:26 smithi190 ceph-mon[32247]: pgmap v2390: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s 2024-06-27T21:51:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:26.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:26.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-06-27T21:51:26.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:26.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:26 smithi192 ceph-mon[32956]: pgmap v2390: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s 2024-06-27T21:51:26.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:26.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:26.443 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:26.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:26.568 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:26.619 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:26.670 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:26.694 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:26.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:26.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:26.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:26.977 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:26.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:27.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:27.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:27.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:27.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:27.343 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:27.367 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:27.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:27.492 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:27.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-06-27T21:51:27.538 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:27.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:27.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:27.727 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:27.752 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:27.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:27.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:27.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:27.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:28.010 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:28.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:28.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:28.238 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:28.259 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:28 smithi151 ceph-mon[28965]: pgmap v2391: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-06-27T21:51:28.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:28 smithi190 ceph-mon[32247]: pgmap v2391: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-06-27T21:51:28.268 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:28.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:28 smithi192 ceph-mon[32956]: pgmap v2391: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-06-27T21:51:28.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:28.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:28.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:28.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:28.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:51:28.538 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:51:28.563 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:51:28.564 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:51:28.600 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:51:28.600 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:51:28.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:28.627 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:51:28.627 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:51:28.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:28.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:51:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:51:28.689 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:51:28.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:51:28.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:51:28.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:51:28.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:51:28.747 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:51:28.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:28.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:51:28.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:51:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.776 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:28.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:28.800 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:51:28.800 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:51:28.830 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:51:28.830 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:51:28.863 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:51:28.864 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:51:28.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:28.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:28.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:29.009 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:29.033 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:29.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:29.156 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:29.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:29.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/661122943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:29.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/661122943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:29.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/661122943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:29.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:29.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:29.421 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:51:29.421 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:51:29.421 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:51:29.421 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:51:29.422 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:51:29.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:29.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:29.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:29.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:29.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:29.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:29.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:29.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:29.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:29.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:29.989 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:30.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:30.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-06-27T21:51:30.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:30.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:30.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:30 smithi151 ceph-mon[28965]: pgmap v2392: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s 2024-06-27T21:51:30.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:30 smithi192 ceph-mon[32956]: pgmap v2392: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s 2024-06-27T21:51:30.303 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:30.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:30.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:30.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:30.484 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:30.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:30 smithi190 ceph-mon[32247]: pgmap v2392: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s 2024-06-27T21:51:30.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:30.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:30.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:30.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:30.842 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:30.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:30.966 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:30.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:31.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:31.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:31.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:31.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:31.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:31.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:31.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:31.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:31.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:31.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:31.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:31.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:31.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:31.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:31.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:31.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:31.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:31.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:32.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:32.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:32.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:32.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:32 smithi151 ceph-mon[28965]: pgmap v2393: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s 2024-06-27T21:51:32.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:32 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:32.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:32 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:32.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:32.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:32 smithi190 ceph-mon[32247]: pgmap v2393: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s 2024-06-27T21:51:32.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:32 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:32.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:32 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:32 smithi192 ceph-mon[32956]: pgmap v2393: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s 2024-06-27T21:51:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:32 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:32.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:32 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:32.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:32.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:32.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:32.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:32.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:32.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:32.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:32.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:32.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:51:32.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:32.864 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:32.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:32.987 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:32.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:33.038 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:51:33.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:33.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:33.186 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:33 smithi190 ceph-mon[32247]: osdmap e972: 12 total, 12 up, 12 in 2024-06-27T21:51:33.186 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3274253744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:59675 8.19 8.fc7e8c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:15.065138+0000", 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 378.118746612, 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83082461900000004, 2024-06-27T21:51:33.188 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 59675 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065138+0000", 2024-06-27T21:51:33.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065138+0000", 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065139+0000", 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939999999999999e-06 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065213+0000", 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3948000000000004e-05 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065228+0000", 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4748e-05 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065252+0000", 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3827000000000001e-05 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065321+0000", 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8992000000000006e-05 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.065335+0000", 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3828e-05 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.895950+0000", 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83061501599999998 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:15.895963+0000", 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2866e-05 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.d 857/838 [PushOp(8:bb5cad6c:::benchmark_data_smithi192_167666_object55735:head, version: 830'3540, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:bb5cad6c:::benchmark_data_smithi192_167666_object55735:head@830'3540, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:05.795373+0000", 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 327.38851155600003, 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30562369099999998, 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795373+0000", 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795373+0000", 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795374+0000", 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8100000000000002e-07 2024-06-27T21:51:33.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:05.795430+0000", 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5999999999999999e-05 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098693+0000", 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.303263055 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098733+0000", 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0574999999999997e-05 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098837+0000", 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103335 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.098851+0000", 2024-06-27T21:51:33.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4251e-05 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.100997+0000", 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021457939999999999 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.16 867/839 [PushOp(8:6f6f41e4:::benchmark_data_smithi192_167666_object25554:head, version: 817'1579, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:6f6f41e4:::benchmark_data_smithi192_167666_object25554:head@817'1579, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:33.198 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:28.857788+0000", 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 304.32609622699999, 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31268284400000002, 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857788+0000", 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857788+0000", 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857789+0000", 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7400000000000003e-07 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:28.857855+0000", 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5889000000000006e-05 2024-06-27T21:51:33.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.168966+0000", 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31111138599999999 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.168994+0000", 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7463000000000001e-05 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.169195+0000", 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020071800000000001 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.169215+0000", 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0236000000000001e-05 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.170471+0000", 2024-06-27T21:51:33.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012564779999999999 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:100065 8.4 8:2399fd11:::benchmark_data_smithi192_167666_object20627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:29.129048+0000", 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 304.05483657799999, 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29965223299999999, 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.203 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 100065 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129048+0000", 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129048+0000", 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129051+0000", 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5940000000000001e-06 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129062+0000", 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1556000000000001e-05 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129101+0000", 2024-06-27T21:51:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8436000000000003e-05 2024-06-27T21:51:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129122+0000", 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1066000000000001e-05 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129245+0000", 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012339 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129373+0000", 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000128217 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,10", 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.129486+0000", 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011226399999999999 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.131155+0000", 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016692510000000001 2024-06-27T21:51:33.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.131180+0000", 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4938000000000001e-05 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.428630+0000", 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29745006400000001 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.209 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.428668+0000", 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7716000000000003e-05 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.428700+0000", 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2740999999999997e-05 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGRecoveryDeleteReply(8.4 e879,873 8:221570d9:::benchmark_data_smithi192_167666_object17847:head,865'4997)", 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 278.4883787, 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.2588125, 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695506+0000", 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4700000000000001e-07 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.695511+0000", 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5020000000000003e-06 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954076+0000", 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25856520100000002 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954083+0000", 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1980000000000004e-06 2024-06-27T21:51:33.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954136+0000", 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2806999999999998e-05 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.954318+0000", 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018224499999999999 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:129217 8.4 8:273ad32a:::benchmark_data_smithi192_167666_object49779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.244361+0000", 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.93952366000002, 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34760285299999999, 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129217 2024-06-27T21:51:33.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244361+0000", 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244361+0000", 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244362+0000", 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0920000000000001e-06 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244367+0000", 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4689999999999999e-06 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244378+0000", 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1639e-05 2024-06-27T21:51:33.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244386+0000", 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3799999999999996e-06 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244433+0000", 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7658999999999999e-05 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244517+0000", 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3786e-05 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 10,11", 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.244588+0000", 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1005000000000002e-05 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246288+0000", 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001699823 2024-06-27T21:51:33.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246855+0000", 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00056720100000000003 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591902+0000", 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34504665400000001 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591935+0000", 2024-06-27T21:51:33.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3105000000000002e-05 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591964+0000", 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.904e-05 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:129217 8.4 e889/881 ondisk, result = 0)", 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.246606+0000", 2024-06-27T21:51:33.221 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.93727833600002, 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34536166299999999, 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246606+0000", 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246606+0000", 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246607+0000", 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1600000000000001e-07 2024-06-27T21:51:33.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.246610+0000", 2024-06-27T21:51:33.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3739999999999998e-06 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591854+0000", 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34524346299999997 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591861+0000", 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.002e-06 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591894+0000", 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2817000000000001e-05 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591898+0000", 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3510000000000004e-06 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.591968+0000", 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.004e-05 2024-06-27T21:51:33.227 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:129225 8.16 e889/876 8:689e3fb0:::benchmark_data_smithi192_167666_object49787:head v 889'5801, mlcod=889'5798)", 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.253857+0000", 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 266.93002721200003, 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.455897161, 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.228 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253857+0000", 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253857+0000", 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253858+0000", 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.51e-07 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.253863+0000", 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9799999999999998e-06 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708079+0000", 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.45421639400000002 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708086+0000", 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6290000000000004e-06 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708130+0000", 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4048000000000002e-05 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.708145+0000", 2024-06-27T21:51:33.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.521e-05 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.709719+0000", 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001573577 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.709755+0000", 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5772000000000002e-05 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:138829 8.11 e895/846 8:8e75fe10:::benchmark_data_smithi192_167666_object59391:head v 895'6216, mlcod=895'6213)", 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:19.559264+0000", 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 253.62462022899999, 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25218020499999999, 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559264+0000", 2024-06-27T21:51:33.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559264+0000", 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559265+0000", 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8800000000000002e-07 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.559271+0000", 2024-06-27T21:51:33.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8329999999999999e-06 2024-06-27T21:51:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.236 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809754+0000", 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.250483067 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809761+0000", 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4489999999999996e-06 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809818+0000", 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7824000000000001e-05 2024-06-27T21:51:33.237 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.809847+0000", 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8424000000000001e-05 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.811416+0000", 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00156879 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.238 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:19.811445+0000", 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9030000000000002e-05 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27262.0:58143 9.a 9:549ccc38:::benchmark_data_smithi192_179902_object58142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:05.178144+0000", 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 148.00574079800001, 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.242020544, 2024-06-27T21:51:33.239 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27262", 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 58143 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178144+0000", 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178144+0000", 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178144+0000", 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4299999999999999e-07 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178166+0000", 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1948000000000001e-05 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178170+0000", 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9179999999999996e-06 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178173+0000", 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8310000000000001e-06 2024-06-27T21:51:33.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.179524+0000", 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001350925 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.179643+0000", 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119103 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.179727+0000", 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4456000000000006e-05 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.180887+0000", 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011596359999999999 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.418908+0000", 2024-06-27T21:51:33.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23802084500000001 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.420101+0000", 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001193251 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.420137+0000", 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5904e-05 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.420164+0000", 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7384000000000001e-05 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27262.0:58144 9.f 9:f83e998a:::benchmark_data_smithi192_179902_object58143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:05.178378+0000", 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 148.005507008, 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.238745708, 2024-06-27T21:51:33.246 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27262", 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 58144 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178378+0000", 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178378+0000", 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178378+0000", 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7899999999999998e-07 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178448+0000", 2024-06-27T21:51:33.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9444999999999997e-05 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178457+0000", 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8479999999999996e-06 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178463+0000", 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5740000000000003e-06 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178513+0000", 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0130000000000003e-05 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178637+0000", 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012381 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,9", 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.178775+0000", 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013819200000000001 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.180482+0000", 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001706342 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.180616+0000", 2024-06-27T21:51:33.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013471100000000001 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.417055+0000", 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23643830099999999 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.417092+0000", 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7101000000000003e-05 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.417123+0000", 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1674999999999997e-05 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:71391 9.d e945/916 9:bf4c3949:::benchmark_data_smithi192_179902_object71390:head v 945'4530, mlcod=945'4528)", 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:16.172019+0000", 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 137.01186599499999, 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14198641000000001, 2024-06-27T21:51:33.253 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.172019+0000", 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.172019+0000", 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.172019+0000", 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8800000000000002e-07 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.172038+0000", 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9142000000000001e-05 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.311795+0000", 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13975652299999999 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.311799+0000", 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5249999999999999e-06 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.311860+0000", 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0055e-05 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.311873+0000", 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3040999999999999e-05 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.313974+0000", 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021016580000000002 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:16.314005+0000", 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0778000000000002e-05 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:101138 9.9 e950/916 9:9dee7e54:::benchmark_data_smithi192_179902_object101137:head v 950'6247, mlcod=950'6244)", 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:47.149871+0000", 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 106.034013988, 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16302217299999999, 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.258 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.149871+0000", 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.149871+0000", 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.149871+0000", 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3300000000000001e-07 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.149925+0000", 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3162999999999999e-05 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.309734+0000", 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.159809797 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.309741+0000", 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1959999999999996e-06 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.309806+0000", 2024-06-27T21:51:33.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4968000000000005e-05 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.309838+0000", 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2224999999999997e-05 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.312857+0000", 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030189610000000001 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:47.312893+0000", 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6029999999999999e-05 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27262.0:126113 9.a 9:5854b4d2:::benchmark_data_smithi192_179902_object16663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:12.543960+0000", 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.639924965999995, 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34553790499999998, 2024-06-27T21:51:33.263 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27262", 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 126113 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.543960+0000", 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.543960+0000", 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.543962+0000", 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.854e-06 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.543968+0000", 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9460000000000001e-06 2024-06-27T21:51:33.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.543985+0000", 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6501e-05 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.543993+0000", 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.259e-06 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.266 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.544045+0000", 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2169000000000001e-05 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.544130+0000", 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4840999999999996e-05 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.544199+0000", 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8564000000000003e-05 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:33.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.547248+0000", 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030496059999999998 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.547479+0000", 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023070600000000001 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889355+0000", 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34187624100000003 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889462+0000", 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106258 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889498+0000", 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5960000000000001e-05 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.269 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27262.0:126113 9.a e955/916 ondisk, result = 0)", 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:12.547464+0000", 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.636420266000002, 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34203887399999999, 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.547464+0000", 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.547464+0000", 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.547465+0000", 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8100000000000003e-07 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.547467+0000", 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5689999999999999e-06 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889158+0000", 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34169026200000002 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889173+0000", 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5656000000000001e-05 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889327+0000", 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153969 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889345+0000", 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7790000000000001e-05 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.889503+0000", 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000158147 2024-06-27T21:51:33.273 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:126123 9.6 e955/916 9:6afd1a94:::benchmark_data_smithi192_179902_object16673:head v 955'7942, mlcod=955'7939)", 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:12.565366+0000", 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.618518561000002, 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.188297616, 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.274 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.565366+0000", 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.565366+0000", 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.565367+0000", 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7999999999999995e-07 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.565371+0000", 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.549e-06 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.749848+0000", 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18447658 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.749855+0000", 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.742e-06 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.749914+0000", 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8958e-05 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.749941+0000", 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7460000000000001e-05 2024-06-27T21:51:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.753631+0000", 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036896229999999999 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.753664+0000", 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3024000000000001e-05 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.278 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:129620 9.d e957/916 9:b1611421:::benchmark_data_smithi192_179902_object20170:head v 957'8154, mlcod=957'8151)", 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:16.776905+0000", 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 76.406979852000006, 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13880276999999999, 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.776905+0000", 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.776905+0000", 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.776906+0000", 2024-06-27T21:51:33.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7100000000000001e-07 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.776911+0000", 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3369999999999999e-06 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.912275+0000", 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13536408799999999 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.912278+0000", 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0929999999999999e-06 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.912321+0000", 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.278e-05 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.282 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.912335+0000", 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3918e-05 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.915662+0000", 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003326917 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:16.915708+0000", 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5865999999999998e-05 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.283 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:143748 9.5 e959/916 9:aadb7ed9:::benchmark_data_smithi192_179902_object34298:head v 959'9008, mlcod=959'9005)", 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:37.013323+0000", 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 56.170561405000001, 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.137990576, 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.013323+0000", 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.013323+0000", 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.013324+0000", 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1e-07 2024-06-27T21:51:33.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.013326+0000", 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4530000000000001e-06 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.146680+0000", 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.133354271 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.146685+0000", 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5770000000000004e-06 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.146730+0000", 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5500000000000001e-05 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.146749+0000", 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8456000000000001e-05 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.151290+0000", 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045407959999999997 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:37.151314+0000", 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4213e-05 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27262.0:168645 9.a 9:5605a825:::benchmark_data_smithi192_179902_object59195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e968)", 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:51:07.558600+0000", 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.625284731000001, 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13305583700000001, 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27262", 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 168645 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.289 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558600+0000", 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558600+0000", 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558601+0000", 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9199999999999999e-07 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558605+0000", 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3109999999999999e-06 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558616+0000", 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.099e-05 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558622+0000", 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9039999999999997e-06 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558660+0000", 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8345e-05 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558756+0000", 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5568000000000004e-05 2024-06-27T21:51:33.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,7", 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.558824+0000", 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7832999999999999e-05 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.562480+0000", 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036566279999999999 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.564232+0000", 2024-06-27T21:51:33.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001751568 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.691583+0000", 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.127350616 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.691622+0000", 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9332999999999998e-05 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.294 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:07.691656+0000", 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3849000000000001e-05 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:186458 9.b e971/916 9:dfea8634:::benchmark_data_smithi192_179902_object77008:head v 971'11624, mlcod=971'11620)", 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:51:31.644966+0000", 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.538919012, 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13483029999999999, 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:33.295 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.644966+0000", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.644966+0000", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.644966+0000", 2024-06-27T21:51:33.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5200000000000002e-07 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.644971+0000", 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.51e-06 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.774898+0000", 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.12992772599999999 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:33.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.774903+0000", 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.882e-06 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.774945+0000", 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1746999999999999e-05 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.774960+0000", 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4671999999999999e-05 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.779748+0000", 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047880070000000004 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:31.779796+0000", 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8303999999999997e-05 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:33.299 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:51:33.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:33 smithi192 ceph-mon[32956]: osdmap e972: 12 total, 12 up, 12 in 2024-06-27T21:51:33.300 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3274253744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:33.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:33.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:33.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:33 smithi151 ceph-mon[28965]: osdmap e972: 12 total, 12 up, 12 in 2024-06-27T21:51:33.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3274253744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:33.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:33.499 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:33.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:33.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:33.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:33.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:33.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:33.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:33.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:33.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:33.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:33.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:34.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:34.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:34.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:34.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:34 smithi190 ceph-mon[32247]: pgmap v2395: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s 2024-06-27T21:51:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:34 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3274253744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:34.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:34 smithi190 ceph-mon[32247]: osdmap e973: 12 total, 12 up, 12 in 2024-06-27T21:51:34.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:34 smithi192 ceph-mon[32956]: pgmap v2395: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s 2024-06-27T21:51:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:34 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3274253744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:34.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:34 smithi192 ceph-mon[32956]: osdmap e973: 12 total, 12 up, 12 in 2024-06-27T21:51:34.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:34.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:34.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-06-27T21:51:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:34 smithi151 ceph-mon[28965]: pgmap v2395: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 717 op/s 2024-06-27T21:51:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:34 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3274253744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:34.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:34 smithi151 ceph-mon[28965]: osdmap e973: 12 total, 12 up, 12 in 2024-06-27T21:51:34.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:34.518 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:34.620 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:34.650 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:34.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:34.779 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:34.817 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:34.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:34.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:34.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:35.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:35.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-06-27T21:51:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.039 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:35.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:35.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:35.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:35.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:35.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:35.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:35.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:35.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:35.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:35.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:35.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:35.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:35.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:35.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:35.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:35.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:36.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:36.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:36.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:51:36.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:36.220 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:36.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:36 smithi190 ceph-mon[32247]: pgmap v2397: 47 pgs: 47 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s 2024-06-27T21:51:36.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:36 smithi192 ceph-mon[32956]: pgmap v2397: 47 pgs: 47 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s 2024-06-27T21:51:36.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:36.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:36 smithi151 ceph-mon[28965]: pgmap v2397: 47 pgs: 47 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s 2024-06-27T21:51:36.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:36.406 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:36.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:36.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:36.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:36.611 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:36.709 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:51:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:51:36.709 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:51:36.710 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:51:36.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.740 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:36.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:36.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:36.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:36.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:36.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:37.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:37.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:37.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:37.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:51:37.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:51:37.208 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:51:37.208 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:51:37.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:51:37.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:51:37.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:37.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:51:37.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:51:37.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:37.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:51:37.289 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:51:37.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:51:37.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:51:37.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:51:37.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:51:37.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:37.379 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:51:37.379 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:51:37.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:37.409 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:51:37.410 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:51:37.439 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:51:37.439 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:51:37.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-06-27T21:51:37.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:51:37.467 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:51:37.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:37.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:37.574 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:37.624 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:51:37.624 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:51:37.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:37.697 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:37.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:37.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:37.864 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:37.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:37.950 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:37.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:38.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:38.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:38.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:38.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:38.246 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:38 smithi190 ceph-mon[32247]: pgmap v2398: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s 2024-06-27T21:51:38.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/414548656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:38.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:38 smithi192 ceph-mon[32956]: pgmap v2398: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s 2024-06-27T21:51:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/414548656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:38.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:38.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:38.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:38 smithi151 ceph-mon[28965]: pgmap v2398: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s 2024-06-27T21:51:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/414548656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:38.455 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:38.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:38.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:38.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:38.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:38.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:38.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:38.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:38.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:38.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:39.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:39.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:39.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:39.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:39.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:39.349 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:39.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:39.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:39.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:39.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:39.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:39.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:39.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:39.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:39.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:39.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:39.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:40.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:40.112 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:40.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:40.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:40.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:40.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:40 smithi190 ceph-mon[32247]: pgmap v2399: 47 pgs: 47 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s 2024-06-27T21:51:40.325 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:51:40.325 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:51:40.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:40.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:40 smithi151 ceph-mon[28965]: pgmap v2399: 47 pgs: 47 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s 2024-06-27T21:51:40.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:40.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:40.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:40.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:40.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:40 smithi192 ceph-mon[32956]: pgmap v2399: 47 pgs: 47 active+clean; 1.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s 2024-06-27T21:51:40.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:40.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:40.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:40.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:40.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:40.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:40.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:40.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:41.019 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.020 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:41.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-06-27T21:51:41.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:41.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:41.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:41.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:41.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:41.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:41.447 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:41.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:41.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:41.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:41.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:41.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:41.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:41.924 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:41.958 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:41.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:42.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:42.062 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:51:42.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:42.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:42.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:42.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:42 smithi190 ceph-mon[32247]: pgmap v2400: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-06-27T21:51:42.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1674181440' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:51:42.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:42 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:51:42.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:42.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:42 smithi151 ceph-mon[28965]: pgmap v2400: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-06-27T21:51:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1674181440' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:51:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:42 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:51:42.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:42.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:42.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:42 smithi192 ceph-mon[32956]: pgmap v2400: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 856 op/s 2024-06-27T21:51:42.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1674181440' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:51:42.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:42 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:51:42.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:42.618 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:42.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:42.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:42.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:42.871 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:42.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:42.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:43.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:43.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:43.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:43.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:43.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:43.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:43 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:43.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:43 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:51:43.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:43 smithi190 ceph-mon[32247]: osdmap e974: 12 total, 12 up, 12 in 2024-06-27T21:51:43.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:43.381 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:43.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:43 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:43.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:43 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:51:43.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:43 smithi151 ceph-mon[28965]: osdmap e974: 12 total, 12 up, 12 in 2024-06-27T21:51:43.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:43.490 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:51:43.491 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:51:43.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:43 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:51:43.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:43 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:51:43.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:43 smithi192 ceph-mon[32956]: osdmap e974: 12 total, 12 up, 12 in 2024-06-27T21:51:43.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:43.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:43.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:43.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:43.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:43.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:44.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:44.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:44.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:51:44.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:44.153 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:44.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:44 smithi190 ceph-mon[32247]: 9.a scrub starts 2024-06-27T21:51:44.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:44 smithi190 ceph-mon[32247]: pgmap v2402: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 810 op/s 2024-06-27T21:51:44.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:44 smithi190 ceph-mon[32247]: 9.a scrub ok 2024-06-27T21:51:44.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-06-27T21:51:44.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:44.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:44.390 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:44.417 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:44.433 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:44.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:44 smithi151 ceph-mon[28965]: 9.a scrub starts 2024-06-27T21:51:44.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:44 smithi151 ceph-mon[28965]: pgmap v2402: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 810 op/s 2024-06-27T21:51:44.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:44 smithi151 ceph-mon[28965]: 9.a scrub ok 2024-06-27T21:51:44.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:44 smithi192 ceph-mon[32956]: 9.a scrub starts 2024-06-27T21:51:44.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:44 smithi192 ceph-mon[32956]: pgmap v2402: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 810 op/s 2024-06-27T21:51:44.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:44 smithi192 ceph-mon[32956]: 9.a scrub ok 2024-06-27T21:51:44.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:44.580 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:44.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:44.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:44.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:51:44.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:44.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:44.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:44.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:44.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:45.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:45.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:45.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: 9.b scrub starts 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: 9.b scrub ok 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: osdmap e975: 12 total, 12 up, 12 in 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: 9.9 scrub starts 2024-06-27T21:51:45.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:45 smithi151 ceph-mon[28965]: 9.9 scrub ok 2024-06-27T21:51:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:45.259 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: 9.b scrub starts 2024-06-27T21:51:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: 9.b scrub ok 2024-06-27T21:51:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: osdmap e975: 12 total, 12 up, 12 in 2024-06-27T21:51:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: 9.9 scrub starts 2024-06-27T21:51:45.260 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:45 smithi190 ceph-mon[32247]: 9.9 scrub ok 2024-06-27T21:51:45.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:45.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:45.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:45.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: 9.b scrub starts 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: 9.b scrub ok 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: osdmap e975: 12 total, 12 up, 12 in 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: 9.9 scrub starts 2024-06-27T21:51:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:45 smithi192 ceph-mon[32956]: 9.9 scrub ok 2024-06-27T21:51:45.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:45.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:45.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:45.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:45.833 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:45.860 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:45.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:45.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:45.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:46.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:46.119 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:46.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:46 smithi190 ceph-mon[32247]: pgmap v2404: 47 pgs: 47 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s 2024-06-27T21:51:46.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:46 smithi190 ceph-mon[32247]: 9.d scrub starts 2024-06-27T21:51:46.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:46 smithi190 ceph-mon[32247]: 9.d scrub ok 2024-06-27T21:51:46.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:46 smithi151 ceph-mon[28965]: pgmap v2404: 47 pgs: 47 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s 2024-06-27T21:51:46.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:46 smithi151 ceph-mon[28965]: 9.d scrub starts 2024-06-27T21:51:46.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:46 smithi151 ceph-mon[28965]: 9.d scrub ok 2024-06-27T21:51:46.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:46.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:46.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:46.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:46.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:51:46.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:51:46.456 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:46.457 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:51:46.457 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:51:46.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:46.491 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:51:46.491 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:51:46.505 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:46.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:51:46.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:51:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:46 smithi192 ceph-mon[32956]: pgmap v2404: 47 pgs: 47 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 850 op/s 2024-06-27T21:51:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:46 smithi192 ceph-mon[32956]: 9.d scrub starts 2024-06-27T21:51:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:46 smithi192 ceph-mon[32956]: 9.d scrub ok 2024-06-27T21:51:46.545 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:51:46.546 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:51:46.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:51:46.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:51:46.578 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-06-27T21:51:46.578 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:51:46.606 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:51:46.606 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:51:46.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:46.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:51:46.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:51:46.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:46.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:51:46.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:51:46.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:51:46.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:51:46.721 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:51:46.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:51:46.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:46.753 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:51:46.754 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:51:46.770 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:46.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:46.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:46.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:47.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:47.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:47.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:47.157 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1450553418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:47.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:47 smithi151 ceph-mon[28965]: 9.3 scrub starts 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1450553418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:47 smithi190 ceph-mon[32247]: 9.3 scrub starts 2024-06-27T21:51:47.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-06-27T21:51:47.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:47.283 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:47.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:47.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:47.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:47.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1450553418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:47.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:47 smithi192 ceph-mon[32956]: 9.3 scrub starts 2024-06-27T21:51:47.597 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:47.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:47.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:47.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:47.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:47.895 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 1, 7, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:51:47.895 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:51:47.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:51:47.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:47.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:47.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:48.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:48.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:48.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:48.187 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:48.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:48 smithi151 ceph-mon[28965]: pgmap v2405: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-27T21:51:48.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:48 smithi151 ceph-mon[28965]: 9.3 scrub ok 2024-06-27T21:51:48.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:48 smithi151 ceph-mon[28965]: 1.1a scrub starts 2024-06-27T21:51:48.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:48 smithi151 ceph-mon[28965]: 1.1a scrub ok 2024-06-27T21:51:48.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:48 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:51:48.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:48 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:51:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:48 smithi190 ceph-mon[32247]: pgmap v2405: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-27T21:51:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:48 smithi190 ceph-mon[32247]: 9.3 scrub ok 2024-06-27T21:51:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:48 smithi190 ceph-mon[32247]: 1.1a scrub starts 2024-06-27T21:51:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:48 smithi190 ceph-mon[32247]: 1.1a scrub ok 2024-06-27T21:51:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:48 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:51:48.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:48 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:51:48.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:48.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:48.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:48.450 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:48 smithi192 ceph-mon[32956]: pgmap v2405: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-06-27T21:51:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:48 smithi192 ceph-mon[32956]: 9.3 scrub ok 2024-06-27T21:51:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:48 smithi192 ceph-mon[32956]: 1.1a scrub starts 2024-06-27T21:51:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:48 smithi192 ceph-mon[32956]: 1.1a scrub ok 2024-06-27T21:51:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:48 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:51:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:48 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:51:48.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:48.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:48.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:48.708 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:48.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:48.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:48.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:48.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:48.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:49.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:49.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:49.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 9.7 scrub starts 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:51:49.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:49 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:51:49.222 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:49.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:49.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:49.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:51:49.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:51:49.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:51:49.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:51:49.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 9.7 scrub starts 2024-06-27T21:51:49.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:51:49.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:49 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:51:49.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:49.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 9.7 scrub starts 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:51:49.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:49 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:51:49.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:49.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:49.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:49.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:49.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:49.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:49.948 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:51:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:51:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:51:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:51:49.948 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:51:49.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:49.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:49.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:50.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:50.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:50.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:50 smithi151 ceph-mon[28965]: pgmap v2406: 47 pgs: 47 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s 2024-06-27T21:51:50.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:50 smithi151 ceph-mon[28965]: 9.4 scrub starts 2024-06-27T21:51:50.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:50 smithi151 ceph-mon[28965]: 9.4 scrub ok 2024-06-27T21:51:50.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:50 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:51:50.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:50 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:51:50.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:50 smithi151 ceph-mon[28965]: 9.7 scrub ok 2024-06-27T21:51:50.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:50.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:50.301 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:50.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:50.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:50.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:50 smithi190 ceph-mon[32247]: pgmap v2406: 47 pgs: 47 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s 2024-06-27T21:51:50.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:50 smithi190 ceph-mon[32247]: 9.4 scrub starts 2024-06-27T21:51:50.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:50 smithi190 ceph-mon[32247]: 9.4 scrub ok 2024-06-27T21:51:50.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:50 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:51:50.456 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:50 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:51:50.456 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:50 smithi190 ceph-mon[32247]: 9.7 scrub ok 2024-06-27T21:51:50.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:51:50.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:50.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:50 smithi192 ceph-mon[32956]: pgmap v2406: 47 pgs: 47 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s 2024-06-27T21:51:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:50 smithi192 ceph-mon[32956]: 9.4 scrub starts 2024-06-27T21:51:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:50 smithi192 ceph-mon[32956]: 9.4 scrub ok 2024-06-27T21:51:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:50 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:51:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:50 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:51:50.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:50 smithi192 ceph-mon[32956]: 9.7 scrub ok 2024-06-27T21:51:50.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:50.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:50.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-06-27T21:51:50.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:50.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:50.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:50.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:50.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:51.044 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.072 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:51.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:51.107 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:51:51.129 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:51:51.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:51.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:51.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:51.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:51 smithi190 ceph-mon[32247]: 9.5 scrub starts 2024-06-27T21:51:51.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2388936413' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:51.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:51 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:51.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:51 smithi151 ceph-mon[28965]: 9.5 scrub starts 2024-06-27T21:51:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2388936413' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:51 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:51.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:51.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:51 smithi192 ceph-mon[32956]: 9.5 scrub starts 2024-06-27T21:51:51.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2388936413' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:51.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:51 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:51:51.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:51.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:51.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:51.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:51.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:51.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:51.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:51.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:51.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:52.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:52.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:52.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:52.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:52.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:52.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: pgmap v2407: 47 pgs: 47 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s 2024-06-27T21:51:52.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: 9.5 scrub ok 2024-06-27T21:51:52.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:51:52.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:51:52.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:52.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: osdmap e976: 12 total, 12 up, 12 in 2024-06-27T21:51:52.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:51:52.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:51:52.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:51:52.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:52 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:51:52.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: pgmap v2407: 47 pgs: 47 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: 9.5 scrub ok 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: osdmap e976: 12 total, 12 up, 12 in 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:51:52.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:51:52.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:52 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:51:52.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:52.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: pgmap v2407: 47 pgs: 47 active+clean; 1.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 954 op/s 2024-06-27T21:51:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: 9.5 scrub ok 2024-06-27T21:51:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:51:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:51:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:51:52.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: osdmap e976: 12 total, 12 up, 12 in 2024-06-27T21:51:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:51:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:51:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:51:52.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:52 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:51:52.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:52.621 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:52.723 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:52.750 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:52.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:52.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:52.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.d 836/821 [PushOp(8:b1be6d51:::benchmark_data_smithi192_167666_object19431:head, version: 812'1214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:b1be6d51:::benchmark_data_smithi192_167666_object19431:head@812'1214, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:16.435593+0000", 2024-06-27T21:51:52.954 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 396.515195517, 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35761547199999999, 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435593+0000", 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435593+0000", 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435594+0000", 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9500000000000002e-07 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.435617+0000", 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2955999999999998e-05 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.791922+0000", 2024-06-27T21:51:52.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35630468100000001 2024-06-27T21:51:52.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.791935+0000", 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3341e-05 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.792038+0000", 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010332600000000001 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.792054+0000", 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.571e-05 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:16.793209+0000", 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011548629999999999 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:80108 8.15 e853/844 8:acb44bb1:::benchmark_data_smithi192_167666_object670:head v 853'4211, mlcod=853'4208)", 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:51.208345+0000", 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 361.74244408599998, 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24365014600000001, 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208345+0000", 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208345+0000", 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208346+0000", 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2024-06-27T21:51:52.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.208355+0000", 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4450000000000002e-06 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450867+0000", 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24251208199999999 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450874+0000", 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6030000000000001e-06 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450916+0000", 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2466999999999998e-05 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.450931+0000", 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4611e-05 2024-06-27T21:51:52.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.451965+0000", 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001034594 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:51.451995+0000", 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9484000000000001e-05 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.964 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:81055 8.15 e855/844 8:ab25a5cb:::benchmark_data_smithi192_167666_object1617:head v 855'4241, mlcod=855'4238)", 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:53.977158+0000", 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 358.97363084800003, 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.369013596, 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977158+0000", 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977158+0000", 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977159+0000", 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.173e-06 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:53.977166+0000", 2024-06-27T21:51:52.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0979999999999996e-06 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344918+0000", 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36775122500000001 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344922+0000", 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.622e-06 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344967+0000", 2024-06-27T21:51:52.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4387e-05 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.344982+0000", 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5218000000000001e-05 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.346142+0000", 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011605179999999999 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:54.346172+0000", 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9354999999999999e-05 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:81708 8.10 8:0a177ce8:::benchmark_data_smithi192_167666_object2270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 357.14354933300001, 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26758338500000001, 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:52.969 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 81708 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807240+0000", 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1800000000000005e-07 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807244+0000", 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.771e-06 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807253+0000", 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1749999999999994e-06 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807258+0000", 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1059999999999999e-06 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807288+0000", 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9498000000000001e-05 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807366+0000", 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8258999999999997e-05 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:51:52.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.807434+0000", 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7651999999999995e-05 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808283+0000", 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00084945300000000001 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808559+0000", 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027518600000000002 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074743+0000", 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.266184639 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074777+0000", 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3547000000000003e-05 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074823+0000", 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6281e-05 2024-06-27T21:51:52.976 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:81708 8.10 e855/854 ondisk, result = 0)", 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 357.14245821700001, 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26651063000000003, 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808331+0000", 2024-06-27T21:51:52.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7700000000000004e-07 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.808335+0000", 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4189999999999998e-06 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074678+0000", 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26634306400000002 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074684+0000", 2024-06-27T21:51:52.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4980000000000003e-06 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074735+0000", 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0504000000000003e-05 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074739+0000", 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6859999999999997e-06 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074841+0000", 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101882 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:81708 8.10 8:0a177ce8:::benchmark_data_smithi192_167666_object2270:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e855)", 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 357.13753683200002, 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26153279400000001, 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for ondisk", 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:52.981 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 81708 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813252+0000", 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0899999999999997e-07 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813254+0000", 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.539e-06 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813259+0000", 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4829999999999999e-06 2024-06-27T21:51:52.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813263+0000", 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1379999999999999e-06 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:52.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813357+0000", 2024-06-27T21:51:52.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4815000000000006e-05 2024-06-27T21:51:52.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for ondisk", 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:55.813391+0000", 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4125000000000003e-05 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:56.074785+0000", 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26139338499999998 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.12 856/836 [PushOp(8:4b894608:::benchmark_data_smithi192_167666_object27253:head, version: 818'1716, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4b894608:::benchmark_data_smithi192_167666_object27253:head@818'1716, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 355.48070424500003, 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46211215900000002, 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470085+0000", 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8100000000000002e-07 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.470131+0000", 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5881999999999999e-05 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931108+0000", 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46097701899999999 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931116+0000", 2024-06-27T21:51:52.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6079999999999998e-06 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931191+0000", 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5159000000000003e-05 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.931205+0000", 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.384e-05 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.932197+0000", 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00099186999999999995 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPull(8.16 e857/839 cost 1049576)", 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:58.249383+0000", 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 354.70140554, 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.737237174, 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.991 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249383+0000", 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249383+0000", 2024-06-27T21:51:52.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249384+0000", 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-07 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.249393+0000", 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6130000000000004e-06 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986132+0000", 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73673969100000003 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986138+0000", 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.592e-06 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986185+0000", 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6563999999999997e-05 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.986621+0000", 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043603799999999997 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.995 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.17 857/836 [PushOp(8:ec39b37b:::benchmark_data_smithi192_167666_object21643:head, version: 814'1359, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ec39b37b:::benchmark_data_smithi192_167666_object21643:head@814'1359, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:59.975389+0000", 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 352.975399805, 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30655363200000002, 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975389+0000", 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975389+0000", 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975391+0000", 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.477e-06 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:59.975450+0000", 2024-06-27T21:51:52.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9800999999999999e-05 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280774+0000", 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30532361099999999 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280792+0000", 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8151e-05 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:52.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280899+0000", 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106951 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.280912+0000", 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3278000000000001e-05 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.281943+0000", 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010303630000000001 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:52.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.17 857/836 [PushOp(8:e964d975:::benchmark_data_smithi192_167666_object23698:head, version: 817'1491, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:e964d975:::benchmark_data_smithi192_167666_object23698:head@817'1491, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:01.507697+0000", 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 351.44309205299999, 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38872538699999998, 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507697+0000", 2024-06-27T21:51:53.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507697+0000", 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507698+0000", 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.446e-06 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.507760+0000", 2024-06-27T21:51:53.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2121999999999999e-05 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895295+0000", 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38753470499999998 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895310+0000", 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5237e-05 2024-06-27T21:51:53.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895392+0000", 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1422999999999996e-05 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.895400+0000", 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4780000000000008e-06 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.896422+0000", 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001021976 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.17 857/836 [PushOp(8:ed779c3f:::benchmark_data_smithi192_167666_object27105:head, version: 818'1718, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ed779c3f:::benchmark_data_smithi192_167666_object27105:head@818'1718, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:03.393984+0000", 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 349.55680499499999, 2024-06-27T21:51:53.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52261519899999997, 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.393984+0000", 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.393984+0000", 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.393985+0000", 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2200000000000003e-07 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.394077+0000", 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2316000000000002e-05 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915130+0000", 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52105278899999996 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915146+0000", 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6432000000000001e-05 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915249+0000", 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010308500000000001 2024-06-27T21:51:53.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.915267+0000", 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8003e-05 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:03.916599+0000", 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013318519999999999 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:83642 8.d e858/838 8:bd2afb08:::benchmark_data_smithi192_167666_object4204:head v 858'5247, mlcod=831'3586)", 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:06.532557+0000", 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 346.41823193900001, 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50356170499999997, 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532557+0000", 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532557+0000", 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532558+0000", 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.78e-07 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:06.532569+0000", 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1034e-05 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034199+0000", 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50163044199999995 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034206+0000", 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2110000000000003e-06 2024-06-27T21:51:53.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034255+0000", 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8973e-05 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.034275+0000", 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0055999999999999e-05 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.036072+0000", 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001797416 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:07.036119+0000", 2024-06-27T21:51:53.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6694999999999999e-05 2024-06-27T21:51:53.014 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.014 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.014 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:94715 8.10 8:0b89cc6a:::benchmark_data_smithi192_167666_object15277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:23.482765+0000", 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 329.46802395700001, 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27123174100000003, 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 94715 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482765+0000", 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482765+0000", 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482766+0000", 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1790000000000001e-06 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482774+0000", 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5020000000000003e-06 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482787+0000", 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3838e-05 2024-06-27T21:51:53.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482795+0000", 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1489999999999997e-06 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482860+0000", 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5247999999999996e-05 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.482988+0000", 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012832699999999999 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 4,10", 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.483102+0000", 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011404900000000001 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:53.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.484543+0000", 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00144103 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.484860+0000", 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031666700000000002 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753930+0000", 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26906978100000001 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753963+0000", 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3531000000000001e-05 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753997+0000", 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3439999999999998e-05 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.021 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:100117 8.17 8:eacbdbc0:::benchmark_data_smithi192_167666_object20679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e868)", 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:29.525411+0000", 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 323.42537775699998, 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33249853000000001, 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.022 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 100117 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525411+0000", 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525411+0000", 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525412+0000", 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2600000000000002e-07 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525416+0000", 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8689999999999997e-06 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525427+0000", 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1401e-05 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525441+0000", 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4335000000000001e-05 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525592+0000", 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015089500000000001 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525698+0000", 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010518699999999999 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,8", 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.525802+0000", 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104598 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.528214+0000", 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024122800000000002 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.529086+0000", 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087144999999999996 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857818+0000", 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.328732207 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857880+0000", 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2353000000000004e-05 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857910+0000", 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9229000000000001e-05 2024-06-27T21:51:53.028 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:100117 8.17 e868/867 ondisk, result = 0)", 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:29.574463+0000", 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 323.37632545100001, 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.283453173, 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.029 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574463+0000", 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574463+0000", 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574464+0000", 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.06e-07 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.574467+0000", 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3969999999999998e-06 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857759+0000", 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.283291505 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857764+0000", 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5310000000000003e-06 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857802+0000", 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8501999999999998e-05 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857811+0000", 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1090000000000004e-06 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:29.857917+0000", 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105523 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:103778 8.12 8:4addb60a:::benchmark_data_smithi192_167666_object24340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e872)", 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:33.717225+0000", 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 319.23356430299998, 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.261469805, 2024-06-27T21:51:53.034 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 103778 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717225+0000", 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717225+0000", 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717226+0000", 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2109999999999999e-06 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717230+0000", 2024-06-27T21:51:53.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000003e-06 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717242+0000", 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2055999999999999e-05 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717248+0000", 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6829999999999996e-06 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717306+0000", 2024-06-27T21:51:53.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7979000000000002e-05 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717438+0000", 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013185200000000001 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 6,10", 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.717562+0000", 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123843 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719019+0000", 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001457161 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719447+0000", 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00042779800000000001 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978505+0000", 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25905842699999998 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978593+0000", 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7960999999999994e-05 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978694+0000", 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010153399999999999 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27031.0:103778 8.12 e872/864 ondisk, result = 0)", 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:33.719231+0000", 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 319.231557775, 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25947442999999998, 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719231+0000", 2024-06-27T21:51:53.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719231+0000", 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719232+0000", 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2900000000000002e-07 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.719236+0000", 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5130000000000001e-06 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978373+0000", 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25913739299999999 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978386+0000", 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2619999999999999e-05 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978475+0000", 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9701000000000001e-05 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978490+0000", 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4399000000000001e-05 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.978706+0000", 2024-06-27T21:51:53.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000215875 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27262.0:58158 9.d 9:b9c4c4ac:::benchmark_data_smithi192_179902_object58157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:05.192894+0000", 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 167.757895042, 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24555985999999999, 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.046 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27262", 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 58158 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.192894+0000", 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.192894+0000", 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.192895+0000", 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.85e-07 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.192951+0000", 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6527000000000003e-05 2024-06-27T21:51:53.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.192963+0000", 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.163e-05 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.192968+0000", 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3759999999999999e-06 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.193015+0000", 2024-06-27T21:51:53.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6993000000000002e-05 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.193103+0000", 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7887999999999996e-05 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,5", 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.193188+0000", 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4997e-05 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.194966+0000", 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001777647 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.195279+0000", 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031349399999999999 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.438370+0000", 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.243090537 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.438406+0000", 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5769000000000003e-05 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:05.438454+0000", 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8217000000000003e-05 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27262.0:126121 9.e 9:78a4ece9:::benchmark_data_smithi192_179902_object16671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e955)", 2024-06-27T21:51:53.052 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:12.561562+0000", 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 100.389226636, 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37173336299999998, 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27262", 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 126121 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561562+0000", 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561562+0000", 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561563+0000", 2024-06-27T21:51:53.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.02e-07 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561567+0000", 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8489999999999999e-06 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561578+0000", 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0959e-05 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561585+0000", 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.391e-06 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561645+0000", 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9438000000000002e-05 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561731+0000", 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5711000000000007e-05 2024-06-27T21:51:53.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,10", 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.561836+0000", 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105701 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.564890+0000", 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030538789999999998 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.565099+0000", 2024-06-27T21:51:53.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000209271 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.933202+0000", 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.368102867 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.933244+0000", 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1811999999999999e-05 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.933296+0000", 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1582999999999997e-05 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.27262.0:126121 9.e e955/916 ondisk, result = 0)", 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:50:12.568237+0000", 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 100.382552007, 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36506554499999999, 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:51:53.059 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.568237+0000", 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.568237+0000", 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.568238+0000", 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2500000000000005e-07 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.568242+0000", 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3680000000000004e-06 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.930674+0000", 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36243160099999999 2024-06-27T21:51:53.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.930678+0000", 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8250000000000004e-06 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.933177+0000", 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024987780000000001 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:51:53.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.933192+0000", 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4422e-05 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:50:12.933302+0000", 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110926 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:51:53.063 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:51:53.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:53.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:53.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:53.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:53.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:53 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:53.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:53 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:53.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:53 smithi190 ceph-mon[32247]: osdmap e977: 12 total, 12 up, 12 in 2024-06-27T21:51:53.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:53 smithi190 ceph-mon[32247]: pgmap v2410: 47 pgs: 47 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s 2024-06-27T21:51:53.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:53 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:53.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:53 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:53.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:53 smithi151 ceph-mon[28965]: osdmap e977: 12 total, 12 up, 12 in 2024-06-27T21:51:53.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:53 smithi151 ceph-mon[28965]: pgmap v2410: 47 pgs: 47 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s 2024-06-27T21:51:53.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:51:53.302 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:51:53.329 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:53.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:53.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:53.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:53 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:53.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:53 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:53.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:53 smithi192 ceph-mon[32956]: osdmap e977: 12 total, 12 up, 12 in 2024-06-27T21:51:53.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:53 smithi192 ceph-mon[32956]: pgmap v2410: 47 pgs: 47 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s 2024-06-27T21:51:53.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:53.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:53.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:53.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:53.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:53.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:53.878 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-06-27T21:51:53.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:51:53.975 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:53.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:51:54.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:54.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:54.126 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2410,"stamp":"2024-06-27T21:51:52.387703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1124008504,"num_objects":17145,"num_object_clones":0,"num_object_copies":51435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201753,"num_write_kb":7004609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163792,"ondisk_log_size":163792,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13026964,"kb_used_data":3454508,"kb_used_omap":110,"kb_used_meta":9572049,"kb_avail":1111914860,"statfs":{"total":1151940427776,"available":1138600816640,"internally_reserved":0,"allocated":3537416192,"data_stored":3533787838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9801779159},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-453378048,"num_objects":-6918,"num_object_clones":0,"num_object_copies":-20754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001643"},"pg_stats":[{"pgid":"9.3","version":"975'12494","reported_seq":12710,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:50.536038+0000","last_change":"2024-06-27T21:51:47.047679+0000","last_active":"2024-06-27T21:51:50.536038+0000","last_peered":"2024-06-27T21:51:50.536038+0000","last_clean":"2024-06-27T21:51:50.536038+0000","last_became_active":"2024-06-27T21:48:11.986220+0000","last_became_peered":"2024-06-27T21:48:11.986220+0000","last_unstale":"2024-06-27T21:51:50.536038+0000","last_undegraded":"2024-06-27T21:51:50.536038+0000","last_fullsized":"2024-06-27T21:51:50.536038+0000","mapping_epoch":916,"log_start":"935'2400","ondisk_log_start":"935'2400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12308","last_scrub_stamp":"2024-06-27T21:51:47.047437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.047437+0000","objects_scrubbed":1271,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:57.425849+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70254592,"num_objects":1072,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12494,"num_write_kb":434112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":239,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.072637+0000","last_change":"2024-06-27T21:50:27.033854+0000","last_active":"2024-06-27T21:51:44.072637+0000","last_peered":"2024-06-27T21:51:44.072637+0000","last_clean":"2024-06-27T21:51:44.072637+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:51:44.072637+0000","last_undegraded":"2024-06-27T21:51:44.072637+0000","last_fullsized":"2024-06-27T21:51:44.072637+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_clean_scrub_stamp":"2024-06-27T21:50:27.033721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:51:56.763040+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"975'12704","reported_seq":12901,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:48.240877+0000","last_change":"2024-06-27T21:50:08.793720+0000","last_active":"2024-06-27T21:51:48.240877+0000","last_peered":"2024-06-27T21:51:48.240877+0000","last_clean":"2024-06-27T21:51:48.240877+0000","last_became_active":"2024-06-27T21:48:12.309522+0000","last_became_peered":"2024-06-27T21:48:12.309522+0000","last_unstale":"2024-06-27T21:51:48.240877+0000","last_undegraded":"2024-06-27T21:51:48.240877+0000","last_fullsized":"2024-06-27T21:51:48.240877+0000","mapping_epoch":916,"log_start":"936'2700","ondisk_log_start":"936'2700","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"955'7754","last_scrub_stamp":"2024-06-27T21:50:08.793623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:50:08.793623+0000","objects_scrubbed":6116,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73924608,"num_objects":1128,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":217,"reported_epoch":977,"state":"active+clean","last_fresh":"2024-06-27T21:51:52.112935+0000","last_change":"2024-06-27T21:51:45.285536+0000","last_active":"2024-06-27T21:51:52.112935+0000","last_peered":"2024-06-27T21:51:52.112935+0000","last_clean":"2024-06-27T21:51:52.112935+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:51:52.112935+0000","last_undegraded":"2024-06-27T21:51:52.112935+0000","last_fullsized":"2024-06-27T21:51:52.112935+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:45.285411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.285411+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:02.452953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"975'12655","reported_seq":12850,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:49.954818+0000","last_change":"2024-06-27T21:50:15.925927+0000","last_active":"2024-06-27T21:51:49.954818+0000","last_peered":"2024-06-27T21:51:49.954818+0000","last_clean":"2024-06-27T21:51:49.954818+0000","last_became_active":"2024-06-27T21:48:11.990332+0000","last_became_peered":"2024-06-27T21:48:11.990332+0000","last_unstale":"2024-06-27T21:51:49.954818+0000","last_undegraded":"2024-06-27T21:51:49.954818+0000","last_fullsized":"2024-06-27T21:51:49.954818+0000","mapping_epoch":916,"log_start":"936'2600","ondisk_log_start":"936'2600","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"956'8072","last_scrub_stamp":"2024-06-27T21:50:15.925815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:50:15.925815+0000","objects_scrubbed":5681,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68616192,"num_objects":1047,"num_object_clones":0,"num_object_copies":3141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":438464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":353,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:46.195531+0000","last_change":"2024-06-27T21:51:46.195566+0000","last_active":"2024-06-27T21:51:46.195531+0000","last_peered":"2024-06-27T21:51:46.195531+0000","last_clean":"2024-06-27T21:51:46.195531+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:51:46.195531+0000","last_undegraded":"2024-06-27T21:51:46.195531+0000","last_fullsized":"2024-06-27T21:51:46.195531+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:46.195469+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:46.195469+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:09.694112+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"975'17188","reported_seq":17551,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:50.176192+0000","last_change":"2024-06-27T21:50:59.459528+0000","last_active":"2024-06-27T21:51:50.176192+0000","last_peered":"2024-06-27T21:51:50.176192+0000","last_clean":"2024-06-27T21:51:50.176192+0000","last_became_active":"2024-06-27T21:48:12.309579+0000","last_became_peered":"2024-06-27T21:48:12.309579+0000","last_unstale":"2024-06-27T21:51:50.176192+0000","last_undegraded":"2024-06-27T21:51:50.176192+0000","last_fullsized":"2024-06-27T21:51:50.176192+0000","mapping_epoch":916,"log_start":"952'7102","ondisk_log_start":"952'7102","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'14609","last_scrub_stamp":"2024-06-27T21:50:59.459390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:50:59.459390+0000","objects_scrubbed":3664,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:02.770866+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70320128,"num_objects":1073,"num_object_clones":0,"num_object_copies":3219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12719,"num_write_kb":441344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1122,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.070079+0000","last_change":"2024-06-27T21:50:57.457002+0000","last_active":"2024-06-27T21:51:44.070079+0000","last_peered":"2024-06-27T21:51:44.070079+0000","last_clean":"2024-06-27T21:51:44.070079+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:51:44.070079+0000","last_undegraded":"2024-06-27T21:51:44.070079+0000","last_fullsized":"2024-06-27T21:51:44.070079+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:57.456869+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:50:57.456869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:07.148632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"975'12488","reported_seq":12722,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:50.198041+0000","last_change":"2024-06-27T21:51:49.291029+0000","last_active":"2024-06-27T21:51:50.198041+0000","last_peered":"2024-06-27T21:51:50.198041+0000","last_clean":"2024-06-27T21:51:50.198041+0000","last_became_active":"2024-06-27T21:48:11.989652+0000","last_became_peered":"2024-06-27T21:48:11.989652+0000","last_unstale":"2024-06-27T21:51:50.198041+0000","last_undegraded":"2024-06-27T21:51:50.198041+0000","last_fullsized":"2024-06-27T21:51:50.198041+0000","mapping_epoch":916,"log_start":"935'2400","ondisk_log_start":"935'2400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12436","last_scrub_stamp":"2024-06-27T21:51:49.290765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:49.290765+0000","objects_scrubbed":1098,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:03.695841+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67895296,"num_objects":1036,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12488,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1739,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:45.127923+0000","last_change":"2024-06-27T21:51:45.127958+0000","last_active":"2024-06-27T21:51:45.127923+0000","last_peered":"2024-06-27T21:51:45.127923+0000","last_clean":"2024-06-27T21:51:45.127923+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:51:45.127923+0000","last_undegraded":"2024-06-27T21:51:45.127923+0000","last_fullsized":"2024-06-27T21:51:45.127923+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:45.127817+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.958636+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.127817+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:55.161037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"975'12761","reported_seq":12969,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:50.504067+0000","last_change":"2024-06-27T21:51:00.166419+0000","last_active":"2024-06-27T21:51:50.504067+0000","last_peered":"2024-06-27T21:51:50.504067+0000","last_clean":"2024-06-27T21:51:50.504067+0000","last_became_active":"2024-06-27T21:48:25.035404+0000","last_became_peered":"2024-06-27T21:48:25.035404+0000","last_unstale":"2024-06-27T21:51:50.504067+0000","last_undegraded":"2024-06-27T21:51:50.504067+0000","last_fullsized":"2024-06-27T21:51:50.504067+0000","mapping_epoch":924,"log_start":"937'2700","ondisk_log_start":"937'2700","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'10253","last_scrub_stamp":"2024-06-27T21:51:00.166232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.166232+0000","objects_scrubbed":3624,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:06.132572+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70844416,"num_objects":1081,"num_object_clones":0,"num_object_copies":3243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12761,"num_write_kb":442944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":714,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.067653+0000","last_change":"2024-06-27T21:51:19.848244+0000","last_active":"2024-06-27T21:51:44.067653+0000","last_peered":"2024-06-27T21:51:44.067653+0000","last_clean":"2024-06-27T21:51:44.067653+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:51:44.067653+0000","last_undegraded":"2024-06-27T21:51:44.067653+0000","last_fullsized":"2024-06-27T21:51:44.067653+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:19.848143+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:51:19.848143+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:33.796830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.b","version":"975'12474","reported_seq":12691,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:48.259516+0000","last_change":"2024-06-27T21:51:43.834598+0000","last_active":"2024-06-27T21:51:48.259516+0000","last_peered":"2024-06-27T21:51:48.259516+0000","last_clean":"2024-06-27T21:51:48.259516+0000","last_became_active":"2024-06-27T21:48:11.989420+0000","last_became_peered":"2024-06-27T21:48:11.989420+0000","last_unstale":"2024-06-27T21:51:48.259516+0000","last_undegraded":"2024-06-27T21:51:48.259516+0000","last_fullsized":"2024-06-27T21:51:48.259516+0000","mapping_epoch":916,"log_start":"934'2400","ondisk_log_start":"934'2400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'12245","last_scrub_stamp":"2024-06-27T21:51:43.834203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:43.834203+0000","objects_scrubbed":1361,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:54.273392+0000","scrub_duration":314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73531392,"num_objects":1122,"num_object_clones":0,"num_object_copies":3366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12474,"num_write_kb":435072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":1111,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.069820+0000","last_change":"2024-06-27T21:50:24.498608+0000","last_active":"2024-06-27T21:51:44.069820+0000","last_peered":"2024-06-27T21:51:44.069820+0000","last_clean":"2024-06-27T21:51:44.069820+0000","last_became_active":"2024-06-27T21:48:18.177749+0000","last_became_peered":"2024-06-27T21:48:18.177749+0000","last_unstale":"2024-06-27T21:51:44.069820+0000","last_undegraded":"2024-06-27T21:51:44.069820+0000","last_fullsized":"2024-06-27T21:51:44.069820+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:24.498490+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:24.498490+0000","last_clean_scrub_stamp":"2024-06-27T21:50:24.498490+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:51:53.834750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"975'12362","reported_seq":12582,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:47.917130+0000","last_change":"2024-06-27T21:51:42.411592+0000","last_active":"2024-06-27T21:51:47.917130+0000","last_peered":"2024-06-27T21:51:47.917130+0000","last_clean":"2024-06-27T21:51:47.917130+0000","last_became_active":"2024-06-27T21:48:12.311072+0000","last_became_peered":"2024-06-27T21:48:12.311072+0000","last_unstale":"2024-06-27T21:51:47.917130+0000","last_undegraded":"2024-06-27T21:51:47.917130+0000","last_fullsized":"2024-06-27T21:51:47.917130+0000","mapping_epoch":916,"log_start":"934'2300","ondisk_log_start":"934'2300","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'12076","last_scrub_stamp":"2024-06-27T21:51:42.411418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:42.411418+0000","objects_scrubbed":1427,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:07.274916+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74186752,"num_objects":1132,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":431808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":345,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.073647+0000","last_change":"2024-06-27T21:51:12.313706+0000","last_active":"2024-06-27T21:51:44.073647+0000","last_peered":"2024-06-27T21:51:44.073647+0000","last_clean":"2024-06-27T21:51:44.073647+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:51:44.073647+0000","last_undegraded":"2024-06-27T21:51:44.073647+0000","last_fullsized":"2024-06-27T21:51:44.073647+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:12.313599+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:51:12.313599+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:25.172124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"975'12526","reported_seq":12741,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:50.162011+0000","last_change":"2024-06-27T21:51:44.838512+0000","last_active":"2024-06-27T21:51:50.162011+0000","last_peered":"2024-06-27T21:51:50.162011+0000","last_clean":"2024-06-27T21:51:50.162011+0000","last_became_active":"2024-06-27T21:48:11.997839+0000","last_became_peered":"2024-06-27T21:48:11.997839+0000","last_unstale":"2024-06-27T21:51:50.162011+0000","last_undegraded":"2024-06-27T21:51:50.162011+0000","last_fullsized":"2024-06-27T21:51:50.162011+0000","mapping_epoch":916,"log_start":"935'2500","ondisk_log_start":"935'2500","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12266","last_scrub_stamp":"2024-06-27T21:51:44.838402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:44.838402+0000","objects_scrubbed":1333,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:02.823839+0000","scrub_duration":252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70254592,"num_objects":1072,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12526,"num_write_kb":435136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":343,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.066456+0000","last_change":"2024-06-27T21:51:03.290233+0000","last_active":"2024-06-27T21:51:44.066456+0000","last_peered":"2024-06-27T21:51:44.066456+0000","last_clean":"2024-06-27T21:51:44.066456+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:51:44.066456+0000","last_undegraded":"2024-06-27T21:51:44.066456+0000","last_fullsized":"2024-06-27T21:51:44.066456+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.290132+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.290132+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:05.358928+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"975'12592","reported_seq":12820,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:47.922586+0000","last_change":"2024-06-27T21:51:08.944383+0000","last_active":"2024-06-27T21:51:47.922586+0000","last_peered":"2024-06-27T21:51:47.922586+0000","last_clean":"2024-06-27T21:51:47.922586+0000","last_became_active":"2024-06-27T21:48:24.958699+0000","last_became_peered":"2024-06-27T21:48:24.958699+0000","last_unstale":"2024-06-27T21:51:47.922586+0000","last_undegraded":"2024-06-27T21:51:47.922586+0000","last_fullsized":"2024-06-27T21:51:47.922586+0000","mapping_epoch":924,"log_start":"935'2500","ondisk_log_start":"935'2500","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'10625","last_scrub_stamp":"2024-06-27T21:51:08.944161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:08.944161+0000","objects_scrubbed":3179,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:17.968158+0000","scrub_duration":1022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12592,"num_write_kb":440704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1446,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.070713+0000","last_change":"2024-06-27T21:51:03.754036+0000","last_active":"2024-06-27T21:51:44.070713+0000","last_peered":"2024-06-27T21:51:44.070713+0000","last_clean":"2024-06-27T21:51:44.070713+0000","last_became_active":"2024-06-27T21:48:18.177860+0000","last_became_peered":"2024-06-27T21:48:18.177860+0000","last_unstale":"2024-06-27T21:51:44.070713+0000","last_undegraded":"2024-06-27T21:51:44.070713+0000","last_fullsized":"2024-06-27T21:51:44.070713+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.753918+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.753918+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:26.192419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"975'12644","reported_seq":12867,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:49.018769+0000","last_change":"2024-06-27T21:51:21.024373+0000","last_active":"2024-06-27T21:51:49.018769+0000","last_peered":"2024-06-27T21:51:49.018769+0000","last_clean":"2024-06-27T21:51:49.018769+0000","last_became_active":"2024-06-27T21:48:12.309446+0000","last_became_peered":"2024-06-27T21:48:12.309446+0000","last_unstale":"2024-06-27T21:51:49.018769+0000","last_undegraded":"2024-06-27T21:51:49.018769+0000","last_fullsized":"2024-06-27T21:51:49.018769+0000","mapping_epoch":916,"log_start":"935'2600","ondisk_log_start":"935'2600","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'11364","last_scrub_stamp":"2024-06-27T21:51:21.024222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:21.024222+0000","objects_scrubbed":2436,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:32.134880+0000","scrub_duration":774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74973184,"num_objects":1144,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12644,"num_write_kb":441216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1520,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:47.688199+0000","last_change":"2024-06-27T21:51:47.688240+0000","last_active":"2024-06-27T21:51:47.688199+0000","last_peered":"2024-06-27T21:51:47.688199+0000","last_clean":"2024-06-27T21:51:47.688199+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:51:47.688199+0000","last_undegraded":"2024-06-27T21:51:47.688199+0000","last_fullsized":"2024-06-27T21:51:47.688199+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:47.687934+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.687934+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:53.007398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"975'14223","reported_seq":14514,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:49.006272+0000","last_change":"2024-06-27T21:51:45.729860+0000","last_active":"2024-06-27T21:51:49.006272+0000","last_peered":"2024-06-27T21:51:49.006272+0000","last_clean":"2024-06-27T21:51:49.006272+0000","last_became_active":"2024-06-27T21:48:12.309793+0000","last_became_peered":"2024-06-27T21:48:12.309793+0000","last_unstale":"2024-06-27T21:51:49.006272+0000","last_undegraded":"2024-06-27T21:51:49.006272+0000","last_fullsized":"2024-06-27T21:51:49.006272+0000","mapping_epoch":916,"log_start":"942'4200","ondisk_log_start":"942'4200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'14053","last_scrub_stamp":"2024-06-27T21:51:45.729739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.729739+0000","objects_scrubbed":1302,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:47.931619+0000","scrub_duration":531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73138200,"num_objects":1117,"num_object_clones":0,"num_object_copies":3351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12713,"num_write_kb":442497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1540,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.070000+0000","last_change":"2024-06-27T21:50:56.782584+0000","last_active":"2024-06-27T21:51:44.070000+0000","last_peered":"2024-06-27T21:51:44.070000+0000","last_clean":"2024-06-27T21:51:44.070000+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:51:44.070000+0000","last_undegraded":"2024-06-27T21:51:44.070000+0000","last_fullsized":"2024-06-27T21:51:44.070000+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.782510+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.782510+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.914888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"977'12681","reported_seq":12917,"reported_epoch":977,"state":"active+clean","last_fresh":"2024-06-27T21:51:52.282230+0000","last_change":"2024-06-27T21:51:00.997686+0000","last_active":"2024-06-27T21:51:52.282230+0000","last_peered":"2024-06-27T21:51:52.282230+0000","last_clean":"2024-06-27T21:51:52.282230+0000","last_became_active":"2024-06-27T21:48:11.986349+0000","last_became_peered":"2024-06-27T21:48:11.986349+0000","last_unstale":"2024-06-27T21:51:52.282230+0000","last_undegraded":"2024-06-27T21:51:52.282230+0000","last_fullsized":"2024-06-27T21:51:52.282230+0000","mapping_epoch":916,"log_start":"936'2600","ondisk_log_start":"936'2600","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'10075","last_scrub_stamp":"2024-06-27T21:51:00.997428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.997428+0000","objects_scrubbed":3581,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:29.403739+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62455808,"num_objects":953,"num_object_clones":0,"num_object_copies":2859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12681,"num_write_kb":436288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":241,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.072948+0000","last_change":"2024-06-27T21:51:01.076257+0000","last_active":"2024-06-27T21:51:44.072948+0000","last_peered":"2024-06-27T21:51:44.072948+0000","last_clean":"2024-06-27T21:51:44.072948+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:51:44.072948+0000","last_undegraded":"2024-06-27T21:51:44.072948+0000","last_fullsized":"2024-06-27T21:51:44.072948+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:01.076151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:01.076151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:26.255270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"977'12620","reported_seq":12852,"reported_epoch":977,"state":"active+clean","last_fresh":"2024-06-27T21:51:52.267452+0000","last_change":"2024-06-27T21:51:00.090300+0000","last_active":"2024-06-27T21:51:52.267452+0000","last_peered":"2024-06-27T21:51:52.267452+0000","last_clean":"2024-06-27T21:51:52.267452+0000","last_became_active":"2024-06-27T21:48:12.309342+0000","last_became_peered":"2024-06-27T21:48:12.309342+0000","last_unstale":"2024-06-27T21:51:52.267452+0000","last_undegraded":"2024-06-27T21:51:52.267452+0000","last_fullsized":"2024-06-27T21:51:52.267452+0000","mapping_epoch":916,"log_start":"936'2600","ondisk_log_start":"936'2600","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'10080","last_scrub_stamp":"2024-06-27T21:51:00.090159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.090159+0000","objects_scrubbed":3520,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:20.197934+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":960,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":142,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:48.643155+0000","last_change":"2024-06-27T21:51:48.643238+0000","last_active":"2024-06-27T21:51:48.643155+0000","last_peered":"2024-06-27T21:51:48.643155+0000","last_clean":"2024-06-27T21:51:48.643155+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:51:48.643155+0000","last_undegraded":"2024-06-27T21:51:48.643155+0000","last_fullsized":"2024-06-27T21:51:48.643155+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.643018+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.643018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:53.260819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"977'16240","reported_seq":16651,"reported_epoch":977,"state":"active+clean","last_fresh":"2024-06-27T21:51:52.206881+0000","last_change":"2024-06-27T21:51:03.946606+0000","last_active":"2024-06-27T21:51:52.206881+0000","last_peered":"2024-06-27T21:51:52.206881+0000","last_clean":"2024-06-27T21:51:52.206881+0000","last_became_active":"2024-06-27T21:48:12.198416+0000","last_became_peered":"2024-06-27T21:48:12.198416+0000","last_unstale":"2024-06-27T21:51:52.206881+0000","last_undegraded":"2024-06-27T21:51:52.206881+0000","last_fullsized":"2024-06-27T21:51:52.206881+0000","mapping_epoch":916,"log_start":"946'6200","ondisk_log_start":"946'6200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'13817","last_scrub_stamp":"2024-06-27T21:51:03.946497+0000","last_deep_scrub":"950'9834","last_deep_scrub_stamp":"2024-06-27T21:49:46.860713+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.946497+0000","objects_scrubbed":3412,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:07.343961+0000","scrub_duration":685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63569920,"num_objects":970,"num_object_clones":0,"num_object_copies":2910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12730,"num_write_kb":438400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":353,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.073600+0000","last_change":"2024-06-27T21:51:09.231702+0000","last_active":"2024-06-27T21:51:44.073600+0000","last_peered":"2024-06-27T21:51:44.073600+0000","last_clean":"2024-06-27T21:51:44.073600+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:51:44.073600+0000","last_undegraded":"2024-06-27T21:51:44.073600+0000","last_fullsized":"2024-06-27T21:51:44.073600+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:09.231581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:09.231581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:30.005850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"975'12590","reported_seq":12822,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:49.986710+0000","last_change":"2024-06-27T21:51:15.994687+0000","last_active":"2024-06-27T21:51:49.986710+0000","last_peered":"2024-06-27T21:51:49.986710+0000","last_clean":"2024-06-27T21:51:49.986710+0000","last_became_active":"2024-06-27T21:48:12.309089+0000","last_became_peered":"2024-06-27T21:48:12.309089+0000","last_unstale":"2024-06-27T21:51:49.986710+0000","last_undegraded":"2024-06-27T21:51:49.986710+0000","last_fullsized":"2024-06-27T21:51:49.986710+0000","mapping_epoch":916,"log_start":"935'2500","ondisk_log_start":"935'2500","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"969'10976","last_scrub_stamp":"2024-06-27T21:51:15.994328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:15.994328+0000","objects_scrubbed":2685,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:45.450916+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69206016,"num_objects":1056,"num_object_clones":0,"num_object_copies":3168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,6],"acting":[11,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1427,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.409737+0000","last_change":"2024-06-27T21:51:01.907123+0000","last_active":"2024-06-27T21:51:44.409737+0000","last_peered":"2024-06-27T21:51:44.409737+0000","last_clean":"2024-06-27T21:51:44.409737+0000","last_became_active":"2024-06-27T21:47:59.079888+0000","last_became_peered":"2024-06-27T21:47:59.079888+0000","last_unstale":"2024-06-27T21:51:44.409737+0000","last_undegraded":"2024-06-27T21:51:44.409737+0000","last_fullsized":"2024-06-27T21:51:44.409737+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:01.907039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:01.907039+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:13.284657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":166,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.071808+0000","last_change":"2024-06-27T21:51:00.903339+0000","last_active":"2024-06-27T21:51:44.071808+0000","last_peered":"2024-06-27T21:51:44.071808+0000","last_clean":"2024-06-27T21:51:44.071808+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:51:44.071808+0000","last_undegraded":"2024-06-27T21:51:44.071808+0000","last_fullsized":"2024-06-27T21:51:44.071808+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:00.903223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.903223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:02.542232+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1133,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:48.107613+0000","last_change":"2024-06-27T21:51:48.107650+0000","last_active":"2024-06-27T21:51:48.107613+0000","last_peered":"2024-06-27T21:51:48.107613+0000","last_clean":"2024-06-27T21:51:48.107613+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:51:48.107613+0000","last_undegraded":"2024-06-27T21:51:48.107613+0000","last_fullsized":"2024-06-27T21:51:48.107613+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.107530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.107530+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:50.921431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1152,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.071731+0000","last_change":"2024-06-27T21:50:10.363865+0000","last_active":"2024-06-27T21:51:44.071731+0000","last_peered":"2024-06-27T21:51:44.071731+0000","last_clean":"2024-06-27T21:51:44.071731+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:51:44.071731+0000","last_undegraded":"2024-06-27T21:51:44.071731+0000","last_fullsized":"2024-06-27T21:51:44.071731+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:10.363748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:50:10.363748+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":233,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.072544+0000","last_change":"2024-06-27T21:50:25.947039+0000","last_active":"2024-06-27T21:51:44.072544+0000","last_peered":"2024-06-27T21:51:44.072544+0000","last_clean":"2024-06-27T21:51:44.072544+0000","last_became_active":"2024-06-27T21:46:34.530850+0000","last_became_peered":"2024-06-27T21:46:34.530850+0000","last_unstale":"2024-06-27T21:51:44.072544+0000","last_undegraded":"2024-06-27T21:51:44.072544+0000","last_fullsized":"2024-06-27T21:51:44.072544+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:25.946926+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.946926+0000","last_clean_scrub_stamp":"2024-06-27T21:50:25.946926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:51:54.992500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1124,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.409793+0000","last_change":"2024-06-27T21:50:21.898413+0000","last_active":"2024-06-27T21:51:44.409793+0000","last_peered":"2024-06-27T21:51:44.409793+0000","last_clean":"2024-06-27T21:51:44.409793+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:51:44.409793+0000","last_undegraded":"2024-06-27T21:51:44.409793+0000","last_fullsized":"2024-06-27T21:51:44.409793+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:21.898293+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:50:21.898293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:51:45.950656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":352,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.073786+0000","last_change":"2024-06-27T21:51:18.316912+0000","last_active":"2024-06-27T21:51:44.073786+0000","last_peered":"2024-06-27T21:51:44.073786+0000","last_clean":"2024-06-27T21:51:44.073786+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:51:44.073786+0000","last_undegraded":"2024-06-27T21:51:44.073786+0000","last_fullsized":"2024-06-27T21:51:44.073786+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:18.316784+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:18.316784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.874908+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":372,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.073097+0000","last_change":"2024-06-27T21:50:56.873549+0000","last_active":"2024-06-27T21:51:44.073097+0000","last_peered":"2024-06-27T21:51:44.073097+0000","last_clean":"2024-06-27T21:51:44.073097+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:51:44.073097+0000","last_undegraded":"2024-06-27T21:51:44.073097+0000","last_fullsized":"2024-06-27T21:51:44.073097+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.873431+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.873431+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:21.297054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":788,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.074197+0000","last_change":"2024-06-27T21:50:57.612741+0000","last_active":"2024-06-27T21:51:44.074197+0000","last_peered":"2024-06-27T21:51:44.074197+0000","last_clean":"2024-06-27T21:51:44.074197+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:51:44.074197+0000","last_undegraded":"2024-06-27T21:51:44.074197+0000","last_fullsized":"2024-06-27T21:51:44.074197+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:57.612612+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:50:57.612612+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:22.461196+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":170,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.071579+0000","last_change":"2024-06-27T21:51:09.954221+0000","last_active":"2024-06-27T21:51:44.071579+0000","last_peered":"2024-06-27T21:51:44.071579+0000","last_clean":"2024-06-27T21:51:44.071579+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:51:44.071579+0000","last_undegraded":"2024-06-27T21:51:44.071579+0000","last_fullsized":"2024-06-27T21:51:44.071579+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:09.954099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:09.954099+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:25.431565+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":974,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.070074+0000","last_change":"2024-06-27T21:50:22.623520+0000","last_active":"2024-06-27T21:51:44.070074+0000","last_peered":"2024-06-27T21:51:44.070074+0000","last_clean":"2024-06-27T21:51:44.070074+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:51:44.070074+0000","last_undegraded":"2024-06-27T21:51:44.070074+0000","last_fullsized":"2024-06-27T21:51:44.070074+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:22.623425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:50:22.623425+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1077,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:47.411901+0000","last_change":"2024-06-27T21:51:47.411923+0000","last_active":"2024-06-27T21:51:47.411901+0000","last_peered":"2024-06-27T21:51:47.411901+0000","last_clean":"2024-06-27T21:51:47.411901+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:51:47.411901+0000","last_undegraded":"2024-06-27T21:51:47.411901+0000","last_fullsized":"2024-06-27T21:51:47.411901+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:47.411842+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.411842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:15.992157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":770,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.073930+0000","last_change":"2024-06-27T21:51:04.535580+0000","last_active":"2024-06-27T21:51:44.073930+0000","last_peered":"2024-06-27T21:51:44.073930+0000","last_clean":"2024-06-27T21:51:44.073930+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:51:44.073930+0000","last_undegraded":"2024-06-27T21:51:44.073930+0000","last_fullsized":"2024-06-27T21:51:44.073930+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:04.535471+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:51:04.535471+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:05.628576+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":788,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.074114+0000","last_change":"2024-06-27T21:50:56.571158+0000","last_active":"2024-06-27T21:51:44.074114+0000","last_peered":"2024-06-27T21:51:44.074114+0000","last_clean":"2024-06-27T21:51:44.074114+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:51:44.074114+0000","last_undegraded":"2024-06-27T21:51:44.074114+0000","last_fullsized":"2024-06-27T21:51:44.074114+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.571030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.571030+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:00.224940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":241,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:44.072844+0000","last_change":"2024-06-27T21:51:03.119223+0000","last_active":"2024-06-27T21:51:44.072844+0000","last_peered":"2024-06-27T21:51:44.072844+0000","last_clean":"2024-06-27T21:51:44.072844+0000","last_became_active":"2024-06-27T21:46:34.528582+0000","last_became_peered":"2024-06-27T21:46:34.528582+0000","last_unstale":"2024-06-27T21:51:44.072844+0000","last_undegraded":"2024-06-27T21:51:44.072844+0000","last_fullsized":"2024-06-27T21:51:44.072844+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.119099+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.119099+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:19.502832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":241,"reported_epoch":975,"state":"active+clean","last_fresh":"2024-06-27T21:51:49.181577+0000","last_change":"2024-06-27T21:51:49.181612+0000","last_active":"2024-06-27T21:51:49.181577+0000","last_peered":"2024-06-27T21:51:49.181577+0000","last_clean":"2024-06-27T21:51:49.181577+0000","last_became_active":"2024-06-27T21:46:34.528359+0000","last_became_peered":"2024-06-27T21:46:34.528359+0000","last_unstale":"2024-06-27T21:51:49.181577+0000","last_undegraded":"2024-06-27T21:51:49.181577+0000","last_fullsized":"2024-06-27T21:51:49.181577+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:49.181498+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:51:49.181498+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:59.400634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1123418136,"num_objects":17143,"num_object_clones":0,"num_object_copies":51429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17143,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201753,"num_write_kb":7004609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3433574400,"data_stored":3433562184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253377,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336444,"kb_used_data":286196,"kb_used_omap":1,"kb_used_meta":1050174,"kb_avail":92408708,"statfs":{"total":95995035648,"available":94626516992,"internally_reserved":0,"allocated":293064704,"data_stored":292764990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1075378629},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417515012,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1343932,"kb_used_data":347892,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92401220,"statfs":{"total":95995035648,"available":94618849280,"internally_reserved":0,"allocated":356241408,"data_stored":355941694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1019935168},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743945,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201596,"kb_used_data":426292,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92543556,"statfs":{"total":95995035648,"available":94764601344,"internally_reserved":0,"allocated":436523008,"data_stored":436223294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":793901504},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763004,"kb_used_data":150132,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92982148,"statfs":{"total":95995035648,"available":95213719552,"internally_reserved":0,"allocated":153735168,"data_stored":153435454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267212,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218760,"kb_used_data":371264,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92526392,"statfs":{"total":95995035648,"available":94747025408,"internally_reserved":0,"allocated":380174336,"data_stored":379866998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":867826122},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898428,"kb_used_data":137012,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92846724,"statfs":{"total":95995035648,"available":95075045376,"internally_reserved":0,"allocated":140300288,"data_stored":140000574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":779680198},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757787,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410824,"kb_used_data":703552,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92334328,"statfs":{"total":95995035648,"available":94550351872,"internally_reserved":0,"allocated":720437248,"data_stored":720129910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161494,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716028,"kb_used_data":145460,"kb_used_omap":92,"kb_used_meta":570403,"kb_avail":93029124,"statfs":{"total":95995035648,"available":95261822976,"internally_reserved":0,"allocated":148951040,"data_stored":148651326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":584092770},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182948,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979672,"kb_used_data":308492,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92765480,"statfs":{"total":95995035648,"available":94991851520,"internally_reserved":0,"allocated":315895808,"data_stored":315587180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":687274439},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411880,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808200,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936952,"statfs":{"total":95995035648,"available":95167438848,"internally_reserved":0,"allocated":978944,"data_stored":675678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575407,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1469568,"kb_used_data":576184,"kb_used_omap":1,"kb_used_meta":893310,"kb_avail":92275584,"statfs":{"total":95995035648,"available":94490198016,"internally_reserved":0,"allocated":590012416,"data_stored":589708630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":914749893},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880508,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864644,"statfs":{"total":95995035648,"available":95093395456,"internally_reserved":0,"allocated":1101824,"data_stored":802110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":587862016,"data_stored":587857944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312868864,"data_stored":312868864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":147849216,"data_stored":147849216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":693047296,"data_stored":693043224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":136183808,"data_stored":136183808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":362942464,"data_stored":362938392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":147324928,"data_stored":147324928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":416677888,"data_stored":416677888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":347275264,"data_stored":347275264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":281542656,"data_stored":281542656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:51:54.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:54.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:54.263 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:51:54.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:54 smithi192 ceph-mon[32956]: from='client.18024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:51:54.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:54 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:51:54.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:54 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:51:54.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:54.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:54 smithi151 ceph-mon[28965]: from='client.18024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:51:54.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:54 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:51:54.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:54 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:51:54.363 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:54.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:54.496 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:54.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:54 smithi190 ceph-mon[32247]: from='client.18024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:51:54.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:54 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:51:54.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:54 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:51:54.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set noscrub 2024-06-27T21:51:54.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:54.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:54.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:54.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:54.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:54.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:54.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:54.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:55.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:55.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:55.144 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:55.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:55 smithi151 ceph-mon[28965]: osdmap e978: 12 total, 12 up, 12 in 2024-06-27T21:51:55.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:55 smithi151 ceph-mon[28965]: pgmap v2412: 47 pgs: 47 active+clean; 972 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-06-27T21:51:55.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:55.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:55.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.403 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:55.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:55.479 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:55 smithi192 ceph-mon[32956]: osdmap e978: 12 total, 12 up, 12 in 2024-06-27T21:51:55.479 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:55 smithi192 ceph-mon[32956]: pgmap v2412: 47 pgs: 47 active+clean; 972 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-06-27T21:51:55.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:55.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:55 smithi190 ceph-mon[32247]: osdmap e978: 12 total, 12 up, 12 in 2024-06-27T21:51:55.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:55 smithi190 ceph-mon[32247]: pgmap v2412: 47 pgs: 47 active+clean; 972 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-06-27T21:51:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.535 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:55.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:55.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:55.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:55.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:55.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:55.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:55.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:55.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:56.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:56.048 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:56.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:56.185 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:56.263 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:51:56.263 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:51:56.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.263 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:207121 9.b 9:dac49b46:::benchmark_data_smithi192_179902_object97671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e978)", 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:51:56.238888+0000", 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.023220029999999999, 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023255702, 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207121 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.264 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238888+0000", 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238888+0000", 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:51:56.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238890+0000", 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.677e-06 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238895+0000", 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5520000000000004e-06 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238910+0000", 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5067e-05 2024-06-27T21:51:56.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238918+0000", 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1919999999999999e-06 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247843+0000", 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0089255069999999992 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247949+0000", 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106047 2024-06-27T21:51:56.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,10", 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.248061+0000", 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111529 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.258528+0000", 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010466962999999999 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:51:56.268 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:207122 9.b 9:d3ac7b76:::benchmark_data_smithi192_179902_object97672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e978)", 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:51:56.238932+0000", 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.023175765000000001, 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023282780999999999, 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207122 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.269 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238932+0000", 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238932+0000", 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:51:56.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238933+0000", 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0500000000000004e-07 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238936+0000", 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8339999999999999e-06 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238943+0000", 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1130000000000003e-06 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.238948+0000", 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.933e-06 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.252946+0000", 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013998195 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.253075+0000", 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012903099999999999 2024-06-27T21:51:56.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,10", 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.253196+0000", 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012102 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:207129 9.b 9:d4a27367:::benchmark_data_smithi192_179902_object97679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e978)", 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:51:56.247749+0000", 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014359532, 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014524293000000001, 2024-06-27T21:51:56.273 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 207129 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247749+0000", 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:51:56.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247749+0000", 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247750+0000", 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4580000000000001e-06 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247755+0000", 2024-06-27T21:51:56.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.933e-06 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247768+0000", 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2909e-05 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.247775+0000", 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5800000000000003e-06 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:51:56.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.258552+0000", 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010776332 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.258651+0000", 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9333000000000006e-05 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 5,10", 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:51:56.258770+0000", 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011915499999999999 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.277 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:51:56.278 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:51:56.278 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 3 2024-06-27T21:51:56.278 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:51:56.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:56.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:56.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.446 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:56.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:51:56.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:56.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:56.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:56.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:56.750 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:51:56.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:51:56.769 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:56.781 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:51:56.781 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:51:56.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:56.813 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:51:56.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:51:56.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:56.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:51:56.839 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:51:56.867 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:51:56.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:51:56.892 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:51:56.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:51:56.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:51:56.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:51:56.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:56.947 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:51:56.947 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:51:56.965 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:56.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:56.973 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:51:56.974 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:51:57.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:51:57.000 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:51:57.029 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:51:57.029 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:51:57.063 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:51:57.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:51:57.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:57.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-06-27T21:51:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:57.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:57.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:57.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:57.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:51:57.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:51:57.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:57.463 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:51:57.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:57.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:51:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.691 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:57.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:51:57.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:57 smithi151 ceph-mon[28965]: pgmap v2413: 47 pgs: 47 active+clean; 856 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s 2024-06-27T21:51:57.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1564902061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:57.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:57 smithi192 ceph-mon[32956]: pgmap v2413: 47 pgs: 47 active+clean; 856 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s 2024-06-27T21:51:57.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1564902061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:57.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:57 smithi190 ceph-mon[32247]: pgmap v2413: 47 pgs: 47 active+clean; 856 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 795 op/s 2024-06-27T21:51:57.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1564902061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:51:57.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:57.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:57.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:57.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:57.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:58.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:58.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:58.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:58.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:58.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:51:58.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:51:58.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:58.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:58.563 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:58.587 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:58.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:51:58.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:58.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:51:58.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:58.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:58.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:51:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:58.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:58.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:51:59.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:59.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:59.130 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 1, 7, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:51:59.130 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:51:59.130 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-06-27T21:51:59.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump -f json-pretty 2024-06-27T21:51:59.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:51:59.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:51:59.283 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:51:59.283 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:51:59.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:59.355 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:59.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:51:59.481 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:51:59.512 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:51:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:59 smithi192 ceph-mon[32956]: pgmap v2414: 47 pgs: 47 active+clean; 737 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-06-27T21:51:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:59 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:51:59.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:51:59 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:51:59.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:51:59.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:51:59.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:59 smithi151 ceph-mon[28965]: pgmap v2414: 47 pgs: 47 active+clean; 737 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-06-27T21:51:59.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:59 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:51:59.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:51:59 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:51:59.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:51:59.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:51:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:51:59.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:59 smithi190 ceph-mon[32247]: pgmap v2414: 47 pgs: 47 active+clean; 737 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-06-27T21:51:59.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:59 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:51:59.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:51:59 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:51:59.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:51:59.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:51:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:51:59.961 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:51:59.991 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:51:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:51:59.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:00.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:00.117 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:00.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:00.240 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-06-27T21:52:00.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:00.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:52:00.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:00.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:00.451 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-06-27T21:52:00.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:00.503 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:00.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:00 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:52:00.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:00 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:52:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2707530549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:52:00.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:00.613 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:00.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:00.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:00 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:52:00.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:00 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:52:00.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2707530549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:52:00.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:00.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:00.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:00 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:52:00.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:00 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:52:00.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2707530549' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-06-27T21:52:00.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:00.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:00.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:00.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:01.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:01.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:01.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:01.244 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:01.273 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:01.374 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:01.403 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:01.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:01.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:01.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:01 smithi151 ceph-mon[28965]: pgmap v2415: 47 pgs: 47 active+clean; 620 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s 2024-06-27T21:52:01.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:01 smithi151 ceph-mon[28965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:01.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2707530549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:52:01.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:01 smithi151 ceph-mon[28965]: osdmap e979: 12 total, 12 up, 12 in 2024-06-27T21:52:01.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:01.663 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:01.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:01 smithi190 ceph-mon[32247]: pgmap v2415: 47 pgs: 47 active+clean; 620 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s 2024-06-27T21:52:01.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:01 smithi190 ceph-mon[32247]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:01.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2707530549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:52:01.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:01 smithi190 ceph-mon[32247]: osdmap e979: 12 total, 12 up, 12 in 2024-06-27T21:52:01.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:01.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:01.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:01 smithi192 ceph-mon[32956]: pgmap v2415: 47 pgs: 47 active+clean; 620 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 970 op/s 2024-06-27T21:52:01.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:01 smithi192 ceph-mon[32956]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:01.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2707530549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-06-27T21:52:01.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:01 smithi192 ceph-mon[32956]: osdmap e979: 12 total, 12 up, 12 in 2024-06-27T21:52:01.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:01.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:01.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:02.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:02.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:02.125 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:02.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:02.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:02.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:02.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 979, 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "784a4e4c-34c8-11ef-bca8-c7b262605968", 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-06-27T21:02:54.280751+0000", 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-06-27T21:52:00.444177+0000", 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-06-27T21:23:43.806967+0000", 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-06-27T21:46:33.505233+0000", 2024-06-27T21:52:02.335 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 27, 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-06-27T21:52:02.336 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 9, 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-06-27T21:52:02.337 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:06:07.812474+0000", 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:52:02.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "789", 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "535", 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:52:02.340 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:52:02.341 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:52:02.342 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.343 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.940000057220459, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 1.940000057220459, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.940000057220459, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.940000057220459, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.79288095235824585, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.79000002145767212, 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-06-27T21:52:02.344 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 9, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_7", 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-06-27T21:48:10.205815+0000", 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-06-27T21:52:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-06-27T21:52:02.346 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "923", 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-06-27T21:52:02.347 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-06-27T21:52:02.348 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-06-27T21:52:02.349 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-06-27T21:52:02.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.2699999809265137, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.2699999809265137, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.82999998331069946, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.8799999952316284, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.8799999952316284, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.79288095235824585, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.79000002145767212, 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.79000002145767212 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.351 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "80c5e023-5c22-463d-86e9-2307159c858f", 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.352 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 872, 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-06-27T21:52:02.353 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.354 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.356 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2519434429 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/2519434429", 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/2519434429", 2024-06-27T21:52:02.357 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/2519434429", 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/2519434429", 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "d6a0fa83-bdab-4c01-b0d0-8996e8d0c083", 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.358 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.1994171142578125, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 895, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-06-27T21:52:02.359 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.362 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 59941028 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/59941028", 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/59941028", 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/59941028", 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/59941028", 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "22115999-54ff-4f34-81cc-9d90bfa4bd62", 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.364 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 909, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.367 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 302970170 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/302970170", 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/302970170", 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/302970170", 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/302970170", 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "88c12fa5-d2af-4144-958e-b887b4474b80", 2024-06-27T21:52:02.370 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 25, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 916, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.372 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.373 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.374 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.374 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3555754662 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/3555754662", 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/3555754662", 2024-06-27T21:52:02.376 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/3555754662", 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/3555754662", 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "97aaddb9-d5bb-4827-8c35-b3c948eae3dc", 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.377 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.3238067626953125, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 31, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 347, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 362, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 920, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 348, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6800", 2024-06-27T21:52:02.378 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6801", 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.379 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6802", 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6803", 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.380 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6806", 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6807", 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6804", 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6805", 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3587059633 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6801/3587059633", 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6803/3587059633", 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6807/3587059633", 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6805/3587059633", 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "7ac9bbd6-0e55-4cab-b406-6f9967eaaf30", 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.383 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.9023590087890625, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 38, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 924, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6808", 2024-06-27T21:52:02.384 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6809", 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.385 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6810", 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6811", 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6814", 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6815", 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.387 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6812", 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6813", 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 888361120 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.388 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6809/888361120", 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6811/888361120", 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6815/888361120", 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6813/888361120", 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "89fae0b2-26a4-456a-9d54-cfbaec80ba6a", 2024-06-27T21:52:02.389 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.0889892578125, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 479, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.390 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6816", 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.391 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6817", 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6818", 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.392 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6819", 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6822", 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.393 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6823", 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6820", 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.394 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6821", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 30028280 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6817/30028280", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6819/30028280", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6823/30028280", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6821/30028280", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.395 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e324ffdc-9842-46ee-a87e-5054ec265484", 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 53, 2024-06-27T21:52:02.396 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 916, 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6824", 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6825", 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.397 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6826", 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.398 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6827", 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6830", 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.399 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6831", 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6828", 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.190:6829", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 849289685 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.190:6825/849289685", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.190:6827/849289685", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.190:6831/849289685", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.190:6829/849289685", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.401 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "43d138bf-2d82-44ea-a98f-b9099e824243", 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 62, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 924, 2024-06-27T21:52:02.402 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6800", 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6801", 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6802", 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6803", 2024-06-27T21:52:02.404 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6806", 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6807", 2024-06-27T21:52:02.405 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6804", 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6805", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2977299942 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6801/2977299942", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6803/2977299942", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2977299942", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2977299942", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "0c86c106-04c9-4eb7-bb3e-d39f563b71fd", 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 69, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 924, 2024-06-27T21:52:02.408 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6808", 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6809", 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.409 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6810", 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6811", 2024-06-27T21:52:02.410 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6814", 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.411 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6815", 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6812", 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.412 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6813", 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3401759799 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6809/3401759799", 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6811/3401759799", 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6815/3401759799", 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6813/3401759799", 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3", 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 76, 2024-06-27T21:52:02.414 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 920, 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6816", 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.415 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6817", 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6818", 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.416 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6819", 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6822", 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6823", 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6820", 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6821", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1426195762 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6817/1426195762", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6819/1426195762", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6823/1426195762", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1426195762", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.419 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "a8a163c4-a14c-4710-bc2e-94e2cb69df0d", 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 84, 2024-06-27T21:52:02.420 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 916, 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6824", 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6825", 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6826", 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6827", 2024-06-27T21:52:02.422 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6830", 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.423 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6831", 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6828", 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.424 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.192:6829", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 517567267 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.192:6825/517567267", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.192:6827/517567267", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.192:6831/517567267", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.192:6829/517567267", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-06-27T21:52:02.425 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:51.585328+0000", 2024-06-27T21:52:02.426 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:01.370736+0000", 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.427 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:43.772724+0000", 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.428 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:08.307017+0000", 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:53.729710+0000", 2024-06-27T21:52:02.429 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 347 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:50:49.772148+0000", 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.430 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:50:53.759295+0000", 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.431 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:50:48.086651+0000", 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:50:54.330842+0000", 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:31.621364+0000", 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-06-27T21:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:17.202704+0000", 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-06-27T21:52:02.434 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540701547738038271, 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-06-27T21:51:06.428554+0000", 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.0", 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: 10 2024-06-27T21:52:02.435 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1", 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: 11 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-06-27T21:52:02.436 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/4089480652": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3356605898": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3908626376": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3261020553": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3654164819": "2024-06-28T21:03:45.726698+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3767407460": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2698418425": "2024-06-28T21:03:22.780955+0000", 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/335837712": "2024-06-28T21:03:22.780955+0000" 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-06-27T21:52:02.437 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-06-27T21:52:02.438 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-06-27T21:52:02.439 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-06-27T21:52:02.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:02.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:02.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:02.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3799220528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:52:02.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:02.602 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:02.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3799220528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:52:02.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:02.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:02.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3799220528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-06-27T21:52:02.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:02.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:02.861 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:52:02.861 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:52:02.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:02.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:02.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:03.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:03.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:03.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:03.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:03.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:03.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:03.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.513 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:03.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:03.553 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-06-27T21:52:03.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd set nodeep-scrub 2024-06-27T21:52:03.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.673 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:03.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:03 smithi190 ceph-mon[32247]: pgmap v2417: 47 pgs: 47 active+clean; 537 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s 2024-06-27T21:52:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:03 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:03.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:03 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:03.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:03.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:03 smithi192 ceph-mon[32956]: pgmap v2417: 47 pgs: 47 active+clean; 537 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s 2024-06-27T21:52:03.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:03 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:03.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:03 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:03.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:03.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:03.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:03.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:03.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:03 smithi151 ceph-mon[28965]: pgmap v2417: 47 pgs: 47 active+clean; 537 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s 2024-06-27T21:52:03.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:03 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:03.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:03 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:04.004 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:04.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:04.060 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:04.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:04.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:04.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:04.325 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:04.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:04.456 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:04.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:04.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:04.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:04.684 INFO:tasks.thrashosds.thrasher:j is {'epoch': 979, 'fsid': '784a4e4c-34c8-11ef-bca8-c7b262605968', 'created': '2024-06-27T21:02:54.280751+0000', 'modified': '2024-06-27T21:52:00.444177+0000', 'last_up_change': '2024-06-27T21:23:43.806967+0000', 'last_in_change': '2024-06-27T21:46:33.505233+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-27T21:06:07.812474+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '789', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '535', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.940000057220459, 'score_stable': 1.940000057220459, 'optimal_score': 1, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.7928809523582458, 'average_primary_affinity': 0.7900000214576721, 'average_primary_affinity_weighted': 0.7900000214576721}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-06-27T21:48:10.205815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '923', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.7928809523582458, 'average_primary_affinity': 0.7900000214576721, 'average_primary_affinity_weighted': 0.7900000214576721}}], 'osds': [{'osd': 0, 'uuid': '80c5e023-5c22-463d-86e9-2307159c858f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 872, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 2519434429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 2519434429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 2519434429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 2519434429}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 2519434429}]}, 'public_addr': '172.21.15.151:6803/2519434429', 'cluster_addr': '172.21.15.151:6805/2519434429', 'heartbeat_back_addr': '172.21.15.151:6809/2519434429', 'heartbeat_front_addr': '172.21.15.151:6807/2519434429', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd6a0fa83-bdab-4c01-b0d0-8996e8d0c083', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1994171142578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 895, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 59941028}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 59941028}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 59941028}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 59941028}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 59941028}]}, 'public_addr': '172.21.15.151:6811/59941028', 'cluster_addr': '172.21.15.151:6813/59941028', 'heartbeat_back_addr': '172.21.15.151:6817/59941028', 'heartbeat_front_addr': '172.21.15.151:6815/59941028', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '22115999-54ff-4f34-81cc-9d90bfa4bd62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 909, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 302970170}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 302970170}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 302970170}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 302970170}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 302970170}]}, 'public_addr': '172.21.15.151:6819/302970170', 'cluster_addr': '172.21.15.151:6821/302970170', 'heartbeat_back_addr': '172.21.15.151:6825/302970170', 'heartbeat_front_addr': '172.21.15.151:6823/302970170', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '88c12fa5-d2af-4144-958e-b887b4474b80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 916, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 3555754662}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 3555754662}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 3555754662}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 3555754662}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 3555754662}]}, 'public_addr': '172.21.15.151:6827/3555754662', 'cluster_addr': '172.21.15.151:6829/3555754662', 'heartbeat_back_addr': '172.21.15.151:6833/3555754662', 'heartbeat_front_addr': '172.21.15.151:6831/3555754662', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '97aaddb9-d5bb-4827-8c35-b3c948eae3dc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3238067626953125, 'last_clean_begin': 31, 'last_clean_end': 347, 'up_from': 362, 'up_thru': 920, 'down_at': 348, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 3587059633}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 3587059633}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 3587059633}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 3587059633}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 3587059633}]}, 'public_addr': '172.21.15.190:6801/3587059633', 'cluster_addr': '172.21.15.190:6803/3587059633', 'heartbeat_back_addr': '172.21.15.190:6807/3587059633', 'heartbeat_front_addr': '172.21.15.190:6805/3587059633', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7ac9bbd6-0e55-4cab-b406-6f9967eaaf30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9023590087890625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 924, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 888361120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 888361120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 888361120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 888361120}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 888361120}]}, 'public_addr': '172.21.15.190:6809/888361120', 'cluster_addr': '172.21.15.190:6811/888361120', 'heartbeat_back_addr': '172.21.15.190:6815/888361120', 'heartbeat_front_addr': '172.21.15.190:6813/888361120', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89fae0b2-26a4-456a-9d54-cfbaec80ba6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0889892578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 479, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 30028280}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 30028280}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 30028280}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 30028280}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 30028280}]}, 'public_addr': '172.21.15.190:6817/30028280', 'cluster_addr': '172.21.15.190:6819/30028280', 'heartbeat_back_addr': '172.21.15.190:6823/30028280', 'heartbeat_front_addr': '172.21.15.190:6821/30028280', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e324ffdc-9842-46ee-a87e-5054ec265484', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 916, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 849289685}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 849289685}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 849289685}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 849289685}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 849289685}]}, 'public_addr': '172.21.15.190:6825/849289685', 'cluster_addr': '172.21.15.190:6827/849289685', 'heartbeat_back_addr': '172.21.15.190:6831/849289685', 'heartbeat_front_addr': '172.21.15.190:6829/849289685', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '43d138bf-2d82-44ea-a98f-b9099e824243', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 924, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2977299942}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2977299942}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2977299942}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2977299942}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2977299942}]}, 'public_addr': '172.21.15.192:6801/2977299942', 'cluster_addr': '172.21.15.192:6803/2977299942', 'heartbeat_back_addr': '172.21.15.192:6807/2977299942', 'heartbeat_front_addr': '172.21.15.192:6805/2977299942', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '0c86c106-04c9-4eb7-bb3e-d39f563b71fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 924, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 3401759799}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 3401759799}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 3401759799}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 3401759799}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 3401759799}]}, 'public_addr': '172.21.15.192:6809/3401759799', 'cluster_addr': '172.21.15.192:6811/3401759799', 'heartbeat_back_addr': '172.21.15.192:6815/3401759799', 'heartbeat_front_addr': '172.21.15.192:6813/3401759799', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 76, 'up_thru': 920, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1426195762}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1426195762}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6823', 'nonce': 1426195762}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1426195762}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1426195762}]}, 'public_addr': '172.21.15.192:6817/1426195762', 'cluster_addr': '172.21.15.192:6819/1426195762', 'heartbeat_back_addr': '172.21.15.192:6823/1426195762', 'heartbeat_front_addr': '172.21.15.192:6821/1426195762', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'a8a163c4-a14c-4710-bc2e-94e2cb69df0d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 84, 'up_thru': 916, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6824', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 517567267}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 517567267}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 517567267}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 517567267}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 517567267}]}, 'public_addr': '172.21.15.192:6825/517567267', 'cluster_addr': '172.21.15.192:6827/517567267', 'heartbeat_back_addr': '172.21.15.192:6831/517567267', 'heartbeat_front_addr': '172.21.15.192:6829/517567267', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:51.585328+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:01.370736+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:43.772724+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:08.307017+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:53.729710+0000', 'dead_epoch': 347}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:50:49.772148+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:50:53.759295+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:50:48.086651+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:50:54.330842+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:31.621364+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:17.202704+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-06-27T21:51:06.428554+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [1, 9, 10]}, {'pgid': '1.1', 'osds': [0, 9, 11]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/4089480652': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3356605898': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3908626376': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:0/3261020553': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3654164819': '2024-06-28T21:03:45.726698+0000', '172.21.15.151:6801/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:6800/3767407460': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/2698418425': '2024-06-28T21:03:22.780955+0000', '172.21.15.151:0/335837712': '2024-06-28T21:03:22.780955+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-06-27T21:52:04.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:52:04.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:04.750 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:04 smithi151 ceph-mon[28965]: osdmap e980: 12 total, 12 up, 12 in 2024-06-27T21:52:04.762 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:04.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:04 smithi190 ceph-mon[32247]: osdmap e980: 12 total, 12 up, 12 in 2024-06-27T21:52:04.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:04 smithi192 ceph-mon[32956]: osdmap e980: 12 total, 12 up, 12 in 2024-06-27T21:52:04.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:04.878 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:04.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:04.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:05.005 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:05.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:05.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:05.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:05.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:05.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:05.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:05.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:05.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:52:05.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:52:05.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:05.523 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:52:05.524 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:52:05.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:52:05.551 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:52:05.582 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:52:05.582 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:52:05.610 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:52:05.610 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:52:05.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:05.640 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:52:05.640 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:52:05.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:05.668 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:52:05.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:52:05.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:52:05.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:52:05.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:52:05.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:52:05.720 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:05 smithi151 ceph-mon[28965]: pgmap v2419: 47 pgs: 47 active+clean; 443 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-06-27T21:52:05.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:52:05.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:52:05.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:05.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:05 smithi190 ceph-mon[32247]: pgmap v2419: 47 pgs: 47 active+clean; 443 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-06-27T21:52:05.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:05.773 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:52:05.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:52:05.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:05 smithi192 ceph-mon[32956]: pgmap v2419: 47 pgs: 47 active+clean; 443 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-06-27T21:52:05.801 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:52:05.801 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:52:05.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:05.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:05.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:05.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:05.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:06.024 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:06.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:06.154 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:06.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:06.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:06.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:06.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:215336 9.2 e980/916 9:4615a693:::benchmark_data_smithi192_179902_object105886:head v 980'13361, mlcod=980'13359)", 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:52:06.448680+0000", 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0067778990000000004, 2024-06-27T21:52:06.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0068047029999999996, 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448680+0000", 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448680+0000", 2024-06-27T21:52:06.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448680+0000", 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-07 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448685+0000", 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9459999999999997e-06 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448730+0000", 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999999e-05 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448733+0000", 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7189999999999999e-06 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448802+0000", 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8442000000000006e-05 2024-06-27T21:52:06.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448820+0000", 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7757e-05 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:215337 9.2 e980/916 9:44a162c2:::benchmark_data_smithi192_179902_object105887:head v 980'13362, mlcod=980'13359)", 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:52:06.448686+0000", 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0067712290000000001, 2024-06-27T21:52:06.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0068495409999999998, 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448686+0000", 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448686+0000", 2024-06-27T21:52:06.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448686+0000", 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.31e-07 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448689+0000", 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6019999999999998e-06 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448749+0000", 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0229999999999998e-05 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.448753+0000", 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6770000000000001e-06 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.452422+0000", 2024-06-27T21:52:06.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036685939999999998 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.452450+0000", 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8002000000000001e-05 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:215339 9.2 e980/916)", 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:52:06.449553+0000", 2024-06-27T21:52:06.464 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0059044199999999996, 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0060252040000000001, 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.449553+0000", 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.449553+0000", 2024-06-27T21:52:06.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.449554+0000", 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0100000000000005e-07 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.449559+0000", 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3190000000000002e-06 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.449586+0000", 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6979999999999999e-05 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.449592+0000", 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3969999999999999e-06 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.467 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:215341 9.2 e980/916)", 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:52:06.453300+0000", 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0021578529999999999, 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023114469999999999, 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.453300+0000", 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.453300+0000", 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.453300+0000", 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5499999999999997e-07 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.453306+0000", 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4700000000000001e-06 2024-06-27T21:52:06.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.453332+0000", 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6446999999999999e-05 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.453339+0000", 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4760000000000003e-06 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:215342 9.c e980/916 9:30893ced:::benchmark_data_smithi192_179902_object105892:head v 980'13413, mlcod=980'13411)", 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:52:06.454470+0000", 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00098704799999999992, 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001176563, 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454470+0000", 2024-06-27T21:52:06.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454470+0000", 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454471+0000", 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9800000000000003e-07 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:06.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454477+0000", 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6910000000000002e-06 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454500+0000", 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3397999999999999e-05 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454506+0000", 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8130000000000001e-06 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454552+0000", 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5473000000000003e-05 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:52:06.454574+0000", 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2126000000000001e-05 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 5 2024-06-27T21:52:06.474 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:52:06.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:06.530 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:06.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:06.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:06.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2371969706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:06.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:06.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2371969706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:06.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:06.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2371969706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:06.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:06.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:06.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:06.957 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:07.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:07.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:07.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:07.168 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:07.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:07.303 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:07.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:07.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:07.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:07.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:07.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:07.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:07 smithi151 ceph-mon[28965]: pgmap v2420: 47 pgs: 47 active+clean; 331 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s 2024-06-27T21:52:07.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:07.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:07 smithi190 ceph-mon[32247]: pgmap v2420: 47 pgs: 47 active+clean; 331 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s 2024-06-27T21:52:07.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:07 smithi192 ceph-mon[32956]: pgmap v2420: 47 pgs: 47 active+clean; 331 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s 2024-06-27T21:52:07.800 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:07.831 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:07.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:07.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:07.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:08.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:08.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:08.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:08.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:08.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:08.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:08.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-06-27T21:52:08.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:08.510 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:08.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:08.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:08.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:08.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:08.821 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:08.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:08.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:08.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:08.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:09.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:09.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:09.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:09.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:09.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:09.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:09.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:09.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-06-27T21:52:09.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:09.522 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:09.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:09 smithi192 ceph-mon[32956]: pgmap v2421: 47 pgs: 47 active+clean; 231 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-06-27T21:52:09.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:09.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:09.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:09 smithi151 ceph-mon[28965]: pgmap v2421: 47 pgs: 47 active+clean; 231 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-06-27T21:52:09.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:09.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:09 smithi190 ceph-mon[32247]: pgmap v2421: 47 pgs: 47 active+clean; 231 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s 2024-06-27T21:52:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:09.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:09.889 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:09.921 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:09.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:10.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:10.052 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:10.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:10.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:10.189 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:10.261 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:52:10.261 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:52:10.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:10.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:10.427 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:10.456 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:10.487 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-06-27T21:52:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:10.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:10 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/335225820' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:10.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.589 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:10.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:10.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:10.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:10.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:10 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/335225820' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:10.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:10.717 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:10.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:10.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:10.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:10 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/335225820' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:10.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:10.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:10.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:10.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:10.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:11.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:11.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:11.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:11.215 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:11.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:11.334 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2421,"stamp":"2024-06-27T21:52:08.390647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":241762872,"num_objects":3683,"num_object_clones":0,"num_object_copies":11049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215215,"num_write_kb":7004609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163554,"ondisk_log_size":163554,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10446956,"kb_used_data":854860,"kb_used_omap":110,"kb_used_meta":9591505,"kb_avail":1114494868,"statfs":{"total":1151940427776,"available":1141242744832,"internally_reserved":0,"allocated":875376640,"data_stored":872127108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9821702103},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-408354816,"num_objects":-6231,"num_object_clones":0,"num_object_copies":-18693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001529"},"pg_stats":[{"pgid":"9.3","version":"980'13302","reported_seq":13528,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:05.566560+0000","last_change":"2024-06-27T21:51:47.047679+0000","last_active":"2024-06-27T21:52:05.566560+0000","last_peered":"2024-06-27T21:52:05.566560+0000","last_clean":"2024-06-27T21:52:05.566560+0000","last_became_active":"2024-06-27T21:48:11.986220+0000","last_became_peered":"2024-06-27T21:48:11.986220+0000","last_unstale":"2024-06-27T21:52:05.566560+0000","last_undegraded":"2024-06-27T21:52:05.566560+0000","last_fullsized":"2024-06-27T21:52:05.566560+0000","mapping_epoch":916,"log_start":"940'3300","ondisk_log_start":"940'3300","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12308","last_scrub_stamp":"2024-06-27T21:51:47.047437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.047437+0000","objects_scrubbed":1271,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:57.425849+0000","scrub_duration":353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":264,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":434112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"21'92","reported_seq":249,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.467641+0000","last_change":"2024-06-27T21:50:27.033854+0000","last_active":"2024-06-27T21:52:03.467641+0000","last_peered":"2024-06-27T21:52:03.467641+0000","last_clean":"2024-06-27T21:52:03.467641+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:52:03.467641+0000","last_undegraded":"2024-06-27T21:52:03.467641+0000","last_fullsized":"2024-06-27T21:52:03.467641+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_clean_scrub_stamp":"2024-06-27T21:50:27.033721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:07.972715+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.4","version":"980'13705","reported_seq":13934,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:08.231781+0000","last_change":"2024-06-27T21:51:48.901368+0000","last_active":"2024-06-27T21:52:08.231781+0000","last_peered":"2024-06-27T21:52:08.231781+0000","last_clean":"2024-06-27T21:52:08.231781+0000","last_became_active":"2024-06-27T21:48:12.309522+0000","last_became_peered":"2024-06-27T21:48:12.309522+0000","last_unstale":"2024-06-27T21:52:08.231781+0000","last_undegraded":"2024-06-27T21:52:08.231781+0000","last_fullsized":"2024-06-27T21:52:08.231781+0000","mapping_epoch":916,"log_start":"941'3700","ondisk_log_start":"941'3700","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12748","last_scrub_stamp":"2024-06-27T21:51:48.901198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.901198+0000","objects_scrubbed":1097,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:00.640327+0000","scrub_duration":393,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13705,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,4],"acting":[8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":223,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.463583+0000","last_change":"2024-06-27T21:51:45.285536+0000","last_active":"2024-06-27T21:52:03.463583+0000","last_peered":"2024-06-27T21:52:03.463583+0000","last_clean":"2024-06-27T21:52:03.463583+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:52:03.463583+0000","last_undegraded":"2024-06-27T21:52:03.463583+0000","last_fullsized":"2024-06-27T21:52:03.463583+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:45.285411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.285411+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:02.452953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"980'13427","reported_seq":13651,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:05.005051+0000","last_change":"2024-06-27T21:51:50.481302+0000","last_active":"2024-06-27T21:52:05.005051+0000","last_peered":"2024-06-27T21:52:05.005051+0000","last_clean":"2024-06-27T21:52:05.005051+0000","last_became_active":"2024-06-27T21:48:11.990332+0000","last_became_peered":"2024-06-27T21:48:11.990332+0000","last_unstale":"2024-06-27T21:52:05.005051+0000","last_undegraded":"2024-06-27T21:52:05.005051+0000","last_fullsized":"2024-06-27T21:52:05.005051+0000","mapping_epoch":916,"log_start":"940'3400","ondisk_log_start":"940'3400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12686","last_scrub_stamp":"2024-06-27T21:51:50.481086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:50.481086+0000","objects_scrubbed":1027,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:12.395502+0000","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18022400,"num_objects":275,"num_object_clones":0,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13427,"num_write_kb":438464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":363,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462909+0000","last_change":"2024-06-27T21:51:46.195566+0000","last_active":"2024-06-27T21:52:03.462909+0000","last_peered":"2024-06-27T21:52:03.462909+0000","last_clean":"2024-06-27T21:52:03.462909+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:52:03.462909+0000","last_undegraded":"2024-06-27T21:52:03.462909+0000","last_fullsized":"2024-06-27T21:52:03.462909+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:46.195469+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:46.195469+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:09.694112+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"980'17989","reported_seq":18362,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:05.188039+0000","last_change":"2024-06-27T21:50:59.459528+0000","last_active":"2024-06-27T21:52:05.188039+0000","last_peered":"2024-06-27T21:52:05.188039+0000","last_clean":"2024-06-27T21:52:05.188039+0000","last_became_active":"2024-06-27T21:48:12.309579+0000","last_became_peered":"2024-06-27T21:48:12.309579+0000","last_unstale":"2024-06-27T21:52:05.188039+0000","last_undegraded":"2024-06-27T21:52:05.188039+0000","last_fullsized":"2024-06-27T21:52:05.188039+0000","mapping_epoch":916,"log_start":"955'7902","ondisk_log_start":"955'7902","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'14609","last_scrub_stamp":"2024-06-27T21:50:59.459390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:50:59.459390+0000","objects_scrubbed":3664,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:08.730492+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13520,"num_write_kb":441344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,1],"acting":[10,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1132,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462016+0000","last_change":"2024-06-27T21:50:57.457002+0000","last_active":"2024-06-27T21:52:03.462016+0000","last_peered":"2024-06-27T21:52:03.462016+0000","last_clean":"2024-06-27T21:52:03.462016+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:52:03.462016+0000","last_undegraded":"2024-06-27T21:52:03.462016+0000","last_fullsized":"2024-06-27T21:52:03.462016+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:57.456869+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:50:57.456869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:07.148632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"980'13251","reported_seq":13495,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:05.210435+0000","last_change":"2024-06-27T21:51:49.291029+0000","last_active":"2024-06-27T21:52:05.210435+0000","last_peered":"2024-06-27T21:52:05.210435+0000","last_clean":"2024-06-27T21:52:05.210435+0000","last_became_active":"2024-06-27T21:48:11.989652+0000","last_became_peered":"2024-06-27T21:48:11.989652+0000","last_unstale":"2024-06-27T21:52:05.210435+0000","last_undegraded":"2024-06-27T21:52:05.210435+0000","last_fullsized":"2024-06-27T21:52:05.210435+0000","mapping_epoch":916,"log_start":"939'3200","ondisk_log_start":"939'3200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12436","last_scrub_stamp":"2024-06-27T21:51:49.290765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:49.290765+0000","objects_scrubbed":1098,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:03.695841+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17891328,"num_objects":273,"num_object_clones":0,"num_object_copies":819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13251,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1749,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.463049+0000","last_change":"2024-06-27T21:51:45.127958+0000","last_active":"2024-06-27T21:52:03.463049+0000","last_peered":"2024-06-27T21:52:03.463049+0000","last_clean":"2024-06-27T21:52:03.463049+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:52:03.463049+0000","last_undegraded":"2024-06-27T21:52:03.463049+0000","last_fullsized":"2024-06-27T21:52:03.463049+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:45.127817+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.958636+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.127817+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:55.161037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"980'13578","reported_seq":13795,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:05.533016+0000","last_change":"2024-06-27T21:51:00.166419+0000","last_active":"2024-06-27T21:52:05.533016+0000","last_peered":"2024-06-27T21:52:05.533016+0000","last_clean":"2024-06-27T21:52:05.533016+0000","last_became_active":"2024-06-27T21:48:25.035404+0000","last_became_peered":"2024-06-27T21:48:25.035404+0000","last_unstale":"2024-06-27T21:52:05.533016+0000","last_undegraded":"2024-06-27T21:52:05.533016+0000","last_fullsized":"2024-06-27T21:52:05.533016+0000","mapping_epoch":924,"log_start":"940'3500","ondisk_log_start":"940'3500","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'10253","last_scrub_stamp":"2024-06-27T21:51:00.166232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.166232+0000","objects_scrubbed":3624,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:06.132572+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301504,"num_objects":264,"num_object_clones":0,"num_object_copies":792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13578,"num_write_kb":442944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":724,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.458744+0000","last_change":"2024-06-27T21:51:19.848244+0000","last_active":"2024-06-27T21:52:03.458744+0000","last_peered":"2024-06-27T21:52:03.458744+0000","last_clean":"2024-06-27T21:52:03.458744+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:52:03.458744+0000","last_undegraded":"2024-06-27T21:52:03.458744+0000","last_fullsized":"2024-06-27T21:52:03.458744+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:19.848143+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:51:19.848143+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:33.796830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.b","version":"980'13466","reported_seq":13692,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:08.282638+0000","last_change":"2024-06-27T21:51:43.834598+0000","last_active":"2024-06-27T21:52:08.282638+0000","last_peered":"2024-06-27T21:52:08.282638+0000","last_clean":"2024-06-27T21:52:08.282638+0000","last_became_active":"2024-06-27T21:48:11.989420+0000","last_became_peered":"2024-06-27T21:48:11.989420+0000","last_unstale":"2024-06-27T21:52:08.282638+0000","last_undegraded":"2024-06-27T21:52:08.282638+0000","last_fullsized":"2024-06-27T21:52:08.282638+0000","mapping_epoch":916,"log_start":"940'3400","ondisk_log_start":"940'3400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'12245","last_scrub_stamp":"2024-06-27T21:51:43.834203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:43.834203+0000","objects_scrubbed":1361,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:54.273392+0000","scrub_duration":314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13466,"num_write_kb":435072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":1129,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.461972+0000","last_change":"2024-06-27T21:51:59.434256+0000","last_active":"2024-06-27T21:52:03.461972+0000","last_peered":"2024-06-27T21:52:03.461972+0000","last_clean":"2024-06-27T21:52:03.461972+0000","last_became_active":"2024-06-27T21:48:18.177749+0000","last_became_peered":"2024-06-27T21:48:18.177749+0000","last_unstale":"2024-06-27T21:52:03.461972+0000","last_undegraded":"2024-06-27T21:52:03.461972+0000","last_fullsized":"2024-06-27T21:52:03.461972+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:59.434153+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:24.498490+0000","last_clean_scrub_stamp":"2024-06-27T21:51:59.434153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:59.595190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"980'13340","reported_seq":13570,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:07.976422+0000","last_change":"2024-06-27T21:51:42.411592+0000","last_active":"2024-06-27T21:52:07.976422+0000","last_peered":"2024-06-27T21:52:07.976422+0000","last_clean":"2024-06-27T21:52:07.976422+0000","last_became_active":"2024-06-27T21:48:12.311072+0000","last_became_peered":"2024-06-27T21:48:12.311072+0000","last_unstale":"2024-06-27T21:52:07.976422+0000","last_undegraded":"2024-06-27T21:52:07.976422+0000","last_fullsized":"2024-06-27T21:52:07.976422+0000","mapping_epoch":916,"log_start":"940'3300","ondisk_log_start":"940'3300","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"974'12076","last_scrub_stamp":"2024-06-27T21:51:42.411418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:42.411418+0000","objects_scrubbed":1427,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:07.274916+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13340,"num_write_kb":431808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":355,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462791+0000","last_change":"2024-06-27T21:51:12.313706+0000","last_active":"2024-06-27T21:52:03.462791+0000","last_peered":"2024-06-27T21:52:03.462791+0000","last_clean":"2024-06-27T21:52:03.462791+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:52:03.462791+0000","last_undegraded":"2024-06-27T21:52:03.462791+0000","last_fullsized":"2024-06-27T21:52:03.462791+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:12.313599+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:51:12.313599+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:25.172124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.9","version":"980'13298","reported_seq":13522,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:05.176346+0000","last_change":"2024-06-27T21:51:44.838512+0000","last_active":"2024-06-27T21:52:05.176346+0000","last_peered":"2024-06-27T21:52:05.176346+0000","last_clean":"2024-06-27T21:52:05.176346+0000","last_became_active":"2024-06-27T21:48:11.997839+0000","last_became_peered":"2024-06-27T21:48:11.997839+0000","last_unstale":"2024-06-27T21:52:05.176346+0000","last_undegraded":"2024-06-27T21:52:05.176346+0000","last_fullsized":"2024-06-27T21:52:05.176346+0000","mapping_epoch":916,"log_start":"938'3200","ondisk_log_start":"938'3200","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'12266","last_scrub_stamp":"2024-06-27T21:51:44.838402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:44.838402+0000","objects_scrubbed":1333,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:02.823839+0000","scrub_duration":252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19660800,"num_objects":300,"num_object_clones":0,"num_object_copies":900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13298,"num_write_kb":435136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":353,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.721839+0000","last_change":"2024-06-27T21:51:03.290233+0000","last_active":"2024-06-27T21:52:03.721839+0000","last_peered":"2024-06-27T21:52:03.721839+0000","last_clean":"2024-06-27T21:52:03.721839+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:52:03.721839+0000","last_undegraded":"2024-06-27T21:52:03.721839+0000","last_fullsized":"2024-06-27T21:52:03.721839+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.290132+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.290132+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:05.358928+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"980'13641","reported_seq":13879,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:07.940139+0000","last_change":"2024-06-27T21:51:08.944383+0000","last_active":"2024-06-27T21:52:07.940139+0000","last_peered":"2024-06-27T21:52:07.940139+0000","last_clean":"2024-06-27T21:52:07.940139+0000","last_became_active":"2024-06-27T21:48:24.958699+0000","last_became_peered":"2024-06-27T21:48:24.958699+0000","last_unstale":"2024-06-27T21:52:07.940139+0000","last_undegraded":"2024-06-27T21:52:07.940139+0000","last_fullsized":"2024-06-27T21:52:07.940139+0000","mapping_epoch":924,"log_start":"941'3600","ondisk_log_start":"941'3600","created":916,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'10625","last_scrub_stamp":"2024-06-27T21:51:08.944161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:08.944161+0000","objects_scrubbed":3179,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:17.968158+0000","scrub_duration":1022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13641,"num_write_kb":440704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1456,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462187+0000","last_change":"2024-06-27T21:51:03.754036+0000","last_active":"2024-06-27T21:52:03.462187+0000","last_peered":"2024-06-27T21:52:03.462187+0000","last_clean":"2024-06-27T21:52:03.462187+0000","last_became_active":"2024-06-27T21:48:18.177860+0000","last_became_peered":"2024-06-27T21:48:18.177860+0000","last_unstale":"2024-06-27T21:52:03.462187+0000","last_undegraded":"2024-06-27T21:52:03.462187+0000","last_fullsized":"2024-06-27T21:52:03.462187+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.753918+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.753918+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:26.192419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"980'13436","reported_seq":13669,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.975460+0000","last_change":"2024-06-27T21:51:21.024373+0000","last_active":"2024-06-27T21:52:03.975460+0000","last_peered":"2024-06-27T21:52:03.975460+0000","last_clean":"2024-06-27T21:52:03.975460+0000","last_became_active":"2024-06-27T21:48:12.309446+0000","last_became_peered":"2024-06-27T21:48:12.309446+0000","last_unstale":"2024-06-27T21:52:03.975460+0000","last_undegraded":"2024-06-27T21:52:03.975460+0000","last_fullsized":"2024-06-27T21:52:03.975460+0000","mapping_epoch":916,"log_start":"940'3400","ondisk_log_start":"940'3400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'11364","last_scrub_stamp":"2024-06-27T21:51:21.024222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:21.024222+0000","objects_scrubbed":2436,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:32.134880+0000","scrub_duration":774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13436,"num_write_kb":441216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1530,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462167+0000","last_change":"2024-06-27T21:51:47.688240+0000","last_active":"2024-06-27T21:52:03.462167+0000","last_peered":"2024-06-27T21:52:03.462167+0000","last_clean":"2024-06-27T21:52:03.462167+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:52:03.462167+0000","last_undegraded":"2024-06-27T21:52:03.462167+0000","last_fullsized":"2024-06-27T21:52:03.462167+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:47.687934+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.687934+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:53.007398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"980'15017","reported_seq":15318,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.970943+0000","last_change":"2024-06-27T21:51:45.729860+0000","last_active":"2024-06-27T21:52:03.970943+0000","last_peered":"2024-06-27T21:52:03.970943+0000","last_clean":"2024-06-27T21:52:03.970943+0000","last_became_active":"2024-06-27T21:48:12.309793+0000","last_became_peered":"2024-06-27T21:48:12.309793+0000","last_unstale":"2024-06-27T21:52:03.970943+0000","last_undegraded":"2024-06-27T21:52:03.970943+0000","last_fullsized":"2024-06-27T21:52:03.970943+0000","mapping_epoch":916,"log_start":"945'5000","ondisk_log_start":"945'5000","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"975'14053","last_scrub_stamp":"2024-06-27T21:51:45.729739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.729739+0000","objects_scrubbed":1302,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:47.931619+0000","scrub_duration":531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102616,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13507,"num_write_kb":442497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1550,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462039+0000","last_change":"2024-06-27T21:50:56.782584+0000","last_active":"2024-06-27T21:52:03.462039+0000","last_peered":"2024-06-27T21:52:03.462039+0000","last_clean":"2024-06-27T21:52:03.462039+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:52:03.462039+0000","last_undegraded":"2024-06-27T21:52:03.462039+0000","last_fullsized":"2024-06-27T21:52:03.462039+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.782510+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.782510+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.914888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.c","version":"980'13460","reported_seq":13702,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:07.262702+0000","last_change":"2024-06-27T21:51:00.997686+0000","last_active":"2024-06-27T21:52:07.262702+0000","last_peered":"2024-06-27T21:52:07.262702+0000","last_clean":"2024-06-27T21:52:07.262702+0000","last_became_active":"2024-06-27T21:48:11.986349+0000","last_became_peered":"2024-06-27T21:48:11.986349+0000","last_unstale":"2024-06-27T21:52:07.262702+0000","last_undegraded":"2024-06-27T21:52:07.262702+0000","last_fullsized":"2024-06-27T21:52:07.262702+0000","mapping_epoch":916,"log_start":"940'3400","ondisk_log_start":"940'3400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'10075","last_scrub_stamp":"2024-06-27T21:51:00.997428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.997428+0000","objects_scrubbed":3581,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:29.403739+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":174,"num_object_clones":0,"num_object_copies":522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13460,"num_write_kb":436288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":251,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.467760+0000","last_change":"2024-06-27T21:51:01.076257+0000","last_active":"2024-06-27T21:52:03.467760+0000","last_peered":"2024-06-27T21:52:03.467760+0000","last_clean":"2024-06-27T21:52:03.467760+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:52:03.467760+0000","last_undegraded":"2024-06-27T21:52:03.467760+0000","last_fullsized":"2024-06-27T21:52:03.467760+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:01.076151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:01.076151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:26.255270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.2","version":"980'13407","reported_seq":13644,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:07.294019+0000","last_change":"2024-06-27T21:51:00.090300+0000","last_active":"2024-06-27T21:52:07.294019+0000","last_peered":"2024-06-27T21:52:07.294019+0000","last_clean":"2024-06-27T21:52:07.294019+0000","last_became_active":"2024-06-27T21:48:12.309342+0000","last_became_peered":"2024-06-27T21:48:12.309342+0000","last_unstale":"2024-06-27T21:52:07.294019+0000","last_undegraded":"2024-06-27T21:52:07.294019+0000","last_fullsized":"2024-06-27T21:52:07.294019+0000","mapping_epoch":916,"log_start":"940'3400","ondisk_log_start":"940'3400","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"966'10080","last_scrub_stamp":"2024-06-27T21:51:00.090159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.090159+0000","objects_scrubbed":3520,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:20.197934+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13407,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":152,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.458575+0000","last_change":"2024-06-27T21:51:48.643238+0000","last_active":"2024-06-27T21:52:03.458575+0000","last_peered":"2024-06-27T21:52:03.458575+0000","last_clean":"2024-06-27T21:52:03.458575+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:52:03.458575+0000","last_undegraded":"2024-06-27T21:52:03.458575+0000","last_fullsized":"2024-06-27T21:52:03.458575+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.643018+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.643018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:53.260819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"980'17023","reported_seq":17440,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:07.280151+0000","last_change":"2024-06-27T21:51:03.946606+0000","last_active":"2024-06-27T21:52:07.280151+0000","last_peered":"2024-06-27T21:52:07.280151+0000","last_clean":"2024-06-27T21:52:07.280151+0000","last_became_active":"2024-06-27T21:48:12.198416+0000","last_became_peered":"2024-06-27T21:48:12.198416+0000","last_unstale":"2024-06-27T21:52:07.280151+0000","last_undegraded":"2024-06-27T21:52:07.280151+0000","last_fullsized":"2024-06-27T21:52:07.280151+0000","mapping_epoch":916,"log_start":"948'7000","ondisk_log_start":"948'7000","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'13817","last_scrub_stamp":"2024-06-27T21:51:03.946497+0000","last_deep_scrub":"950'9834","last_deep_scrub_stamp":"2024-06-27T21:49:46.860713+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.946497+0000","objects_scrubbed":3412,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:07.343961+0000","scrub_duration":685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13513,"num_write_kb":438400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":363,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462576+0000","last_change":"2024-06-27T21:51:09.231702+0000","last_active":"2024-06-27T21:52:03.462576+0000","last_peered":"2024-06-27T21:52:03.462576+0000","last_clean":"2024-06-27T21:52:03.462576+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:52:03.462576+0000","last_undegraded":"2024-06-27T21:52:03.462576+0000","last_fullsized":"2024-06-27T21:52:03.462576+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:09.231581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:09.231581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:30.005850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"980'13364","reported_seq":13606,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:04.968154+0000","last_change":"2024-06-27T21:51:15.994687+0000","last_active":"2024-06-27T21:52:04.968154+0000","last_peered":"2024-06-27T21:52:04.968154+0000","last_clean":"2024-06-27T21:52:04.968154+0000","last_became_active":"2024-06-27T21:48:12.309089+0000","last_became_peered":"2024-06-27T21:48:12.309089+0000","last_unstale":"2024-06-27T21:52:04.968154+0000","last_undegraded":"2024-06-27T21:52:04.968154+0000","last_fullsized":"2024-06-27T21:52:04.968154+0000","mapping_epoch":916,"log_start":"939'3300","ondisk_log_start":"939'3300","created":916,"last_epoch_clean":917,"parent":"0.0","parent_split_bits":0,"last_scrub":"969'10976","last_scrub_stamp":"2024-06-27T21:51:15.994328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:48:10.967611+0000","last_clean_scrub_stamp":"2024-06-27T21:51:15.994328+0000","objects_scrubbed":2685,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:45.450916+0000","scrub_duration":831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":282,"num_object_clones":0,"num_object_copies":846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13364,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,6],"acting":[11,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1437,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.722513+0000","last_change":"2024-06-27T21:51:01.907123+0000","last_active":"2024-06-27T21:52:03.722513+0000","last_peered":"2024-06-27T21:52:03.722513+0000","last_clean":"2024-06-27T21:52:03.722513+0000","last_became_active":"2024-06-27T21:47:59.079888+0000","last_became_peered":"2024-06-27T21:47:59.079888+0000","last_unstale":"2024-06-27T21:52:03.722513+0000","last_undegraded":"2024-06-27T21:52:03.722513+0000","last_fullsized":"2024-06-27T21:52:03.722513+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:01.907039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:01.907039+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:13.284657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":176,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462942+0000","last_change":"2024-06-27T21:51:00.903339+0000","last_active":"2024-06-27T21:52:03.462942+0000","last_peered":"2024-06-27T21:52:03.462942+0000","last_clean":"2024-06-27T21:52:03.462942+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:52:03.462942+0000","last_undegraded":"2024-06-27T21:52:03.462942+0000","last_fullsized":"2024-06-27T21:52:03.462942+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:00.903223+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:51:00.903223+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1143,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.721775+0000","last_change":"2024-06-27T21:51:48.107650+0000","last_active":"2024-06-27T21:52:03.721775+0000","last_peered":"2024-06-27T21:52:03.721775+0000","last_clean":"2024-06-27T21:52:03.721775+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:52:03.721775+0000","last_undegraded":"2024-06-27T21:52:03.721775+0000","last_fullsized":"2024-06-27T21:52:03.721775+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.107530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.107530+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:50.921431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1170,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.463311+0000","last_change":"2024-06-27T21:51:51.181305+0000","last_active":"2024-06-27T21:52:03.463311+0000","last_peered":"2024-06-27T21:52:03.463311+0000","last_clean":"2024-06-27T21:52:03.463311+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:52:03.463311+0000","last_undegraded":"2024-06-27T21:52:03.463311+0000","last_fullsized":"2024-06-27T21:52:03.463311+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:51.181073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:51.181073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:00.572984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":251,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.467592+0000","last_change":"2024-06-27T21:51:59.040911+0000","last_active":"2024-06-27T21:52:03.467592+0000","last_peered":"2024-06-27T21:52:03.467592+0000","last_clean":"2024-06-27T21:52:03.467592+0000","last_became_active":"2024-06-27T21:46:34.530850+0000","last_became_peered":"2024-06-27T21:46:34.530850+0000","last_unstale":"2024-06-27T21:52:03.467592+0000","last_undegraded":"2024-06-27T21:52:03.467592+0000","last_fullsized":"2024-06-27T21:52:03.467592+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:59.040816+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.946926+0000","last_clean_scrub_stamp":"2024-06-27T21:51:59.040816+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:26.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1142,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.722534+0000","last_change":"2024-06-27T21:51:48.835296+0000","last_active":"2024-06-27T21:52:03.722534+0000","last_peered":"2024-06-27T21:52:03.722534+0000","last_clean":"2024-06-27T21:52:03.722534+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:52:03.722534+0000","last_undegraded":"2024-06-27T21:52:03.722534+0000","last_fullsized":"2024-06-27T21:52:03.722534+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.835191+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.835191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:07.619688+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":362,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462791+0000","last_change":"2024-06-27T21:51:18.316912+0000","last_active":"2024-06-27T21:52:03.462791+0000","last_peered":"2024-06-27T21:52:03.462791+0000","last_clean":"2024-06-27T21:52:03.462791+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:52:03.462791+0000","last_undegraded":"2024-06-27T21:52:03.462791+0000","last_fullsized":"2024-06-27T21:52:03.462791+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:18.316784+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:18.316784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.874908+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":383,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.467239+0000","last_change":"2024-06-27T21:50:56.873549+0000","last_active":"2024-06-27T21:52:03.467239+0000","last_peered":"2024-06-27T21:52:03.467239+0000","last_clean":"2024-06-27T21:52:03.467239+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:52:03.467239+0000","last_undegraded":"2024-06-27T21:52:03.467239+0000","last_fullsized":"2024-06-27T21:52:03.467239+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.873431+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.873431+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:21.297054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":798,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.463792+0000","last_change":"2024-06-27T21:50:57.612741+0000","last_active":"2024-06-27T21:52:03.463792+0000","last_peered":"2024-06-27T21:52:03.463792+0000","last_clean":"2024-06-27T21:52:03.463792+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:52:03.463792+0000","last_undegraded":"2024-06-27T21:52:03.463792+0000","last_fullsized":"2024-06-27T21:52:03.463792+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:57.612612+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:50:57.612612+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:22.461196+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":180,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462932+0000","last_change":"2024-06-27T21:51:09.954221+0000","last_active":"2024-06-27T21:52:03.462932+0000","last_peered":"2024-06-27T21:52:03.462932+0000","last_clean":"2024-06-27T21:52:03.462932+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:52:03.462932+0000","last_undegraded":"2024-06-27T21:52:03.462932+0000","last_fullsized":"2024-06-27T21:52:03.462932+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:09.954099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:09.954099+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:25.431565+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":992,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.462072+0000","last_change":"2024-06-27T21:51:51.683171+0000","last_active":"2024-06-27T21:52:03.462072+0000","last_peered":"2024-06-27T21:52:03.462072+0000","last_clean":"2024-06-27T21:52:03.462072+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:52:03.462072+0000","last_undegraded":"2024-06-27T21:52:03.462072+0000","last_fullsized":"2024-06-27T21:52:03.462072+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:51.683075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:51.683075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:20.402839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1087,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.461962+0000","last_change":"2024-06-27T21:51:47.411923+0000","last_active":"2024-06-27T21:52:03.461962+0000","last_peered":"2024-06-27T21:52:03.461962+0000","last_clean":"2024-06-27T21:52:03.461962+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:52:03.461962+0000","last_undegraded":"2024-06-27T21:52:03.461962+0000","last_fullsized":"2024-06-27T21:52:03.461962+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:47.411842+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.411842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:15.992157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":781,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.463787+0000","last_change":"2024-06-27T21:51:04.535580+0000","last_active":"2024-06-27T21:52:03.463787+0000","last_peered":"2024-06-27T21:52:03.463787+0000","last_clean":"2024-06-27T21:52:03.463787+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:52:03.463787+0000","last_undegraded":"2024-06-27T21:52:03.463787+0000","last_fullsized":"2024-06-27T21:52:03.463787+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:04.535471+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:51:04.535471+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:05.628576+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":799,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.463933+0000","last_change":"2024-06-27T21:50:56.571158+0000","last_active":"2024-06-27T21:52:03.463933+0000","last_peered":"2024-06-27T21:52:03.463933+0000","last_clean":"2024-06-27T21:52:03.463933+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:52:03.463933+0000","last_undegraded":"2024-06-27T21:52:03.463933+0000","last_fullsized":"2024-06-27T21:52:03.463933+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.571030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.571030+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:06.704244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":251,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.467583+0000","last_change":"2024-06-27T21:51:03.119223+0000","last_active":"2024-06-27T21:52:03.467583+0000","last_peered":"2024-06-27T21:52:03.467583+0000","last_clean":"2024-06-27T21:52:03.467583+0000","last_became_active":"2024-06-27T21:46:34.528582+0000","last_became_peered":"2024-06-27T21:46:34.528582+0000","last_unstale":"2024-06-27T21:52:03.467583+0000","last_undegraded":"2024-06-27T21:52:03.467583+0000","last_fullsized":"2024-06-27T21:52:03.467583+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.119099+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.119099+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:19.502832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":251,"reported_epoch":980,"state":"active+clean","last_fresh":"2024-06-27T21:52:03.467584+0000","last_change":"2024-06-27T21:51:49.181612+0000","last_active":"2024-06-27T21:52:03.467584+0000","last_peered":"2024-06-27T21:52:03.467584+0000","last_clean":"2024-06-27T21:52:03.467584+0000","last_became_active":"2024-06-27T21:46:34.528359+0000","last_became_peered":"2024-06-27T21:46:34.528359+0000","last_unstale":"2024-06-27T21:52:03.467584+0000","last_undegraded":"2024-06-27T21:52:03.467584+0000","last_fullsized":"2024-06-27T21:52:03.467584+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:49.181498+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:51:49.181498+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:59.400634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":241172504,"num_objects":3681,"num_object_clones":0,"num_object_copies":11043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215215,"num_write_kb":7004609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":788213760,"data_stored":788201544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160702,"ondisk_log_size":160702,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253380,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121836,"kb_used_data":75808,"kb_used_omap":1,"kb_used_meta":1046014,"kb_avail":92623316,"statfs":{"total":95995035648,"available":94846275584,"internally_reserved":0,"allocated":77627392,"data_stored":77357765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1071118789},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417515015,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094828,"kb_used_data":98784,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92650324,"statfs":{"total":95995035648,"available":94873931776,"internally_reserved":0,"allocated":101154816,"data_stored":100885189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1019935168},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743948,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887212,"kb_used_data":111904,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92857940,"statfs":{"total":95995035648,"available":95086530560,"internally_reserved":0,"allocated":114589696,"data_stored":114320069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":793901504},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972880,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624808,"kb_used_data":20256,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93120344,"statfs":{"total":95995035648,"available":95355232256,"internally_reserved":0,"allocated":20742144,"data_stored":20472517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267215,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967292,"kb_used_data":119648,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92777860,"statfs":{"total":95995035648,"available":95004528640,"internally_reserved":0,"allocated":122519552,"data_stored":122246733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":867826122},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789356,"kb_used_data":36256,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92955796,"statfs":{"total":95995035648,"available":95186735104,"internally_reserved":0,"allocated":37126144,"data_stored":36856517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":771160518},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757791,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840628,"kb_used_data":109676,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92904524,"statfs":{"total":95995035648,"available":95134232576,"internally_reserved":0,"allocated":112308224,"data_stored":112030973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161497,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630556,"kb_used_data":51732,"kb_used_omap":92,"kb_used_meta":578723,"kb_avail":93114596,"statfs":{"total":95995035648,"available":95349346304,"internally_reserved":0,"allocated":52973568,"data_stored":52708373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":592612450},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182951,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730920,"kb_used_data":59744,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93014232,"statfs":{"total":95995035648,"available":95246573568,"internally_reserved":0,"allocated":61177856,"data_stored":60908229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":687274439},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411884,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808264,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936888,"statfs":{"total":95995035648,"available":95167373312,"internally_reserved":0,"allocated":892928,"data_stored":619749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575410,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070828,"kb_used_data":169188,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92674324,"statfs":{"total":95995035648,"available":94898507776,"internally_reserved":0,"allocated":173248512,"data_stored":172974813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738935,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880428,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864724,"statfs":{"total":95995035648,"available":95093477376,"internally_reserved":0,"allocated":1015808,"data_stored":746181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170266624,"data_stored":170262552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58982400,"data_stored":58982400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44957696,"data_stored":44957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90443776,"data_stored":90439704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32899072,"data_stored":32899072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":106565632,"data_stored":106561560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16842752,"data_stored":16842752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":100073472,"data_stored":100073472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":91160576,"data_stored":91160576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":76021760,"data_stored":76021760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:52:11.337 INFO:tasks.thrashosds.thrasher:Setting 1.1e to [7, 6, 10, 11, 8, 0] 2024-06-27T21:52:11.337 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.1e', '7', '6', '10', '11', '8', '0'] 2024-06-27T21:52:11.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pg-upmap-items 1.1e 7 6 10 11 8 0 2024-06-27T21:52:11.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:11.440 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:52:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:52:11.440 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:52:11.441 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:52:11.441 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:52:11.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:11.463 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:11.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:11.586 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:11.643 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Removed 109447 objects 2024-06-27T21:52:11.643 INFO:tasks.radosbench.radosbench.2.smithi192.stdout:Clean up completed and total clean up time :135.723 2024-06-27T21:52:11.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: from='client.18060 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: pgmap v2422: 47 pgs: 47 active+clean; 124 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/335225820' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: osdmap e981: 12 total, 12 up, 12 in 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:11.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:11.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:11.699 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:11 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:11.712 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: from='client.18060 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: pgmap v2422: 47 pgs: 47 active+clean; 124 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/335225820' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: osdmap e981: 12 total, 12 up, 12 in 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:11.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:11 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:11.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: from='client.18060 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:52:11.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: pgmap v2422: 47 pgs: 47 active+clean; 124 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/335225820' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: osdmap e981: 12 total, 12 up, 12 in 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:11.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:11 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:11.814 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:11.835 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:11.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:11.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:11.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:12.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:12.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:12.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-06-27T21:52:12.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:12.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:12.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:12.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:12.351 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:12.436 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:12.458 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:12.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:12.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:12.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:12.709 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:12.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:12 smithi190 ceph-mon[32247]: osdmap e982: 12 total, 12 up, 12 in 2024-06-27T21:52:12.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:12 smithi192 ceph-mon[32956]: osdmap e982: 12 total, 12 up, 12 in 2024-06-27T21:52:12.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:12.834 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:12.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:12 smithi151 ceph-mon[28965]: osdmap e982: 12 total, 12 up, 12 in 2024-06-27T21:52:12.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:12.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:12.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:13.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:13.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:13.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:13.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:13.280 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:13.296 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:52:13.297 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:52:13.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:13.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:52:13.322 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:52:13.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:13.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:52:13.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:52:13.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:52:13.361 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:52:13.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:52:13.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:52:13.404 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:52:13.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:52:13.426 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:52:13.426 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:52:13.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:13.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:52:13.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:52:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:13.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:13.471 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:52:13.471 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:52:13.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:52:13.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:52:13.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:52:13.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:52:13.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:52:13.541 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:52:13.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:13.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:13.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:13.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:13.736 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:13 smithi192 ceph-mon[32956]: pgmap v2425: 47 pgs: 47 active+clean; 59 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-06-27T21:52:13.736 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3322707146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:13.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:13 smithi190 ceph-mon[32247]: pgmap v2425: 47 pgs: 47 active+clean; 59 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-06-27T21:52:13.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3322707146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:13.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:13.829 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:13.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:13 smithi151 ceph-mon[28965]: pgmap v2425: 47 pgs: 47 active+clean; 59 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s 2024-06-27T21:52:13.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3322707146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:13.934 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:13.960 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:13.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:14.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:14.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:14.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:14.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:14.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:14.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:14.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:14.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:14.533 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:52:14.533 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:52:14.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:14.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:14.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:14.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:14.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:14.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:14.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:14.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:14.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:15.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:52:15.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:15.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:15.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-06-27T21:52:15.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:15.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:15.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:15.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:15.391 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:15.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:15.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:15.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:15.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:15.601 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:15.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:15 smithi151 ceph-mon[28965]: pgmap v2426: 47 pgs: 47 active+clean; 13 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-06-27T21:52:15.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:15.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:15.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:15 smithi190 ceph-mon[32247]: pgmap v2426: 47 pgs: 47 active+clean; 13 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-06-27T21:52:15.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:15 smithi192 ceph-mon[32956]: pgmap v2426: 47 pgs: 47 active+clean; 13 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 636 op/s 2024-06-27T21:52:15.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:15.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:15.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:15.979 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:15.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:16.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:16.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:16.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:16.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:16.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:16.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:16.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:16.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:16.501 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.1e pg_upmap_items mapping to [7->6,10->11,8->0] 2024-06-27T21:52:16.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-06-27T21:52:16.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:16.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:16.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:16 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:16.659 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:16 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:16.660 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1896262609' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [7, 6, 10, 11, 8, 0]}]: dispatch 2024-06-27T21:52:16.674 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-06-27T21:52:16.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-06-27T21:52:16.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:16 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:16.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:16 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:16.696 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1896262609' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [7, 6, 10, 11, 8, 0]}]: dispatch 2024-06-27T21:52:16.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:16.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:16.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:16 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:16 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:16.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1896262609' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [7, 6, 10, 11, 8, 0]}]: dispatch 2024-06-27T21:52:16.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:16.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:16.880 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:52:16.880 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:52:16.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:16.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:16.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:17.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:17.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:17.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:17.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:17.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:17.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:17.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:17.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:17.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-06-27T21:52:17.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:17.644 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:17.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:17 smithi151 ceph-mon[28965]: pgmap v2427: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s 2024-06-27T21:52:17.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:17 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1896262609' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [7, 6, 10, 11, 8, 0]}]': finished 2024-06-27T21:52:17.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:17 smithi151 ceph-mon[28965]: osdmap e983: 12 total, 12 up, 12 in 2024-06-27T21:52:17.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:17 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:52:17.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:17.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:17 smithi190 ceph-mon[32247]: pgmap v2427: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s 2024-06-27T21:52:17.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:17 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1896262609' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [7, 6, 10, 11, 8, 0]}]': finished 2024-06-27T21:52:17.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:17 smithi190 ceph-mon[32247]: osdmap e983: 12 total, 12 up, 12 in 2024-06-27T21:52:17.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:17 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:52:17.766 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:17.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:17 smithi192 ceph-mon[32956]: pgmap v2427: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 460 op/s 2024-06-27T21:52:17.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:17 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1896262609' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1e", "id": [7, 6, 10, 11, 8, 0]}]': finished 2024-06-27T21:52:17.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:17 smithi192 ceph-mon[32956]: osdmap e983: 12 total, 12 up, 12 in 2024-06-27T21:52:17.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:17 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:52:17.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:17.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:17.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:17.978 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:17.983 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:17.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:18.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:18.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:18.135 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:18.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:18.258 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:18.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:18.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:18.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:18.519 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:52:18.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:18 smithi192 ceph-mon[32956]: osdmap e984: 12 total, 12 up, 12 in 2024-06-27T21:52:18.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:18 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3671701186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:52:18.584 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:18.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:18.629 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:18.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:18 smithi151 ceph-mon[28965]: osdmap e984: 12 total, 12 up, 12 in 2024-06-27T21:52:18.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:18 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3671701186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:52:18.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:18.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:18.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:18 smithi190 ceph-mon[32247]: osdmap e984: 12 total, 12 up, 12 in 2024-06-27T21:52:18.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:18 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3671701186' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:52:18.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:18.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:18.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:18.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:18.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:19.000 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:19.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:19.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:19.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:19.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:19.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:19.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:19.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:19.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:19.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:19.620 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:19.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:19.729 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:19 smithi192 ceph-mon[32956]: pgmap v2430: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 211 op/s 2024-06-27T21:52:19.729 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:19 smithi192 ceph-mon[32956]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:19.729 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3671701186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:52:19.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:19 smithi192 ceph-mon[32956]: osdmap e985: 12 total, 12 up, 12 in 2024-06-27T21:52:19.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:19 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:19.730 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:19 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:19.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:19.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.869 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:19.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:19 smithi151 ceph-mon[28965]: pgmap v2430: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 211 op/s 2024-06-27T21:52:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:19 smithi151 ceph-mon[28965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3671701186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:52:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:19 smithi151 ceph-mon[28965]: osdmap e985: 12 total, 12 up, 12 in 2024-06-27T21:52:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:19 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:19.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:19 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:19.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4 2024-06-27T21:52:19.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:19.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.4' 2024-06-27T21:52:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:19 smithi190 ceph-mon[32247]: pgmap v2430: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 211 op/s 2024-06-27T21:52:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:19 smithi190 ceph-mon[32247]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-06-27T21:52:20.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3671701186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:52:20.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:19 smithi190 ceph-mon[32247]: osdmap e985: 12 total, 12 up, 12 in 2024-06-27T21:52:20.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:19 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:20.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:19 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:20.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0 2024-06-27T21:52:20.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.0' 2024-06-27T21:52:20.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:20.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:20.292 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:52:20.292 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:52:20.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:20.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:20.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11 2024-06-27T21:52:20.485 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.11' 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:52:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:20.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:52:20.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:52:20.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:52:20.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:20 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:20.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:20.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:20.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:20.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:20.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:52:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:52:20.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:52:20.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:20 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:20.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:20.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-06-27T21:52:20.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7 2024-06-27T21:52:20.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:20.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.7' 2024-06-27T21:52:21.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:52:21.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:52:21.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:20 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:52:21.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:21.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:21.117 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:21.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:21.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:21.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:21.356 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:21.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6 2024-06-27T21:52:21.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.6' 2024-06-27T21:52:21.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:21.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:21 smithi151 ceph-mon[28965]: 9.f scrub starts 2024-06-27T21:52:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:21 smithi151 ceph-mon[28965]: 9.f scrub ok 2024-06-27T21:52:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:21 smithi151 ceph-mon[28965]: pgmap v2432: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 32 op/s 2024-06-27T21:52:21.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:21 smithi151 ceph-mon[28965]: osdmap e986: 12 total, 12 up, 12 in 2024-06-27T21:52:21.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:21.731 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:21.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:21.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:21 smithi192 ceph-mon[32956]: 9.f scrub starts 2024-06-27T21:52:21.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:21 smithi192 ceph-mon[32956]: 9.f scrub ok 2024-06-27T21:52:21.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:21 smithi192 ceph-mon[32956]: pgmap v2432: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 32 op/s 2024-06-27T21:52:21.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:21 smithi192 ceph-mon[32956]: osdmap e986: 12 total, 12 up, 12 in 2024-06-27T21:52:21.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:21.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:21.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:21.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:21.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:22.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:21 smithi190 ceph-mon[32247]: 9.f scrub starts 2024-06-27T21:52:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:21 smithi190 ceph-mon[32247]: 9.f scrub ok 2024-06-27T21:52:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:21 smithi190 ceph-mon[32247]: pgmap v2432: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 32 op/s 2024-06-27T21:52:22.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:21 smithi190 ceph-mon[32247]: osdmap e986: 12 total, 12 up, 12 in 2024-06-27T21:52:22.078 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.100 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:22.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:22.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:22.228 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:22.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9 2024-06-27T21:52:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.352 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:22.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.9' 2024-06-27T21:52:22.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:22.460 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 9, 10, 2, 8, 1, 7, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 4] 2024-06-27T21:52:22.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-06-27T21:52:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:22.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:22.532 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_7' removed 2024-06-27T21:52:22.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-06-27T21:52:22.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-06-27T21:52:22.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1 2024-06-27T21:52:22.600 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-06-27T21:52:22.600 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-06-27T21:52:22.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.1' 2024-06-27T21:52:22.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-06-27T21:52:22.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-06-27T21:52:22.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-06-27T21:52:22.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-06-27T21:52:22.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-06-27T21:52:22.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-06-27T21:52:22.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-06-27T21:52:22.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-06-27T21:52:22.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:52:22.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5 2024-06-27T21:52:22.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-06-27T21:52:22.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-06-27T21:52:22.728 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-06-27T21:52:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.5' 2024-06-27T21:52:22.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-06-27T21:52:22.733 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-06-27T21:52:22.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-06-27T21:52:22.755 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-06-27T21:52:22.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-06-27T21:52:22.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-06-27T21:52:22.792 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-06-27T21:52:22.792 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-06-27T21:52:22.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1946070978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:22.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2720831527' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:52:22.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:22 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:52:22.792 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:22 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:52:22.800 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-06-27T21:52:22.800 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-06-27T21:52:22.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1946070978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:22.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2720831527' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:52:22.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:22 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:52:22.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:22 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:52:22.821 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-06-27T21:52:22.821 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-06-27T21:52:22.829 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:22.851 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:22.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2 2024-06-27T21:52:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:22.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:22.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.2' 2024-06-27T21:52:23.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1946070978' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-06-27T21:52:23.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2720831527' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-27T21:52:23.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:22 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:52:23.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:22 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:52:23.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10 2024-06-27T21:52:23.097 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:23.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.10' 2024-06-27T21:52:23.199 DEBUG:teuthology.orchestra.run.smithi192:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8 2024-06-27T21:52:23.221 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: docker: command not found 2024-06-27T21:52:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:23.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi192 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.8' 2024-06-27T21:52:23.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3 2024-06-27T21:52:23.347 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-06-27T21:52:23.357 INFO:tasks.thrashosds:joining thrashosds 2024-06-27T21:52:23.357 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-06-27T21:52:23.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:52:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T21:52:23.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-06-27T21:52:23.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd.3' 2024-06-27T21:52:23.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-06-27T21:52:23.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:52:23.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:52:23.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: pgmap v2434: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:23.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2720831527' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:52:23.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: osdmap e987: 12 total, 12 up, 12 in 2024-06-27T21:52:23.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:52:23.649 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:23 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: pgmap v2434: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2720831527' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: osdmap e987: 12 total, 12 up, 12 in 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:52:23.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:23 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:52:23.795 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:24.005 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:24.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:52:24.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:52:24.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: pgmap v2434: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:24.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2720831527' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-06-27T21:52:24.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: osdmap e987: 12 total, 12 up, 12 in 2024-06-27T21:52:24.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:52:24.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:23 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:52:24.716 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:24.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:24.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:24.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:24.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:25.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:25.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:25.406 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:52:25.407 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:52:25.407 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-06-27T21:52:25.407 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:52:25.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:25 smithi192 ceph-mon[32956]: pgmap v2436: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:25.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:25 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:52:25.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:25 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:52:25.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:25 smithi192 ceph-mon[32956]: osdmap e988: 12 total, 12 up, 12 in 2024-06-27T21:52:25.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:25 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:52:25.795 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:25 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:52:25.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:25 smithi151 ceph-mon[28965]: pgmap v2436: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:25.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:25 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:52:25.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:25 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:52:25.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:25 smithi151 ceph-mon[28965]: osdmap e988: 12 total, 12 up, 12 in 2024-06-27T21:52:25.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:25 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:52:25.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:25 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:52:26.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:25 smithi190 ceph-mon[32247]: pgmap v2436: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:26.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:25 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:52:26.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:25 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:52:26.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:25 smithi190 ceph-mon[32247]: osdmap e988: 12 total, 12 up, 12 in 2024-06-27T21:52:26.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:25 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:52:26.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:25 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:52:26.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-06-27T21:52:26.315 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:26.720 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:26 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:52:26.720 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:26 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:52:26.720 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:26 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:52:26.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:26 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:52:26.726 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:26 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:52:26.726 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:26 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:52:26.726 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:26 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:52:26.727 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:26 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:52:27.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:26 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:52:27.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:26 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:52:27.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:26 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:52:27.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:26 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:52:27.155 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:52:27.156 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:52:27.555 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:52:27.674 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: pgmap v2438: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:52:27.675 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3660043532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:27.927 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:52:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-06-27T21:52:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-06-27T21:52:27.927 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-06-27T21:52:27.927 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:52:27.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: pgmap v2438: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:52:27.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3660043532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: pgmap v2438: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:52:28.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3660043532' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:28.475 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2438,"stamp":"2024-06-27T21:52:26.394677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9656564,"kb_used_data":10640,"kb_used_omap":110,"kb_used_meta":9645649,"kb_avail":1115285260,"statfs":{"total":1151940427776,"available":1142052106240,"internally_reserved":0,"allocated":10895360,"data_stored":7254008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9877145559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-218896,"num_write_kb":-7004609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001370"},"pg_stats":[{"pgid":"1.b","version":"21'92","reported_seq":263,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.592046+0000","last_change":"2024-06-27T21:50:27.033854+0000","last_active":"2024-06-27T21:52:22.592046+0000","last_peered":"2024-06-27T21:52:22.592046+0000","last_clean":"2024-06-27T21:52:22.592046+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:52:22.592046+0000","last_undegraded":"2024-06-27T21:52:22.592046+0000","last_fullsized":"2024-06-27T21:52:22.592046+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_clean_scrub_stamp":"2024-06-27T21:50:27.033721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":235,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:21.005962+0000","last_change":"2024-06-27T21:51:45.285536+0000","last_active":"2024-06-27T21:52:21.005962+0000","last_peered":"2024-06-27T21:52:21.005962+0000","last_clean":"2024-06-27T21:52:21.005962+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:52:21.005962+0000","last_undegraded":"2024-06-27T21:52:21.005962+0000","last_fullsized":"2024-06-27T21:52:21.005962+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:45.285411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.285411+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:02.452953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":377,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.983404+0000","last_change":"2024-06-27T21:51:46.195566+0000","last_active":"2024-06-27T21:52:22.983404+0000","last_peered":"2024-06-27T21:52:22.983404+0000","last_clean":"2024-06-27T21:52:22.983404+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:52:22.983404+0000","last_undegraded":"2024-06-27T21:52:22.983404+0000","last_fullsized":"2024-06-27T21:52:22.983404+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:46.195469+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:46.195469+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:09.694112+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1147,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.534464+0000","last_change":"2024-06-27T21:50:57.457002+0000","last_active":"2024-06-27T21:52:22.534464+0000","last_peered":"2024-06-27T21:52:22.534464+0000","last_clean":"2024-06-27T21:52:22.534464+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:52:22.534464+0000","last_undegraded":"2024-06-27T21:52:22.534464+0000","last_fullsized":"2024-06-27T21:52:22.534464+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:57.456869+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:50:57.456869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":1761,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:21.006956+0000","last_change":"2024-06-27T21:51:45.127958+0000","last_active":"2024-06-27T21:52:21.006956+0000","last_peered":"2024-06-27T21:52:21.006956+0000","last_clean":"2024-06-27T21:52:21.006956+0000","last_became_active":"2024-06-27T21:30:30.815000+0000","last_became_peered":"2024-06-27T21:30:30.815000+0000","last_unstale":"2024-06-27T21:52:21.006956+0000","last_undegraded":"2024-06-27T21:52:21.006956+0000","last_fullsized":"2024-06-27T21:52:21.006956+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:45.127817+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.958636+0000","last_clean_scrub_stamp":"2024-06-27T21:51:45.127817+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:55.161037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":740,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-06-27T21:52:24.541391+0000","last_change":"2024-06-27T21:51:19.848244+0000","last_active":"2024-06-27T21:52:24.541391+0000","last_peered":"2024-06-27T21:52:24.541391+0000","last_clean":"2024-06-27T21:52:24.541391+0000","last_became_active":"2024-06-27T21:44:35.940325+0000","last_became_peered":"2024-06-27T21:44:35.940325+0000","last_unstale":"2024-06-27T21:52:24.541391+0000","last_undegraded":"2024-06-27T21:52:24.541391+0000","last_fullsized":"2024-06-27T21:52:24.541391+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:19.848143+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:51:19.848143+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:33.796830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":1143,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.534197+0000","last_change":"2024-06-27T21:51:59.434256+0000","last_active":"2024-06-27T21:52:22.534197+0000","last_peered":"2024-06-27T21:52:22.534197+0000","last_clean":"2024-06-27T21:52:22.534197+0000","last_became_active":"2024-06-27T21:48:18.177749+0000","last_became_peered":"2024-06-27T21:48:18.177749+0000","last_unstale":"2024-06-27T21:52:22.534197+0000","last_undegraded":"2024-06-27T21:52:22.534197+0000","last_fullsized":"2024-06-27T21:52:22.534197+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:59.434153+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:24.498490+0000","last_clean_scrub_stamp":"2024-06-27T21:51:59.434153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:59.595190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":369,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.982670+0000","last_change":"2024-06-27T21:51:12.313706+0000","last_active":"2024-06-27T21:52:22.982670+0000","last_peered":"2024-06-27T21:52:22.982670+0000","last_clean":"2024-06-27T21:52:22.982670+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:52:22.982670+0000","last_undegraded":"2024-06-27T21:52:22.982670+0000","last_fullsized":"2024-06-27T21:52:22.982670+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:12.313599+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:51:12.313599+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:25.172124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":377,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-06-27T21:52:24.744056+0000","last_change":"2024-06-27T21:52:22.171248+0000","last_active":"2024-06-27T21:52:24.744056+0000","last_peered":"2024-06-27T21:52:24.744056+0000","last_clean":"2024-06-27T21:52:24.744056+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:52:24.744056+0000","last_undegraded":"2024-06-27T21:52:24.744056+0000","last_fullsized":"2024-06-27T21:52:24.744056+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:52:22.171169+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:52:22.171169+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:23.935928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1471,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.302695+0000","last_change":"2024-06-27T21:51:03.754036+0000","last_active":"2024-06-27T21:52:23.302695+0000","last_peered":"2024-06-27T21:52:23.302695+0000","last_clean":"2024-06-27T21:52:23.302695+0000","last_became_active":"2024-06-27T21:48:18.177860+0000","last_became_peered":"2024-06-27T21:48:18.177860+0000","last_unstale":"2024-06-27T21:52:23.302695+0000","last_undegraded":"2024-06-27T21:52:23.302695+0000","last_fullsized":"2024-06-27T21:52:23.302695+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.753918+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.753918+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:26.192419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1544,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.303199+0000","last_change":"2024-06-27T21:51:47.688240+0000","last_active":"2024-06-27T21:52:23.303199+0000","last_peered":"2024-06-27T21:52:23.303199+0000","last_clean":"2024-06-27T21:52:23.303199+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:52:23.303199+0000","last_undegraded":"2024-06-27T21:52:23.303199+0000","last_fullsized":"2024-06-27T21:52:23.303199+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:47.687934+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.687934+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:53.007398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1564,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.302556+0000","last_change":"2024-06-27T21:50:56.782584+0000","last_active":"2024-06-27T21:52:23.302556+0000","last_peered":"2024-06-27T21:52:23.302556+0000","last_clean":"2024-06-27T21:52:23.302556+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:52:23.302556+0000","last_undegraded":"2024-06-27T21:52:23.302556+0000","last_fullsized":"2024-06-27T21:52:23.302556+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:56.782510+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:50:56.782510+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.914888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":265,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.592028+0000","last_change":"2024-06-27T21:51:01.076257+0000","last_active":"2024-06-27T21:52:22.592028+0000","last_peered":"2024-06-27T21:52:22.592028+0000","last_clean":"2024-06-27T21:52:22.592028+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:52:22.592028+0000","last_undegraded":"2024-06-27T21:52:22.592028+0000","last_fullsized":"2024-06-27T21:52:22.592028+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:01.076151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:01.076151+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:26.255270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":168,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-06-27T21:52:24.541436+0000","last_change":"2024-06-27T21:51:48.643238+0000","last_active":"2024-06-27T21:52:24.541436+0000","last_peered":"2024-06-27T21:52:24.541436+0000","last_clean":"2024-06-27T21:52:24.541436+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:52:24.541436+0000","last_undegraded":"2024-06-27T21:52:24.541436+0000","last_fullsized":"2024-06-27T21:52:24.541436+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.643018+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:40.622325+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.643018+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:53.260819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":377,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.982556+0000","last_change":"2024-06-27T21:51:09.231702+0000","last_active":"2024-06-27T21:52:22.982556+0000","last_peered":"2024-06-27T21:52:22.982556+0000","last_clean":"2024-06-27T21:52:22.982556+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:52:22.982556+0000","last_undegraded":"2024-06-27T21:52:22.982556+0000","last_fullsized":"2024-06-27T21:52:22.982556+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:09.231581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:09.231581+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:30.005850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1449,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:20.902993+0000","last_change":"2024-06-27T21:51:01.907123+0000","last_active":"2024-06-27T21:52:20.902993+0000","last_peered":"2024-06-27T21:52:20.902993+0000","last_clean":"2024-06-27T21:52:20.902993+0000","last_became_active":"2024-06-27T21:47:59.079888+0000","last_became_peered":"2024-06-27T21:47:59.079888+0000","last_unstale":"2024-06-27T21:52:20.902993+0000","last_undegraded":"2024-06-27T21:52:20.902993+0000","last_fullsized":"2024-06-27T21:52:20.902993+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:01.907039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:01.907039+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.824614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":196,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:21.006488+0000","last_change":"2024-06-27T21:52:19.309139+0000","last_active":"2024-06-27T21:52:21.006488+0000","last_peered":"2024-06-27T21:52:21.006488+0000","last_clean":"2024-06-27T21:52:21.006488+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:52:21.006488+0000","last_undegraded":"2024-06-27T21:52:21.006488+0000","last_fullsized":"2024-06-27T21:52:21.006488+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:52:19.309059+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:31:32.931223+0000","last_clean_scrub_stamp":"2024-06-27T21:52:19.309059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:32.307760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1159,"reported_epoch":988,"state":"active+clean","last_fresh":"2024-06-27T21:52:24.743986+0000","last_change":"2024-06-27T21:51:48.107650+0000","last_active":"2024-06-27T21:52:24.743986+0000","last_peered":"2024-06-27T21:52:24.743986+0000","last_clean":"2024-06-27T21:52:24.743986+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:52:24.743986+0000","last_undegraded":"2024-06-27T21:52:24.743986+0000","last_fullsized":"2024-06-27T21:52:24.743986+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.107530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.107530+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:50.921431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1182,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:21.006816+0000","last_change":"2024-06-27T21:51:51.181305+0000","last_active":"2024-06-27T21:52:21.006816+0000","last_peered":"2024-06-27T21:52:21.006816+0000","last_clean":"2024-06-27T21:52:21.006816+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:52:21.006816+0000","last_undegraded":"2024-06-27T21:52:21.006816+0000","last_fullsized":"2024-06-27T21:52:21.006816+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:51.181073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:51.181073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:00.572984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":265,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.591949+0000","last_change":"2024-06-27T21:51:59.040911+0000","last_active":"2024-06-27T21:52:22.591949+0000","last_peered":"2024-06-27T21:52:22.591949+0000","last_clean":"2024-06-27T21:52:22.591949+0000","last_became_active":"2024-06-27T21:46:34.530850+0000","last_became_peered":"2024-06-27T21:46:34.530850+0000","last_unstale":"2024-06-27T21:52:22.591949+0000","last_undegraded":"2024-06-27T21:52:22.591949+0000","last_fullsized":"2024-06-27T21:52:22.591949+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:59.040816+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.946926+0000","last_clean_scrub_stamp":"2024-06-27T21:51:59.040816+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:26.476473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1154,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:20.903027+0000","last_change":"2024-06-27T21:51:48.835296+0000","last_active":"2024-06-27T21:52:20.903027+0000","last_peered":"2024-06-27T21:52:20.903027+0000","last_clean":"2024-06-27T21:52:20.903027+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:52:20.903027+0000","last_undegraded":"2024-06-27T21:52:20.903027+0000","last_fullsized":"2024-06-27T21:52:20.903027+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:48.835191+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:51:48.835191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:07.619688+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":377,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.982716+0000","last_change":"2024-06-27T21:51:18.316912+0000","last_active":"2024-06-27T21:52:22.982716+0000","last_peered":"2024-06-27T21:52:22.982716+0000","last_clean":"2024-06-27T21:52:22.982716+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:52:22.982716+0000","last_undegraded":"2024-06-27T21:52:22.982716+0000","last_fullsized":"2024-06-27T21:52:22.982716+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:18.316784+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:18.316784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:23.874908+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":405,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.034677+0000","last_change":"2024-06-27T21:52:21.928248+0000","last_active":"2024-06-27T21:52:23.034677+0000","last_peered":"2024-06-27T21:52:23.034677+0000","last_clean":"2024-06-27T21:52:23.034677+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:52:23.034677+0000","last_undegraded":"2024-06-27T21:52:23.034677+0000","last_fullsized":"2024-06-27T21:52:23.034677+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:52:21.928156+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:52:21.928156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:46.685827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":812,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.083691+0000","last_change":"2024-06-27T21:50:57.612741+0000","last_active":"2024-06-27T21:52:23.083691+0000","last_peered":"2024-06-27T21:52:23.083691+0000","last_clean":"2024-06-27T21:52:23.083691+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:52:23.083691+0000","last_undegraded":"2024-06-27T21:52:23.083691+0000","last_fullsized":"2024-06-27T21:52:23.083691+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:50:57.612612+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:50:57.612612+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":192,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-06-27T21:52:21.006367+0000","last_change":"2024-06-27T21:51:09.954221+0000","last_active":"2024-06-27T21:52:21.006367+0000","last_peered":"2024-06-27T21:52:21.006367+0000","last_clean":"2024-06-27T21:52:21.006367+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:52:21.006367+0000","last_undegraded":"2024-06-27T21:52:21.006367+0000","last_fullsized":"2024-06-27T21:52:21.006367+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:09.954099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:09.954099+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:52:25.431565+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":1006,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.302533+0000","last_change":"2024-06-27T21:51:51.683171+0000","last_active":"2024-06-27T21:52:23.302533+0000","last_peered":"2024-06-27T21:52:23.302533+0000","last_clean":"2024-06-27T21:52:23.302533+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:52:23.302533+0000","last_undegraded":"2024-06-27T21:52:23.302533+0000","last_fullsized":"2024-06-27T21:52:23.302533+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:51.683075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:51:51.683075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:20.402839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1101,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.534193+0000","last_change":"2024-06-27T21:51:47.411923+0000","last_active":"2024-06-27T21:52:22.534193+0000","last_peered":"2024-06-27T21:52:22.534193+0000","last_clean":"2024-06-27T21:52:22.534193+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:52:22.534193+0000","last_undegraded":"2024-06-27T21:52:22.534193+0000","last_fullsized":"2024-06-27T21:52:22.534193+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:47.411842+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:48:52.674762+0000","last_clean_scrub_stamp":"2024-06-27T21:51:47.411842+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:15.992157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":795,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.083566+0000","last_change":"2024-06-27T21:51:04.535580+0000","last_active":"2024-06-27T21:52:23.083566+0000","last_peered":"2024-06-27T21:52:23.083566+0000","last_clean":"2024-06-27T21:52:23.083566+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:52:23.083566+0000","last_undegraded":"2024-06-27T21:52:23.083566+0000","last_fullsized":"2024-06-27T21:52:23.083566+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:04.535471+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:51:04.535471+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":821,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:23.085095+0000","last_change":"2024-06-27T21:52:23.085125+0000","last_active":"2024-06-27T21:52:23.085095+0000","last_peered":"2024-06-27T21:52:23.085095+0000","last_clean":"2024-06-27T21:52:23.085095+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:52:23.085095+0000","last_undegraded":"2024-06-27T21:52:23.085095+0000","last_fullsized":"2024-06-27T21:52:23.085095+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:52:23.085020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:52:23.085020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:30.943534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":265,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.592383+0000","last_change":"2024-06-27T21:51:03.119223+0000","last_active":"2024-06-27T21:52:22.592383+0000","last_peered":"2024-06-27T21:52:22.592383+0000","last_clean":"2024-06-27T21:52:22.592383+0000","last_became_active":"2024-06-27T21:46:34.528582+0000","last_became_peered":"2024-06-27T21:46:34.528582+0000","last_unstale":"2024-06-27T21:52:22.592383+0000","last_undegraded":"2024-06-27T21:52:22.592383+0000","last_fullsized":"2024-06-27T21:52:22.592383+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:03.119099+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:51:03.119099+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":271,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-06-27T21:52:22.592152+0000","last_change":"2024-06-27T21:52:17.604443+0000","last_active":"2024-06-27T21:52:22.592152+0000","last_peered":"2024-06-27T21:52:22.592152+0000","last_clean":"2024-06-27T21:52:22.592152+0000","last_became_active":"2024-06-27T21:52:17.603944+0000","last_became_peered":"2024-06-27T21:52:17.603944+0000","last_unstale":"2024-06-27T21:52:22.592152+0000","last_undegraded":"2024-06-27T21:52:22.592152+0000","last_fullsized":"2024-06-27T21:52:22.592152+0000","mapping_epoch":983,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:51:49.181498+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:51:49.181498+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:53:14.772982+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253384,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055112,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1054334,"kb_avail":92690040,"statfs":{"total":95995035648,"available":94914600960,"internally_reserved":0,"allocated":778240,"data_stored":473631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1079638469},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":76,"seq":326417515019,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002376,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92742776,"statfs":{"total":95995035648,"available":94968602624,"internally_reserved":0,"allocated":778240,"data_stored":473631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1025636800},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785808,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959344,"statfs":{"total":95995035648,"available":95190368256,"internally_reserved":0,"allocated":790528,"data_stored":481159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":803862976},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609464,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93135688,"statfs":{"total":95995035648,"available":95370944512,"internally_reserved":0,"allocated":761856,"data_stored":465780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267219,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":856780,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":92888372,"statfs":{"total":95995035648,"available":95117692928,"internally_reserved":0,"allocated":983040,"data_stored":674879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":876345802},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758344,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986808,"statfs":{"total":95995035648,"available":95218491392,"internally_reserved":0,"allocated":1105920,"data_stored":801311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757794,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740156,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004996,"statfs":{"total":95995035648,"available":95237115904,"internally_reserved":0,"allocated":966656,"data_stored":667028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161501,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583748,"kb_used_data":760,"kb_used_omap":92,"kb_used_meta":582883,"kb_avail":93161404,"statfs":{"total":95995035648,"available":95397277696,"internally_reserved":0,"allocated":778240,"data_stored":473631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":596872290},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182954,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659764,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085388,"statfs":{"total":95995035648,"available":95319437312,"internally_reserved":0,"allocated":1089536,"data_stored":793460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":674494919},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808192,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936960,"statfs":{"total":95995035648,"available":95167447040,"internally_reserved":0,"allocated":966656,"data_stored":667028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575414,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916292,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828860,"statfs":{"total":95995035648,"available":95056752640,"internally_reserved":0,"allocated":778240,"data_stored":473631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738939,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880528,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864624,"statfs":{"total":95995035648,"available":95093374976,"internally_reserved":0,"allocated":1118208,"data_stored":808839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:52:28.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: from='client.18096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3660043532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:28.621 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:28 smithi192 ceph-mon[32956]: osdmap e989: 12 total, 12 up, 12 in 2024-06-27T21:52:28.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: from='client.18096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3660043532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:28.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:28 smithi151 ceph-mon[28965]: osdmap e989: 12 total, 12 up, 12 in 2024-06-27T21:52:28.908 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.c/config 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: from='client.18096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3660043532' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:29.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:28 smithi190 ceph-mon[32247]: osdmap e989: 12 total, 12 up, 12 in 2024-06-27T21:52:29.195 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:29.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:29 smithi151 ceph-mon[28965]: pgmap v2440: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:29.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:29 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:52:29.742 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:29 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:52:29.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:29 smithi192 ceph-mon[32956]: pgmap v2440: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:29.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:29 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:52:29.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:29 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:52:30.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:29 smithi190 ceph-mon[32247]: pgmap v2440: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:29 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:52:30.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:29 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:52:30.623 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-06-27T21:52:30.623 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-06-27T21:52:30.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-06-27T21:52:30.623 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-06-27T21:52:30.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.26656.0:181754 7.c 7:37e9dc29:::benchmark_data_smithi192_150129_object78449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e780)", 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:43:12.044102+0000", 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 558.57387008499995, 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.72435616000000003, 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.26656", 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2309556478", 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 181754 2024-06-27T21:52:30.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044102+0000", 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044102+0000", 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044104+0000", 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1829999999999999e-06 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044110+0000", 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.117e-06 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044124+0000", 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4555e-05 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044133+0000", 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3140000000000004e-06 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044182+0000", 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9536999999999997e-05 2024-06-27T21:52:30.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044301+0000", 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000118688 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2,6", 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.044417+0000", 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000115975 2024-06-27T21:52:30.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.048619+0000", 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042019550000000003 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.048806+0000", 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018761099999999999 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:30.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768388+0000", 2024-06-27T21:52:30.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.71958176699999998 2024-06-27T21:52:30.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:52:30.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768427+0000", 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9039999999999999e-05 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768459+0000", 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1418000000000003e-05 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.26656.0:181754 7.c e780/779 ondisk, result = 0)", 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:43:12.048199+0000", 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 558.56977308099999, 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.72026291200000003, 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.048199+0000", 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.048199+0000", 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.048200+0000", 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8400000000000005e-07 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.048203+0000", 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1389999999999999e-06 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768300+0000", 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.72009680300000001 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768306+0000", 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768377+0000", 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1228000000000003e-05 2024-06-27T21:52:30.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768383+0000", 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8239999999999996e-06 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:43:12.768462+0000", 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9394000000000006e-05 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(8.f 821/816 [PushReplyOp(8:febdef11:::benchmark_data_smithi192_167666_object2315:head)])", 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:49.945441+0000", 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 460.67253124600001, 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79455450000000005, 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.945441+0000", 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.945441+0000", 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.945442+0000", 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.945444+0000", 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.65e-06 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.738343+0000", 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79289879399999996 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.738351+0000", 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8099999999999998e-06 2024-06-27T21:52:30.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.738426+0000", 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4680999999999999e-05 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.739996+0000", 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015701949999999999 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(8.7 821/816 [PushReplyOp(8:e0bc957d:::benchmark_data_smithi192_167666_object3389:head)])", 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:44:49.949838+0000", 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 460.668134189, 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79561932700000004, 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.949838+0000", 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.949838+0000", 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.949839+0000", 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2699999999999998e-07 2024-06-27T21:52:30.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:49.949843+0000", 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2969999999999997e-06 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.744076+0000", 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.79423292300000004 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.744090+0000", 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3803e-05 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.744244+0000", 2024-06-27T21:52:30.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015391 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:44:50.745458+0000", 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001213567 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59314 8.16 8.9de680d6 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e833)", 2024-06-27T21:52:30.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:12.911362+0000", 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 437.70661036199999, 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.989349437, 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59314 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:12.911362+0000", 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:12.911362+0000", 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:12.911363+0000", 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-06-27T21:52:30.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:12.911385+0000", 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2575e-05 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:12.911394+0000", 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5609999999999999e-06 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:12.911400+0000", 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.263e-06 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.037302+0000", 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.12590179000000001 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.037309+0000", 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2880000000000004e-06 2024-06-27T21:52:30.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.900680+0000", 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8633706569999999 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.900712+0000", 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1683999999999997e-05 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59345 8.f 8:fe6be046:::benchmark_data_smithi192_167666_object59344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e833)", 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:13.140433+0000", 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 437.47753944700003, 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.80458616900000002, 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.651 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59345 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140433+0000", 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140433+0000", 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140435+0000", 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.807e-06 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140458+0000", 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2863000000000001e-05 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140474+0000", 2024-06-27T21:52:30.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5959000000000001e-05 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140481+0000", 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5100000000000001e-06 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140532+0000", 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0627999999999997e-05 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140653+0000", 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000120927 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,2", 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.140750+0000", 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7695999999999994e-05 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.142508+0000", 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017574400000000001 2024-06-27T21:52:30.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.143060+0000", 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00055186600000000003 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.944794+0000", 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.801734524 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.944971+0000", 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000177164 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.945019+0000", 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7784999999999997e-05 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.657 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59350 8.16 8.8211e0d6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:13.952111+0000", 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.66586112700003, 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94901392799999995, 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59350 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.952111+0000", 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.952111+0000", 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.952113+0000", 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.824e-06 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.952147+0000", 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3649999999999998e-05 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.952158+0000", 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1345000000000001e-05 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.952168+0000", 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.028e-05 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.955601+0000", 2024-06-27T21:52:30.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034322939999999998 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.955609+0000", 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7450000000000001e-06 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901117+0000", 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.94550723599999997 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901125+0000", 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5539999999999998e-06 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59387 8.16 8.e0851f96 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:13.983066+0000", 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.634906048, 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91811336499999996, 2024-06-27T21:52:30.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59387 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983066+0000", 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983066+0000", 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983068+0000", 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2139999999999999e-06 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983092+0000", 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3989999999999999e-05 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983104+0000", 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2036e-05 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983111+0000", 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1849999999999998e-06 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983164+0000", 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3563000000000002e-05 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.983172+0000", 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3429999999999998e-06 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901174+0000", 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91800189300000001 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901180+0000", 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1410000000000002e-06 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59396 8.16 8.965dd7d6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:13.987317+0000", 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.63065584999998, 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91389140499999999, 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59396 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.669 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.987317+0000", 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.987317+0000", 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.987317+0000", 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2200000000000003e-07 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.987369+0000", 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1379999999999999e-05 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.987379+0000", 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0244999999999999e-05 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.987386+0000", 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.781e-06 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.988687+0000", 2024-06-27T21:52:30.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013013110000000001 2024-06-27T21:52:30.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:13.988694+0000", 2024-06-27T21:52:30.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3270000000000003e-06 2024-06-27T21:52:30.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901201+0000", 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.91250668700000004 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901208+0000", 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8519999999999998e-06 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59443 8.16 8.280a9836 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:14.022285+0000", 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.59568720599998, 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87895008799999996, 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59443 2024-06-27T21:52:30.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022285+0000", 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022285+0000", 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022286+0000", 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022325+0000", 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8717000000000002e-05 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022336+0000", 2024-06-27T21:52:30.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0744000000000001e-05 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022341+0000", 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0649999999999998e-06 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022626+0000", 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028531599999999998 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.022635+0000", 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6999999999999997e-06 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901229+0000", 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87859403700000005 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901235+0000", 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6290000000000004e-06 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.679 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59449 8.16 8.ab5caf36 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:14.033939+0000", 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.58403389900002, 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86732437500000004, 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59449 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.033939+0000", 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.033939+0000", 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.033940+0000", 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0020000000000001e-06 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.033986+0000", 2024-06-27T21:52:30.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-05 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.033997+0000", 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1236999999999999e-05 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.034002+0000", 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3890000000000004e-06 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.034045+0000", 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2758e-05 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.034051+0000", 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6119999999999998e-06 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901256+0000", 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.86720533300000002 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901263+0000", 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8739999999999997e-06 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59517 8.16 8.ac3c1db6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:14.250471+0000", 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.36750132899999, 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65081919700000002, 2024-06-27T21:52:30.685 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59517 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250471+0000", 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250471+0000", 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250473+0000", 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5e-06 2024-06-27T21:52:30.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250504+0000", 2024-06-27T21:52:30.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1454000000000001e-05 2024-06-27T21:52:30.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250520+0000", 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6299000000000001e-05 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250529+0000", 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4379999999999995e-06 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250582+0000", 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3295999999999999e-05 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.250590+0000", 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6599999999999995e-06 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901283+0000", 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.65069360499999995 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901290+0000", 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9450000000000004e-06 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59521 8.16 8.89af6a96 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:14.253407+0000", 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.36456559999999, 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64794788800000003, 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59521 2024-06-27T21:52:30.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253407+0000", 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253407+0000", 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253408+0000", 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3600000000000003e-07 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253429+0000", 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1684000000000001e-05 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253439+0000", 2024-06-27T21:52:30.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6309999999999998e-06 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253445+0000", 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3790000000000003e-06 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253489+0000", 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3483999999999999e-05 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.253495+0000", 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7669999999999998e-06 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901312+0000", 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64781630999999995 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901355+0000", 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2997000000000002e-05 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59524 8.16 8.9a09df76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:14.255522+0000", 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.36245049399997, 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64587798600000002, 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.698 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59524 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255522+0000", 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255522+0000", 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255523+0000", 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2190000000000001e-06 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255542+0000", 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8919999999999998e-05 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255551+0000", 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2189999999999993e-06 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255556+0000", 2024-06-27T21:52:30.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0180000000000001e-06 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255604+0000", 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7333999999999998e-05 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.255611+0000", 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0010000000000004e-06 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901379+0000", 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.64576845500000002 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901400+0000", 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0820000000000001e-05 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27031.0:59538 8.16 8.22191876 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:14.267502+0000", 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 436.35047087300001, 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63392826300000005, 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27031", 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:30.704 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 59538 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267502+0000", 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267502+0000", 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267503+0000", 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7409999999999999e-06 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267534+0000", 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0230999999999999e-05 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267545+0000", 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1287999999999999e-05 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267551+0000", 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7450000000000001e-06 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267608+0000", 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7315000000000003e-05 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.267617+0000", 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4329999999999993e-06 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901421+0000", 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.63380403699999999 2024-06-27T21:52:30.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:14.901430+0000", 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.473e-06 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(8.d 857/838 [PushReplyOp(8:b30a1e48:::benchmark_data_smithi192_167666_object39205:head)])", 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:45:58.752890+0000", 2024-06-27T21:52:30.709 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 391.86508262500001, 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97462259799999995, 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.752890+0000", 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.752890+0000", 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.752890+0000", 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8699999999999995e-07 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:58.752895+0000", 2024-06-27T21:52:30.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2339999999999996e-06 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.725413+0000", 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.97251817200000001 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.725438+0000", 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5080999999999999e-05 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.725509+0000", 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0945999999999995e-05 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:45:59.727512+0000", 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020035780000000002 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.713 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "MOSDPGPushReply(8.d 857/838 [PushReplyOp(8:be4b72cd:::benchmark_data_smithi192_167666_object48329:head)])", 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:46:02.573341+0000", 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 388.044631208, 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87546249700000001, 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:02.573341+0000", 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:02.573341+0000", 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:02.573342+0000", 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3200000000000005e-07 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:02.573346+0000", 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8979999999999998e-06 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:03.447140+0000", 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.87379442699999998 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:03.447155+0000", 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.491e-05 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:03.447309+0000", 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015399000000000001 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:46:03.448804+0000", 2024-06-27T21:52:30.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00149464 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.27262.0:71346 9.5 e945/916 ondisk, result = 0)", 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:49:16.155017+0000", 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 194.46295501899999, 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.22313633899999999, 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.718 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.155017+0000", 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.155017+0000", 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.155018+0000", 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9099999999999991e-07 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.155022+0000", 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5049999999999999e-06 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.378016+0000", 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.222993687 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.378021+0000", 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3009999999999997e-06 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.378058+0000", 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6986000000000001e-05 2024-06-27T21:52:30.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.378063+0000", 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6530000000000002e-06 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:49:16.378154+0000", 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1216000000000002e-05 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.722 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.27262.0:126101 9.8 e955/924 9:191c6516:::benchmark_data_smithi192_179902_object16651:head v 955'7969, mlcod=955'7966)", 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:50:12.504136+0000", 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 138.11383651599999, 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.34088561299999998, 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.504136+0000", 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.504136+0000", 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.504137+0000", 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1200000000000002e-07 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.504142+0000", 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5960000000000003e-06 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.841570+0000", 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.33742730999999998 2024-06-27T21:52:30.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.841588+0000", 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8133000000000001e-05 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.841708+0000", 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000119805 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.841750+0000", 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2015999999999999e-05 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.844982+0000", 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032325919999999998 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:12.845022+0000", 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9449000000000001e-05 2024-06-27T21:52:30.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.27262.0:149885 9.0 9:096a63a7:::benchmark_data_smithi192_179902_object40435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e961)", 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-06-27T21:50:45.680364+0000", 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 104.937608858, 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.37436899899999998, 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-06-27T21:52:30.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.27262", 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/2253233959", 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 149885 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680364+0000", 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680364+0000", 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680365+0000", 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.294e-06 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680370+0000", 2024-06-27T21:52:30.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.826e-06 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680383+0000", 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3658e-05 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680391+0000", 2024-06-27T21:52:30.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8310000000000001e-06 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680438+0000", 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7113000000000001e-05 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680542+0000", 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010361099999999999 2024-06-27T21:52:30.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1,6", 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.680637+0000", 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4709000000000004e-05 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.685181+0000", 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045442649999999996 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:45.685338+0000", 2024-06-27T21:52:30.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000157035 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:46.054637+0000", 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.36929872800000002 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:46.054689+0000", 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2349000000000003e-05 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-06-27T21:52:30.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-06-27T21:50:46.054733+0000", 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3579999999999999e-05 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-06-27T21:52:30.735 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-06-27T21:52:30.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:30 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:52:30.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:30 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:52:30.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:30 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:52:30.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:30 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:52:31.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:30 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:52:31.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:30 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:52:31.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset noscrub 2024-06-27T21:52:31.425 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:31.564 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pgp_num to 31 2024-06-27T21:52:31.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-06-27T21:52:31.894 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:52:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:31 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:52:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:31 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:52:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:31 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:52:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:31 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:52:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:31 smithi151 ceph-mon[28965]: pgmap v2441: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:31.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:31 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1489706005' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:52:32.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:31 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:52:32.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:31 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:52:32.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:31 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:52:32.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:31 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:52:32.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:31 smithi190 ceph-mon[32247]: pgmap v2441: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:32.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:31 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1489706005' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:52:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:31 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:52:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:31 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:52:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:31 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:52:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:31 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:52:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:31 smithi192 ceph-mon[32956]: pgmap v2441: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:32.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:31 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1489706005' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]: dispatch 2024-06-27T21:52:32.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1489706005' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:52:32.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:32 smithi190 ceph-mon[32247]: osdmap e990: 12 total, 12 up, 12 in 2024-06-27T21:52:32.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1489706005' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:52:32.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:32 smithi151 ceph-mon[28965]: osdmap e990: 12 total, 12 up, 12 in 2024-06-27T21:52:33.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1489706005' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-06-27T21:52:33.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:32 smithi192 ceph-mon[32956]: osdmap e990: 12 total, 12 up, 12 in 2024-06-27T21:52:33.666 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:52:33.666 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:52:33.666 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-06-27T21:52:33.667 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:52:33.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:33 smithi151 ceph-mon[28965]: pgmap v2443: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:34.013 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:33 smithi190 ceph-mon[32247]: pgmap v2443: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:34.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:33 smithi192 ceph-mon[32956]: pgmap v2443: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:34.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-06-27T21:52:34.592 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-06-27T21:52:34.632 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:52:34.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:34 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:52:34.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:34 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:52:34.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:34 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3996093583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:52:34.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:34 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:52:34.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:34 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:52:34.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:34 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3996093583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:52:35.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:34 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:52:35.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:34 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:52:35.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:34 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3996093583' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-06-27T21:52:35.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd unset nodeep-scrub 2024-06-27T21:52:35.674 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:35.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:35 smithi151 ceph-mon[28965]: pgmap v2444: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:35.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3996093583' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:52:35.698 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:35 smithi151 ceph-mon[28965]: osdmap e991: 12 total, 12 up, 12 in 2024-06-27T21:52:35.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:35 smithi190 ceph-mon[32247]: pgmap v2444: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:35.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3996093583' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:52:35.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:35 smithi190 ceph-mon[32247]: osdmap e991: 12 total, 12 up, 12 in 2024-06-27T21:52:36.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:35 smithi192 ceph-mon[32956]: pgmap v2444: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:36.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3996093583' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-06-27T21:52:36.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:35 smithi192 ceph-mon[32956]: osdmap e991: 12 total, 12 up, 12 in 2024-06-27T21:52:36.342 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:52:36.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-06-27T21:52:36.342 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-06-27T21:52:36.342 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-06-27T21:52:36.342 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:52:36.876 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:52:36.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:52:36.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:36 smithi151 ceph-mon[28965]: osdmap e992: 12 total, 12 up, 12 in 2024-06-27T21:52:36.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:52:36.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:52:36.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:36 smithi190 ceph-mon[32247]: osdmap e992: 12 total, 12 up, 12 in 2024-06-27T21:52:36.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-06-27T21:52:37.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:52:37.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:52:37.045 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:36 smithi192 ceph-mon[32956]: osdmap e992: 12 total, 12 up, 12 in 2024-06-27T21:52:37.200 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.b/config 2024-06-27T21:52:37.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-06-27T21:52:37.573 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:38.022 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-06-27T21:52:38.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:38 smithi190 ceph-mon[32247]: pgmap v2447: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:38.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:38 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:52:38.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:38 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:52:38.275 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4278971492' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:38.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:38 smithi192 ceph-mon[32956]: pgmap v2447: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:38 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:52:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:38 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:52:38.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4278971492' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:38 smithi151 ceph-mon[28965]: pgmap v2447: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:38 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:52:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:38 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:52:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4278971492' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-06-27T21:52:38.787 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:23402 8.0 e815/797 8:0cf7414f:::benchmark_data_smithi192_167666_object23401:head v 815'1399, mlcod=814'1396)", 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:34.897033+0000", 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 483.88934334800001, 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27473579100000001, 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.788 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897033+0000", 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897033+0000", 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897034+0000", 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9500000000000002e-07 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:34.897149+0000", 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011548 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170582+0000", 2024-06-27T21:52:38.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27343230699999999 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170594+0000", 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2294e-05 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170749+0000", 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000155322 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.170790+0000", 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0772999999999997e-05 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.171735+0000", 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00094516900000000004 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.171769+0000", 2024-06-27T21:52:38.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3850999999999998e-05 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:23416 8.2 e815/797 8:4f4be8b9:::benchmark_data_smithi192_167666_object23415:head v 815'1485, mlcod=814'1482)", 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:35.040870+0000", 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 483.74550655600001, 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31619940000000002, 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.793 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040870+0000", 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040870+0000", 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040871+0000", 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0890000000000001e-06 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.040902+0000", 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.042e-05 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356031+0000", 2024-06-27T21:52:38.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31512962300000003 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356037+0000", 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5720000000000002e-06 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356119+0000", 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1866999999999994e-05 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.356135+0000", 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6730000000000001e-05 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.357034+0000", 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00089823599999999996 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:35.357070+0000", 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5862999999999999e-05 2024-06-27T21:52:38.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:38264 8.2 e820/797 8:4283fde4:::benchmark_data_smithi192_167666_object38263:head v 820'2407, mlcod=820'2405)", 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:44:49.059774+0000", 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 469.72660321500001, 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23295091000000001, 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.059774+0000", 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.059774+0000", 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.059774+0000", 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0399999999999995e-07 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.059815+0000", 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1060000000000003e-05 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.291156+0000", 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.23134090199999999 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.291164+0000", 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2519999999999999e-06 2024-06-27T21:52:38.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.291252+0000", 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7639000000000006e-05 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.291280+0000", 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8008999999999999e-05 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.292685+0000", 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001404422 2024-06-27T21:52:38.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:44:49.292724+0000", 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9922000000000002e-05 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:79406 8.e e850/797 8:7c278eaa:::benchmark_data_smithi192_167666_object79405:head v 850'5002, mlcod=850'4998)", 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:48.644288+0000", 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 410.14208899300002, 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48320776999999998, 2024-06-27T21:52:38.802 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:48.644288+0000", 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:48.644288+0000", 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:48.644292+0000", 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4889999999999997e-06 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:48.644353+0000", 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1019e-05 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:49.125366+0000", 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.481012247 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:49.125372+0000", 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6069999999999996e-06 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:49.125443+0000", 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0751000000000001e-05 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:49.125460+0000", 2024-06-27T21:52:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7385999999999999e-05 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:49.127468+0000", 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020075399999999999 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:49.127496+0000", 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7730999999999999e-05 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.f 856/839 [PushOp(8:f11e06da:::benchmark_data_smithi192_167666_object4901:head, version: 806'315, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:f11e06da:::benchmark_data_smithi192_167666_object4901:head@806'315, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:57.765094+0000", 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 401.02128294099998, 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30587955500000003, 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.765094+0000", 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.765094+0000", 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.765094+0000", 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2600000000000002e-07 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:57.765163+0000", 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8606e-05 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.069032+0000", 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30386851999999998 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.069043+0000", 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1566000000000001e-05 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.069132+0000", 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9260999999999998e-05 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.069143+0000", 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0239e-05 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.070973+0000", 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001830737 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.f 857/839 [PushOp(8:f44f1be3:::benchmark_data_smithi192_167666_object4940:head, version: 806'317, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:f44f1be3:::benchmark_data_smithi192_167666_object4940:head@806'317, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:45:58.085313+0000", 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 400.70106346, 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.86690102099999999, 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.811 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.085313+0000", 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.085313+0000", 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.085314+0000", 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2300000000000001e-07 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.085377+0000", 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3583000000000004e-05 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.950346+0000", 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.86496839199999997 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.950356+0000", 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0431000000000001e-05 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.950445+0000", 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8535000000000005e-05 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.950458+0000", 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3390000000000001e-05 2024-06-27T21:52:38.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:45:58.952214+0000", 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001756067 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(8.f 857/839 [PushOp(8:feb12822:::benchmark_data_smithi192_167666_object7297:head, version: 807'461, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:feb12822:::benchmark_data_smithi192_167666_object7297:head@807'461, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:00.852895+0000", 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 397.933482228, 2024-06-27T21:52:38.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91954493900000001, 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.852895+0000", 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.852895+0000", 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.852895+0000", 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4099999999999998e-07 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:00.852945+0000", 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0149999999999999e-05 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.770427+0000", 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91748148600000001 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.770454+0000", 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6783000000000001e-05 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.770574+0000", 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120556 2024-06-27T21:52:38.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.770583+0000", 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6400000000000003e-06 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:01.772439+0000", 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001856583 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:94715 8.10 e865/854 8:0b89cc6a:::benchmark_data_smithi192_167666_object15277:head v 865'4732, mlcod=865'4729)", 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:23.483178+0000", 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 375.30319887399997, 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27064298199999998, 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.483178+0000", 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.483178+0000", 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.483179+0000", 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7800000000000001e-07 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.483184+0000", 2024-06-27T21:52:38.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7429999999999999e-06 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.752554+0000", 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26936945000000001 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.752559+0000", 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9409999999999998e-06 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.752600+0000", 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1502999999999999e-05 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.752613+0000", 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3186e-05 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753792+0000", 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001178433 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:23.753821+0000", 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9048e-05 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:103712 8.18 e871/854 8:1fc4146c:::benchmark_data_smithi192_167666_object24274:head v 871'5140, mlcod=871'5137)", 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:33.462364+0000", 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 365.32401226600001, 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25407904100000001, 2024-06-27T21:52:38.824 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.462364+0000", 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.462364+0000", 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.462365+0000", 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3900000000000004e-07 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.462370+0000", 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3709999999999999e-06 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.714818+0000", 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.252447009 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.714823+0000", 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3000000000000001e-06 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.714913+0000", 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9906000000000003e-05 2024-06-27T21:52:38.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.714934+0000", 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1545000000000001e-05 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.716414+0000", 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014801840000000001 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:33.716444+0000", 2024-06-27T21:52:38.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9087000000000001e-05 2024-06-27T21:52:38.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.829 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.27031.0:120322 8.14 8:2beaca9a:::benchmark_data_smithi192_167666_object40884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e879)", 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:46:54.629868+0000", 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 344.15650876000001, 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.268931166, 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.27031", 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.192:0/576735386", 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 120322 2024-06-27T21:52:38.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629868+0000", 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629868+0000", 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629869+0000", 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3969999999999999e-06 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629874+0000", 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7600000000000002e-06 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629886+0000", 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1498000000000001e-05 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629891+0000", 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1329999999999998e-06 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.629942+0000", 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1138999999999999e-05 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.630029+0000", 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6710000000000002e-05 2024-06-27T21:52:38.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.630099+0000", 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0856000000000001e-05 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.631625+0000", 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001525909 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:38.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.632444+0000", 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00081840000000000005 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.898696+0000", 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.266252247 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.898752+0000", 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6020000000000002e-05 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:46:54.898799+0000", 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7097e-05 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:129369 8.b e889/823 8:d4ed83f2:::benchmark_data_smithi192_167666_object49931:head v 889'7981, mlcod=889'7978)", 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:06.896149+0000", 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 331.89022746500001, 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18991404000000001, 2024-06-27T21:52:38.836 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.896149+0000", 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.896149+0000", 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.896150+0000", 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5799999999999999e-07 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:06.896154+0000", 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5970000000000002e-06 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:07.083037+0000", 2024-06-27T21:52:38.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.186882091 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:07.083041+0000", 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5220000000000001e-06 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:07.083088+0000", 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6644999999999998e-05 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:07.083109+0000", 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1483000000000001e-05 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:07.086028+0000", 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029184969999999999 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:07.086063+0000", 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5646999999999999e-05 2024-06-27T21:52:38.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27031.0:156425 8.19 e905/868 8:9e81eff6:::benchmark_data_smithi192_167666_object76987:head v 905'6722, mlcod=905'6719)", 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:47:41.458610+0000", 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 297.32776685900001, 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.174544699, 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.458610+0000", 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.458610+0000", 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.458611+0000", 2024-06-27T21:52:38.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4300000000000003e-07 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.458615+0000", 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8500000000000002e-06 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.631025+0000", 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.17240997899999999 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.631031+0000", 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5300000000000004e-06 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.631076+0000", 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5009000000000002e-05 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.631099+0000", 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3439e-05 2024-06-27T21:52:38.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.633127+0000", 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020274720000000002 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:47:41.633155+0000", 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7776999999999998e-05 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:1408 9.4 e925/916 9:2f65122a:::benchmark_data_smithi192_179902_object1407:head v 925'84, mlcod=925'81)", 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:48:25.513938+0000", 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 253.27243831999999, 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.17303073499999999, 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.513938+0000", 2024-06-27T21:52:38.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.513938+0000", 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.513939+0000", 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999996e-07 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.513996+0000", 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6827000000000003e-05 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.686538+0000", 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.172541897 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.686545+0000", 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2710000000000003e-06 2024-06-27T21:52:38.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.686598+0000", 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3162999999999999e-05 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.686615+0000", 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6903999999999999e-05 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.686938+0000", 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032306099999999998 2024-06-27T21:52:38.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:25.686969+0000", 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1007999999999999e-05 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:32918 9.c e933/916 9:3468f359:::benchmark_data_smithi192_179902_object32917:head v 933'2024, mlcod=933'2021)", 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:48:45.516493+0000", 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 233.26988390599999, 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.145293482, 2024-06-27T21:52:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.516493+0000", 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.516493+0000", 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.516494+0000", 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.85e-07 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.516596+0000", 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000102806 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.660391+0000", 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14379462300000001 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.660406+0000", 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5193e-05 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.660529+0000", 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012242000000000001 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.660570+0000", 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0995000000000003e-05 2024-06-27T21:52:38.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.661749+0000", 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011795079999999999 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:48:45.661786+0000", 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7151999999999999e-05 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.854 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:75232 9.4 e945/916 9:298d384e:::benchmark_data_smithi192_179902_object75231:head v 945'4837, mlcod=945'4834)", 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:19.823925+0000", 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 198.96245140600001, 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22356799599999999, 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:19.823925+0000", 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:19.823925+0000", 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:19.823927+0000", 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2529999999999999e-06 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:19.823972+0000", 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5034999999999998e-05 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:20.045251+0000", 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.221279594 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:20.045258+0000", 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9140000000000002e-06 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:20.045315+0000", 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7253999999999998e-05 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:20.045340+0000", 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4301999999999999e-05 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:20.047446+0000", 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002105998 2024-06-27T21:52:38.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:20.047493+0000", 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7645999999999997e-05 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:79249 9.4 e945/916 9:29694c75:::benchmark_data_smithi192_179902_object79248:head v 945'5060, mlcod=945'5057)", 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:23.572906+0000", 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 195.21347085900001, 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18448289800000001, 2024-06-27T21:52:38.859 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.572906+0000", 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.572906+0000", 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.572907+0000", 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.133e-06 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.572933+0000", 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6049000000000001e-05 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.755443+0000", 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.18250978500000001 2024-06-27T21:52:38.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.755449+0000", 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.246e-06 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.755494+0000", 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.507e-05 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.755508+0000", 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4003e-05 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.757358+0000", 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001849954 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.757389+0000", 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0657999999999997e-05 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.863 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:79335 9.4 e945/916 9:2b445a9f:::benchmark_data_smithi192_179902_object79334:head v 945'5062, mlcod=945'5059)", 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:49:23.758456+0000", 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 195.02792081499999, 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.22206704999999999, 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.758456+0000", 2024-06-27T21:52:38.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.758456+0000", 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.758456+0000", 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5699999999999998e-07 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.758552+0000", 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5749999999999996e-05 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.975983+0000", 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21743119699999999 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.975989+0000", 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.959e-06 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.976046+0000", 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6650000000000001e-05 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.976067+0000", 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1002e-05 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.980481+0000", 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044142199999999999 2024-06-27T21:52:38.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:49:23.980523+0000", 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1814999999999999e-05 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:172890 9.c e969/916 9:3d26ae27:::benchmark_data_smithi192_179902_object63440:head v 969'10759, mlcod=969'10756)", 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:51:12.500011+0000", 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 86.286366044000005, 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21626610800000001, 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.868 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.500011+0000", 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.500011+0000", 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.500012+0000", 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9600000000000008e-07 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.500019+0000", 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2280000000000001e-06 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.712253+0000", 2024-06-27T21:52:38.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21223376399999999 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.712260+0000", 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3329999999999999e-06 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.712314+0000", 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3706999999999999e-05 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.712333+0000", 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9361e-05 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.716227+0000", 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038938390000000001 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:12.716277+0000", 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9879999999999997e-05 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.27262.0:200959 9.c e975/916 9:3fb81b46:::benchmark_data_smithi192_179902_object91509:head v 975'12522, mlcod=975'12519)", 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-06-27T21:51:48.995550+0000", 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 49.790826506000002, 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102316131, 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:48.995550+0000", 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:48.995550+0000", 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:48.995551+0000", 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.246e-06 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:48.995558+0000", 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5899999999999996e-06 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:49.093416+0000", 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.097857484999999994 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:49.093423+0000", 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.525e-06 2024-06-27T21:52:38.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:49.093490+0000", 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6472999999999999e-05 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:49.093512+0000", 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2693000000000001e-05 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:49.097812+0000", 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042995719999999998 2024-06-27T21:52:38.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-06-27T21:51:49.097866+0000", 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4546999999999998e-05 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-06-27T21:52:38.877 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-06-27T21:52:39.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4278971492' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:39.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:39 smithi151 ceph-mon[28965]: osdmap e993: 12 total, 12 up, 12 in 2024-06-27T21:52:39.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4278971492' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:39.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:39 smithi190 ceph-mon[32247]: osdmap e993: 12 total, 12 up, 12 in 2024-06-27T21:52:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4278971492' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-06-27T21:52:39.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:39 smithi192 ceph-mon[32956]: osdmap e993: 12 total, 12 up, 12 in 2024-06-27T21:52:40.222 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:40 smithi151 ceph-mon[28965]: pgmap v2449: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:40.223 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a: {} 2024-06-27T21:52:40.223 INFO:teuthology.orchestra.run.smithi151.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-27T21:52:40.237 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c: {} 2024-06-27T21:52:40.237 INFO:teuthology.orchestra.run.smithi151.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-27T21:52:40.251 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b: {} 2024-06-27T21:52:40.251 INFO:teuthology.orchestra.run.smithi151.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-06-27T21:52:40.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:40 smithi192 ceph-mon[32956]: pgmap v2449: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:40.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:40 smithi190 ceph-mon[32247]: pgmap v2449: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:41.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 0 1 2024-06-27T21:52:41.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4227565905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:52:41.308 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4227565905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:52:41.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4227565905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-27T21:52:42.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:42 smithi192 ceph-mon[32956]: pgmap v2450: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:42.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:42 smithi151 ceph-mon[28965]: pgmap v2450: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:42 smithi190 ceph-mon[32247]: pgmap v2450: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:43.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:43 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:52:43.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:43 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:52:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:43 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:52:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:43 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:52:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:43 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:52:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:43 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:52:44.048 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-06-27T21:52:44.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:44 smithi151 ceph-mon[28965]: pgmap v2451: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:44.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:44 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2701218165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:52:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:44 smithi190 ceph-mon[32247]: pgmap v2451: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:44 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2701218165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:52:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:44 smithi192 ceph-mon[32956]: pgmap v2451: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:44 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2701218165' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:52:44.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 0 1 2024-06-27T21:52:44.924 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:45.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2701218165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:52:45.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:45 smithi151 ceph-mon[28965]: osdmap e994: 12 total, 12 up, 12 in 2024-06-27T21:52:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2701218165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:52:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:45 smithi190 ceph-mon[32247]: osdmap e994: 12 total, 12 up, 12 in 2024-06-27T21:52:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2701218165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:52:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:45 smithi192 ceph-mon[32956]: osdmap e994: 12 total, 12 up, 12 in 2024-06-27T21:52:46.305 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:46 smithi151 ceph-mon[28965]: pgmap v2453: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:46 smithi190 ceph-mon[32247]: pgmap v2453: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:46 smithi192 ceph-mon[32956]: pgmap v2453: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:47.060 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-06-27T21:52:47.398 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:47 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/196341324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:52:47.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:47 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/196341324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:52:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:47 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/196341324' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-06-27T21:52:47.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 1 1 2024-06-27T21:52:47.972 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:48.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:48 smithi151 ceph-mon[28965]: pgmap v2454: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:48.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:48 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/196341324' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:52:48.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:48 smithi151 ceph-mon[28965]: osdmap e995: 12 total, 12 up, 12 in 2024-06-27T21:52:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:48 smithi190 ceph-mon[32247]: pgmap v2454: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:48.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:48 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/196341324' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:52:48.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:48 smithi190 ceph-mon[32247]: osdmap e995: 12 total, 12 up, 12 in 2024-06-27T21:52:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:48 smithi192 ceph-mon[32956]: pgmap v2454: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:48 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/196341324' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-06-27T21:52:48.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:48 smithi192 ceph-mon[32956]: osdmap e995: 12 total, 12 up, 12 in 2024-06-27T21:52:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-06-27T21:52:50.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:50 smithi151 ceph-mon[28965]: pgmap v2456: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:50.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3153832091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:52:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:50 smithi190 ceph-mon[32247]: pgmap v2456: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:50.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3153832091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:52:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:50 smithi192 ceph-mon[32956]: pgmap v2456: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3153832091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:52:50.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 1 1 2024-06-27T21:52:51.006 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:51.385 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:51 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3153832091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:52:51.385 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:51 smithi151 ceph-mon[28965]: osdmap e996: 12 total, 12 up, 12 in 2024-06-27T21:52:51.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:51 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3153832091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:52:51.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:51 smithi190 ceph-mon[32247]: osdmap e996: 12 total, 12 up, 12 in 2024-06-27T21:52:51.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:51 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3153832091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:52:51.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:51 smithi192 ceph-mon[32956]: osdmap e996: 12 total, 12 up, 12 in 2024-06-27T21:52:52.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:52 smithi151 ceph-mon[28965]: pgmap v2458: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:52.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:52 smithi190 ceph-mon[32247]: pgmap v2458: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:52.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:52 smithi192 ceph-mon[32956]: pgmap v2458: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:53.098 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-06-27T21:52:53.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1022029677' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:52:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1022029677' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:52:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1022029677' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-06-27T21:52:53.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 2 1 2024-06-27T21:52:53.995 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:54.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:54 smithi151 ceph-mon[28965]: pgmap v2459: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:54.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:54 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1022029677' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:52:54.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:54 smithi151 ceph-mon[28965]: osdmap e997: 12 total, 12 up, 12 in 2024-06-27T21:52:54.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:54 smithi190 ceph-mon[32247]: pgmap v2459: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:54.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:54 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1022029677' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:52:54.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:54 smithi190 ceph-mon[32247]: osdmap e997: 12 total, 12 up, 12 in 2024-06-27T21:52:54.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:54 smithi192 ceph-mon[32956]: pgmap v2459: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:54.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:54 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1022029677' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-06-27T21:52:54.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:54 smithi192 ceph-mon[32956]: osdmap e997: 12 total, 12 up, 12 in 2024-06-27T21:52:55.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:55 smithi151 ceph-mon[28965]: osdmap e998: 12 total, 12 up, 12 in 2024-06-27T21:52:55.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:55 smithi190 ceph-mon[32247]: osdmap e998: 12 total, 12 up, 12 in 2024-06-27T21:52:55.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:55 smithi192 ceph-mon[32956]: osdmap e998: 12 total, 12 up, 12 in 2024-06-27T21:52:56.122 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-06-27T21:52:56.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:56 smithi151 ceph-mon[28965]: pgmap v2462: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:56.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:56 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-27T21:52:56.446 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/210066279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:52:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:56 smithi190 ceph-mon[32247]: pgmap v2462: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:56 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-27T21:52:56.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/210066279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:52:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:56 smithi192 ceph-mon[32956]: pgmap v2462: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:56.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:56 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-06-27T21:52:56.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/210066279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:52:56.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 2 1 2024-06-27T21:52:57.150 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:52:57.403 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/210066279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:52:57.403 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:57 smithi151 ceph-mon[28965]: osdmap e999: 12 total, 12 up, 12 in 2024-06-27T21:52:57.403 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:57 smithi151 ceph-mon[28965]: pgmap v2464: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/210066279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:52:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:57 smithi190 ceph-mon[32247]: osdmap e999: 12 total, 12 up, 12 in 2024-06-27T21:52:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:57 smithi190 ceph-mon[32247]: pgmap v2464: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/210066279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:52:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:57 smithi192 ceph-mon[32956]: osdmap e999: 12 total, 12 up, 12 in 2024-06-27T21:52:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:57 smithi192 ceph-mon[32956]: pgmap v2464: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:58.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:58 smithi151 ceph-mon[28965]: 1.a deep-scrub starts 2024-06-27T21:52:58.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:58 smithi151 ceph-mon[28965]: 1.a deep-scrub ok 2024-06-27T21:52:58.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:58 smithi151 ceph-mon[28965]: 1.19 scrub starts 2024-06-27T21:52:58.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:58 smithi151 ceph-mon[28965]: 1.19 scrub ok 2024-06-27T21:52:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:58 smithi190 ceph-mon[32247]: 1.a deep-scrub starts 2024-06-27T21:52:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:58 smithi190 ceph-mon[32247]: 1.a deep-scrub ok 2024-06-27T21:52:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:58 smithi190 ceph-mon[32247]: 1.19 scrub starts 2024-06-27T21:52:58.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:58 smithi190 ceph-mon[32247]: 1.19 scrub ok 2024-06-27T21:52:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:58 smithi192 ceph-mon[32956]: 1.a deep-scrub starts 2024-06-27T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:58 smithi192 ceph-mon[32956]: 1.a deep-scrub ok 2024-06-27T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:58 smithi192 ceph-mon[32956]: 1.19 scrub starts 2024-06-27T21:52:58.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:58 smithi192 ceph-mon[32956]: 1.19 scrub ok 2024-06-27T21:52:59.149 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-06-27T21:52:59.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:59 smithi151 ceph-mon[28965]: pgmap v2465: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:59.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:52:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2277683853' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:52:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:59 smithi190 ceph-mon[32247]: pgmap v2465: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:59.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:52:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2277683853' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:52:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:59 smithi192 ceph-mon[32956]: pgmap v2465: 31 pgs: 4 peering, 27 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:52:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:52:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2277683853' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-06-27T21:52:59.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 3 1 2024-06-27T21:53:00.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2277683853' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:53:00.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:00 smithi151 ceph-mon[28965]: osdmap e1000: 12 total, 12 up, 12 in 2024-06-27T21:53:00.214 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:00.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2277683853' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:53:00.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:00 smithi190 ceph-mon[32247]: osdmap e1000: 12 total, 12 up, 12 in 2024-06-27T21:53:00.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2277683853' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-06-27T21:53:00.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:00 smithi192 ceph-mon[32956]: osdmap e1000: 12 total, 12 up, 12 in 2024-06-27T21:53:01.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:01 smithi151 ceph-mon[28965]: pgmap v2467: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:01.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:01 smithi190 ceph-mon[32247]: pgmap v2467: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:01.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:01 smithi192 ceph-mon[32956]: pgmap v2467: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:02.157 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-06-27T21:53:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:02 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-06-27T21:53:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:02 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:53:02.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:02 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1814671472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:53:02.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:02 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-06-27T21:53:02.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:02 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:53:02.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:02 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1814671472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:53:02.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:02 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 4 pgs peering) 2024-06-27T21:53:02.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:02 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:53:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:02 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1814671472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:53:02.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 3 1 2024-06-27T21:53:03.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:03 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:53:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:03 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:53:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1814671472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:53:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:03 smithi151 ceph-mon[28965]: osdmap e1001: 12 total, 12 up, 12 in 2024-06-27T21:53:03.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:03 smithi151 ceph-mon[28965]: pgmap v2469: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:03.233 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:03 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:53:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:03 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:53:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1814671472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:53:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:03 smithi190 ceph-mon[32247]: osdmap e1001: 12 total, 12 up, 12 in 2024-06-27T21:53:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:03 smithi190 ceph-mon[32247]: pgmap v2469: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:03 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:53:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:03 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:53:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1814671472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:53:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:03 smithi192 ceph-mon[32956]: osdmap e1001: 12 total, 12 up, 12 in 2024-06-27T21:53:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:03 smithi192 ceph-mon[32956]: pgmap v2469: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:04.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:04 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:53:04.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:04 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:53:04.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:04 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:53:04.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:04 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:53:04.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:04 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:53:04.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:04 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:53:05.175 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-06-27T21:53:05.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:05 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:53:05.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:05 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:53:05.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:05 smithi151 ceph-mon[28965]: pgmap v2470: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:05.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:05 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1412110328' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:53:05.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:05 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:53:05.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:05 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:53:05.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:05 smithi190 ceph-mon[32247]: pgmap v2470: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:05.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:05 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1412110328' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:53:05.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:05 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:53:05.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:05 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:53:05.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:05 smithi192 ceph-mon[32956]: pgmap v2470: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:05.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:05 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1412110328' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-06-27T21:53:05.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 5 1 2024-06-27T21:53:06.197 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:06.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1412110328' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:53:06.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:06 smithi151 ceph-mon[28965]: osdmap e1002: 12 total, 12 up, 12 in 2024-06-27T21:53:06.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:06 smithi151 ceph-mon[28965]: 1.1a deep-scrub starts 2024-06-27T21:53:06.443 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:06 smithi151 ceph-mon[28965]: 1.1a deep-scrub ok 2024-06-27T21:53:06.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1412110328' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:53:06.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:06 smithi190 ceph-mon[32247]: osdmap e1002: 12 total, 12 up, 12 in 2024-06-27T21:53:06.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:06 smithi190 ceph-mon[32247]: 1.1a deep-scrub starts 2024-06-27T21:53:06.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:06 smithi190 ceph-mon[32247]: 1.1a deep-scrub ok 2024-06-27T21:53:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1412110328' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-06-27T21:53:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:06 smithi192 ceph-mon[32956]: osdmap e1002: 12 total, 12 up, 12 in 2024-06-27T21:53:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:06 smithi192 ceph-mon[32956]: 1.1a deep-scrub starts 2024-06-27T21:53:06.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:06 smithi192 ceph-mon[32956]: 1.1a deep-scrub ok 2024-06-27T21:53:07.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:07 smithi151 ceph-mon[28965]: pgmap v2472: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:07.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:07 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:07.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:07 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:07.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:07 smithi190 ceph-mon[32247]: pgmap v2472: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:07.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:07 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:07.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:07 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:07.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:07 smithi192 ceph-mon[32956]: pgmap v2472: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:07.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:07 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:07.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:07 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:08.190 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-06-27T21:53:08.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:08 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:08 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:08 smithi151 ceph-mon[28965]: osdmap e1003: 12 total, 12 up, 12 in 2024-06-27T21:53:08.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:08 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4199178365' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:53:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:08 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:08 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:08 smithi190 ceph-mon[32247]: osdmap e1003: 12 total, 12 up, 12 in 2024-06-27T21:53:08.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:08 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4199178365' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:53:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:08 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:08 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:08 smithi192 ceph-mon[32956]: osdmap e1003: 12 total, 12 up, 12 in 2024-06-27T21:53:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:08 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4199178365' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:53:08.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 5 1 2024-06-27T21:53:09.103 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:09 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4199178365' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:53:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:09 smithi151 ceph-mon[28965]: osdmap e1004: 12 total, 12 up, 12 in 2024-06-27T21:53:09.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:09 smithi151 ceph-mon[28965]: pgmap v2475: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:09.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:09 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4199178365' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:53:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:09 smithi190 ceph-mon[32247]: osdmap e1004: 12 total, 12 up, 12 in 2024-06-27T21:53:09.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:09 smithi190 ceph-mon[32247]: pgmap v2475: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:09.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:09 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4199178365' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:53:09.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:09 smithi192 ceph-mon[32956]: osdmap e1004: 12 total, 12 up, 12 in 2024-06-27T21:53:09.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:09 smithi192 ceph-mon[32956]: pgmap v2475: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:12.026 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-06-27T21:53:12.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:12 smithi192 ceph-mon[32956]: pgmap v2476: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:12.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:12 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/156087957' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:53:12.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:12 smithi151 ceph-mon[28965]: pgmap v2476: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:12.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:12 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/156087957' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:53:12.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:12 smithi190 ceph-mon[32247]: pgmap v2476: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:12.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:12 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/156087957' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-06-27T21:53:12.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 6 1 2024-06-27T21:53:12.993 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:13.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:13 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:53:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:13 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:53:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:13 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/156087957' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:53:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:13 smithi192 ceph-mon[32956]: osdmap e1005: 12 total, 12 up, 12 in 2024-06-27T21:53:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:13 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:53:13.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:13 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:53:13.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:13 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:53:13.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:13 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:53:13.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:13 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/156087957' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:53:13.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:13 smithi151 ceph-mon[28965]: osdmap e1005: 12 total, 12 up, 12 in 2024-06-27T21:53:13.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:13 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:53:13.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:13 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:53:13.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:13 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:53:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:13 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:53:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:13 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/156087957' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-06-27T21:53:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:13 smithi190 ceph-mon[32247]: osdmap e1005: 12 total, 12 up, 12 in 2024-06-27T21:53:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:13 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:53:13.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:13 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:53:14.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:14 smithi151 ceph-mon[28965]: pgmap v2478: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:14.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:14 smithi151 ceph-mon[28965]: osdmap e1006: 12 total, 12 up, 12 in 2024-06-27T21:53:14.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:14 smithi192 ceph-mon[32956]: pgmap v2478: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:14.295 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:14 smithi192 ceph-mon[32956]: osdmap e1006: 12 total, 12 up, 12 in 2024-06-27T21:53:14.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:14 smithi190 ceph-mon[32247]: pgmap v2478: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:14.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:14 smithi190 ceph-mon[32247]: osdmap e1006: 12 total, 12 up, 12 in 2024-06-27T21:53:15.040 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-06-27T21:53:15.374 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:15 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:53:15.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:15 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:53:15.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:15 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4035791887' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:53:15.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:15 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:53:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:15 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:53:15.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:15 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4035791887' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:53:15.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:15 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:53:15.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:15 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:53:15.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:15 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4035791887' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:53:15.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 6 1 2024-06-27T21:53:16.054 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:16.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:16 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:53:16.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:16 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:53:16.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:16 smithi151 ceph-mon[28965]: pgmap v2480: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:16.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:16 smithi151 ceph-mon[28965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:53:16.330 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:16 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4035791887' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:53:16.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:16 smithi151 ceph-mon[28965]: osdmap e1007: 12 total, 12 up, 12 in 2024-06-27T21:53:16.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:16 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:53:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:16 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:53:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:16 smithi190 ceph-mon[32247]: pgmap v2480: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:16 smithi190 ceph-mon[32247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:53:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:16 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4035791887' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:53:16.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:16 smithi190 ceph-mon[32247]: osdmap e1007: 12 total, 12 up, 12 in 2024-06-27T21:53:16.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:16 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:53:16.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:16 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:53:16.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:16 smithi192 ceph-mon[32956]: pgmap v2480: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:16.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:16 smithi192 ceph-mon[32956]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-27T21:53:16.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:16 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4035791887' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:53:16.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:16 smithi192 ceph-mon[32956]: osdmap e1007: 12 total, 12 up, 12 in 2024-06-27T21:53:18.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:18 smithi151 ceph-mon[28965]: pgmap v2482: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:18.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:18 smithi190 ceph-mon[32247]: pgmap v2482: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:18.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:18 smithi192 ceph-mon[32956]: pgmap v2482: 31 pgs: 1 peering, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:19.061 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-06-27T21:53:19.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:19 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:53:19.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:19 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:53:19.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:19 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2702861002' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:53:19.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:19 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:53:19.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:19 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:53:19.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:19 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2702861002' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:53:19.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:19 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:53:19.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:19 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:53:19.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:19 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2702861002' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-06-27T21:53:19.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 7 1 2024-06-27T21:53:20.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:20.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:20 smithi151 ceph-mon[28965]: pgmap v2483: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:20.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:20 smithi151 ceph-mon[28965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:53:20.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:20 smithi151 ceph-mon[28965]: Cluster is now healthy 2024-06-27T21:53:20.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:20 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2702861002' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:53:20.412 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:20 smithi151 ceph-mon[28965]: osdmap e1008: 12 total, 12 up, 12 in 2024-06-27T21:53:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:20 smithi190 ceph-mon[32247]: pgmap v2483: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:20 smithi190 ceph-mon[32247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:53:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:20 smithi190 ceph-mon[32247]: Cluster is now healthy 2024-06-27T21:53:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:20 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2702861002' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:53:20.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:20 smithi190 ceph-mon[32247]: osdmap e1008: 12 total, 12 up, 12 in 2024-06-27T21:53:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:20 smithi192 ceph-mon[32956]: pgmap v2483: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:20 smithi192 ceph-mon[32956]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-27T21:53:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:20 smithi192 ceph-mon[32956]: Cluster is now healthy 2024-06-27T21:53:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:20 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2702861002' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-06-27T21:53:20.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:20 smithi192 ceph-mon[32956]: osdmap e1008: 12 total, 12 up, 12 in 2024-06-27T21:53:21.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:21 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:21 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:21 smithi151 ceph-mon[28965]: osdmap e1009: 12 total, 12 up, 12 in 2024-06-27T21:53:21.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:21 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:53:21.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:21 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:21.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:21 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:21.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:21 smithi190 ceph-mon[32247]: osdmap e1009: 12 total, 12 up, 12 in 2024-06-27T21:53:21.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:21 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:53:21.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:21 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:21.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:21 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:21.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:21 smithi192 ceph-mon[32956]: osdmap e1009: 12 total, 12 up, 12 in 2024-06-27T21:53:21.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:21 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:53:22.424 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-06-27T21:53:22.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:22 smithi151 ceph-mon[28965]: pgmap v2486: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:22.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:53:22.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:53:22.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:53:22.425 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:22 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3960596695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:53:22.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:22 smithi190 ceph-mon[32247]: pgmap v2486: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:53:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:53:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:53:22.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:22 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3960596695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:53:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:22 smithi192 ceph-mon[32956]: pgmap v2486: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:53:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:53:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:53:22.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:22 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3960596695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:53:23.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 7 1 2024-06-27T21:53:23.351 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:23.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:23 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:53:23.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:23 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:53:23.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:23 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3960596695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:53:23.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:23 smithi151 ceph-mon[28965]: osdmap e1010: 12 total, 12 up, 12 in 2024-06-27T21:53:23.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:23 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:53:23.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:23 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:53:23.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:23 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3960596695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:53:23.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:23 smithi190 ceph-mon[32247]: osdmap e1010: 12 total, 12 up, 12 in 2024-06-27T21:53:23.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:23 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:53:23.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:23 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:53:23.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:23 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3960596695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:53:23.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:23 smithi192 ceph-mon[32956]: osdmap e1010: 12 total, 12 up, 12 in 2024-06-27T21:53:24.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:24 smithi151 ceph-mon[28965]: pgmap v2487: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:24.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:24 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:24.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:24 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:24.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:24 smithi190 ceph-mon[32247]: pgmap v2487: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:24.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:24 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:24.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:24 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:24.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:24 smithi192 ceph-mon[32956]: pgmap v2487: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:24.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:24 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:24.577 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:24 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:25.228 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:25 smithi151 ceph-mon[28965]: osdmap e1011: 12 total, 12 up, 12 in 2024-06-27T21:53:25.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:25 smithi190 ceph-mon[32247]: osdmap e1011: 12 total, 12 up, 12 in 2024-06-27T21:53:25.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:25 smithi192 ceph-mon[32956]: osdmap e1011: 12 total, 12 up, 12 in 2024-06-27T21:53:26.095 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-06-27T21:53:26.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:26 smithi151 ceph-mon[28965]: pgmap v2490: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:26.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:26 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3972474290' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:53:26.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:26 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:26.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:26 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:26.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:26 smithi190 ceph-mon[32247]: pgmap v2490: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:26.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:26 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3972474290' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:53:26.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:26 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:26.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:26 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:26 smithi192 ceph-mon[32956]: pgmap v2490: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:26 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3972474290' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-06-27T21:53:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:26 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:26 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:26.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 8 1 2024-06-27T21:53:27.156 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:27.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:27 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3972474290' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:53:27.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:27 smithi151 ceph-mon[28965]: osdmap e1012: 12 total, 12 up, 12 in 2024-06-27T21:53:27.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:27 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:27.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:27 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:27.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:27 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3972474290' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:53:27.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:27 smithi190 ceph-mon[32247]: osdmap e1012: 12 total, 12 up, 12 in 2024-06-27T21:53:27.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:27 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:27.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:27 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:27.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:27 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3972474290' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-06-27T21:53:27.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:27 smithi192 ceph-mon[32956]: osdmap e1012: 12 total, 12 up, 12 in 2024-06-27T21:53:27.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:27 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:27.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:27 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:28.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:28 smithi151 ceph-mon[28965]: pgmap v2492: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:28.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:28 smithi151 ceph-mon[28965]: osdmap e1013: 12 total, 12 up, 12 in 2024-06-27T21:53:28.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:28 smithi151 ceph-mon[28965]: 1.10 scrub starts 2024-06-27T21:53:28.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:28 smithi151 ceph-mon[28965]: 1.10 scrub ok 2024-06-27T21:53:28.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:28 smithi190 ceph-mon[32247]: pgmap v2492: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:28.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:28 smithi190 ceph-mon[32247]: osdmap e1013: 12 total, 12 up, 12 in 2024-06-27T21:53:28.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:28 smithi190 ceph-mon[32247]: 1.10 scrub starts 2024-06-27T21:53:28.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:28 smithi190 ceph-mon[32247]: 1.10 scrub ok 2024-06-27T21:53:28.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:28 smithi192 ceph-mon[32956]: pgmap v2492: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:28.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:28 smithi192 ceph-mon[32956]: osdmap e1013: 12 total, 12 up, 12 in 2024-06-27T21:53:28.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:28 smithi192 ceph-mon[32956]: 1.10 scrub starts 2024-06-27T21:53:28.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:28 smithi192 ceph-mon[32956]: 1.10 scrub ok 2024-06-27T21:53:29.121 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-06-27T21:53:29.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:29 smithi151 ceph-mon[28965]: 1.13 scrub starts 2024-06-27T21:53:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:29 smithi151 ceph-mon[28965]: 1.13 scrub ok 2024-06-27T21:53:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:29 smithi151 ceph-mon[28965]: pgmap v2494: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:29 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:29 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:29.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:29 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2531185208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:53:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:29 smithi190 ceph-mon[32247]: 1.13 scrub starts 2024-06-27T21:53:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:29 smithi190 ceph-mon[32247]: 1.13 scrub ok 2024-06-27T21:53:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:29 smithi190 ceph-mon[32247]: pgmap v2494: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:29 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:29.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:29 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:29.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:29 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2531185208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:53:29.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:29 smithi192 ceph-mon[32956]: 1.13 scrub starts 2024-06-27T21:53:29.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:29 smithi192 ceph-mon[32956]: 1.13 scrub ok 2024-06-27T21:53:29.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:29 smithi192 ceph-mon[32956]: pgmap v2494: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:29.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:29 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:29.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:29 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:29.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:29 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2531185208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:53:29.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 8 1 2024-06-27T21:53:30.120 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:30.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:30 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:53:30.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:30 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:53:30.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:30 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2531185208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:53:30.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:30 smithi151 ceph-mon[28965]: osdmap e1014: 12 total, 12 up, 12 in 2024-06-27T21:53:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:30 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:53:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:30 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:53:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:30 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2531185208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:53:30.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:30 smithi190 ceph-mon[32247]: osdmap e1014: 12 total, 12 up, 12 in 2024-06-27T21:53:30.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:30 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:53:30.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:30 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:53:30.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:30 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2531185208' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:53:30.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:30 smithi192 ceph-mon[32956]: osdmap e1014: 12 total, 12 up, 12 in 2024-06-27T21:53:31.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:31 smithi151 ceph-mon[28965]: pgmap v2496: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:31.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:31 smithi190 ceph-mon[32247]: pgmap v2496: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:31.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:31 smithi192 ceph-mon[32956]: pgmap v2496: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:32.131 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-06-27T21:53:32.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:32 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:53:32.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:32 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:53:32.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:32 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:53:32.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:32 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:53:32.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:32 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/916426397' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:53:32.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:32 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:53:32.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:32 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:53:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:32 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:53:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:32 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:53:32.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:32 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/916426397' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:53:32.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:32 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:53:32.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:32 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:53:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:32 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:53:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:32 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:53:32.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:32 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/916426397' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-06-27T21:53:32.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 9 1 2024-06-27T21:53:33.133 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:33.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:33 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/916426397' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:53:33.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:33 smithi151 ceph-mon[28965]: osdmap e1015: 12 total, 12 up, 12 in 2024-06-27T21:53:33.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:33 smithi151 ceph-mon[28965]: pgmap v2498: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:33.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:33 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/916426397' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:53:33.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:33 smithi190 ceph-mon[32247]: osdmap e1015: 12 total, 12 up, 12 in 2024-06-27T21:53:33.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:33 smithi190 ceph-mon[32247]: pgmap v2498: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:33.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:33 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/916426397' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-06-27T21:53:33.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:33 smithi192 ceph-mon[32956]: osdmap e1015: 12 total, 12 up, 12 in 2024-06-27T21:53:33.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:33 smithi192 ceph-mon[32956]: pgmap v2498: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:34.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:34 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:34.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:34 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:34.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:34 smithi151 ceph-mon[28965]: osdmap e1016: 12 total, 12 up, 12 in 2024-06-27T21:53:34.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:34 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:34.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:34 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:34.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:34 smithi190 ceph-mon[32247]: osdmap e1016: 12 total, 12 up, 12 in 2024-06-27T21:53:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:34 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:34 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:34 smithi192 ceph-mon[32956]: osdmap e1016: 12 total, 12 up, 12 in 2024-06-27T21:53:35.152 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-06-27T21:53:35.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:35 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:53:35.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:35 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:53:35.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:35 smithi151 ceph-mon[28965]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:35.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:35 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/500665278' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:53:35.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:35 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:53:35.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:35 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:53:35.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:35 smithi190 ceph-mon[32247]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:35.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:35 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/500665278' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:53:35.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:35 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:53:35.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:35 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:53:35.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:35 smithi192 ceph-mon[32956]: pgmap v2500: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:35.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:35 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/500665278' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:53:35.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 9 1 2024-06-27T21:53:36.194 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:36 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/500665278' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:53:36.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:36 smithi151 ceph-mon[28965]: osdmap e1017: 12 total, 12 up, 12 in 2024-06-27T21:53:36.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:36.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:36.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:36 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/500665278' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:53:36.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:36 smithi190 ceph-mon[32247]: osdmap e1017: 12 total, 12 up, 12 in 2024-06-27T21:53:36.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:36.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:36.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:36 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/500665278' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:53:36.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:36 smithi192 ceph-mon[32956]: osdmap e1017: 12 total, 12 up, 12 in 2024-06-27T21:53:37.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:37 smithi151 ceph-mon[28965]: pgmap v2502: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:37.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:37 smithi190 ceph-mon[32247]: pgmap v2502: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:37 smithi192 ceph-mon[32956]: pgmap v2502: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:38.160 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-06-27T21:53:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:38 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:38 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:38 smithi151 ceph-mon[28965]: osdmap e1018: 12 total, 12 up, 12 in 2024-06-27T21:53:38.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:38 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2977720327' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:53:38.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:38 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:38.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:38 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:38.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:38 smithi190 ceph-mon[32247]: osdmap e1018: 12 total, 12 up, 12 in 2024-06-27T21:53:38.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:38 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2977720327' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:53:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:38 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:38 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:38 smithi192 ceph-mon[32956]: osdmap e1018: 12 total, 12 up, 12 in 2024-06-27T21:53:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:38 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2977720327' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-06-27T21:53:38.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 10 1 2024-06-27T21:53:39.158 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:39.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:39 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2977720327' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:53:39.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:39 smithi151 ceph-mon[28965]: osdmap e1019: 12 total, 12 up, 12 in 2024-06-27T21:53:39.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:39 smithi151 ceph-mon[28965]: pgmap v2505: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:39.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:39 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2977720327' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:53:39.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:39 smithi190 ceph-mon[32247]: osdmap e1019: 12 total, 12 up, 12 in 2024-06-27T21:53:39.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:39 smithi190 ceph-mon[32247]: pgmap v2505: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:39.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:39 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2977720327' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-06-27T21:53:39.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:39 smithi192 ceph-mon[32956]: osdmap e1019: 12 total, 12 up, 12 in 2024-06-27T21:53:39.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:39 smithi192 ceph-mon[32956]: pgmap v2505: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:42.045 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-06-27T21:53:42.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:42 smithi151 ceph-mon[28965]: pgmap v2506: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:42.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:42 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:53:42.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:42 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:53:42.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:42 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4103787227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:53:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:42 smithi190 ceph-mon[32247]: pgmap v2506: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:42 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:53:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:42 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:53:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:42 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4103787227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:53:42.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:42 smithi192 ceph-mon[32956]: pgmap v2506: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:42.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:42 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:53:42.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:42 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:53:42.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:42 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4103787227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:53:42.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 10 1 2024-06-27T21:53:43.086 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:43.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:43 smithi151 ceph-mon[28965]: 1.17 scrub starts 2024-06-27T21:53:43.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:43 smithi151 ceph-mon[28965]: 1.17 scrub ok 2024-06-27T21:53:43.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4103787227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:53:43.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:43 smithi151 ceph-mon[28965]: osdmap e1020: 12 total, 12 up, 12 in 2024-06-27T21:53:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:43 smithi190 ceph-mon[32247]: 1.17 scrub starts 2024-06-27T21:53:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:43 smithi190 ceph-mon[32247]: 1.17 scrub ok 2024-06-27T21:53:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4103787227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:53:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:43 smithi190 ceph-mon[32247]: osdmap e1020: 12 total, 12 up, 12 in 2024-06-27T21:53:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:43 smithi192 ceph-mon[32956]: 1.17 scrub starts 2024-06-27T21:53:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:43 smithi192 ceph-mon[32956]: 1.17 scrub ok 2024-06-27T21:53:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4103787227' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:53:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:43 smithi192 ceph-mon[32956]: osdmap e1020: 12 total, 12 up, 12 in 2024-06-27T21:53:44.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:44 smithi151 ceph-mon[28965]: pgmap v2508: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:44.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:44 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:44.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:44 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:44 smithi190 ceph-mon[32247]: pgmap v2508: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:44 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:44.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:44 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:44 smithi192 ceph-mon[32956]: pgmap v2508: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:44 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:44 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:45.063 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-06-27T21:53:45.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:45 smithi151 ceph-mon[28965]: osdmap e1021: 12 total, 12 up, 12 in 2024-06-27T21:53:45.341 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:45 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1871312127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:53:45.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:45 smithi190 ceph-mon[32247]: osdmap e1021: 12 total, 12 up, 12 in 2024-06-27T21:53:45.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:45 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1871312127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:53:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:45 smithi192 ceph-mon[32956]: osdmap e1021: 12 total, 12 up, 12 in 2024-06-27T21:53:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:45 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1871312127' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-06-27T21:53:45.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 11 1 2024-06-27T21:53:46.096 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:46.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:46 smithi151 ceph-mon[28965]: pgmap v2510: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:46.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:46 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:53:46.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:46 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:53:46.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:46 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1871312127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:53:46.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:46 smithi151 ceph-mon[28965]: osdmap e1022: 12 total, 12 up, 12 in 2024-06-27T21:53:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:46 smithi190 ceph-mon[32247]: pgmap v2510: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:46 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:53:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:46 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:53:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:46 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1871312127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:53:46.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:46 smithi190 ceph-mon[32247]: osdmap e1022: 12 total, 12 up, 12 in 2024-06-27T21:53:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:46 smithi192 ceph-mon[32956]: pgmap v2510: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:46 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:53:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:46 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:53:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:46 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1871312127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-06-27T21:53:46.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:46 smithi192 ceph-mon[32956]: osdmap e1022: 12 total, 12 up, 12 in 2024-06-27T21:53:47.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:47 smithi151 ceph-mon[28965]: 1.4 scrub starts 2024-06-27T21:53:47.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:47 smithi151 ceph-mon[28965]: 1.4 scrub ok 2024-06-27T21:53:47.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:47 smithi190 ceph-mon[32247]: 1.4 scrub starts 2024-06-27T21:53:47.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:47 smithi190 ceph-mon[32247]: 1.4 scrub ok 2024-06-27T21:53:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:47 smithi192 ceph-mon[32956]: 1.4 scrub starts 2024-06-27T21:53:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:47 smithi192 ceph-mon[32956]: 1.4 scrub ok 2024-06-27T21:53:48.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:48 smithi151 ceph-mon[28965]: pgmap v2512: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:48.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:48 smithi151 ceph-mon[28965]: 1.1b scrub starts 2024-06-27T21:53:48.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:48 smithi151 ceph-mon[28965]: 1.1b scrub ok 2024-06-27T21:53:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:48 smithi190 ceph-mon[32247]: pgmap v2512: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:48 smithi190 ceph-mon[32247]: 1.1b scrub starts 2024-06-27T21:53:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:48 smithi190 ceph-mon[32247]: 1.1b scrub ok 2024-06-27T21:53:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:48 smithi192 ceph-mon[32956]: pgmap v2512: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:48 smithi192 ceph-mon[32956]: 1.1b scrub starts 2024-06-27T21:53:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:48 smithi192 ceph-mon[32956]: 1.1b scrub ok 2024-06-27T21:53:49.089 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-06-27T21:53:49.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:49 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2223866270' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:53:49.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:49 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2223866270' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:53:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:49 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2223866270' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:53:49.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 11 1 2024-06-27T21:53:50.114 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:50.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:50 smithi151 ceph-mon[28965]: pgmap v2513: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:50.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2223866270' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:53:50.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:50 smithi151 ceph-mon[28965]: osdmap e1023: 12 total, 12 up, 12 in 2024-06-27T21:53:50.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:53:50.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:53:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:50 smithi190 ceph-mon[32247]: pgmap v2513: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2223866270' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:53:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:50 smithi190 ceph-mon[32247]: osdmap e1023: 12 total, 12 up, 12 in 2024-06-27T21:53:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:53:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:53:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:50 smithi192 ceph-mon[32956]: pgmap v2513: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2223866270' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:53:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:50 smithi192 ceph-mon[32956]: osdmap e1023: 12 total, 12 up, 12 in 2024-06-27T21:53:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:53:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:53:51.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:51 smithi151 ceph-mon[28965]: osdmap e1024: 12 total, 12 up, 12 in 2024-06-27T21:53:51.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:51 smithi190 ceph-mon[32247]: osdmap e1024: 12 total, 12 up, 12 in 2024-06-27T21:53:51.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:51 smithi192 ceph-mon[32956]: osdmap e1024: 12 total, 12 up, 12 in 2024-06-27T21:53:52.101 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-06-27T21:53:52.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:52 smithi151 ceph-mon[28965]: pgmap v2516: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:52.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:52 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2989138838' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:53:52.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:52 smithi190 ceph-mon[32247]: pgmap v2516: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:52.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:52 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2989138838' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:53:52.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:52 smithi192 ceph-mon[32956]: pgmap v2516: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:52.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:52 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2989138838' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-06-27T21:53:52.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd reweight 4 1 2024-06-27T21:53:53.149 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:53.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2989138838' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:53:53.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:53 smithi151 ceph-mon[28965]: osdmap e1025: 12 total, 12 up, 12 in 2024-06-27T21:53:53.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:53 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:53:53.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:53 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:53:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2989138838' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:53:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:53 smithi190 ceph-mon[32247]: osdmap e1025: 12 total, 12 up, 12 in 2024-06-27T21:53:53.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:53 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:53:53.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:53 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:53:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2989138838' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-06-27T21:53:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:53 smithi192 ceph-mon[32956]: osdmap e1025: 12 total, 12 up, 12 in 2024-06-27T21:53:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:53 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:53:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:53 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:53:54.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:54 smithi151 ceph-mon[28965]: pgmap v2518: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:54.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:54 smithi190 ceph-mon[32247]: pgmap v2518: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:54.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:54 smithi192 ceph-mon[32956]: pgmap v2518: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:55.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:55 smithi151 ceph-mon[28965]: 1.15 scrub starts 2024-06-27T21:53:55.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:55 smithi151 ceph-mon[28965]: 1.15 scrub ok 2024-06-27T21:53:55.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:55 smithi190 ceph-mon[32247]: 1.15 scrub starts 2024-06-27T21:53:55.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:55 smithi190 ceph-mon[32247]: 1.15 scrub ok 2024-06-27T21:53:55.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:55 smithi192 ceph-mon[32956]: 1.15 scrub starts 2024-06-27T21:53:55.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:55 smithi192 ceph-mon[32956]: 1.15 scrub ok 2024-06-27T21:53:56.120 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-06-27T21:53:56.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:56 smithi151 ceph-mon[28965]: pgmap v2519: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:56.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/692574035' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:53:56.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:56 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:53:56.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:56 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:53:56.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:56 smithi190 ceph-mon[32247]: pgmap v2519: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:56.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/692574035' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:53:56.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:56 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:53:56.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:56 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:53:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:56 smithi192 ceph-mon[32956]: pgmap v2519: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/692574035' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:53:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:56 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:53:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:56 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:53:56.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd primary-affinity 4 1 2024-06-27T21:53:57.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/692574035' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:53:57.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: osdmap e1026: 12 total, 12 up, 12 in 2024-06-27T21:53:57.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: 1.5 scrub starts 2024-06-27T21:53:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: 1.5 scrub ok 2024-06-27T21:53:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: pgmap v2521: 31 pgs: 2 active+clean+scrubbing, 29 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: 1.9 scrub starts 2024-06-27T21:53:57.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:57 smithi151 ceph-mon[28965]: 1.9 scrub ok 2024-06-27T21:53:57.242 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:53:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/692574035' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:53:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: osdmap e1026: 12 total, 12 up, 12 in 2024-06-27T21:53:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: 1.5 scrub starts 2024-06-27T21:53:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: 1.5 scrub ok 2024-06-27T21:53:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: pgmap v2521: 31 pgs: 2 active+clean+scrubbing, 29 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: 1.9 scrub starts 2024-06-27T21:53:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:57 smithi190 ceph-mon[32247]: 1.9 scrub ok 2024-06-27T21:53:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/692574035' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:53:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: osdmap e1026: 12 total, 12 up, 12 in 2024-06-27T21:53:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: 1.5 scrub starts 2024-06-27T21:53:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: 1.5 scrub ok 2024-06-27T21:53:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: pgmap v2521: 31 pgs: 2 active+clean+scrubbing, 29 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: 1.9 scrub starts 2024-06-27T21:53:57.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:57 smithi192 ceph-mon[32956]: 1.9 scrub ok 2024-06-27T21:53:59.467 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-06-27T21:53:59.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:59 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:53:59.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:59 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:53:59.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:59 smithi190 ceph-mon[32247]: pgmap v2522: 31 pgs: 2 active+clean+scrubbing, 29 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:59.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:53:59 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/255974513' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:53:59.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:59 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:53:59.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:59 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:53:59.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:59 smithi151 ceph-mon[28965]: pgmap v2522: 31 pgs: 2 active+clean+scrubbing, 29 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:59.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:53:59 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/255974513' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:53:59.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:59 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:53:59.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:59 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:53:59.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:59 smithi192 ceph-mon[32956]: pgmap v2522: 31 pgs: 2 active+clean+scrubbing, 29 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:53:59.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:53:59 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/255974513' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-06-27T21:54:00.118 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-06-27T21:54:00.119 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-06-27T21:54:00.119 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-06-27T21:54:00.119 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-06-27T21:54:00.119 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-27T21:54:00.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump --format=json 2024-06-27T21:54:00.369 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:00.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:00 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/255974513' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:54:00.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:00 smithi190 ceph-mon[32247]: osdmap e1027: 12 total, 12 up, 12 in 2024-06-27T21:54:00.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:00 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:54:00.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:00 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:54:00.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:00 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/255974513' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:54:00.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:00 smithi192 ceph-mon[32956]: osdmap e1027: 12 total, 12 up, 12 in 2024-06-27T21:54:00.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:00 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:54:00.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:00 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:54:00.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:00 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/255974513' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-06-27T21:54:00.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:00 smithi151 ceph-mon[28965]: osdmap e1027: 12 total, 12 up, 12 in 2024-06-27T21:54:00.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:00 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:54:00.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:00 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:54:02.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:02 smithi151 ceph-mon[28965]: pgmap v2524: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:02.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:02 smithi192 ceph-mon[32956]: pgmap v2524: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:02.365 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:54:02.365 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1027,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","created":"2024-06-27T21:02:54.280751+0000","modified":"2024-06-27T21:53:59.466046+0000","last_up_change":"2024-06-27T21:23:43.806967+0000","last_in_change":"2024-06-27T21:46:33.505233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-27T21:06:07.812474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"990","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"535","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80c5e023-5c22-463d-86e9-2307159c858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":872,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6803","nonce":2519434429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6805","nonce":2519434429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6809","nonce":2519434429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6807","nonce":2519434429}]},"public_addr":"172.21.15.151:6803/2519434429","cluster_addr":"172.21.15.151:6805/2519434429","heartbeat_back_addr":"172.21.15.151:6809/2519434429","heartbeat_front_addr":"172.21.15.151:6807/2519434429","state":["exists","up"]},{"osd":1,"uuid":"d6a0fa83-bdab-4c01-b0d0-8996e8d0c083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6811","nonce":59941028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6813","nonce":59941028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6817","nonce":59941028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6815","nonce":59941028}]},"public_addr":"172.21.15.151:6811/59941028","cluster_addr":"172.21.15.151:6813/59941028","heartbeat_back_addr":"172.21.15.151:6817/59941028","heartbeat_front_addr":"172.21.15.151:6815/59941028","state":["exists","up"]},{"osd":2,"uuid":"22115999-54ff-4f34-81cc-9d90bfa4bd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":909,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6819","nonce":302970170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6821","nonce":302970170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6825","nonce":302970170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6823","nonce":302970170}]},"public_addr":"172.21.15.151:6819/302970170","cluster_addr":"172.21.15.151:6821/302970170","heartbeat_back_addr":"172.21.15.151:6825/302970170","heartbeat_front_addr":"172.21.15.151:6823/302970170","state":["exists","up"]},{"osd":3,"uuid":"88c12fa5-d2af-4144-958e-b887b4474b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6827","nonce":3555754662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6829","nonce":3555754662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6833","nonce":3555754662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6831","nonce":3555754662}]},"public_addr":"172.21.15.151:6827/3555754662","cluster_addr":"172.21.15.151:6829/3555754662","heartbeat_back_addr":"172.21.15.151:6833/3555754662","heartbeat_front_addr":"172.21.15.151:6831/3555754662","state":["exists","up"]},{"osd":4,"uuid":"97aaddb9-d5bb-4827-8c35-b3c948eae3dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":347,"up_from":362,"up_thru":920,"down_at":348,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6801","nonce":3587059633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6803","nonce":3587059633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6807","nonce":3587059633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6805","nonce":3587059633}]},"public_addr":"172.21.15.190:6801/3587059633","cluster_addr":"172.21.15.190:6803/3587059633","heartbeat_back_addr":"172.21.15.190:6807/3587059633","heartbeat_front_addr":"172.21.15.190:6805/3587059633","state":["exists","up"]},{"osd":5,"uuid":"7ac9bbd6-0e55-4cab-b406-6f9967eaaf30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1005,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6809","nonce":888361120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6811","nonce":888361120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6815","nonce":888361120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6813","nonce":888361120}]},"public_addr":"172.21.15.190:6809/888361120","cluster_addr":"172.21.15.190:6811/888361120","heartbeat_back_addr":"172.21.15.190:6815/888361120","heartbeat_front_addr":"172.21.15.190:6813/888361120","state":["exists","up"]},{"osd":6,"uuid":"89fae0b2-26a4-456a-9d54-cfbaec80ba6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1008,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6817","nonce":30028280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6819","nonce":30028280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6823","nonce":30028280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6821","nonce":30028280}]},"public_addr":"172.21.15.190:6817/30028280","cluster_addr":"172.21.15.190:6819/30028280","heartbeat_back_addr":"172.21.15.190:6823/30028280","heartbeat_front_addr":"172.21.15.190:6821/30028280","state":["exists","up"]},{"osd":7,"uuid":"e324ffdc-9842-46ee-a87e-5054ec265484","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6825","nonce":849289685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6827","nonce":849289685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6831","nonce":849289685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6829","nonce":849289685}]},"public_addr":"172.21.15.190:6825/849289685","cluster_addr":"172.21.15.190:6827/849289685","heartbeat_back_addr":"172.21.15.190:6831/849289685","heartbeat_front_addr":"172.21.15.190:6829/849289685","state":["exists","up"]},{"osd":8,"uuid":"43d138bf-2d82-44ea-a98f-b9099e824243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":924,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6801","nonce":2977299942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6803","nonce":2977299942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6807","nonce":2977299942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6805","nonce":2977299942}]},"public_addr":"172.21.15.192:6801/2977299942","cluster_addr":"172.21.15.192:6803/2977299942","heartbeat_back_addr":"172.21.15.192:6807/2977299942","heartbeat_front_addr":"172.21.15.192:6805/2977299942","state":["exists","up"]},{"osd":9,"uuid":"0c86c106-04c9-4eb7-bb3e-d39f563b71fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":924,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6809","nonce":3401759799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6811","nonce":3401759799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6815","nonce":3401759799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6813","nonce":3401759799}]},"public_addr":"172.21.15.192:6809/3401759799","cluster_addr":"172.21.15.192:6811/3401759799","heartbeat_back_addr":"172.21.15.192:6815/3401759799","heartbeat_front_addr":"172.21.15.192:6813/3401759799","state":["exists","up"]},{"osd":10,"uuid":"ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":920,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6817","nonce":1426195762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6819","nonce":1426195762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6823","nonce":1426195762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6821","nonce":1426195762}]},"public_addr":"172.21.15.192:6817/1426195762","cluster_addr":"172.21.15.192:6819/1426195762","heartbeat_back_addr":"172.21.15.192:6823/1426195762","heartbeat_front_addr":"172.21.15.192:6821/1426195762","state":["exists","up"]},{"osd":11,"uuid":"a8a163c4-a14c-4710-bc2e-94e2cb69df0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":983,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6825","nonce":517567267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6827","nonce":517567267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6831","nonce":517567267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6829","nonce":517567267}]},"public_addr":"172.21.15.192:6825/517567267","cluster_addr":"172.21.15.192:6827/517567267","heartbeat_back_addr":"172.21.15.192:6831/517567267","heartbeat_front_addr":"172.21.15.192:6829/517567267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:06.879037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:32.913880+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:06.566380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:20.044462+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:23.350143+0000","dead_epoch":347},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:25.788819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:49.555970+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:43.486691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:26.187438+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:37.099644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:28.938368+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:34.532031+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[1,9,10]},{"pgid":"1.1","osds":[0,9,11]}],"pg_upmap_items":[{"pgid":"1.1e","mappings":[{"from":7,"to":6},{"from":8,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/4089480652":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3356605898":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3908626376":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3261020553":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/2698418425":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/335837712":"2024-06-28T21:03:22.780955+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-27T21:54:02.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:02 smithi190 ceph-mon[32247]: pgmap v2524: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:02.976 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-27T21:54:02.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd dump --format=json 2024-06-27T21:54:03.228 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:03.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:03 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:54:03.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:03 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:54:03.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:03 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2220981358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:54:03.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:03 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:54:03.350 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:03 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:54:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:03 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2220981358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:54:03.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:03 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:54:03.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:03 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:54:03.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:03 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2220981358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:54:04.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:04 smithi192 ceph-mon[32956]: pgmap v2525: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:04.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:04 smithi151 ceph-mon[28965]: pgmap v2525: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:04.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:04 smithi190 ceph-mon[32247]: pgmap v2525: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:05.222 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:54:05.223 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1027,"fsid":"784a4e4c-34c8-11ef-bca8-c7b262605968","created":"2024-06-27T21:02:54.280751+0000","modified":"2024-06-27T21:53:59.466046+0000","last_up_change":"2024-06-27T21:23:43.806967+0000","last_in_change":"2024-06-27T21:46:33.505233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-27T21:06:07.812474+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"990","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"535","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80c5e023-5c22-463d-86e9-2307159c858f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":872,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6803","nonce":2519434429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6805","nonce":2519434429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6809","nonce":2519434429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":2519434429},{"type":"v1","addr":"172.21.15.151:6807","nonce":2519434429}]},"public_addr":"172.21.15.151:6803/2519434429","cluster_addr":"172.21.15.151:6805/2519434429","heartbeat_back_addr":"172.21.15.151:6809/2519434429","heartbeat_front_addr":"172.21.15.151:6807/2519434429","state":["exists","up"]},{"osd":1,"uuid":"d6a0fa83-bdab-4c01-b0d0-8996e8d0c083","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":997,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6811","nonce":59941028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6813","nonce":59941028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6817","nonce":59941028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":59941028},{"type":"v1","addr":"172.21.15.151:6815","nonce":59941028}]},"public_addr":"172.21.15.151:6811/59941028","cluster_addr":"172.21.15.151:6813/59941028","heartbeat_back_addr":"172.21.15.151:6817/59941028","heartbeat_front_addr":"172.21.15.151:6815/59941028","state":["exists","up"]},{"osd":2,"uuid":"22115999-54ff-4f34-81cc-9d90bfa4bd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":909,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6819","nonce":302970170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6821","nonce":302970170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6825","nonce":302970170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":302970170},{"type":"v1","addr":"172.21.15.151:6823","nonce":302970170}]},"public_addr":"172.21.15.151:6819/302970170","cluster_addr":"172.21.15.151:6821/302970170","heartbeat_back_addr":"172.21.15.151:6825/302970170","heartbeat_front_addr":"172.21.15.151:6823/302970170","state":["exists","up"]},{"osd":3,"uuid":"88c12fa5-d2af-4144-958e-b887b4474b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6827","nonce":3555754662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6829","nonce":3555754662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6833","nonce":3555754662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":3555754662},{"type":"v1","addr":"172.21.15.151:6831","nonce":3555754662}]},"public_addr":"172.21.15.151:6827/3555754662","cluster_addr":"172.21.15.151:6829/3555754662","heartbeat_back_addr":"172.21.15.151:6833/3555754662","heartbeat_front_addr":"172.21.15.151:6831/3555754662","state":["exists","up"]},{"osd":4,"uuid":"97aaddb9-d5bb-4827-8c35-b3c948eae3dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":347,"up_from":362,"up_thru":920,"down_at":348,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6801","nonce":3587059633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6803","nonce":3587059633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6807","nonce":3587059633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":3587059633},{"type":"v1","addr":"172.21.15.190:6805","nonce":3587059633}]},"public_addr":"172.21.15.190:6801/3587059633","cluster_addr":"172.21.15.190:6803/3587059633","heartbeat_back_addr":"172.21.15.190:6807/3587059633","heartbeat_front_addr":"172.21.15.190:6805/3587059633","state":["exists","up"]},{"osd":5,"uuid":"7ac9bbd6-0e55-4cab-b406-6f9967eaaf30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":1005,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6809","nonce":888361120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6811","nonce":888361120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6815","nonce":888361120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":888361120},{"type":"v1","addr":"172.21.15.190:6813","nonce":888361120}]},"public_addr":"172.21.15.190:6809/888361120","cluster_addr":"172.21.15.190:6811/888361120","heartbeat_back_addr":"172.21.15.190:6815/888361120","heartbeat_front_addr":"172.21.15.190:6813/888361120","state":["exists","up"]},{"osd":6,"uuid":"89fae0b2-26a4-456a-9d54-cfbaec80ba6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1008,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6817","nonce":30028280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6819","nonce":30028280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6823","nonce":30028280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":30028280},{"type":"v1","addr":"172.21.15.190:6821","nonce":30028280}]},"public_addr":"172.21.15.190:6817/30028280","cluster_addr":"172.21.15.190:6819/30028280","heartbeat_back_addr":"172.21.15.190:6823/30028280","heartbeat_front_addr":"172.21.15.190:6821/30028280","state":["exists","up"]},{"osd":7,"uuid":"e324ffdc-9842-46ee-a87e-5054ec265484","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":916,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6825","nonce":849289685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6827","nonce":849289685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6831","nonce":849289685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":849289685},{"type":"v1","addr":"172.21.15.190:6829","nonce":849289685}]},"public_addr":"172.21.15.190:6825/849289685","cluster_addr":"172.21.15.190:6827/849289685","heartbeat_back_addr":"172.21.15.190:6831/849289685","heartbeat_front_addr":"172.21.15.190:6829/849289685","state":["exists","up"]},{"osd":8,"uuid":"43d138bf-2d82-44ea-a98f-b9099e824243","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":924,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6801","nonce":2977299942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6803","nonce":2977299942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6807","nonce":2977299942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2977299942},{"type":"v1","addr":"172.21.15.192:6805","nonce":2977299942}]},"public_addr":"172.21.15.192:6801/2977299942","cluster_addr":"172.21.15.192:6803/2977299942","heartbeat_back_addr":"172.21.15.192:6807/2977299942","heartbeat_front_addr":"172.21.15.192:6805/2977299942","state":["exists","up"]},{"osd":9,"uuid":"0c86c106-04c9-4eb7-bb3e-d39f563b71fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":924,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6809","nonce":3401759799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6811","nonce":3401759799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6815","nonce":3401759799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":3401759799},{"type":"v1","addr":"172.21.15.192:6813","nonce":3401759799}]},"public_addr":"172.21.15.192:6809/3401759799","cluster_addr":"172.21.15.192:6811/3401759799","heartbeat_back_addr":"172.21.15.192:6815/3401759799","heartbeat_front_addr":"172.21.15.192:6813/3401759799","state":["exists","up"]},{"osd":10,"uuid":"ec4e9f06-538e-4e15-b8b8-a9018ccd8ef3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":920,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6817","nonce":1426195762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6819","nonce":1426195762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6823","nonce":1426195762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1426195762},{"type":"v1","addr":"172.21.15.192:6821","nonce":1426195762}]},"public_addr":"172.21.15.192:6817/1426195762","cluster_addr":"172.21.15.192:6819/1426195762","heartbeat_back_addr":"172.21.15.192:6823/1426195762","heartbeat_front_addr":"172.21.15.192:6821/1426195762","state":["exists","up"]},{"osd":11,"uuid":"a8a163c4-a14c-4710-bc2e-94e2cb69df0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":983,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6825","nonce":517567267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6827","nonce":517567267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6831","nonce":517567267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":517567267},{"type":"v1","addr":"172.21.15.192:6829","nonce":517567267}]},"public_addr":"172.21.15.192:6825/517567267","cluster_addr":"172.21.15.192:6827/517567267","heartbeat_back_addr":"172.21.15.192:6831/517567267","heartbeat_front_addr":"172.21.15.192:6829/517567267","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:06.879037+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:32.913880+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:06.566380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:20.044462+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:23.350143+0000","dead_epoch":347},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:25.788819+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:49.555970+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:43.486691+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:26.187438+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:37.099644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:28.938368+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-27T21:53:34.532031+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[1,9,10]},{"pgid":"1.1","osds":[0,9,11]}],"pg_upmap_items":[{"pgid":"1.1e","mappings":[{"from":7,"to":6},{"from":8,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/4089480652":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3356605898":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3908626376":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:0/3261020553":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3654164819":"2024-06-28T21:03:45.726698+0000","172.21.15.151:6801/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:6800/3767407460":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/2698418425":"2024-06-28T21:03:22.780955+0000","172.21.15.151:0/335837712":"2024-06-28T21:03:22.780955+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-27T21:54:05.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-27T21:54:05.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-27T21:54:05.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-27T21:54:05.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-27T21:54:05.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-27T21:54:05.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-27T21:54:05.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-27T21:54:05.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-27T21:54:05.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-27T21:54:05.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-27T21:54:05.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-27T21:54:05.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-27T21:54:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:06 smithi151 ceph-mon[28965]: pgmap v2526: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:06.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:06 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2904165327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:54:06.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:06 smithi190 ceph-mon[32247]: pgmap v2526: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:06.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:06 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2904165327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:54:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:06 smithi192 ceph-mon[32956]: pgmap v2526: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:06 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2904165327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-27T21:54:06.629 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:06.635 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:06.639 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:07.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:07 smithi151 ceph-mon[28965]: 1.19 deep-scrub starts 2024-06-27T21:54:07.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:07 smithi151 ceph-mon[28965]: 1.19 deep-scrub ok 2024-06-27T21:54:07.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:07 smithi190 ceph-mon[32247]: 1.19 deep-scrub starts 2024-06-27T21:54:07.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:07 smithi190 ceph-mon[32247]: 1.19 deep-scrub ok 2024-06-27T21:54:07.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:07 smithi192 ceph-mon[32956]: 1.19 deep-scrub starts 2024-06-27T21:54:07.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:07 smithi192 ceph-mon[32956]: 1.19 deep-scrub ok 2024-06-27T21:54:08.022 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.023 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:08 smithi151 ceph-mon[28965]: pgmap v2527: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:08.254 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.254 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.255 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.255 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.255 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:08.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:08 smithi190 ceph-mon[32247]: pgmap v2527: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:08 smithi192 ceph-mon[32956]: pgmap v2527: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:10.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:10 smithi151 ceph-mon[28965]: pgmap v2528: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:10 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:10 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:10.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:10 smithi151 ceph-mon[28965]: osdmap e1028: 12 total, 12 up, 12 in 2024-06-27T21:54:10.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:10 smithi190 ceph-mon[32247]: pgmap v2528: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:10.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:10 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:10.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:10 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:10.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:10 smithi190 ceph-mon[32247]: osdmap e1028: 12 total, 12 up, 12 in 2024-06-27T21:54:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:10 smithi192 ceph-mon[32956]: pgmap v2528: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:10 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:10 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:10 smithi192 ceph-mon[32956]: osdmap e1028: 12 total, 12 up, 12 in 2024-06-27T21:54:12.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:12 smithi151 ceph-mon[28965]: pgmap v2530: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:12.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:12 smithi190 ceph-mon[32247]: pgmap v2530: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:12.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:12 smithi192 ceph-mon[32956]: pgmap v2530: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:13.394 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:14.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:14 smithi151 ceph-mon[28965]: pgmap v2531: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:14.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:14 smithi190 ceph-mon[32247]: pgmap v2531: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:14.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:14 smithi192 ceph-mon[32956]: pgmap v2531: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:16.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:16 smithi151 ceph-mon[28965]: pgmap v2532: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:16.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:16 smithi190 ceph-mon[32247]: pgmap v2532: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:16.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:16 smithi192 ceph-mon[32956]: pgmap v2532: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:17.919 INFO:teuthology.orchestra.run.smithi151.stdout:360777253407 2024-06-27T21:54:17.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-27T21:54:18.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:18 smithi151 ceph-mon[28965]: pgmap v2533: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:18.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:18 smithi151 ceph-mon[28965]: 1.c scrub starts 2024-06-27T21:54:18.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:18 smithi151 ceph-mon[28965]: 1.c scrub ok 2024-06-27T21:54:18.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:18 smithi190 ceph-mon[32247]: pgmap v2533: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:18.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:18 smithi190 ceph-mon[32247]: 1.c scrub starts 2024-06-27T21:54:18.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:18 smithi190 ceph-mon[32247]: 1.c scrub ok 2024-06-27T21:54:18.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:18 smithi192 ceph-mon[32956]: pgmap v2533: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:18.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:18 smithi192 ceph-mon[32956]: 1.c scrub starts 2024-06-27T21:54:18.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:18 smithi192 ceph-mon[32956]: 1.c scrub ok 2024-06-27T21:54:19.681 INFO:teuthology.orchestra.run.smithi151.stdout:34359738960 2024-06-27T21:54:19.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-27T21:54:20.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:20 smithi151 ceph-mon[28965]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:20.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:20 smithi151 ceph-mon[28965]: 1.a scrub starts 2024-06-27T21:54:20.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:20 smithi151 ceph-mon[28965]: 1.a scrub ok 2024-06-27T21:54:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:20 smithi190 ceph-mon[32247]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:20 smithi190 ceph-mon[32247]: 1.a scrub starts 2024-06-27T21:54:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:20 smithi190 ceph-mon[32247]: 1.a scrub ok 2024-06-27T21:54:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:20 smithi192 ceph-mon[32956]: pgmap v2534: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:20 smithi192 ceph-mon[32956]: 1.a scrub starts 2024-06-27T21:54:20.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:20 smithi192 ceph-mon[32956]: 1.a scrub ok 2024-06-27T21:54:22.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:22 smithi151 ceph-mon[28965]: pgmap v2535: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:22.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:22 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:54:22.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:22 smithi151 ceph-mon[28965]: 1.d scrub starts 2024-06-27T21:54:22.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:22 smithi151 ceph-mon[28965]: 1.d scrub ok 2024-06-27T21:54:22.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:22 smithi190 ceph-mon[32247]: pgmap v2535: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:22.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:22 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:54:22.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:22 smithi190 ceph-mon[32247]: 1.d scrub starts 2024-06-27T21:54:22.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:22 smithi190 ceph-mon[32247]: 1.d scrub ok 2024-06-27T21:54:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:22 smithi192 ceph-mon[32956]: pgmap v2535: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:22 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:54:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:22 smithi192 ceph-mon[32956]: 1.d scrub starts 2024-06-27T21:54:22.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:22 smithi192 ceph-mon[32956]: 1.d scrub ok 2024-06-27T21:54:23.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:23 smithi151 ceph-mon[28965]: 1.1a deep-scrub starts 2024-06-27T21:54:23.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:23 smithi151 ceph-mon[28965]: 1.1a deep-scrub ok 2024-06-27T21:54:23.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:54:23.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:54:23.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:23 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:54:23.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:23 smithi190 ceph-mon[32247]: 1.1a deep-scrub starts 2024-06-27T21:54:23.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:23 smithi190 ceph-mon[32247]: 1.1a deep-scrub ok 2024-06-27T21:54:23.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:54:23.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:54:23.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:23 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:54:23.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:23 smithi192 ceph-mon[32956]: 1.1a deep-scrub starts 2024-06-27T21:54:23.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:23 smithi192 ceph-mon[32956]: 1.1a deep-scrub ok 2024-06-27T21:54:23.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:54:23.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:54:23.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:23 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:54:24.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:24 smithi151 ceph-mon[28965]: pgmap v2536: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:24.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:24 smithi190 ceph-mon[32247]: pgmap v2536: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:24.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:24 smithi192 ceph-mon[32956]: pgmap v2536: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:24.770 INFO:teuthology.orchestra.run.smithi151.stdout:77309411911 2024-06-27T21:54:24.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-27T21:54:26.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:26 smithi151 ceph-mon[28965]: pgmap v2537: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:26.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:26 smithi151 ceph-mon[28965]: 1.f scrub starts 2024-06-27T21:54:26.299 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:26 smithi151 ceph-mon[28965]: 1.f scrub ok 2024-06-27T21:54:26.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:26 smithi190 ceph-mon[32247]: pgmap v2537: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:26.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:26 smithi190 ceph-mon[32247]: 1.f scrub starts 2024-06-27T21:54:26.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:26 smithi190 ceph-mon[32247]: 1.f scrub ok 2024-06-27T21:54:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:26 smithi192 ceph-mon[32956]: pgmap v2537: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:26 smithi192 ceph-mon[32956]: 1.f scrub starts 2024-06-27T21:54:26.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:26 smithi192 ceph-mon[32956]: 1.f scrub ok 2024-06-27T21:54:27.083 INFO:teuthology.orchestra.run.smithi151.stdout:163208757819 2024-06-27T21:54:27.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-27T21:54:28.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:28 smithi151 ceph-mon[28965]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:28.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:28 smithi190 ceph-mon[32247]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:28.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:28 smithi192 ceph-mon[32956]: pgmap v2538: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:28.593 INFO:teuthology.orchestra.run.smithi151.stdout:326417515044 2024-06-27T21:54:28.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-27T21:54:29.026 INFO:teuthology.orchestra.run.smithi151.stdout:266287972908 2024-06-27T21:54:29.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-27T21:54:29.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:29 smithi151 ceph-mon[28965]: 1.1e scrub starts 2024-06-27T21:54:29.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:29 smithi151 ceph-mon[28965]: 1.1e scrub ok 2024-06-27T21:54:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:29 smithi190 ceph-mon[32247]: 1.1e scrub starts 2024-06-27T21:54:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:29 smithi190 ceph-mon[32247]: 1.1e scrub ok 2024-06-27T21:54:29.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:29 smithi192 ceph-mon[32956]: 1.1e scrub starts 2024-06-27T21:54:29.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:29 smithi192 ceph-mon[32956]: 1.1e scrub ok 2024-06-27T21:54:30.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:30 smithi151 ceph-mon[28965]: pgmap v2539: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:30.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:30 smithi151 ceph-mon[28965]: 1.12 scrub starts 2024-06-27T21:54:30.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:30 smithi151 ceph-mon[28965]: 1.12 scrub ok 2024-06-27T21:54:30.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:30 smithi190 ceph-mon[32247]: pgmap v2539: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:30.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:30 smithi190 ceph-mon[32247]: 1.12 scrub starts 2024-06-27T21:54:30.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:30 smithi190 ceph-mon[32247]: 1.12 scrub ok 2024-06-27T21:54:30.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:30 smithi192 ceph-mon[32956]: pgmap v2539: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:30.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:30 smithi192 ceph-mon[32956]: 1.12 scrub starts 2024-06-27T21:54:30.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:30 smithi192 ceph-mon[32956]: 1.12 scrub ok 2024-06-27T21:54:30.851 INFO:teuthology.orchestra.run.smithi151.stdout:55834575439 2024-06-27T21:54:30.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-27T21:54:31.671 INFO:teuthology.orchestra.run.smithi151.stdout:296352743977 2024-06-27T21:54:31.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-27T21:54:31.794 INFO:teuthology.orchestra.run.smithi151.stdout:227633267243 2024-06-27T21:54:31.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-27T21:54:32.147 INFO:teuthology.orchestra.run.smithi151.stdout:193273528885 2024-06-27T21:54:32.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-27T21:54:32.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:32 smithi151 ceph-mon[28965]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:32.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:32 smithi151 ceph-mon[28965]: 1.10 deep-scrub starts 2024-06-27T21:54:32.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:32 smithi151 ceph-mon[28965]: 1.10 deep-scrub ok 2024-06-27T21:54:32.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:32 smithi190 ceph-mon[32247]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:32.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:32 smithi190 ceph-mon[32247]: 1.10 deep-scrub starts 2024-06-27T21:54:32.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:32 smithi190 ceph-mon[32247]: 1.10 deep-scrub ok 2024-06-27T21:54:32.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:32 smithi192 ceph-mon[32956]: pgmap v2540: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:32.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:32 smithi192 ceph-mon[32956]: 1.10 deep-scrub starts 2024-06-27T21:54:32.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:32 smithi192 ceph-mon[32956]: 1.10 deep-scrub ok 2024-06-27T21:54:32.677 INFO:teuthology.orchestra.run.smithi151.stdout:107374182981 2024-06-27T21:54:32.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-27T21:54:32.696 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:32.697 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:32.763 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:32.788 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:33.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:33 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:33.170 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:33 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:33.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:33 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:33.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:33 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:33.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:33 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:33.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:33 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:34.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:34 smithi151 ceph-mon[28965]: pgmap v2541: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:34.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:34 smithi151 ceph-mon[28965]: osdmap e1029: 12 total, 12 up, 12 in 2024-06-27T21:54:34.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:34 smithi190 ceph-mon[32247]: pgmap v2541: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:34.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:34 smithi190 ceph-mon[32247]: osdmap e1029: 12 total, 12 up, 12 in 2024-06-27T21:54:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:34 smithi192 ceph-mon[32956]: pgmap v2541: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:34.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:34 smithi192 ceph-mon[32956]: osdmap e1029: 12 total, 12 up, 12 in 2024-06-27T21:54:35.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:35 smithi151 ceph-mon[28965]: 1.11 scrub starts 2024-06-27T21:54:35.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:35 smithi151 ceph-mon[28965]: 1.11 scrub ok 2024-06-27T21:54:35.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:35 smithi190 ceph-mon[32247]: 1.11 scrub starts 2024-06-27T21:54:35.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:35 smithi190 ceph-mon[32247]: 1.11 scrub ok 2024-06-27T21:54:35.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:35 smithi192 ceph-mon[32956]: 1.11 scrub starts 2024-06-27T21:54:35.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:35 smithi192 ceph-mon[32956]: 1.11 scrub ok 2024-06-27T21:54:36.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:36 smithi151 ceph-mon[28965]: pgmap v2543: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:36.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:36 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:36.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:36 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:36.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:36 smithi190 ceph-mon[32247]: pgmap v2543: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:36.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:36 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:36.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:36 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:36.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:36 smithi192 ceph-mon[32956]: pgmap v2543: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:36.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:36 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:36.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:36 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:37.024 INFO:teuthology.orchestra.run.smithi151.stdout:34359738965 2024-06-27T21:54:37.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:37 smithi151 ceph-mon[28965]: osdmap e1030: 12 total, 12 up, 12 in 2024-06-27T21:54:37.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:37 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:37.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:37 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:37.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:37 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3724292517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-27T21:54:37.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:37 smithi190 ceph-mon[32247]: osdmap e1030: 12 total, 12 up, 12 in 2024-06-27T21:54:37.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:37 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:37.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:37 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:37.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:37 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3724292517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-27T21:54:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:37 smithi192 ceph-mon[32956]: osdmap e1030: 12 total, 12 up, 12 in 2024-06-27T21:54:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:37 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:37 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:37.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:37 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3724292517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-27T21:54:37.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738960 got 34359738965 for osd.0 2024-06-27T21:54:37.781 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:38.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:38 smithi151 ceph-mon[28965]: pgmap v2545: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:38.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:38 smithi151 ceph-mon[28965]: osdmap e1031: 12 total, 12 up, 12 in 2024-06-27T21:54:38.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:38 smithi190 ceph-mon[32247]: pgmap v2545: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:38.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:38 smithi190 ceph-mon[32247]: osdmap e1031: 12 total, 12 up, 12 in 2024-06-27T21:54:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:38 smithi192 ceph-mon[32956]: pgmap v2545: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:38.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:38 smithi192 ceph-mon[32956]: osdmap e1031: 12 total, 12 up, 12 in 2024-06-27T21:54:38.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:38.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:38.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:38.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:39.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:39 smithi151 ceph-mon[28965]: pgmap v2547: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:39.457 INFO:teuthology.orchestra.run.smithi151.stdout:1554778161528 2024-06-27T21:54:39.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-27T21:54:39.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:39 smithi190 ceph-mon[32247]: pgmap v2547: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:39.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:39 smithi192 ceph-mon[32956]: pgmap v2547: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:40.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:40.175 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:40.330 INFO:teuthology.orchestra.run.smithi151.stdout:360777253411 2024-06-27T21:54:40.376 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:40 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:40.376 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:40 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:40.376 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:40 smithi151 ceph-mon[28965]: osdmap e1032: 12 total, 12 up, 12 in 2024-06-27T21:54:40.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:40 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:40.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:40 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:40.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:40 smithi190 ceph-mon[32247]: osdmap e1032: 12 total, 12 up, 12 in 2024-06-27T21:54:40.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:40 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:40.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:40 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:40.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:40 smithi192 ceph-mon[32956]: osdmap e1032: 12 total, 12 up, 12 in 2024-06-27T21:54:41.088 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777253407 got 360777253411 for osd.11 2024-06-27T21:54:41.088 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:41.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:41 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4034181085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-27T21:54:41.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:41 smithi151 ceph-mon[28965]: pgmap v2549: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:41.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:41 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4034181085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-27T21:54:41.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:41 smithi190 ceph-mon[32247]: pgmap v2549: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:41.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:41 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4034181085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-27T21:54:41.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:41 smithi192 ceph-mon[32956]: pgmap v2549: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:41.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:41.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:54:42.398 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:42 smithi151 ceph-mon[28965]: 1.3 scrub starts 2024-06-27T21:54:42.398 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:42 smithi151 ceph-mon[28965]: 1.3 scrub ok 2024-06-27T21:54:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:42 smithi190 ceph-mon[32247]: 1.3 scrub starts 2024-06-27T21:54:42.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:42 smithi190 ceph-mon[32247]: 1.3 scrub ok 2024-06-27T21:54:42.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:42 smithi192 ceph-mon[32956]: 1.3 scrub starts 2024-06-27T21:54:42.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:42 smithi192 ceph-mon[32956]: 1.3 scrub ok 2024-06-27T21:54:43.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:43 smithi151 ceph-mon[28965]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:43.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:43 smithi151 ceph-mon[28965]: 1.14 scrub starts 2024-06-27T21:54:43.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:43 smithi151 ceph-mon[28965]: 1.14 scrub ok 2024-06-27T21:54:43.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:43 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3367098126' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-27T21:54:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:43 smithi190 ceph-mon[32247]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:43 smithi190 ceph-mon[32247]: 1.14 scrub starts 2024-06-27T21:54:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:43 smithi190 ceph-mon[32247]: 1.14 scrub ok 2024-06-27T21:54:43.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:43 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3367098126' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-27T21:54:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:43 smithi192 ceph-mon[32956]: pgmap v2550: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:43 smithi192 ceph-mon[32956]: 1.14 scrub starts 2024-06-27T21:54:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:43 smithi192 ceph-mon[32956]: 1.14 scrub ok 2024-06-27T21:54:43.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:43 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3367098126' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-27T21:54:44.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:44 smithi151 ceph-mon[28965]: 1.b scrub starts 2024-06-27T21:54:44.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:44 smithi151 ceph-mon[28965]: 1.b scrub ok 2024-06-27T21:54:44.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:44 smithi151 ceph-mon[28965]: 1.6 scrub starts 2024-06-27T21:54:44.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:44 smithi151 ceph-mon[28965]: 1.6 scrub ok 2024-06-27T21:54:44.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:44 smithi151 ceph-mon[28965]: 1.16 scrub starts 2024-06-27T21:54:44.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:44 smithi151 ceph-mon[28965]: 1.16 scrub ok 2024-06-27T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:44 smithi190 ceph-mon[32247]: 1.b scrub starts 2024-06-27T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:44 smithi190 ceph-mon[32247]: 1.b scrub ok 2024-06-27T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:44 smithi190 ceph-mon[32247]: 1.6 scrub starts 2024-06-27T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:44 smithi190 ceph-mon[32247]: 1.6 scrub ok 2024-06-27T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:44 smithi190 ceph-mon[32247]: 1.16 scrub starts 2024-06-27T21:54:44.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:44 smithi190 ceph-mon[32247]: 1.16 scrub ok 2024-06-27T21:54:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:44 smithi192 ceph-mon[32956]: 1.b scrub starts 2024-06-27T21:54:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:44 smithi192 ceph-mon[32956]: 1.b scrub ok 2024-06-27T21:54:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:44 smithi192 ceph-mon[32956]: 1.6 scrub starts 2024-06-27T21:54:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:44 smithi192 ceph-mon[32956]: 1.6 scrub ok 2024-06-27T21:54:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:44 smithi192 ceph-mon[32956]: 1.16 scrub starts 2024-06-27T21:54:44.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:44 smithi192 ceph-mon[32956]: 1.16 scrub ok 2024-06-27T21:54:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:45.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: pgmap v2551: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2712706207,config=2353167246,health=861809967,kv=495348086}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2991191792,logm=2228487374}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995752308}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047929286}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2584616389}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871567679}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763115302}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408009228}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646485091}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161380945}) 2024-06-27T21:54:45.449 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698700928}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603840655}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912861862}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051935271}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527957599}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582192447}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628422289}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750823936}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168816337}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809853797}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676646364}) 2024-06-27T21:54:45.450 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406313211}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4292172411}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131065231}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008654823}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1335054727}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2672351517}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3653658594}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871218403}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823804788}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646477113}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482106313}) 2024-06-27T21:54:45.451 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371152223}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193923959}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=951289900}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458079484}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496422241}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598538290}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3350440647}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356803606}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2924519278}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901101801}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753795826}) 2024-06-27T21:54:45.452 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601501503}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3094436812}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264995768}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=828026006}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477128197}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2018220633}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299447858}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243762605}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4091867227}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155294655}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797496463}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243693871}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=729207664}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=411355406}) 2024-06-27T21:54:45.453 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287415137}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199257238}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650026714}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2038932949}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1081544603}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3839206959}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133677981}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986690582}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109000632}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454830675}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675310522}) 2024-06-27T21:54:45.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246838859}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558583419}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430810964}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3130917779}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2557302212}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949993628}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836022400}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2038270679}) 2024-06-27T21:54:45.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2283610172}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1585500112}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991095899}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769676582}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2988224356}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830917015}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=607360469}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867546138}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2116138878}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714191282}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3033577360}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621282832}) 2024-06-27T21:54:45.456 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799531788}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2027524467}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178759560}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1920278793}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726434351}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512598295}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {logm=21,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=45} crc {logm=1561904703,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=724853318,mon_config_key=2408433602}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3987801981}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1628771256}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=46,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=36} crc {mon_config_key=2191107312,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=4127304935,osdmap=654399554}) 2024-06-27T21:54:45.457 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:45 smithi151 ceph-mon[28965]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=2549734775}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: pgmap v2551: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2712706207,config=2353167246,health=861809967,kv=495348086}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2991191792,logm=2228487374}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995752308}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047929286}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2584616389}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871567679}) 2024-06-27T21:54:45.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763115302}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408009228}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646485091}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161380945}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698700928}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603840655}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912861862}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051935271}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527957599}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582192447}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628422289}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750823936}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168816337}) 2024-06-27T21:54:45.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809853797}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676646364}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406313211}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4292172411}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131065231}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008654823}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1335054727}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2672351517}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3653658594}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871218403}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823804788}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646477113}) 2024-06-27T21:54:45.516 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482106313}) 2024-06-27T21:54:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371152223}) 2024-06-27T21:54:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193923959}) 2024-06-27T21:54:45.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=951289900}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458079484}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496422241}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598538290}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3350440647}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356803606}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2924519278}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901101801}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753795826}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601501503}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3094436812}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264995768}) 2024-06-27T21:54:45.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=828026006}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477128197}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2018220633}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299447858}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243762605}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4091867227}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155294655}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797496463}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243693871}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=729207664}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=411355406}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287415137}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199257238}) 2024-06-27T21:54:45.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650026714}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2038932949}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1081544603}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3839206959}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133677981}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986690582}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109000632}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454830675}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675310522}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246838859}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558583419}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430810964}) 2024-06-27T21:54:45.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3130917779}) 2024-06-27T21:54:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2557302212}) 2024-06-27T21:54:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949993628}) 2024-06-27T21:54:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836022400}) 2024-06-27T21:54:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2038270679}) 2024-06-27T21:54:45.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2283610172}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1585500112}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991095899}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769676582}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2988224356}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830917015}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=607360469}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867546138}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2116138878}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714191282}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3033577360}) 2024-06-27T21:54:45.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621282832}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799531788}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2027524467}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178759560}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1920278793}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726434351}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512598295}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {logm=21,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=45} crc {logm=1561904703,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=724853318,mon_config_key=2408433602}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3987801981}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1628771256}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=46,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=36} crc {mon_config_key=2191107312,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=4127304935,osdmap=654399554}) 2024-06-27T21:54:45.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:45 smithi190 ceph-mon[32247]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=2549734775}) 2024-06-27T21:54:45.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: pgmap v2551: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=58} crc {auth=2712706207,config=2353167246,health=861809967,kv=495348086}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2991191792,logm=2228487374}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2995752308}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1047929286}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2584616389}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871567679}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763115302}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408009228}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1646485091}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161380945}) 2024-06-27T21:54:45.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=698700928}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603840655}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=912861862}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4051935271}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2527957599}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582192447}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628422289}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1750823936}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1168816337}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809853797}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676646364}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406313211}) 2024-06-27T21:54:45.546 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4292172411}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2131065231}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2008654823}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1335054727}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2672351517}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3653658594}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=871218403}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823804788}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646477113}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482106313}) 2024-06-27T21:54:45.547 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3371152223}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193923959}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=951289900}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=458079484}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496422241}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=598538290}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3350440647}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356803606}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2924519278}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2901101801}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1753795826}) 2024-06-27T21:54:45.548 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601501503}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3094436812}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3264995768}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=828026006}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2477128197}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2018220633}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1299447858}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4243762605}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4091867227}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155294655}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=797496463}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3243693871}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=729207664}) 2024-06-27T21:54:45.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=411355406}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=287415137}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3199257238}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=650026714}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2038932949}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1081544603}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3839206959}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133677981}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986690582}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109000632}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2454830675}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675310522}) 2024-06-27T21:54:45.550 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246838859}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558583419}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3430810964}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3130917779}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2557302212}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949993628}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836022400}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2038270679}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2283610172}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1585500112}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991095899}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2769676582}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2988224356}) 2024-06-27T21:54:45.551 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830917015}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=607360469}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867546138}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2116138878}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1714191282}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3033577360}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621282832}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799531788}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2027524467}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1178759560}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1920278793}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3726434351}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3512598295}) 2024-06-27T21:54:45.552 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {logm=21,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=45} crc {logm=1561904703,mdsmap=2339142170,mgr=434099168,mgr_command_descs=3040664224,mgr_metadata=1799985202,mgrstat=724853318,mon_config_key=2408433602}) 2024-06-27T21:54:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3987801981}) 2024-06-27T21:54:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1628771256}) 2024-06-27T21:54:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=46,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=36} crc {mon_config_key=2191107312,monmap=3596043196,osd_metadata=3144634705,osd_pg_creating=4127304935,osdmap=654399554}) 2024-06-27T21:54:45.553 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:45 smithi192 ceph-mon[32956]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=2549734775}) 2024-06-27T21:54:46.184 INFO:teuthology.orchestra.run.smithi151.stdout:266287972911 2024-06-27T21:54:46.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:46 smithi151 ceph-mon[28965]: osdmap e1033: 12 total, 12 up, 12 in 2024-06-27T21:54:46.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:46 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:46.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:46 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:46 smithi190 ceph-mon[32247]: osdmap e1033: 12 total, 12 up, 12 in 2024-06-27T21:54:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:46 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:46.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:46 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:46 smithi192 ceph-mon[32956]: osdmap e1033: 12 total, 12 up, 12 in 2024-06-27T21:54:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:46 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:46.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:46 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:46.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972908 got 266287972911 for osd.8 2024-06-27T21:54:46.891 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:47.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:47 smithi151 ceph-mon[28965]: osdmap e1034: 12 total, 12 up, 12 in 2024-06-27T21:54:47.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:47 smithi151 ceph-mon[28965]: pgmap v2554: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:47.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:47 smithi190 ceph-mon[32247]: osdmap e1034: 12 total, 12 up, 12 in 2024-06-27T21:54:47.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:47 smithi190 ceph-mon[32247]: pgmap v2554: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:47 smithi192 ceph-mon[32956]: osdmap e1034: 12 total, 12 up, 12 in 2024-06-27T21:54:47.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:47 smithi192 ceph-mon[32956]: pgmap v2554: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:48.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:48 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:48.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:48 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:48.447 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:48 smithi151 ceph-mon[28965]: osdmap e1035: 12 total, 12 up, 12 in 2024-06-27T21:54:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:48 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:48 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:48.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:48 smithi190 ceph-mon[32247]: osdmap e1035: 12 total, 12 up, 12 in 2024-06-27T21:54:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:48 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:48 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:48.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:48 smithi192 ceph-mon[32956]: osdmap e1035: 12 total, 12 up, 12 in 2024-06-27T21:54:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:49 smithi151 ceph-mon[28965]: pgmap v2556: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:49.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:49 smithi190 ceph-mon[32247]: pgmap v2556: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:49.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:49 smithi192 ceph-mon[32956]: pgmap v2556: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:50.037 INFO:teuthology.orchestra.run.smithi151.stdout:1554778161530 2024-06-27T21:54:50.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:50 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:50.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:50 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:50.309 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:50 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1661707464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-27T21:54:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:50 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:50 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:50.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:50 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1661707464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-27T21:54:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:50 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:50 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:50.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:50 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1661707464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-27T21:54:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:51 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:51 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:51 smithi151 ceph-mon[28965]: osdmap e1036: 12 total, 12 up, 12 in 2024-06-27T21:54:51.448 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:51 smithi151 ceph-mon[28965]: pgmap v2558: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:51.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:51 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:51.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:51 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:51.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:51 smithi190 ceph-mon[32247]: osdmap e1036: 12 total, 12 up, 12 in 2024-06-27T21:54:51.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:51 smithi190 ceph-mon[32247]: pgmap v2558: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:51.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:51 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:51.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:51 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:51.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:51 smithi192 ceph-mon[32956]: osdmap e1036: 12 total, 12 up, 12 in 2024-06-27T21:54:51.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:51 smithi192 ceph-mon[32956]: pgmap v2558: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:52.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:52 smithi151 ceph-mon[28965]: purged_snaps scrub starts 2024-06-27T21:54:52.267 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:52 smithi151 ceph-mon[28965]: purged_snaps scrub ok 2024-06-27T21:54:52.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:52 smithi190 ceph-mon[32247]: purged_snaps scrub starts 2024-06-27T21:54:52.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:52 smithi190 ceph-mon[32247]: purged_snaps scrub ok 2024-06-27T21:54:52.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:52 smithi192 ceph-mon[32956]: purged_snaps scrub starts 2024-06-27T21:54:52.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:52 smithi192 ceph-mon[32956]: purged_snaps scrub ok 2024-06-27T21:54:52.628 INFO:teuthology.orchestra.run.smithi151.stdout:163208757824 2024-06-27T21:54:52.847 INFO:teuthology.orchestra.run.smithi151.stdout:193273528890 2024-06-27T21:54:53.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:53 smithi151 ceph-mon[28965]: osdmap e1037: 12 total, 12 up, 12 in 2024-06-27T21:54:53.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:53 smithi151 ceph-mon[28965]: pgmap v2560: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:53.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1389092896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-27T21:54:53.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:53 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3111458610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-27T21:54:53.394 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757819 got 163208757824 for osd.5 2024-06-27T21:54:53.394 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:53 smithi190 ceph-mon[32247]: osdmap e1037: 12 total, 12 up, 12 in 2024-06-27T21:54:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:53 smithi190 ceph-mon[32247]: pgmap v2560: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1389092896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-27T21:54:53.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:53 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3111458610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-27T21:54:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:53 smithi192 ceph-mon[32956]: osdmap e1037: 12 total, 12 up, 12 in 2024-06-27T21:54:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:53 smithi192 ceph-mon[32956]: pgmap v2560: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1389092896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-27T21:54:53.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:53 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3111458610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-27T21:54:53.977 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1554778161528 got 1554778161530 for osd.4 2024-06-27T21:54:53.977 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:55.796 INFO:teuthology.orchestra.run.smithi151.stdout:55834575445 2024-06-27T21:54:56.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:56 smithi151 ceph-mon[28965]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:56.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:56 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/2724626962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-27T21:54:56.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:56 smithi190 ceph-mon[32247]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:56.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:56 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/2724626962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-27T21:54:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:56 smithi192 ceph-mon[32956]: pgmap v2561: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:56.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:56 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/2724626962' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-27T21:54:57.106 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528885 got 193273528890 for osd.6 2024-06-27T21:54:57.106 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:57.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:57 smithi151 ceph-mon[28965]: 1.8 scrub starts 2024-06-27T21:54:57.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:57 smithi151 ceph-mon[28965]: 1.8 scrub ok 2024-06-27T21:54:57.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:57 smithi151 ceph-mon[28965]: 1.1 scrub starts 2024-06-27T21:54:57.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:57 smithi151 ceph-mon[28965]: 1.1 scrub ok 2024-06-27T21:54:57.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:57 smithi151 ceph-mon[28965]: 1.13 deep-scrub starts 2024-06-27T21:54:57.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:57 smithi151 ceph-mon[28965]: 1.13 deep-scrub ok 2024-06-27T21:54:57.466 INFO:teuthology.orchestra.run.smithi151.stdout:326417515050 2024-06-27T21:54:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:57 smithi190 ceph-mon[32247]: 1.8 scrub starts 2024-06-27T21:54:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:57 smithi190 ceph-mon[32247]: 1.8 scrub ok 2024-06-27T21:54:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:57 smithi190 ceph-mon[32247]: 1.1 scrub starts 2024-06-27T21:54:57.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:57 smithi190 ceph-mon[32247]: 1.1 scrub ok 2024-06-27T21:54:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:57 smithi190 ceph-mon[32247]: 1.13 deep-scrub starts 2024-06-27T21:54:57.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:57 smithi190 ceph-mon[32247]: 1.13 deep-scrub ok 2024-06-27T21:54:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:57 smithi192 ceph-mon[32956]: 1.8 scrub starts 2024-06-27T21:54:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:57 smithi192 ceph-mon[32956]: 1.8 scrub ok 2024-06-27T21:54:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:57 smithi192 ceph-mon[32956]: 1.1 scrub starts 2024-06-27T21:54:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:57 smithi192 ceph-mon[32956]: 1.1 scrub ok 2024-06-27T21:54:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:57 smithi192 ceph-mon[32956]: 1.13 deep-scrub starts 2024-06-27T21:54:57.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:57 smithi192 ceph-mon[32956]: 1.13 deep-scrub ok 2024-06-27T21:54:57.655 INFO:teuthology.orchestra.run.smithi151.stdout:77309411918 2024-06-27T21:54:58.405 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:58 smithi151 ceph-mon[28965]: pgmap v2562: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:58.405 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/1797976851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-27T21:54:58.405 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:58 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/4186166906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-27T21:54:58.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411911 got 77309411918 for osd.2 2024-06-27T21:54:58.436 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:58 smithi190 ceph-mon[32247]: pgmap v2562: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/1797976851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-27T21:54:58.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:58 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/4186166906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-27T21:54:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:58 smithi192 ceph-mon[32956]: pgmap v2562: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:54:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/1797976851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-27T21:54:58.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:58 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/4186166906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-27T21:54:58.872 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575439 got 55834575445 for osd.1 2024-06-27T21:54:58.873 DEBUG:teuthology.parallel:result is None 2024-06-27T21:54:59.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:59 smithi151 ceph-mon[28965]: 1.1d scrub starts 2024-06-27T21:54:59.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:54:59 smithi151 ceph-mon[28965]: 1.1d scrub ok 2024-06-27T21:54:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:59 smithi190 ceph-mon[32247]: 1.1d scrub starts 2024-06-27T21:54:59.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:54:59 smithi190 ceph-mon[32247]: 1.1d scrub ok 2024-06-27T21:54:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:59 smithi192 ceph-mon[32956]: 1.1d scrub starts 2024-06-27T21:54:59.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:54:59 smithi192 ceph-mon[32956]: 1.1d scrub ok 2024-06-27T21:55:00.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:00 smithi151 ceph-mon[28965]: pgmap v2563: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:00.420 INFO:teuthology.orchestra.run.smithi151.stdout:227633267250 2024-06-27T21:55:00.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:00 smithi190 ceph-mon[32247]: pgmap v2563: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:00.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:00 smithi192 ceph-mon[32956]: pgmap v2563: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:00.643 INFO:teuthology.orchestra.run.smithi151.stdout:296352743983 2024-06-27T21:55:00.814 INFO:teuthology.orchestra.run.smithi151.stdout:107374182987 2024-06-27T21:55:01.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3833403640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-27T21:55:01.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3196062192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-27T21:55:01.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:01 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/3920342396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-27T21:55:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3833403640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-27T21:55:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3196062192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-27T21:55:01.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:01 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/3920342396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-27T21:55:01.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3833403640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-27T21:55:01.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3196062192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-27T21:55:01.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:01 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/3920342396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-27T21:55:01.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182981 got 107374182987 for osd.3 2024-06-27T21:55:01.596 DEBUG:teuthology.parallel:result is None 2024-06-27T21:55:02.060 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743977 got 296352743983 for osd.9 2024-06-27T21:55:02.060 DEBUG:teuthology.parallel:result is None 2024-06-27T21:55:02.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:02 smithi151 ceph-mon[28965]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:02.445 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:02 smithi192 ceph-mon[32956]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:02.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:02 smithi190 ceph-mon[32247]: pgmap v2564: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:02.776 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633267243 got 227633267250 for osd.7 2024-06-27T21:55:02.777 DEBUG:teuthology.parallel:result is None 2024-06-27T21:55:02.898 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417515044 got 326417515050 for osd.10 2024-06-27T21:55:02.898 DEBUG:teuthology.parallel:result is None 2024-06-27T21:55:02.899 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-06-27T21:55:02.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:55:03.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:55:03.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.18 scrub starts 2024-06-27T21:55:03.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.18 scrub ok 2024-06-27T21:55:03.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.7 scrub starts 2024-06-27T21:55:03.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.7 scrub ok 2024-06-27T21:55:03.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.1c scrub starts 2024-06-27T21:55:03.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.1c scrub ok 2024-06-27T21:55:03.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.2 scrub starts 2024-06-27T21:55:03.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:03 smithi151 ceph-mon[28965]: 1.2 scrub ok 2024-06-27T21:55:03.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.18 scrub starts 2024-06-27T21:55:03.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.18 scrub ok 2024-06-27T21:55:03.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.7 scrub starts 2024-06-27T21:55:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.7 scrub ok 2024-06-27T21:55:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.1c scrub starts 2024-06-27T21:55:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.1c scrub ok 2024-06-27T21:55:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.2 scrub starts 2024-06-27T21:55:03.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:03 smithi190 ceph-mon[32247]: 1.2 scrub ok 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.18 scrub starts 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.18 scrub ok 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.7 scrub starts 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.7 scrub ok 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.1c scrub starts 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.1c scrub ok 2024-06-27T21:55:03.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.2 scrub starts 2024-06-27T21:55:03.545 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:03 smithi192 ceph-mon[32956]: 1.2 scrub ok 2024-06-27T21:55:04.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:04 smithi151 ceph-mon[28965]: pgmap v2565: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:04.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:04 smithi190 ceph-mon[32247]: pgmap v2565: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:04.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:04 smithi192 ceph-mon[32956]: pgmap v2565: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:05.071 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:55:05.072 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:55:05.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:05 smithi151 ceph-mon[28965]: 1.e scrub starts 2024-06-27T21:55:05.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:05 smithi151 ceph-mon[28965]: 1.e scrub ok 2024-06-27T21:55:05.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:05 smithi190 ceph-mon[32247]: 1.e scrub starts 2024-06-27T21:55:05.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:05 smithi190 ceph-mon[32247]: 1.e scrub ok 2024-06-27T21:55:05.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:05 smithi192 ceph-mon[32956]: 1.e scrub starts 2024-06-27T21:55:05.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:05 smithi192 ceph-mon[32956]: 1.e scrub ok 2024-06-27T21:55:05.826 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2566,"stamp":"2024-06-27T21:55:04.428168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9639776,"kb_used_data":10536,"kb_used_omap":110,"kb_used_meta":9629009,"kb_avail":1115302048,"statfs":{"total":1151940427776,"available":1142069297152,"internally_reserved":0,"allocated":10788864,"data_stored":6983340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9860106199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002304"},"pg_stats":[{"pgid":"1.b","version":"21'92","reported_seq":389,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529656+0000","last_change":"2024-06-27T21:54:43.156349+0000","last_active":"2024-06-27T21:54:52.529656+0000","last_peered":"2024-06-27T21:54:52.529656+0000","last_clean":"2024-06-27T21:54:52.529656+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:54:52.529656+0000","last_undegraded":"2024-06-27T21:54:52.529656+0000","last_fullsized":"2024-06-27T21:54:52.529656+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:43.156256+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_clean_scrub_stamp":"2024-06-27T21:54:43.156256+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:50.379979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":355,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:53.255230+0000","last_change":"2024-06-27T21:54:16.529576+0000","last_active":"2024-06-27T21:54:53.255230+0000","last_peered":"2024-06-27T21:54:53.255230+0000","last_clean":"2024-06-27T21:54:53.255230+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:54:53.255230+0000","last_undegraded":"2024-06-27T21:54:53.255230+0000","last_fullsized":"2024-06-27T21:54:53.255230+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:16.529488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:16.529488+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:28.909961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529550+0000","last_change":"2024-06-27T21:54:21.550785+0000","last_active":"2024-06-27T21:54:52.529550+0000","last_peered":"2024-06-27T21:54:52.529550+0000","last_clean":"2024-06-27T21:54:52.529550+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:54:52.529550+0000","last_undegraded":"2024-06-27T21:54:52.529550+0000","last_fullsized":"2024-06-27T21:54:52.529550+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:21.550689+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:54:21.550689+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:29.505992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1265,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197282+0000","last_change":"2024-06-27T21:53:53.024311+0000","last_active":"2024-06-27T21:54:52.197282+0000","last_peered":"2024-06-27T21:54:52.197282+0000","last_clean":"2024-06-27T21:54:52.197282+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:54:52.197282+0000","last_undegraded":"2024-06-27T21:54:52.197282+0000","last_fullsized":"2024-06-27T21:54:52.197282+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:53.024179+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:53:53.024179+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:03.223689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":878,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.530092+0000","last_change":"2024-06-27T21:54:25.134299+0000","last_active":"2024-06-27T21:54:52.530092+0000","last_peered":"2024-06-27T21:54:52.530092+0000","last_clean":"2024-06-27T21:54:52.530092+0000","last_became_active":"2024-06-27T21:53:20.074424+0000","last_became_peered":"2024-06-27T21:53:20.074424+0000","last_unstale":"2024-06-27T21:54:52.530092+0000","last_undegraded":"2024-06-27T21:54:52.530092+0000","last_fullsized":"2024-06-27T21:54:52.530092+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:25.134185+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.958636+0000","last_clean_scrub_stamp":"2024-06-27T21:54:25.134185+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:38.465041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":435,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.458788+0000","last_change":"2024-06-27T21:54:01.864531+0000","last_active":"2024-06-27T21:54:52.458788+0000","last_peered":"2024-06-27T21:54:52.458788+0000","last_clean":"2024-06-27T21:54:52.458788+0000","last_became_active":"2024-06-27T21:52:54.105254+0000","last_became_peered":"2024-06-27T21:52:54.105254+0000","last_unstale":"2024-06-27T21:54:52.458788+0000","last_undegraded":"2024-06-27T21:54:52.458788+0000","last_fullsized":"2024-06-27T21:54:52.458788+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:01.863767+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:54:01.863767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:06.984943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,10],"acting":[1,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":849,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.458789+0000","last_change":"2024-06-27T21:54:41.749332+0000","last_active":"2024-06-27T21:54:52.458789+0000","last_peered":"2024-06-27T21:54:52.458789+0000","last_clean":"2024-06-27T21:54:52.458789+0000","last_became_active":"2024-06-27T21:52:54.104890+0000","last_became_peered":"2024-06-27T21:52:54.104890+0000","last_unstale":"2024-06-27T21:54:52.458789+0000","last_undegraded":"2024-06-27T21:54:52.458789+0000","last_fullsized":"2024-06-27T21:54:52.458789+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:41.749243+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:24.498490+0000","last_clean_scrub_stamp":"2024-06-27T21:54:41.749243+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:09.584794+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:02.563526+0000","last_change":"2024-06-27T21:55:02.563553+0000","last_active":"2024-06-27T21:55:02.563526+0000","last_peered":"2024-06-27T21:55:02.563526+0000","last_clean":"2024-06-27T21:55:02.563526+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:55:02.563526+0000","last_undegraded":"2024-06-27T21:55:02.563526+0000","last_fullsized":"2024-06-27T21:55:02.563526+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:02.563464+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:55:02.563464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:14.881065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":493,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:55.806890+0000","last_change":"2024-06-27T21:54:55.806890+0000","last_active":"2024-06-27T21:54:55.806890+0000","last_peered":"2024-06-27T21:54:55.806890+0000","last_clean":"2024-06-27T21:54:55.806890+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:54:55.806890+0000","last_undegraded":"2024-06-27T21:54:55.806890+0000","last_fullsized":"2024-06-27T21:54:55.806890+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:55.806766+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:54:55.806766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:16.841648+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1388,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.530091+0000","last_change":"2024-06-27T21:53:58.219575+0000","last_active":"2024-06-27T21:54:52.530091+0000","last_peered":"2024-06-27T21:54:52.530091+0000","last_clean":"2024-06-27T21:54:52.530091+0000","last_became_active":"2024-06-27T21:53:20.075149+0000","last_became_peered":"2024-06-27T21:53:20.075149+0000","last_unstale":"2024-06-27T21:54:52.530091+0000","last_undegraded":"2024-06-27T21:54:52.530091+0000","last_fullsized":"2024-06-27T21:54:52.530091+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:58.219481+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:53:58.219481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:54:59.315219+0000","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1662,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197266+0000","last_change":"2024-06-27T21:54:43.207720+0000","last_active":"2024-06-27T21:54:52.197266+0000","last_peered":"2024-06-27T21:54:52.197266+0000","last_clean":"2024-06-27T21:54:52.197266+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:54:52.197266+0000","last_undegraded":"2024-06-27T21:54:52.197266+0000","last_fullsized":"2024-06-27T21:54:52.197266+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:43.207632+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:54:43.207632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:52.701537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1683,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197154+0000","last_change":"2024-06-27T21:53:56.129437+0000","last_active":"2024-06-27T21:54:52.197154+0000","last_peered":"2024-06-27T21:54:52.197154+0000","last_clean":"2024-06-27T21:54:52.197154+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:54:52.197154+0000","last_undegraded":"2024-06-27T21:54:52.197154+0000","last_fullsized":"2024-06-27T21:54:52.197154+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:56.129341+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:53:56.129341+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:19.714843+0000","scrub_duration":990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":384,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529726+0000","last_change":"2024-06-27T21:53:46.023494+0000","last_active":"2024-06-27T21:54:52.529726+0000","last_peered":"2024-06-27T21:54:52.529726+0000","last_clean":"2024-06-27T21:54:52.529726+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:54:52.529726+0000","last_undegraded":"2024-06-27T21:54:52.529726+0000","last_fullsized":"2024-06-27T21:54:52.529726+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:46.023357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:53:46.023357+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:14.912211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":285,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529736+0000","last_change":"2024-06-27T21:54:19.862951+0000","last_active":"2024-06-27T21:54:52.529736+0000","last_peered":"2024-06-27T21:54:52.529736+0000","last_clean":"2024-06-27T21:54:52.529736+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:54:52.529736+0000","last_undegraded":"2024-06-27T21:54:52.529736+0000","last_fullsized":"2024-06-27T21:54:52.529736+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:19.862853+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:52:57.593103+0000","last_clean_scrub_stamp":"2024-06-27T21:54:19.862853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:45.935453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529334+0000","last_change":"2024-06-27T21:53:56.479283+0000","last_active":"2024-06-27T21:54:52.529334+0000","last_peered":"2024-06-27T21:54:52.529334+0000","last_clean":"2024-06-27T21:54:52.529334+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:54:52.529334+0000","last_undegraded":"2024-06-27T21:54:52.529334+0000","last_fullsized":"2024-06-27T21:54:52.529334+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:56.479161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:53:56.479161+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:17.553205+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1352,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:55.499629+0000","last_change":"2024-06-27T21:54:55.499658+0000","last_active":"2024-06-27T21:54:55.499629+0000","last_peered":"2024-06-27T21:54:55.499629+0000","last_clean":"2024-06-27T21:54:55.499629+0000","last_became_active":"2024-06-27T21:53:13.043416+0000","last_became_peered":"2024-06-27T21:53:13.043416+0000","last_unstale":"2024-06-27T21:54:55.499629+0000","last_undegraded":"2024-06-27T21:54:55.499629+0000","last_fullsized":"2024-06-27T21:54:55.499629+0000","mapping_epoch":1005,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":1006,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:55.499562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:55.499562+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:11.180614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":317,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529720+0000","last_change":"2024-06-27T21:54:31.539044+0000","last_active":"2024-06-27T21:54:52.529720+0000","last_peered":"2024-06-27T21:54:52.529720+0000","last_clean":"2024-06-27T21:54:52.529720+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:54:52.529720+0000","last_undegraded":"2024-06-27T21:54:52.529720+0000","last_fullsized":"2024-06-27T21:54:52.529720+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:31.538940+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:31.538940+0000","last_clean_scrub_stamp":"2024-06-27T21:54:31.538940+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:42.591413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1274,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.744700+0000","last_change":"2024-06-27T21:54:33.771955+0000","last_active":"2024-06-27T21:54:52.744700+0000","last_peered":"2024-06-27T21:54:52.744700+0000","last_clean":"2024-06-27T21:54:52.744700+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:54:52.744700+0000","last_undegraded":"2024-06-27T21:54:52.744700+0000","last_fullsized":"2024-06-27T21:54:52.744700+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:33.771876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:33.771876+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:41.400319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1302,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529628+0000","last_change":"2024-06-27T21:54:29.542467+0000","last_active":"2024-06-27T21:54:52.529628+0000","last_peered":"2024-06-27T21:54:52.529628+0000","last_clean":"2024-06-27T21:54:52.529628+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:54:52.529628+0000","last_undegraded":"2024-06-27T21:54:52.529628+0000","last_fullsized":"2024-06-27T21:54:52.529628+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:29.542337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:29.542337+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:35.307339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":195,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:56.690824+0000","last_change":"2024-06-27T21:54:56.690845+0000","last_active":"2024-06-27T21:54:56.690824+0000","last_peered":"2024-06-27T21:54:56.690824+0000","last_clean":"2024-06-27T21:54:56.690824+0000","last_became_active":"2024-06-27T21:52:54.105376+0000","last_became_peered":"2024-06-27T21:52:54.105376+0000","last_unstale":"2024-06-27T21:54:56.690824+0000","last_undegraded":"2024-06-27T21:54:56.690824+0000","last_fullsized":"2024-06-27T21:54:56.690824+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:56.690765+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:56.690765+0000","last_clean_scrub_stamp":"2024-06-27T21:54:56.690765+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:19.912827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1274,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.196295+0000","last_change":"2024-06-27T21:54:42.639228+0000","last_active":"2024-06-27T21:54:52.196295+0000","last_peered":"2024-06-27T21:54:52.196295+0000","last_clean":"2024-06-27T21:54:52.196295+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:54:52.196295+0000","last_undegraded":"2024-06-27T21:54:52.196295+0000","last_fullsized":"2024-06-27T21:54:52.196295+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:42.639105+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:54:42.639105+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:46.217330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529542+0000","last_change":"2024-06-27T21:53:54.344369+0000","last_active":"2024-06-27T21:54:52.529542+0000","last_peered":"2024-06-27T21:54:52.529542+0000","last_clean":"2024-06-27T21:54:52.529542+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:54:52.529542+0000","last_undegraded":"2024-06-27T21:54:52.529542+0000","last_fullsized":"2024-06-27T21:54:52.529542+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:54.344252+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:53:54.344252+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:20.732420+0000","scrub_duration":873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":523,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.458829+0000","last_change":"2024-06-27T21:54:43.728524+0000","last_active":"2024-06-27T21:54:52.458829+0000","last_peered":"2024-06-27T21:54:52.458829+0000","last_clean":"2024-06-27T21:54:52.458829+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:54:52.458829+0000","last_undegraded":"2024-06-27T21:54:52.458829+0000","last_fullsized":"2024-06-27T21:54:52.458829+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:43.728437+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:54:43.728437+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:51.885786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":930,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529438+0000","last_change":"2024-06-27T21:53:41.991403+0000","last_active":"2024-06-27T21:54:52.529438+0000","last_peered":"2024-06-27T21:54:52.529438+0000","last_clean":"2024-06-27T21:54:52.529438+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:54:52.529438+0000","last_undegraded":"2024-06-27T21:54:52.529438+0000","last_fullsized":"2024-06-27T21:54:52.529438+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:41.991282+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:53:41.991282+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:11.090888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":320,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:01.481329+0000","last_change":"2024-06-27T21:55:01.481384+0000","last_active":"2024-06-27T21:55:01.481329+0000","last_peered":"2024-06-27T21:55:01.481329+0000","last_clean":"2024-06-27T21:55:01.481329+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:55:01.481329+0000","last_undegraded":"2024-06-27T21:55:01.481329+0000","last_fullsized":"2024-06-27T21:55:01.481329+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:01.481210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:55:01.481210+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:11.166258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":1125,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197277+0000","last_change":"2024-06-27T21:54:06.190722+0000","last_active":"2024-06-27T21:54:52.197277+0000","last_peered":"2024-06-27T21:54:52.197277+0000","last_clean":"2024-06-27T21:54:52.197277+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:54:52.197277+0000","last_undegraded":"2024-06-27T21:54:52.197277+0000","last_fullsized":"2024-06-27T21:54:52.197277+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:06.190625+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:06.190625+0000","last_clean_scrub_stamp":"2024-06-27T21:54:06.190625+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:16.627653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1219,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197186+0000","last_change":"2024-06-27T21:54:21.220589+0000","last_active":"2024-06-27T21:54:52.197186+0000","last_peered":"2024-06-27T21:54:52.197186+0000","last_clean":"2024-06-27T21:54:52.197186+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:54:52.197186+0000","last_undegraded":"2024-06-27T21:54:52.197186+0000","last_fullsized":"2024-06-27T21:54:52.197186+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:21.220458+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:21.220458+0000","last_clean_scrub_stamp":"2024-06-27T21:54:21.220458+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:31.163537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":912,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529360+0000","last_change":"2024-06-27T21:53:47.013435+0000","last_active":"2024-06-27T21:54:52.529360+0000","last_peered":"2024-06-27T21:54:52.529360+0000","last_clean":"2024-06-27T21:54:52.529360+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:54:52.529360+0000","last_undegraded":"2024-06-27T21:54:52.529360+0000","last_fullsized":"2024-06-27T21:54:52.529360+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:47.013309+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:53:47.013309+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:12.235799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":939,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:02.346752+0000","last_change":"2024-06-27T21:55:02.346782+0000","last_active":"2024-06-27T21:55:02.346752+0000","last_peered":"2024-06-27T21:55:02.346752+0000","last_clean":"2024-06-27T21:55:02.346752+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:55:02.346752+0000","last_undegraded":"2024-06-27T21:55:02.346752+0000","last_fullsized":"2024-06-27T21:55:02.346752+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:02.346683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:55:02.346683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:31.459269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":195,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:58.677740+0000","last_change":"2024-06-27T21:54:58.677762+0000","last_active":"2024-06-27T21:54:58.677740+0000","last_peered":"2024-06-27T21:54:58.677740+0000","last_clean":"2024-06-27T21:54:58.677740+0000","last_became_active":"2024-06-27T21:52:54.107704+0000","last_became_peered":"2024-06-27T21:52:54.107704+0000","last_unstale":"2024-06-27T21:54:58.677740+0000","last_undegraded":"2024-06-27T21:54:58.677740+0000","last_fullsized":"2024-06-27T21:54:58.677740+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:58.677680+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:54:58.677680+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:16.750817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":390,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529909+0000","last_change":"2024-06-27T21:54:28.009731+0000","last_active":"2024-06-27T21:54:52.529909+0000","last_peered":"2024-06-27T21:54:52.529909+0000","last_clean":"2024-06-27T21:54:52.529909+0000","last_became_active":"2024-06-27T21:52:17.603944+0000","last_became_peered":"2024-06-27T21:52:17.603944+0000","last_unstale":"2024-06-27T21:54:52.529909+0000","last_undegraded":"2024-06-27T21:54:52.529909+0000","last_fullsized":"2024-06-27T21:54:52.529909+0000","mapping_epoch":983,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:28.009606+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:54:28.009606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:39.352178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253416,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055092,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":1054334,"kb_avail":92690060,"statfs":{"total":95995035648,"available":94914621440,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1079638469},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417515051,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002360,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92742792,"statfs":{"total":95995035648,"available":94968619008,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1025636800},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785784,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959368,"statfs":{"total":95995035648,"available":95190392832,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":803862976},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972916,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609464,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93135688,"statfs":{"total":95995035648,"available":95370944512,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267252,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92905028,"statfs":{"total":95995035648,"available":95134748672,"internally_reserved":0,"allocated":970752,"data_stored":650961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":859306442},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758328,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986824,"statfs":{"total":95995035648,"available":95218507776,"internally_reserved":0,"allocated":1093632,"data_stored":777393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740156,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004996,"statfs":{"total":95995035648,"available":95237115904,"internally_reserved":0,"allocated":970752,"data_stored":650961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583740,"kb_used_data":748,"kb_used_omap":92,"kb_used_meta":582883,"kb_avail":93161412,"statfs":{"total":95995035648,"available":95397285888,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":596872290},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659764,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085388,"statfs":{"total":95995035648,"available":95319437312,"internally_reserved":0,"allocated":1093632,"data_stored":777393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":674494919},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411920,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808188,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936964,"statfs":{"total":95995035648,"available":95167451136,"internally_reserved":0,"allocated":970752,"data_stored":650961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575447,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916276,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828876,"statfs":{"total":95995035648,"available":95056769024,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738971,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880500,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864652,"statfs":{"total":95995035648,"available":95093403648,"internally_reserved":0,"allocated":1093632,"data_stored":777393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:55:05.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph pg dump --format=json 2024-06-27T21:55:06.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:55:06.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:06 smithi151 ceph-mon[28965]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:06.334 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:06 smithi151 ceph-mon[28965]: from='client.18438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:55:06.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:06 smithi190 ceph-mon[32247]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:06.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:06 smithi190 ceph-mon[32247]: from='client.18438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:55:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:06 smithi192 ceph-mon[32956]: pgmap v2566: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:06.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:06 smithi192 ceph-mon[32956]: from='client.18438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:55:07.906 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-06-27T21:55:07.907 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-06-27T21:55:08.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:08 smithi151 ceph-mon[28965]: pgmap v2567: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:08.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:08 smithi151 ceph-mon[28965]: 1.0 scrub starts 2024-06-27T21:55:08.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:08 smithi151 ceph-mon[28965]: 1.0 scrub ok 2024-06-27T21:55:08.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:08 smithi190 ceph-mon[32247]: pgmap v2567: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:08.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:08 smithi190 ceph-mon[32247]: 1.0 scrub starts 2024-06-27T21:55:08.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:08 smithi190 ceph-mon[32247]: 1.0 scrub ok 2024-06-27T21:55:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:08 smithi192 ceph-mon[32956]: pgmap v2567: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:08 smithi192 ceph-mon[32956]: 1.0 scrub starts 2024-06-27T21:55:08.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:08 smithi192 ceph-mon[32956]: 1.0 scrub ok 2024-06-27T21:55:08.642 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2567,"stamp":"2024-06-27T21:55:06.428738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9639768,"kb_used_data":10536,"kb_used_omap":110,"kb_used_meta":9629009,"kb_avail":1115302056,"statfs":{"total":1151940427776,"available":1142069305344,"internally_reserved":0,"allocated":10788864,"data_stored":6983340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112681,"internal_metadata":9860106199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002487"},"pg_stats":[{"pgid":"1.b","version":"21'92","reported_seq":389,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529656+0000","last_change":"2024-06-27T21:54:43.156349+0000","last_active":"2024-06-27T21:54:52.529656+0000","last_peered":"2024-06-27T21:54:52.529656+0000","last_clean":"2024-06-27T21:54:52.529656+0000","last_became_active":"2024-06-27T21:47:59.091491+0000","last_became_peered":"2024-06-27T21:47:59.091491+0000","last_unstale":"2024-06-27T21:54:52.529656+0000","last_undegraded":"2024-06-27T21:54:52.529656+0000","last_fullsized":"2024-06-27T21:54:52.529656+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:43.156256+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:27.033721+0000","last_clean_scrub_stamp":"2024-06-27T21:54:43.156256+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:50.379979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"21'92","reported_seq":355,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:53.255230+0000","last_change":"2024-06-27T21:54:16.529576+0000","last_active":"2024-06-27T21:54:53.255230+0000","last_peered":"2024-06-27T21:54:53.255230+0000","last_clean":"2024-06-27T21:54:53.255230+0000","last_became_active":"2024-06-27T21:47:11.206300+0000","last_became_peered":"2024-06-27T21:47:11.206300+0000","last_unstale":"2024-06-27T21:54:53.255230+0000","last_undegraded":"2024-06-27T21:54:53.255230+0000","last_fullsized":"2024-06-27T21:54:53.255230+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:16.529488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:16.529488+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:28.909961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529550+0000","last_change":"2024-06-27T21:54:21.550785+0000","last_active":"2024-06-27T21:54:52.529550+0000","last_peered":"2024-06-27T21:54:52.529550+0000","last_clean":"2024-06-27T21:54:52.529550+0000","last_became_active":"2024-06-27T21:44:49.893425+0000","last_became_peered":"2024-06-27T21:44:49.893425+0000","last_unstale":"2024-06-27T21:54:52.529550+0000","last_undegraded":"2024-06-27T21:54:52.529550+0000","last_fullsized":"2024-06-27T21:54:52.529550+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:21.550689+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:54:21.550689+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:29.505992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"21'92","reported_seq":1265,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197282+0000","last_change":"2024-06-27T21:53:53.024311+0000","last_active":"2024-06-27T21:54:52.197282+0000","last_peered":"2024-06-27T21:54:52.197282+0000","last_clean":"2024-06-27T21:54:52.197282+0000","last_became_active":"2024-06-27T21:44:49.902553+0000","last_became_peered":"2024-06-27T21:44:49.902553+0000","last_unstale":"2024-06-27T21:54:52.197282+0000","last_undegraded":"2024-06-27T21:54:52.197282+0000","last_fullsized":"2024-06-27T21:54:52.197282+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:53.024179+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:53:53.024179+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:03.223689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"21'92","reported_seq":878,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.530092+0000","last_change":"2024-06-27T21:54:25.134299+0000","last_active":"2024-06-27T21:54:52.530092+0000","last_peered":"2024-06-27T21:54:52.530092+0000","last_clean":"2024-06-27T21:54:52.530092+0000","last_became_active":"2024-06-27T21:53:20.074424+0000","last_became_peered":"2024-06-27T21:53:20.074424+0000","last_unstale":"2024-06-27T21:54:52.530092+0000","last_undegraded":"2024-06-27T21:54:52.530092+0000","last_fullsized":"2024-06-27T21:54:52.530092+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:25.134185+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:25.958636+0000","last_clean_scrub_stamp":"2024-06-27T21:54:25.134185+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:38.465041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":435,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.458788+0000","last_change":"2024-06-27T21:54:01.864531+0000","last_active":"2024-06-27T21:54:52.458788+0000","last_peered":"2024-06-27T21:54:52.458788+0000","last_clean":"2024-06-27T21:54:52.458788+0000","last_became_active":"2024-06-27T21:52:54.105254+0000","last_became_peered":"2024-06-27T21:52:54.105254+0000","last_unstale":"2024-06-27T21:54:52.458788+0000","last_undegraded":"2024-06-27T21:54:52.458788+0000","last_fullsized":"2024-06-27T21:54:52.458788+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:01.863767+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:41:04.707163+0000","last_clean_scrub_stamp":"2024-06-27T21:54:01.863767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:06.984943+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,10],"acting":[1,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":849,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.458789+0000","last_change":"2024-06-27T21:54:41.749332+0000","last_active":"2024-06-27T21:54:52.458789+0000","last_peered":"2024-06-27T21:54:52.458789+0000","last_clean":"2024-06-27T21:54:52.458789+0000","last_became_active":"2024-06-27T21:52:54.104890+0000","last_became_peered":"2024-06-27T21:52:54.104890+0000","last_unstale":"2024-06-27T21:54:52.458789+0000","last_undegraded":"2024-06-27T21:54:52.458789+0000","last_fullsized":"2024-06-27T21:54:52.458789+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:41.749243+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:50:24.498490+0000","last_clean_scrub_stamp":"2024-06-27T21:54:41.749243+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:09.584794+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:02.563526+0000","last_change":"2024-06-27T21:55:02.563553+0000","last_active":"2024-06-27T21:55:02.563526+0000","last_peered":"2024-06-27T21:55:02.563526+0000","last_clean":"2024-06-27T21:55:02.563526+0000","last_became_active":"2024-06-27T21:44:49.892601+0000","last_became_peered":"2024-06-27T21:44:49.892601+0000","last_unstale":"2024-06-27T21:55:02.563526+0000","last_undegraded":"2024-06-27T21:55:02.563526+0000","last_fullsized":"2024-06-27T21:55:02.563526+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:02.563464+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:42.388603+0000","last_clean_scrub_stamp":"2024-06-27T21:55:02.563464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:14.881065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":493,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:55.806890+0000","last_change":"2024-06-27T21:54:55.806890+0000","last_active":"2024-06-27T21:54:55.806890+0000","last_peered":"2024-06-27T21:54:55.806890+0000","last_clean":"2024-06-27T21:54:55.806890+0000","last_became_active":"2024-06-27T21:46:34.528304+0000","last_became_peered":"2024-06-27T21:46:34.528304+0000","last_unstale":"2024-06-27T21:54:55.806890+0000","last_undegraded":"2024-06-27T21:54:55.806890+0000","last_fullsized":"2024-06-27T21:54:55.806890+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:55.806766+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:40.396776+0000","last_clean_scrub_stamp":"2024-06-27T21:54:55.806766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:16.841648+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,11],"acting":[0,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":1396,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:02.231246+0000","last_change":"2024-06-27T21:55:02.231273+0000","last_active":"2024-06-27T21:55:02.231246+0000","last_peered":"2024-06-27T21:55:02.231246+0000","last_clean":"2024-06-27T21:55:02.231246+0000","last_became_active":"2024-06-27T21:53:20.075149+0000","last_became_peered":"2024-06-27T21:53:20.075149+0000","last_unstale":"2024-06-27T21:55:02.231246+0000","last_undegraded":"2024-06-27T21:55:02.231246+0000","last_fullsized":"2024-06-27T21:55:02.231246+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:02.231177+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:26:22.770782+0000","last_clean_scrub_stamp":"2024-06-27T21:55:02.231177+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:23.434018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":1662,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197266+0000","last_change":"2024-06-27T21:54:43.207720+0000","last_active":"2024-06-27T21:54:52.197266+0000","last_peered":"2024-06-27T21:54:52.197266+0000","last_clean":"2024-06-27T21:54:52.197266+0000","last_became_active":"2024-06-27T21:36:51.809720+0000","last_became_peered":"2024-06-27T21:36:51.809720+0000","last_unstale":"2024-06-27T21:54:52.197266+0000","last_undegraded":"2024-06-27T21:54:52.197266+0000","last_fullsized":"2024-06-27T21:54:52.197266+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:43.207632+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:54:43.207632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:52.701537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":1683,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197154+0000","last_change":"2024-06-27T21:53:56.129437+0000","last_active":"2024-06-27T21:54:52.197154+0000","last_peered":"2024-06-27T21:54:52.197154+0000","last_clean":"2024-06-27T21:54:52.197154+0000","last_became_active":"2024-06-27T21:48:18.177911+0000","last_became_peered":"2024-06-27T21:48:18.177911+0000","last_unstale":"2024-06-27T21:54:52.197154+0000","last_undegraded":"2024-06-27T21:54:52.197154+0000","last_fullsized":"2024-06-27T21:54:52.197154+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:56.129341+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:00.224085+0000","last_clean_scrub_stamp":"2024-06-27T21:53:56.129341+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:19.714843+0000","scrub_duration":990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":384,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529726+0000","last_change":"2024-06-27T21:53:46.023494+0000","last_active":"2024-06-27T21:54:52.529726+0000","last_peered":"2024-06-27T21:54:52.529726+0000","last_clean":"2024-06-27T21:54:52.529726+0000","last_became_active":"2024-06-27T21:46:34.528332+0000","last_became_peered":"2024-06-27T21:46:34.528332+0000","last_unstale":"2024-06-27T21:54:52.529726+0000","last_undegraded":"2024-06-27T21:54:52.529726+0000","last_fullsized":"2024-06-27T21:54:52.529726+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:46.023357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:53:46.023357+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:14.912211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":285,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529736+0000","last_change":"2024-06-27T21:54:19.862951+0000","last_active":"2024-06-27T21:54:52.529736+0000","last_peered":"2024-06-27T21:54:52.529736+0000","last_clean":"2024-06-27T21:54:52.529736+0000","last_became_active":"2024-06-27T21:48:24.958901+0000","last_became_peered":"2024-06-27T21:48:24.958901+0000","last_unstale":"2024-06-27T21:54:52.529736+0000","last_undegraded":"2024-06-27T21:54:52.529736+0000","last_fullsized":"2024-06-27T21:54:52.529736+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:19.862853+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:52:57.593103+0000","last_clean_scrub_stamp":"2024-06-27T21:54:19.862853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:45.935453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529334+0000","last_change":"2024-06-27T21:53:56.479283+0000","last_active":"2024-06-27T21:54:52.529334+0000","last_peered":"2024-06-27T21:54:52.529334+0000","last_clean":"2024-06-27T21:54:52.529334+0000","last_became_active":"2024-06-27T21:44:49.891144+0000","last_became_peered":"2024-06-27T21:44:49.891144+0000","last_unstale":"2024-06-27T21:54:52.529334+0000","last_undegraded":"2024-06-27T21:54:52.529334+0000","last_fullsized":"2024-06-27T21:54:52.529334+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:56.479161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:53:56.479161+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:17.553205+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":1352,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:55.499629+0000","last_change":"2024-06-27T21:54:55.499658+0000","last_active":"2024-06-27T21:54:55.499629+0000","last_peered":"2024-06-27T21:54:55.499629+0000","last_clean":"2024-06-27T21:54:55.499629+0000","last_became_active":"2024-06-27T21:53:13.043416+0000","last_became_peered":"2024-06-27T21:53:13.043416+0000","last_unstale":"2024-06-27T21:54:55.499629+0000","last_undegraded":"2024-06-27T21:54:55.499629+0000","last_fullsized":"2024-06-27T21:54:55.499629+0000","mapping_epoch":1005,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":1006,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:55.499562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:55.499562+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:11.180614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"21'92","reported_seq":317,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529720+0000","last_change":"2024-06-27T21:54:31.539044+0000","last_active":"2024-06-27T21:54:52.529720+0000","last_peered":"2024-06-27T21:54:52.529720+0000","last_clean":"2024-06-27T21:54:52.529720+0000","last_became_active":"2024-06-27T21:47:59.155826+0000","last_became_peered":"2024-06-27T21:47:59.155826+0000","last_unstale":"2024-06-27T21:54:52.529720+0000","last_undegraded":"2024-06-27T21:54:52.529720+0000","last_fullsized":"2024-06-27T21:54:52.529720+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:31.538940+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:31.538940+0000","last_clean_scrub_stamp":"2024-06-27T21:54:31.538940+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:42.591413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"21'92","reported_seq":1274,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.744700+0000","last_change":"2024-06-27T21:54:33.771955+0000","last_active":"2024-06-27T21:54:52.744700+0000","last_peered":"2024-06-27T21:54:52.744700+0000","last_clean":"2024-06-27T21:54:52.744700+0000","last_became_active":"2024-06-27T21:36:51.811158+0000","last_became_peered":"2024-06-27T21:36:51.811158+0000","last_unstale":"2024-06-27T21:54:52.744700+0000","last_undegraded":"2024-06-27T21:54:52.744700+0000","last_fullsized":"2024-06-27T21:54:52.744700+0000","mapping_epoch":632,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:33.771876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:33.771876+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:41.400319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"21'92","reported_seq":1302,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529628+0000","last_change":"2024-06-27T21:54:29.542467+0000","last_active":"2024-06-27T21:54:52.529628+0000","last_peered":"2024-06-27T21:54:52.529628+0000","last_clean":"2024-06-27T21:54:52.529628+0000","last_became_active":"2024-06-27T21:44:35.890637+0000","last_became_peered":"2024-06-27T21:44:35.890637+0000","last_unstale":"2024-06-27T21:54:52.529628+0000","last_undegraded":"2024-06-27T21:54:52.529628+0000","last_fullsized":"2024-06-27T21:54:52.529628+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:29.542337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:54:29.542337+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:35.307339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"21'92","reported_seq":195,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:56.690824+0000","last_change":"2024-06-27T21:54:56.690845+0000","last_active":"2024-06-27T21:54:56.690824+0000","last_peered":"2024-06-27T21:54:56.690824+0000","last_clean":"2024-06-27T21:54:56.690824+0000","last_became_active":"2024-06-27T21:52:54.105376+0000","last_became_peered":"2024-06-27T21:52:54.105376+0000","last_unstale":"2024-06-27T21:54:56.690824+0000","last_undegraded":"2024-06-27T21:54:56.690824+0000","last_fullsized":"2024-06-27T21:54:56.690824+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:56.690765+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:56.690765+0000","last_clean_scrub_stamp":"2024-06-27T21:54:56.690765+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:19.912827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"21'92","reported_seq":1274,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.196295+0000","last_change":"2024-06-27T21:54:42.639228+0000","last_active":"2024-06-27T21:54:52.196295+0000","last_peered":"2024-06-27T21:54:52.196295+0000","last_clean":"2024-06-27T21:54:52.196295+0000","last_became_active":"2024-06-27T21:44:49.889869+0000","last_became_peered":"2024-06-27T21:44:49.889869+0000","last_unstale":"2024-06-27T21:54:52.196295+0000","last_undegraded":"2024-06-27T21:54:52.196295+0000","last_fullsized":"2024-06-27T21:54:52.196295+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":477,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:42.639105+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:43:38.265889+0000","last_clean_scrub_stamp":"2024-06-27T21:54:42.639105+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:46.217330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"21'92","reported_seq":494,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529542+0000","last_change":"2024-06-27T21:53:54.344369+0000","last_active":"2024-06-27T21:54:52.529542+0000","last_peered":"2024-06-27T21:54:52.529542+0000","last_clean":"2024-06-27T21:54:52.529542+0000","last_became_active":"2024-06-27T21:44:49.891715+0000","last_became_peered":"2024-06-27T21:44:49.891715+0000","last_unstale":"2024-06-27T21:54:52.529542+0000","last_undegraded":"2024-06-27T21:54:52.529542+0000","last_fullsized":"2024-06-27T21:54:52.529542+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:54.344252+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:53:54.344252+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:20.732420+0000","scrub_duration":873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"21'92","reported_seq":523,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.458829+0000","last_change":"2024-06-27T21:54:43.728524+0000","last_active":"2024-06-27T21:54:52.458829+0000","last_peered":"2024-06-27T21:54:52.458829+0000","last_clean":"2024-06-27T21:54:52.458829+0000","last_became_active":"2024-06-27T21:47:11.208884+0000","last_became_peered":"2024-06-27T21:47:11.208884+0000","last_unstale":"2024-06-27T21:54:52.458829+0000","last_undegraded":"2024-06-27T21:54:52.458829+0000","last_fullsized":"2024-06-27T21:54:52.458829+0000","mapping_epoch":892,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":893,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:43.728437+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:23:20.126469+0000","last_clean_scrub_stamp":"2024-06-27T21:54:43.728437+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:51.885786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"21'92","reported_seq":930,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529438+0000","last_change":"2024-06-27T21:53:41.991403+0000","last_active":"2024-06-27T21:54:52.529438+0000","last_peered":"2024-06-27T21:54:52.529438+0000","last_clean":"2024-06-27T21:54:52.529438+0000","last_became_active":"2024-06-27T21:47:59.078819+0000","last_became_peered":"2024-06-27T21:47:59.078819+0000","last_unstale":"2024-06-27T21:54:52.529438+0000","last_undegraded":"2024-06-27T21:54:52.529438+0000","last_fullsized":"2024-06-27T21:54:52.529438+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:41.991282+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:33:03.234726+0000","last_clean_scrub_stamp":"2024-06-27T21:53:41.991282+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:11.090888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"21'92","reported_seq":320,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:01.481329+0000","last_change":"2024-06-27T21:55:01.481384+0000","last_active":"2024-06-27T21:55:01.481329+0000","last_peered":"2024-06-27T21:55:01.481329+0000","last_clean":"2024-06-27T21:55:01.481329+0000","last_became_active":"2024-06-27T21:47:59.113172+0000","last_became_peered":"2024-06-27T21:47:59.113172+0000","last_unstale":"2024-06-27T21:55:01.481329+0000","last_undegraded":"2024-06-27T21:55:01.481329+0000","last_fullsized":"2024-06-27T21:55:01.481329+0000","mapping_epoch":909,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":910,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:01.481210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:55:01.481210+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:11.166258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1408736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"21'92","reported_seq":1125,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197277+0000","last_change":"2024-06-27T21:54:06.190722+0000","last_active":"2024-06-27T21:54:52.197277+0000","last_peered":"2024-06-27T21:54:52.197277+0000","last_clean":"2024-06-27T21:54:52.197277+0000","last_became_active":"2024-06-27T21:33:01.781024+0000","last_became_peered":"2024-06-27T21:33:01.781024+0000","last_unstale":"2024-06-27T21:54:52.197277+0000","last_undegraded":"2024-06-27T21:54:52.197277+0000","last_fullsized":"2024-06-27T21:54:52.197277+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:06.190625+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:06.190625+0000","last_clean_scrub_stamp":"2024-06-27T21:54:06.190625+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:16.627653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"21'92","reported_seq":1219,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.197186+0000","last_change":"2024-06-27T21:54:21.220589+0000","last_active":"2024-06-27T21:54:52.197186+0000","last_peered":"2024-06-27T21:54:52.197186+0000","last_clean":"2024-06-27T21:54:52.197186+0000","last_became_active":"2024-06-27T21:44:49.893786+0000","last_became_peered":"2024-06-27T21:44:49.893786+0000","last_unstale":"2024-06-27T21:54:52.197186+0000","last_undegraded":"2024-06-27T21:54:52.197186+0000","last_fullsized":"2024-06-27T21:54:52.197186+0000","mapping_epoch":820,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":821,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:21.220458+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:54:21.220458+0000","last_clean_scrub_stamp":"2024-06-27T21:54:21.220458+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:31.163537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"21'92","reported_seq":912,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529360+0000","last_change":"2024-06-27T21:53:47.013435+0000","last_active":"2024-06-27T21:54:52.529360+0000","last_peered":"2024-06-27T21:54:52.529360+0000","last_clean":"2024-06-27T21:54:52.529360+0000","last_became_active":"2024-06-27T21:48:24.958429+0000","last_became_peered":"2024-06-27T21:48:24.958429+0000","last_unstale":"2024-06-27T21:54:52.529360+0000","last_undegraded":"2024-06-27T21:54:52.529360+0000","last_fullsized":"2024-06-27T21:54:52.529360+0000","mapping_epoch":924,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:53:47.013309+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:46:36.721315+0000","last_clean_scrub_stamp":"2024-06-27T21:53:47.013309+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:12.235799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"21'92","reported_seq":939,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:55:02.346752+0000","last_change":"2024-06-27T21:55:02.346782+0000","last_active":"2024-06-27T21:55:02.346752+0000","last_peered":"2024-06-27T21:55:02.346752+0000","last_clean":"2024-06-27T21:55:02.346752+0000","last_became_active":"2024-06-27T21:44:35.890267+0000","last_became_peered":"2024-06-27T21:44:35.890267+0000","last_unstale":"2024-06-27T21:55:02.346752+0000","last_undegraded":"2024-06-27T21:55:02.346752+0000","last_fullsized":"2024-06-27T21:55:02.346752+0000","mapping_epoch":815,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":816,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:55:02.346683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-27T21:06:08.276613+0000","last_clean_scrub_stamp":"2024-06-27T21:55:02.346683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:31.459269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"21'92","reported_seq":195,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:58.677740+0000","last_change":"2024-06-27T21:54:58.677762+0000","last_active":"2024-06-27T21:54:58.677740+0000","last_peered":"2024-06-27T21:54:58.677740+0000","last_clean":"2024-06-27T21:54:58.677740+0000","last_became_active":"2024-06-27T21:52:54.107704+0000","last_became_peered":"2024-06-27T21:52:54.107704+0000","last_unstale":"2024-06-27T21:54:58.677740+0000","last_undegraded":"2024-06-27T21:54:58.677740+0000","last_fullsized":"2024-06-27T21:54:58.677740+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:58.677680+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:21:18.604618+0000","last_clean_scrub_stamp":"2024-06-27T21:54:58.677680+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:56:16.750817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"21'92","reported_seq":390,"reported_epoch":1037,"state":"active+clean","last_fresh":"2024-06-27T21:54:52.529909+0000","last_change":"2024-06-27T21:54:28.009731+0000","last_active":"2024-06-27T21:54:52.529909+0000","last_peered":"2024-06-27T21:54:52.529909+0000","last_clean":"2024-06-27T21:54:52.529909+0000","last_became_active":"2024-06-27T21:52:17.603944+0000","last_became_peered":"2024-06-27T21:52:17.603944+0000","last_unstale":"2024-06-27T21:54:52.529909+0000","last_undegraded":"2024-06-27T21:54:52.529909+0000","last_fullsized":"2024-06-27T21:54:52.529909+0000","mapping_epoch":983,"log_start":"0'0","ondisk_log_start":"0'0","created":535,"last_epoch_clean":984,"parent":"0.0","parent_split_bits":5,"last_scrub":"21'92","last_scrub_stamp":"2024-06-27T21:54:28.009606+0000","last_deep_scrub":"21'92","last_deep_scrub_stamp":"2024-06-27T21:39:45.256884+0000","last_clean_scrub_stamp":"2024-06-27T21:54:28.009606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-27T21:55:39.352178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,6],"acting":[11,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1736416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777253417,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055092,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":1054334,"kb_avail":92690060,"statfs":{"total":95995035648,"available":94914621440,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1079638469},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417515052,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1002356,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":1001598,"kb_avail":92742796,"statfs":{"total":95995035648,"available":94968623104,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":1025636800},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743985,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785784,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959368,"statfs":{"total":95995035648,"available":95190392832,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":803862976},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972916,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609464,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93135688,"statfs":{"total":95995035648,"available":95370944512,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633267252,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92905028,"statfs":{"total":95995035648,"available":95134748672,"internally_reserved":0,"allocated":970752,"data_stored":650961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":859306442},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528894,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758324,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986828,"statfs":{"total":95995035648,"available":95218511872,"internally_reserved":0,"allocated":1093632,"data_stored":777393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740156,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004996,"statfs":{"total":95995035648,"available":95237115904,"internally_reserved":0,"allocated":970752,"data_stored":650961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":362,"seq":1554778161534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583740,"kb_used_data":748,"kb_used_omap":92,"kb_used_meta":582883,"kb_avail":93161412,"statfs":{"total":95995035648,"available":95397285888,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95134,"internal_metadata":596872290},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659764,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93085388,"statfs":{"total":95995035648,"available":95319437312,"internally_reserved":0,"allocated":1093632,"data_stored":777393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":674494919},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411920,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808188,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936964,"statfs":{"total":95995035648,"available":95167451136,"internally_reserved":0,"allocated":970752,"data_stored":650961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575447,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916276,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828876,"statfs":{"total":95995035648,"available":95056769024,"internally_reserved":0,"allocated":765952,"data_stored":449713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738972,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880500,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864652,"statfs":{"total":95995035648,"available":95093403648,"internally_reserved":0,"allocated":1093632,"data_stored":777393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-27T21:55:08.644 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-06-27T21:55:08.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122 shell --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-06-27T21:55:08.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/mon.a/config 2024-06-27T21:55:09.316 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:09 smithi151 ceph-mon[28965]: from='client.18444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:55:09.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:09 smithi190 ceph-mon[32247]: from='client.18444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:55:09.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:09 smithi192 ceph-mon[32956]: from='client.18444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-27T21:55:10.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:10 smithi151 ceph-mon[28965]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:10.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:10 smithi190 ceph-mon[32247]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:10.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:10 smithi192 ceph-mon[32956]: pgmap v2568: 31 pgs: 31 active+clean; 577 KiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-27T21:55:11.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/435277477' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-06-27T21:55:11.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:11 smithi190 ceph-mon[32247]: from='client.? 172.21.15.151:0/435277477' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-06-27T21:55:11.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:55:11.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/435277477' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-06-27T21:55:11.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 ceph-mon[28965]: from='client.? 172.21.15.151:0/435277477' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-06-27T21:55:11.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:55:11.197 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:55:11.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:55:11.198 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 ceph-mon[28965]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:55:11.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:55:11.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:55:11.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:11 smithi190 ceph-mon[32247]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:55:11.521 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-27T21:55:11.540 INFO:tasks.cephadm:Teardown begin 2024-06-27T21:55:11.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:55:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/435277477' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-06-27T21:55:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:11 smithi192 ceph-mon[32956]: from='client.? 172.21.15.151:0/435277477' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-06-27T21:55:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-27T21:55:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-27T21:55:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-27T21:55:11.544 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:11 smithi192 ceph-mon[32956]: from='mgr.14150 172.21.15.151:0/2099242946' entity='mgr.y' 2024-06-27T21:55:11.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:55:11.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:55:11.633 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-27T21:55:11.634 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-27T21:55:11.656 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-27T21:55:11.685 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-27T21:55:11.702 INFO:tasks.cephadm:Stopping all daemons... 2024-06-27T21:55:11.703 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-27T21:55:11.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a 2024-06-27T21:55:12.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:11 smithi151 systemd[1]: Stopping Ceph mon.a for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:12.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:12 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:55:12.032+0000 7f15dc32a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:12.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:12 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a[28961]: 2024-06-27T21:55:12.032+0000 7f15dc32a640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-27T21:55:12.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:12 smithi151 podman[306510]: 2024-06-27 21:55:12.08375135 +0000 UTC m=+0.170120549 container died 2adefe13d0e02bd30496252c93edad231fdfb85a9c71157dfa6cbbc8224d2769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:55:12.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:12 smithi151 podman[306510]: 2024-06-27 21:55:12.557631012 +0000 UTC m=+0.644000214 container cleanup 2adefe13d0e02bd30496252c93edad231fdfb85a9c71157dfa6cbbc8224d2769 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD) 2024-06-27T21:55:12.697 INFO:journalctl@ceph.mon.a.smithi151.stdout:Jun 27 21:55:12 smithi151 bash[306510]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-a 2024-06-27T21:55:12.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.a.service' 2024-06-27T21:55:13.044 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:13.044 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-27T21:55:13.044 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-06-27T21:55:13.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.b 2024-06-27T21:55:13.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:13 smithi190 systemd[1]: Stopping Ceph mon.b for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:13.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:13 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-b[32243]: 2024-06-27T21:55:13.455+0000 7f980c7ae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:13.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:13 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-b[32243]: 2024-06-27T21:55:13.455+0000 7f980c7ae640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-27T21:55:13.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:13 smithi190 podman[197764]: 2024-06-27 21:55:13.619132577 +0000 UTC m=+0.285052170 container died bbb336e9e7a78e5ded01419c8bf4ca05a741f62c4bcd5382c16860b53fbf0fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-b, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-27T21:55:14.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:14 smithi190 podman[197764]: 2024-06-27 21:55:14.081421987 +0000 UTC m=+0.747341622 container cleanup bbb336e9e7a78e5ded01419c8bf4ca05a741f62c4bcd5382c16860b53fbf0fa8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625) 2024-06-27T21:55:14.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Jun 27 21:55:14 smithi190 bash[197764]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-b 2024-06-27T21:55:14.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.b.service' 2024-06-27T21:55:14.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:14.559 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-06-27T21:55:14.559 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-06-27T21:55:14.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.c 2024-06-27T21:55:15.000 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:14 smithi192 systemd[1]: Stopping Ceph mon.c for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:15.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-c[32938]: 2024-06-27T21:55:14.999+0000 7fdfa6fc3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:15.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-c[32938]: 2024-06-27T21:55:15.000+0000 7fdfa6fc3640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-27T21:55:15.294 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:15 smithi192 podman[194659]: 2024-06-27 21:55:15.09258784 +0000 UTC m=+0.232196624 container died 812c6218d9b80c975c300dbb90d35f80bddcd5f508afa99163a93e81236c5b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:15.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:15 smithi192 podman[194659]: 2024-06-27 21:55:15.667796185 +0000 UTC m=+0.807405011 container cleanup 812c6218d9b80c975c300dbb90d35f80bddcd5f508afa99163a93e81236c5b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:55:15.794 INFO:journalctl@ceph.mon.c.smithi192.stdout:Jun 27 21:55:15 smithi192 bash[194659]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mon-c 2024-06-27T21:55:16.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mon.c.service' 2024-06-27T21:55:16.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:16.115 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-06-27T21:55:16.115 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-27T21:55:16.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y 2024-06-27T21:55:16.395 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:55:16 smithi151 systemd[1]: Stopping Ceph mgr.y for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:16.687 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:55:16 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y[29189]: 2024-06-27T21:55:16.395+0000 7fc37035f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:16.687 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:55:16 smithi151 podman[306632]: 2024-06-27 21:55:16.426708956 +0000 UTC m=+0.118187015 container died d869d5399341d57439d3dcc5c335a637da176ac7db62e06c3c9c677e741078ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-27T21:55:16.947 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:55:16 smithi151 podman[306632]: 2024-06-27 21:55:16.904274104 +0000 UTC m=+0.595752161 container cleanup d869d5399341d57439d3dcc5c335a637da176ac7db62e06c3c9c677e741078ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f51acc18cc950586c82c58d666661d4391fc1122, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True) 2024-06-27T21:55:16.947 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Jun 27 21:55:16 smithi151 bash[306632]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-y 2024-06-27T21:55:17.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.y.service' 2024-06-27T21:55:17.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:17.365 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-27T21:55:17.365 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-27T21:55:17.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.x 2024-06-27T21:55:17.730 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:55:17 smithi190 systemd[1]: Stopping Ceph mgr.x for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:18.013 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:55:17 smithi190 podman[197888]: 2024-06-27 21:55:17.731708276 +0000 UTC m=+0.170247841 container died e30e311f29287962ba88f6563a17828263e25c2277aa6cc0fab43a410df4096c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-x, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-27T21:55:18.353 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:55:18 smithi190 podman[197888]: 2024-06-27 21:55:18.111001086 +0000 UTC m=+0.549540648 container cleanup e30e311f29287962ba88f6563a17828263e25c2277aa6cc0fab43a410df4096c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-x, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-27T21:55:18.353 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Jun 27 21:55:18 smithi190 bash[197888]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-mgr-x 2024-06-27T21:55:18.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@mgr.x.service' 2024-06-27T21:55:18.510 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:18.510 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-27T21:55:18.510 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-27T21:55:18.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.0 2024-06-27T21:55:18.808 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:18 smithi151 systemd[1]: Stopping Ceph osd.0 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:19.197 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:18 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0[40567]: 2024-06-27T21:55:18.807+0000 7f59ae5e2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:19.197 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:18 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0[40567]: 2024-06-27T21:55:18.807+0000 7f59ae5e2640 -1 osd.0 1037 *** Got signal Terminated *** 2024-06-27T21:55:19.197 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:18 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0[40567]: 2024-06-27T21:55:18.807+0000 7f59ae5e2640 -1 osd.0 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:55:24.778 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:24 smithi151 podman[306755]: 2024-06-27 21:55:24.451815108 +0000 UTC m=+5.747183193 container died 0a2a78f7cfef7d5aacb34e5358a88e1ea4d17d1e0d229be2f3d363af1c7f1488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-27T21:55:25.197 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:24 smithi151 podman[306755]: 2024-06-27 21:55:24.98374095 +0000 UTC m=+6.279109036 container cleanup 0a2a78f7cfef7d5aacb34e5358a88e1ea4d17d1e0d229be2f3d363af1c7f1488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, ceph=True, io.buildah.version=1.36.0) 2024-06-27T21:55:25.197 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:24 smithi151 bash[306755]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0 2024-06-27T21:55:25.697 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:25 smithi151 podman[306783]: 2024-06-27 21:55:25.285472473 +0000 UTC m=+0.825617397 container remove 0a2a78f7cfef7d5aacb34e5358a88e1ea4d17d1e0d229be2f3d363af1c7f1488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:26.037 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:25 smithi151 podman[306874]: 2024-06-27 21:55:25.664282502 +0000 UTC m=+0.017488920 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:55:26.037 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:25 smithi151 podman[306874]: 2024-06-27 21:55:25.794783435 +0000 UTC m=+0.147989847 container create 669dd2256905553296a4fa476b33ca86a86375bc8d06277da740b028eba41a22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:55:26.038 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:26 smithi151 podman[306874]: 2024-06-27 21:55:26.038617891 +0000 UTC m=+0.391824304 container init 669dd2256905553296a4fa476b33ca86a86375bc8d06277da740b028eba41a22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-27T21:55:26.305 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:26 smithi151 podman[306874]: 2024-06-27 21:55:26.042338879 +0000 UTC m=+0.395545298 container start 669dd2256905553296a4fa476b33ca86a86375bc8d06277da740b028eba41a22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-27T21:55:26.306 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:26 smithi151 podman[306874]: 2024-06-27 21:55:26.130355758 +0000 UTC m=+0.483562171 container attach 669dd2256905553296a4fa476b33ca86a86375bc8d06277da740b028eba41a22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD) 2024-06-27T21:55:26.306 INFO:journalctl@ceph.osd.0.smithi151.stdout:Jun 27 21:55:26 smithi151 podman[306874]: 2024-06-27 21:55:26.306492144 +0000 UTC m=+0.659698561 container died 669dd2256905553296a4fa476b33ca86a86375bc8d06277da740b028eba41a22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , ceph=True) 2024-06-27T21:55:26.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.0.service' 2024-06-27T21:55:26.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:26.967 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-27T21:55:26.967 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-27T21:55:26.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.1 2024-06-27T21:55:27.197 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:27 smithi151 systemd[1]: Stopping Ceph osd.1 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:27.697 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1[46181]: 2024-06-27T21:55:27.311+0000 7efefee3f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:27.697 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1[46181]: 2024-06-27T21:55:27.311+0000 7efefee3f640 -1 osd.1 1037 *** Got signal Terminated *** 2024-06-27T21:55:27.697 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:27 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1[46181]: 2024-06-27T21:55:27.311+0000 7efefee3f640 -1 osd.1 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:55:33.143 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:32 smithi151 podman[306981]: 2024-06-27 21:55:32.857105262 +0000 UTC m=+5.667015120 container died 5375e7898cd08f5eafdd033cfbec42388df9c005232fef866222a3d3e6c3e4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS) 2024-06-27T21:55:33.447 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:33 smithi151 podman[306981]: 2024-06-27 21:55:33.312095849 +0000 UTC m=+6.122005704 container cleanup 5375e7898cd08f5eafdd033cfbec42388df9c005232fef866222a3d3e6c3e4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-27T21:55:33.447 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:33 smithi151 bash[306981]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1 2024-06-27T21:55:33.926 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:33 smithi151 podman[307001]: 2024-06-27 21:55:33.596996129 +0000 UTC m=+0.741804430 container remove 5375e7898cd08f5eafdd033cfbec42388df9c005232fef866222a3d3e6c3e4d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-27T21:55:34.197 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:33 smithi151 podman[307089]: 2024-06-27 21:55:33.829835653 +0000 UTC m=+0.018005336 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:55:34.197 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:34 smithi151 podman[307089]: 2024-06-27 21:55:34.148197219 +0000 UTC m=+0.336366933 container create c6e22a219551e89099e6138009bbc679cddbccc82b4b42e891df1b044d9795ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, ceph=True) 2024-06-27T21:55:34.572 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:34 smithi151 podman[307089]: 2024-06-27 21:55:34.375676979 +0000 UTC m=+0.563846658 container init c6e22a219551e89099e6138009bbc679cddbccc82b4b42e891df1b044d9795ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:34.572 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:34 smithi151 podman[307089]: 2024-06-27 21:55:34.379042751 +0000 UTC m=+0.567212423 container start c6e22a219551e89099e6138009bbc679cddbccc82b4b42e891df1b044d9795ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240625, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:55:34.573 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:34 smithi151 podman[307089]: 2024-06-27 21:55:34.441936678 +0000 UTC m=+0.630106372 container attach c6e22a219551e89099e6138009bbc679cddbccc82b4b42e891df1b044d9795ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True) 2024-06-27T21:55:34.947 INFO:journalctl@ceph.osd.1.smithi151.stdout:Jun 27 21:55:34 smithi151 podman[307089]: 2024-06-27 21:55:34.573853802 +0000 UTC m=+0.762023477 container died c6e22a219551e89099e6138009bbc679cddbccc82b4b42e891df1b044d9795ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-27T21:55:35.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.1.service' 2024-06-27T21:55:35.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:35.347 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-27T21:55:35.347 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-27T21:55:35.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.2 2024-06-27T21:55:35.697 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:35 smithi151 systemd[1]: Stopping Ceph osd.2 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:36.083 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:35 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2[52157]: 2024-06-27T21:55:35.705+0000 7fdaf99cb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:36.083 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:35 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2[52157]: 2024-06-27T21:55:35.705+0000 7fdaf99cb640 -1 osd.2 1037 *** Got signal Terminated *** 2024-06-27T21:55:36.083 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:35 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2[52157]: 2024-06-27T21:55:35.705+0000 7fdaf99cb640 -1 osd.2 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:55:41.564 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:41 smithi151 podman[307193]: 2024-06-27 21:55:41.280935048 +0000 UTC m=+5.700459406 container died bb0fb2084b81a7bbf3d8feef8ea3736982c5f7cc6ffcdf9dad66e1e1cd603fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:41.906 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:41 smithi151 podman[307193]: 2024-06-27 21:55:41.748210675 +0000 UTC m=+6.167735075 container cleanup bb0fb2084b81a7bbf3d8feef8ea3736982c5f7cc6ffcdf9dad66e1e1cd603fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:55:41.907 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:41 smithi151 bash[307193]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2 2024-06-27T21:55:42.197 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:41 smithi151 podman[307213]: 2024-06-27 21:55:41.907759856 +0000 UTC m=+0.629803556 container remove bb0fb2084b81a7bbf3d8feef8ea3736982c5f7cc6ffcdf9dad66e1e1cd603fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:42.618 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:42 smithi151 podman[307300]: 2024-06-27 21:55:42.217689138 +0000 UTC m=+0.017128342 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:55:42.618 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:42 smithi151 podman[307300]: 2024-06-27 21:55:42.325185842 +0000 UTC m=+0.124625033 container create 2cf94359e1a6b112f2e33f0beec401094d7225d3cdcd45e8af6ea4fe9f0b40b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-27T21:55:42.618 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:42 smithi151 podman[307300]: 2024-06-27 21:55:42.544026623 +0000 UTC m=+0.343465817 container init 2cf94359e1a6b112f2e33f0beec401094d7225d3cdcd45e8af6ea4fe9f0b40b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-27T21:55:42.618 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:42 smithi151 podman[307300]: 2024-06-27 21:55:42.548050704 +0000 UTC m=+0.347489898 container start 2cf94359e1a6b112f2e33f0beec401094d7225d3cdcd45e8af6ea4fe9f0b40b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:42.618 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:42 smithi151 podman[307300]: 2024-06-27 21:55:42.618903832 +0000 UTC m=+0.418343032 container attach 2cf94359e1a6b112f2e33f0beec401094d7225d3cdcd45e8af6ea4fe9f0b40b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-27T21:55:42.947 INFO:journalctl@ceph.osd.2.smithi151.stdout:Jun 27 21:55:42 smithi151 podman[307300]: 2024-06-27 21:55:42.734925348 +0000 UTC m=+0.534364544 container died 2cf94359e1a6b112f2e33f0beec401094d7225d3cdcd45e8af6ea4fe9f0b40b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:55:43.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.2.service' 2024-06-27T21:55:43.547 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:43.548 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-27T21:55:43.548 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-27T21:55:43.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.3 2024-06-27T21:55:43.883 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:43 smithi151 systemd[1]: Stopping Ceph osd.3 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:44.197 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:43 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:55:43.883+0000 7f023d794640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:44.197 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:43 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:55:43.883+0000 7f023d794640 -1 osd.3 1037 *** Got signal Terminated *** 2024-06-27T21:55:44.197 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:43 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:55:43.883+0000 7f023d794640 -1 osd.3 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:55:45.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:45.261+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:46.013 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:45 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:45.715+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:46.447 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:46 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:55:46.132+0000 7f0235da2640 -1 osd.3 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.478446+0000 front 2024-06-27T21:55:22.478317+0000 (oldest deadline 2024-06-27T21:55:45.279172+0000) 2024-06-27T21:55:46.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:46.238+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:47.013 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:46 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:46.676+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:47.447 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:47 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:55:47.159+0000 7f0235da2640 -1 osd.3 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.478446+0000 front 2024-06-27T21:55:22.478317+0000 (oldest deadline 2024-06-27T21:55:45.279172+0000) 2024-06-27T21:55:47.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:47.267+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:47.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:47.366+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:47.714 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:47 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:47.461+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:48.013 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:47 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:47.714+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:48.447 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:48 smithi151 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3[58294]: 2024-06-27T21:55:48.130+0000 7f0235da2640 -1 osd.3 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.478446+0000 front 2024-06-27T21:55:22.478317+0000 (oldest deadline 2024-06-27T21:55:45.279172+0000) 2024-06-27T21:55:48.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:48 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:48.698+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:48.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:48 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:48.493+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:48.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:48.391+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:48.794 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:48.312+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:49.649 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:49 smithi151 podman[307402]: 2024-06-27 21:55:49.351098394 +0000 UTC m=+5.573810816 container died be152698fd994169fd1de130ac305ba45785eeb81cbac08728823fa8f44e2ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:55:49.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:49 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:49.675+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:49.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:49 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:49.445+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:49.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:49.412+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:49.794 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:49.338+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:49.947 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:49 smithi151 podman[307402]: 2024-06-27 21:55:49.80830264 +0000 UTC m=+6.031015120 container cleanup be152698fd994169fd1de130ac305ba45785eeb81cbac08728823fa8f44e2ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:55:49.947 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:49 smithi151 bash[307402]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3 2024-06-27T21:55:50.360 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:50.071+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:50.417 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:49 smithi151 podman[307430]: 2024-06-27 21:55:49.992690275 +0000 UTC m=+0.618526120 container remove be152698fd994169fd1de130ac305ba45785eeb81cbac08728823fa8f44e2ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD) 2024-06-27T21:55:50.669 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:50 smithi151 podman[307520]: 2024-06-27 21:55:50.418415664 +0000 UTC m=+0.113640868 container create ab013fa83ac30ce5bc8663311a95666eb483a6ff25d5da32f56694e84ed5e855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-06-27T21:55:50.669 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:50 smithi151 podman[307520]: 2024-06-27 21:55:50.32156851 +0000 UTC m=+0.016793690 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:55:50.669 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:50 smithi151 podman[307520]: 2024-06-27 21:55:50.670556744 +0000 UTC m=+0.365781919 container init ab013fa83ac30ce5bc8663311a95666eb483a6ff25d5da32f56694e84ed5e855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625) 2024-06-27T21:55:50.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:50 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:50.466+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:50.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:50 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:50.660+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:50.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:50.429+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:50.794 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:50.361+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:50.947 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:50 smithi151 podman[307520]: 2024-06-27 21:55:50.67394214 +0000 UTC m=+0.369167306 container start ab013fa83ac30ce5bc8663311a95666eb483a6ff25d5da32f56694e84ed5e855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3-deactivate, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-06-27T21:55:50.948 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:50 smithi151 podman[307520]: 2024-06-27 21:55:50.754097377 +0000 UTC m=+0.449322556 container attach ab013fa83ac30ce5bc8663311a95666eb483a6ff25d5da32f56694e84ed5e855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-27T21:55:50.948 INFO:journalctl@ceph.osd.3.smithi151.stdout:Jun 27 21:55:50 smithi151 podman[307520]: 2024-06-27 21:55:50.871529971 +0000 UTC m=+0.566755137 container died ab013fa83ac30ce5bc8663311a95666eb483a6ff25d5da32f56694e84ed5e855 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-3-deactivate, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:55:51.387 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:51.345+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:51.388 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:51.107+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:51.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.3.service' 2024-06-27T21:55:51.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:51.559 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-27T21:55:51.560 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-27T21:55:51.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4 2024-06-27T21:55:51.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:51 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:51.434+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:51.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:51 smithi190 systemd[1]: Stopping Ceph osd.4 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:55:51.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:51 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:51.619+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:51.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:51.388+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:52.392 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:52.363+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:52.393 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:52.136+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:52.422 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:52.129+0000 7f20af8af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:55:52.422 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:52.129+0000 7f20af8af640 -1 osd.4 1037 *** Got signal Terminated *** 2024-06-27T21:55:52.423 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:52.129+0000 7f20af8af640 -1 osd.4 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:55:52.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:52.612+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:52.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:52 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:52.423+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:52.793 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:52.393+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:53.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:53 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:53.181+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:53.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:53 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:53.418+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:53.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:53 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:53.351+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:53.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:53 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:53.402+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:53.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:53 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:53.611+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:54.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:54 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:54.391+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:54.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:54 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:54.328+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:54.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:54 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:54.184+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:54.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:54 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:54.184+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:55:54.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:54 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:54.415+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:54.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:54 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:54.593+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:54.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:55:54 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:55:54.416+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:55:55.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:55 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:55.162+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:55.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:55 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:55.162+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:55:55.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:55 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:55.316+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:55.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:55 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:55.378+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:55.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:55 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:55.378+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:55:55.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:55:55 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:55:55.407+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:55:55.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:55 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:55.398+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:55.763 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:55 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:55.577+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:55.764 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:55 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:55.577+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.767317+0000 front 2024-06-27T21:55:31.767269+0000 (oldest deadline 2024-06-27T21:55:55.267350+0000) 2024-06-27T21:55:56.178 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:55:55 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:55:55.909+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:55:56.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:56.178+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:56.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:56.178+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:55:56.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:56.369+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:56.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:56.369+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:55:56.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:56.357+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:56.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:56.357+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:55:56.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:55:56 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:55:56.410+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:55:56.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:56 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:56.446+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:56.764 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:56 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:56.583+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:20.665338+0000 front 2024-06-27T21:55:20.665397+0000 (oldest deadline 2024-06-27T21:55:44.765509+0000) 2024-06-27T21:55:56.764 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:56 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4[100254]: 2024-06-27T21:55:56.583+0000 7f20a7ebd640 -1 osd.4 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.767317+0000 front 2024-06-27T21:55:31.767269+0000 (oldest deadline 2024-06-27T21:55:55.267350+0000) 2024-06-27T21:55:57.211 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:55:56 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:55:56.937+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:55:57.463 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:55:57 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:55:57.389+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:55:57.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:57.398+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:57.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:57.398+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:55:57.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:57.211+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:57.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:57.211+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:55:57.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:57.352+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:57.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:57.352+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:55:57.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:55:57 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:55:57.537+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:55:57.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:57 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:57.463+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:57.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:57 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:57.463+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:55:57.764 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:57 smithi190 podman[198026]: 2024-06-27 21:55:57.545678142 +0000 UTC m=+5.780829212 container died 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=) 2024-06-27T21:55:58.205 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:55:57 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:55:57.956+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:55:58.238 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:57 smithi190 podman[198026]: 2024-06-27 21:55:57.946825756 +0000 UTC m=+6.181976826 container cleanup 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-27T21:55:58.238 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:57 smithi190 bash[198026]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4 2024-06-27T21:55:58.513 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:55:58 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:55:58.415+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:55:58.513 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:58 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:58.416+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:58.513 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:58 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:58.416+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:55:58.513 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:58 smithi190 podman[198041]: 2024-06-27 21:55:58.239679293 +0000 UTC m=+0.663871924 container remove 91fc9063268d435a8f923293566bd663fddcfc8dc5bb409f0f50ccfc318fd35a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-27T21:55:58.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:58.423+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:58.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:58.423+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:55:58.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:58.206+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:58.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:58.206+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:55:58.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:58.397+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:58.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:58.397+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:55:58.850 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:55:58 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:55:58.558+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:55:58.850 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:58 smithi190 podman[198131]: 2024-06-27 21:55:58.557372031 +0000 UTC m=+0.111674448 container create d4bb04b2b508678cf5c62edcc8a8b32680d4d2867ce0ce36caf93c60520f2d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-27T21:55:58.850 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:58 smithi190 podman[198131]: 2024-06-27 21:55:58.463009763 +0000 UTC m=+0.017312147 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:55:58.851 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:58 smithi190 podman[198131]: 2024-06-27 21:55:58.768364817 +0000 UTC m=+0.322667206 container init d4bb04b2b508678cf5c62edcc8a8b32680d4d2867ce0ce36caf93c60520f2d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-27T21:55:58.851 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:58 smithi190 podman[198131]: 2024-06-27 21:55:58.771684076 +0000 UTC m=+0.325986465 container start d4bb04b2b508678cf5c62edcc8a8b32680d4d2867ce0ce36caf93c60520f2d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True) 2024-06-27T21:55:58.851 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:58 smithi190 podman[198131]: 2024-06-27 21:55:58.851500854 +0000 UTC m=+0.405803240 container attach d4bb04b2b508678cf5c62edcc8a8b32680d4d2867ce0ce36caf93c60520f2d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-27T21:55:59.218 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:55:58 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:55:58.949+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:55:59.263 INFO:journalctl@ceph.osd.4.smithi190.stdout:Jun 27 21:55:59 smithi190 podman[198131]: 2024-06-27 21:55:59.010153815 +0000 UTC m=+0.564456194 container died d4bb04b2b508678cf5c62edcc8a8b32680d4d2867ce0ce36caf93c60520f2d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-27T21:55:59.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:59.218+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:55:59.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:55:59.218+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:55:59.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:59.419+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:55:59.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:55:59.419+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:55:59.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:59.382+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:55:59.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:55:59.382+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:55:59.553 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:55:59 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:55:59.444+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:55:59.553 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:59 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:59.394+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:55:59.553 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:55:59 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:55:59.394+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:55:59.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.4.service' 2024-06-27T21:55:59.693 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:55:59.693 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-27T21:55:59.693 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-27T21:55:59.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.5 2024-06-27T21:56:00.013 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:55:59 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:55:59.553+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:56:00.013 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:55:59 smithi190 systemd[1]: Stopping Ceph osd.5 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:00.250 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:55:59 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:55:59.921+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:00.385 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:00.074+0000 7fc1a66ae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:00.385 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:00.074+0000 7fc1a66ae640 -1 osd.5 1037 *** Got signal Terminated *** 2024-06-27T21:56:00.386 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:00.074+0000 7fc1a66ae640 -1 osd.5 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:00.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:00.251+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:00.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:00.251+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:00.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:00.417+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:00.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:00.417+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:00.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:00.417+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:00.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:00.417+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:00.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:00.574+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:56:00.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:00.469+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:00.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:00.385+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:00.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:00 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:00.385+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:01.224 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:00 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:00.944+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:01.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:01.225+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:01.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:01.225+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:01.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:01.436+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:01.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:01.436+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:01.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:01.386+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:01.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:01.386+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:01.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:01.565+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:56:01.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:01.444+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:01.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:01.385+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:01.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:01 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:01.385+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:02.201 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:01.928+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:02.201 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:01 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:01.928+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:02.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:02.202+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:02.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:02.202+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:02.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:02.411+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:02.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:02.411+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:02.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:02.408+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:02.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:02.408+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:02.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:02.408+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:02.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:02 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:02.600+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:56:02.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:02 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:02.426+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:02.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:02 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:02.378+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:02.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:02 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:02.378+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:03.241 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:02.885+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:03.241 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:02 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:02.885+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:03.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:03.459+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:03.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:03.459+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:03.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:03.242+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:03.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:03.242+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:03.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:03.242+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:03.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:03.440+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:03.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:03.440+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:03.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:03.440+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:03.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:03.582+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:56:03.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:03.582+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.885573+0000 front 2024-06-27T21:55:36.885519+0000 (oldest deadline 2024-06-27T21:56:02.785594+0000) 2024-06-27T21:56:03.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:03.436+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:03.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:03.412+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:03.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:03.412+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:03.764 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:03 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:03.412+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:04.234 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:03.898+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:04.234 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:03 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:03.898+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:04.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:04.466+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:04.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:04.466+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:04.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:04.466+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:04.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:04.234+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:04.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:04.234+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:04.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:04.234+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:04.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:04.438+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:04.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:04.438+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:04.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:04.438+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:04.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:04.548+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.585242+0000 front 2024-06-27T21:55:31.585276+0000 (oldest deadline 2024-06-27T21:55:56.885043+0000) 2024-06-27T21:56:04.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5[42471]: 2024-06-27T21:56:04.548+0000 7fc19ecbc640 -1 osd.5 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.885573+0000 front 2024-06-27T21:55:36.885519+0000 (oldest deadline 2024-06-27T21:56:02.785594+0000) 2024-06-27T21:56:04.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:04.438+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:04.764 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:04.438+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:04.764 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:04.445+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:04.764 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:04.446+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:04.764 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:04 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:04.446+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:05.245 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:04.929+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:05.246 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:04 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:04.929+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:05.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:05.486+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:05.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:05.486+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:05.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:05.486+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:05.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:05.399+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:05.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:05.399+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:05.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:05.399+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:05.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:05.247+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:05.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:05.247+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:05.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:05.247+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:05.706 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:05 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:05.472+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:05.706 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:05 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:05.472+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:05.706 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:05 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:05.397+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:05.707 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:05 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:05.397+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:05.707 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:05 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:05.397+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:05.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:05 smithi190 podman[198244]: 2024-06-27 21:56:05.706975435 +0000 UTC m=+5.752877035 container died 0b3199f038b8fac4e01ece4a17d8f58835996adc01632082498741b50ee0988e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-27T21:56:06.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 podman[198244]: 2024-06-27 21:56:06.075063529 +0000 UTC m=+6.120965133 container cleanup 0b3199f038b8fac4e01ece4a17d8f58835996adc01632082498741b50ee0988e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:06.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 bash[198244]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5 2024-06-27T21:56:06.272 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:05.925+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:06.272 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:05 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:05.925+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:06.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:06.519+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:06.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:06.519+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:06.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:06.519+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:06.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:06.367+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:06.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:06.367+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:06.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:06.367+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:06.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:06.273+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:06.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:06.273+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:06.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:06.273+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:06.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 podman[198259]: 2024-06-27 21:56:06.376620629 +0000 UTC m=+0.663490131 container remove 0b3199f038b8fac4e01ece4a17d8f58835996adc01632082498741b50ee0988e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:06.709 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:06 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:06.478+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:06.709 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:06 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:06.478+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:06.709 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:06 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:06.444+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:06.709 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:06 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:06.444+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:06.709 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:06 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:06.444+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:07.013 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 podman[198349]: 2024-06-27 21:56:06.612105046 +0000 UTC m=+0.017403557 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:07.013 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 podman[198349]: 2024-06-27 21:56:06.741088492 +0000 UTC m=+0.146386987 container create 990f8bbe1af20cdfa09c59c8fb6c475dc3deb071ebe59411496e6fd306ab3e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD) 2024-06-27T21:56:07.013 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 podman[198349]: 2024-06-27 21:56:06.955156396 +0000 UTC m=+0.360454908 container init 990f8bbe1af20cdfa09c59c8fb6c475dc3deb071ebe59411496e6fd306ab3e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-06-27T21:56:07.013 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:06 smithi190 podman[198349]: 2024-06-27 21:56:06.958484794 +0000 UTC m=+0.363783293 container start 990f8bbe1af20cdfa09c59c8fb6c475dc3deb071ebe59411496e6fd306ab3e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-27T21:56:07.228 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:06.915+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:07.228 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:06 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:06.915+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:07.432 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:07 smithi190 podman[198349]: 2024-06-27 21:56:07.049483679 +0000 UTC m=+0.454782175 container attach 990f8bbe1af20cdfa09c59c8fb6c475dc3deb071ebe59411496e6fd306ab3e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240625) 2024-06-27T21:56:07.432 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:07 smithi190 podman[198349]: 2024-06-27 21:56:07.152566914 +0000 UTC m=+0.557865416 container died 990f8bbe1af20cdfa09c59c8fb6c475dc3deb071ebe59411496e6fd306ab3e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-27T21:56:07.520 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:07.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:07.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:07.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:07.521 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:07.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:07.521 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:07.374+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:07.521 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:07.374+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:07.521 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:07.374+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:07.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.5.service' 2024-06-27T21:56:07.705 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:07 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:07.467+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:07.705 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:07 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:07.467+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:07 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:07.489+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:07 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:07.489+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:07 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:07.489+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:07 smithi190 podman[198372]: 2024-06-27 21:56:07.666986767 +0000 UTC m=+0.508748264 container remove 990f8bbe1af20cdfa09c59c8fb6c475dc3deb071ebe59411496e6fd306ab3e96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:07 smithi190 systemd[1]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.5.service: Deactivated successfully. 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:07 smithi190 systemd[1]: Stopped Ceph osd.5 for 784a4e4c-34c8-11ef-bca8-c7b262605968. 2024-06-27T21:56:07.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Jun 27 21:56:07 smithi190 systemd[1]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.5.service: Consumed 26min 9.044s CPU time. 2024-06-27T21:56:07.720 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:07.720 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-27T21:56:07.720 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-27T21:56:07.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.6 2024-06-27T21:56:07.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:07.521+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:07.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:07.521+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:07.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:07.521+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:08.013 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:07 smithi190 systemd[1]: Stopping Ceph osd.6 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:08.188 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:07.883+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:08.188 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:07 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:07.883+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:08.429 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:08.061+0000 7f7725500640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:08.429 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:08.061+0000 7f7725500640 -1 osd.6 1037 *** Got signal Terminated *** 2024-06-27T21:56:08.429 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:08.061+0000 7f7725500640 -1 osd.6 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:08.429 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:08.430+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:08.429 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:08.430+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:08.515 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:08.188+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:08.516 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:08.188+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:08.516 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:08.188+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:08.516 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:08.382+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:08.516 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:08.382+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:08.516 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:08.382+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:08.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:08.500+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:08.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:08.500+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:08.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:08 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:08.500+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:08.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:08.515+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:08.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:08.515+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:08.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:08.515+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:09.182 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:08.900+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:09.182 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:08 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:08.900+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:09.507 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:09.182+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:09.507 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:09.182+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:09.507 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:09.182+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:09.508 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:09.399+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:09.508 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:09.399+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:09.508 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:09.399+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:09.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:09 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:09.462+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:09.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:09 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:09.462+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:09.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:09 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:09.520+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:09.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:09 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:09.520+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:09.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:09 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:09.520+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:09.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:09.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:09.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:09.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:09.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:09.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:10.213 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:09.864+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:10.214 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:09 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:09.864+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:10.472 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:10.214+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:10.472 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:10.214+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:10.472 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:10.214+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:10.472 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:10.436+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:10.472 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:10.436+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:10.472 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:10.436+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:10.425+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:10.425+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:10.425+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:44.502613+0000 front 2024-06-27T21:55:44.502645+0000 (oldest deadline 2024-06-27T21:56:09.802582+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:10.530+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:10.530+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:10.530+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:10.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:10 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:10.530+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:10.793 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:10.472+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:10.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:10.472+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:10.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:10.472+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:11.235 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:10.815+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:11.235 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:10 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:10.815+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:11.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:11.429+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:11.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:11.429+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:11.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:11.429+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:11.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:11.235+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:11.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:11.235+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:11.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:11.235+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:11.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:11.474+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:11.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:11.474+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:11.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:11.474+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:11.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:11.474+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:11.469+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:11.469+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:11.469+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:44.502613+0000 front 2024-06-27T21:55:44.502645+0000 (oldest deadline 2024-06-27T21:56:09.802582+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:11.564+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:11.564+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:11.564+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:11.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:11 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:11.564+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:12.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:11.790+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:12.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:11 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:11.790+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:12.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:12.424+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:12.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:12.424+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:12.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:12.424+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:12.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:12.424+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:12.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:12.382+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:12.382+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:12.382+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:12.382+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:12.246+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:12.246+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:12.246+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:12.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:12.246+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:12.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:12.497+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:28.199586+0000 front 2024-06-27T21:55:28.199739+0000 (oldest deadline 2024-06-27T21:55:54.099659+0000) 2024-06-27T21:56:12.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:12.497+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.501495+0000 front 2024-06-27T21:55:40.501580+0000 (oldest deadline 2024-06-27T21:56:04.001517+0000) 2024-06-27T21:56:12.763 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6[48405]: 2024-06-27T21:56:12.497+0000 7f7721b16640 -1 osd.6 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:44.502613+0000 front 2024-06-27T21:55:44.502645+0000 (oldest deadline 2024-06-27T21:56:09.802582+0000) 2024-06-27T21:56:12.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:12.581+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:12.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:12.581+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:12.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:12.581+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:12.764 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:12 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:12.581+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:13.288 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:12.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:13.288 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:12 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:12.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:13.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:13.471+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:13.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:13.471+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:13.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:13.471+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:13.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:13.471+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:13.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:13.288+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:13.288+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:13.288+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:13.288+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:13.370+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:13.370+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:13.370+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:13.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:13.370+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:13.894 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:13 smithi190 podman[198452]: 2024-06-27 21:56:13.628813753 +0000 UTC m=+5.684075047 container died db14eaa6771ceac79a2721f85e4785c52bf3b45fd30d977e5dd979507eae1427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-27T21:56:13.895 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:13 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:13.608+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:13.895 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:13 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:13.608+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:13.895 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:13 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:13.608+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:13.895 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:13 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:13.608+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:14.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:13.790+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:14.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:13.790+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:14.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:13 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:13.790+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:14.263 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:13 smithi190 podman[198452]: 2024-06-27 21:56:13.978074039 +0000 UTC m=+6.033335333 container cleanup db14eaa6771ceac79a2721f85e4785c52bf3b45fd30d977e5dd979507eae1427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:56:14.263 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:13 smithi190 bash[198452]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6 2024-06-27T21:56:14.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:14.494+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:14.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:14.494+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:14.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:14.494+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:14.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:14.494+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:14.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:14.260+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:14.260+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:14.260+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:14.260+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:14.403+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:14.403+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:14.403+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:14.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:14.403+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:14.562 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198466]: 2024-06-27 21:56:14.27073664 +0000 UTC m=+0.640829687 container remove db14eaa6771ceac79a2721f85e4785c52bf3b45fd30d977e5dd979507eae1427 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-27T21:56:14.834 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:14 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:14.598+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:14.834 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:14 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:14.598+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:14.834 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:14 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:14.598+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:14.834 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:14 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:14.598+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:14.835 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198555]: 2024-06-27 21:56:14.56337434 +0000 UTC m=+0.092451201 container create 162878e54c6c5d0c5b68048f94bcaf78e484e9fc29a880580cb45992b808d9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-27T21:56:14.835 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198555]: 2024-06-27 21:56:14.488041102 +0000 UTC m=+0.017117932 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:14.835 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198555]: 2024-06-27 21:56:14.765743127 +0000 UTC m=+0.294819956 container init 162878e54c6c5d0c5b68048f94bcaf78e484e9fc29a880580cb45992b808d9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6-deactivate, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:56:14.835 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198555]: 2024-06-27 21:56:14.769189749 +0000 UTC m=+0.298266577 container start 162878e54c6c5d0c5b68048f94bcaf78e484e9fc29a880580cb45992b808d9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:14.835 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198555]: 2024-06-27 21:56:14.832464092 +0000 UTC m=+0.361540912 container attach 162878e54c6c5d0c5b68048f94bcaf78e484e9fc29a880580cb45992b808d9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:15.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:14.775+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:15.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:14.775+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:15.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:14 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:14.775+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:15.250 INFO:journalctl@ceph.osd.6.smithi190.stdout:Jun 27 21:56:14 smithi190 podman[198555]: 2024-06-27 21:56:14.956407411 +0000 UTC m=+0.485484230 container died 162878e54c6c5d0c5b68048f94bcaf78e484e9fc29a880580cb45992b808d9c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, ceph=True) 2024-06-27T21:56:15.446 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:15.274+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:15.446 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:15.274+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:15.446 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:15.274+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:15.446 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:15.274+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:15.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.6.service' 2024-06-27T21:56:15.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:15.581 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-27T21:56:15.581 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-27T21:56:15.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.7 2024-06-27T21:56:15.750 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:15.502+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:15.750 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:15.502+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:15.750 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:15.502+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:15.751 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:15.502+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:15.751 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:15.446+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:15.751 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:15.446+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:15.751 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:15.446+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:15.751 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:15.446+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:15.922 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.636+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:15.922 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.636+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:15.922 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.636+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:15.922 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.636+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:15.922 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 systemd[1]: Stopping Ceph osd.7 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:15.923 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.921+0000 7f5cc1f2a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:15.923 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.921+0000 7f5cc1f2a640 -1 osd.7 1037 *** Got signal Terminated *** 2024-06-27T21:56:15.923 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:15 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:15.921+0000 7f5cc1f2a640 -1 osd.7 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:16.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:15.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:16.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:15.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:16.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:15 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:15.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:16.487+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:16.487+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:16.487+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:16.487+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:16.311+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:16.311+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:16.311+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:16.741 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:16.311+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:16.742 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:16.449+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:16.742 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:16.449+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:16.742 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:16.449+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:16.742 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:16.449+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:16.898 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:16 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:16.615+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:16.898 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:16 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:16.615+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:16.898 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:16 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:16.615+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:16.898 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:16 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:16.615+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:17.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:16.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:17.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:16.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:17.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:16 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:16.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:17.703 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:17.458+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:17.703 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:17.458+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:17.703 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:17.458+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:17.703 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:17.458+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:17.703 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:17.296+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:17.296+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:17.296+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:17.296+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:17.467+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:17.467+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:17.467+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:17.704 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:17.467+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:18.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:17 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:17.587+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:18.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:17 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:17.587+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:18.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:17 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:17.587+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:18.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:17 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:17.587+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:18.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:17.703+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:18.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:17.703+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:18.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:17 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:17.703+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:18.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:18.493+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:18.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:18.493+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:18.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:18.493+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:18.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:18.493+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:18.265+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:18.265+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:18.265+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:18.265+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:18.458+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:18.458+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:18.458+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:18.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:18.458+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:19.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:18 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:18.542+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:19.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:18 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:18.542+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:19.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:18 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:18.542+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:19.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:18 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:18.542+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:19.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:18 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:18.542+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.864080+0000 front 2024-06-27T21:55:53.863937+0000 (oldest deadline 2024-06-27T21:56:17.864668+0000) 2024-06-27T21:56:19.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:18.673+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:19.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:18.673+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:19.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:18 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:18.673+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:19.533 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:19.271+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:19.533 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:19.271+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:19.271+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:19.271+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:19.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:19.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:19.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:19.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:19.534 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:19.507+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:19.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:19.638+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:19.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:19.638+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:19.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:19.638+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:19.795 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:19.533+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:19.795 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:19.533+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:19.795 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:19.533+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:19.795 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:19 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:19.533+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:20.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:19 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:19.584+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:20.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:19 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:19.584+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:20.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:19 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:19.584+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:20.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:19 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:19.584+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:20.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:19 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:19.584+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.864080+0000 front 2024-06-27T21:55:53.863937+0000 (oldest deadline 2024-06-27T21:56:17.864668+0000) 2024-06-27T21:56:20.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:20.290+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:20.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:20.290+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:20.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:20.290+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:20.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:20.290+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:20.544 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:20.489+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:20.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:20.489+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:20.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:20.489+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:20.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:20.489+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:20.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:20.489+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:21.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:20 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:20.567+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:23.559442+0000 front 2024-06-27T21:55:23.559476+0000 (oldest deadline 2024-06-27T21:55:46.959929+0000) 2024-06-27T21:56:21.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:20 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:20.567+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.260915+0000 front 2024-06-27T21:55:32.261007+0000 (oldest deadline 2024-06-27T21:55:56.961108+0000) 2024-06-27T21:56:21.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:20 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:20.567+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:36.961639+0000 front 2024-06-27T21:55:36.961817+0000 (oldest deadline 2024-06-27T21:56:02.861526+0000) 2024-06-27T21:56:21.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:20 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:20.567+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863134+0000 front 2024-06-27T21:55:46.863158+0000 (oldest deadline 2024-06-27T21:56:10.362976+0000) 2024-06-27T21:56:21.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:20 smithi190 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7[54065]: 2024-06-27T21:56:20.567+0000 7f5cba538640 -1 osd.7 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.864080+0000 front 2024-06-27T21:55:53.863937+0000 (oldest deadline 2024-06-27T21:56:17.864668+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:20.647+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:20.647+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:20.647+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:20.547+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:20.547+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:20.547+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:21.044 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:20 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:20.547+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:21.671 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:21.504+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:21.504+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:21.504+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:21.504+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:21.530+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:21.530+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:21.530+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:21.672 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:21.530+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:21.673 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:21.530+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:21.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:21.325+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:21.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:21.325+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:21.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:21.325+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:21.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:21.325+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:21.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:21.325+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:21.763 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:21 smithi190 podman[198658]: 2024-06-27 21:56:21.481530564 +0000 UTC m=+5.663361665 container died 6987499b9c987736bc349c138ad0dfbbbc9ac917bee17034213926ba6cd5db68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:22.038 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:21 smithi190 podman[198658]: 2024-06-27 21:56:21.839013681 +0000 UTC m=+6.020844827 container cleanup 6987499b9c987736bc349c138ad0dfbbbc9ac917bee17034213926ba6cd5db68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7, ceph=True, org.label-schema.build-date=20240625, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:22.038 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:21 smithi190 bash[198658]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7 2024-06-27T21:56:22.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:21.672+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:22.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:21.672+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:22.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:21 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:21.672+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:22.443 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198685]: 2024-06-27 21:56:22.039740317 +0000 UTC m=+0.533523412 container remove 6987499b9c987736bc349c138ad0dfbbbc9ac917bee17034213926ba6cd5db68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:22.700 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198776]: 2024-06-27 21:56:22.346853266 +0000 UTC m=+0.017346239 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:22.700 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198776]: 2024-06-27 21:56:22.4489396 +0000 UTC m=+0.119432558 container create 4aa97303e345e7bac418c38bd77e35ac5766a881d0a3ff4e89afd3da9d4c74e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-27T21:56:22.700 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198776]: 2024-06-27 21:56:22.701462009 +0000 UTC m=+0.371954981 container init 4aa97303e345e7bac418c38bd77e35ac5766a881d0a3ff4e89afd3da9d4c74e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:22.717 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:22.295+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:22.717 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:22.295+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:22.717 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:22.295+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:22.295+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:22.295+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:22.536+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:22.536+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:22.536+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:22.536+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:22.536+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:22.718 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:22.495+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:22.719 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:22.495+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:22.719 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:22.495+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:22.719 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:22.495+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:22.719 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:22.495+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:23.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198776]: 2024-06-27 21:56:22.70481311 +0000 UTC m=+0.375306070 container start 4aa97303e345e7bac418c38bd77e35ac5766a881d0a3ff4e89afd3da9d4c74e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-06-27T21:56:23.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198776]: 2024-06-27 21:56:22.776307829 +0000 UTC m=+0.446800806 container attach 4aa97303e345e7bac418c38bd77e35ac5766a881d0a3ff4e89afd3da9d4c74e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:23.013 INFO:journalctl@ceph.osd.7.smithi190.stdout:Jun 27 21:56:22 smithi190 podman[198776]: 2024-06-27 21:56:22.895950765 +0000 UTC m=+0.566443729 container died 4aa97303e345e7bac418c38bd77e35ac5766a881d0a3ff4e89afd3da9d4c74e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-06-27T21:56:23.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:22.718+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:23.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:22.718+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:23.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:22 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:22.718+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:23.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.7.service' 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:23.519+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:23.519+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:23.519+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:23.519+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:23.519+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:23.286+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:23.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:23.286+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:23.286+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:23.286+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:23.286+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.501+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.501+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.501+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.501+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:23.545 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.501+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:23.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:23.571 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-27T21:56:23.571 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-06-27T21:56:23.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.8 2024-06-27T21:56:23.892 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 systemd[1]: Stopping Ceph osd.8 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:23.893 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:23.747+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:23.893 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:23.747+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:23.893 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:23.747+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:23.893 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:23.747+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:24.283 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.893+0000 7fa938db0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:24.284 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.893+0000 7fa938db0640 -1 osd.8 1037 *** Got signal Terminated *** 2024-06-27T21:56:24.284 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:23 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:23.893+0000 7fa938db0640 -1 osd.8 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:24.488+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:24.488+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:24.488+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:24.488+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:24.488+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:24.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:24.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:24.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:24.536 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:24.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:24.537 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:24.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:24.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:24.761+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:24.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:24.761+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:24.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:24.761+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:24.794 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:24.761+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:24.795 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:24.535+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:24.795 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:24.535+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:24.795 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:24.535+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:24.795 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:24.535+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:24.795 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:24 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:24.535+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:25.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:25.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:25.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:25.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:25.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:25.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:25.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:25.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:25.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:25.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:25.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:25.525+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:25.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:25.525+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:25.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:25.525+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:25.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:25.525+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:25.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:25.525+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:26.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:25.729+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:26.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:25.729+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:26.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:25.729+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:26.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:25.729+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:26.044 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:25.571+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:26.044 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:25.571+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:26.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:25.571+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:26.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:25.571+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:26.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:25 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:25.571+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:26.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:26.289+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:26.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:26.289+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:26.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:26.289+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:26.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:26.289+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:26.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:26.289+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:27.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:26.739+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:27.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:26.739+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:27.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:26.739+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:27.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:26.739+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:27.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:26.739+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:26.550+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:26.550+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:26.550+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:26.550+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:26.550+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:26.584+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:26.584+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:27.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:26.584+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:27.046 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:26.584+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:27.046 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:26.584+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:27.046 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:26 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:26.584+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:01.152376+0000 front 2024-06-27T21:56:01.152469+0000 (oldest deadline 2024-06-27T21:56:25.852329+0000) 2024-06-27T21:56:27.619 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:27.528+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:27.619 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:27.528+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:27.619 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:27.528+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:27.619 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:27.528+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:27.528+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:27.328+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:27.328+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:27.328+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:27.328+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:27.328+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:27.620 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:27.328+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:28.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:27.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:28.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:27.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:28.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:27.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:28.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:27.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:28.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:27.750+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:28.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:27.619+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:28.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:27.619+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:28.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:27.619+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:28.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:27.619+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:28.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:27.619+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:28.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:27 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:27.619+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:01.152376+0000 front 2024-06-27T21:56:01.152469+0000 (oldest deadline 2024-06-27T21:56:25.852329+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:28.479+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:28.479+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:28.479+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:28.479+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:28.479+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:28.343+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:28.600 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:28.343+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:28.601 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:28.343+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:28.601 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:28.343+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:28.601 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:28.343+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:28.601 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:28.343+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:29.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:28.792+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:29.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:28.792+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:29.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:28.792+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:29.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:28.792+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:29.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:28.792+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:29.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:28.600+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.445618+0000 front 2024-06-27T21:55:22.445754+0000 (oldest deadline 2024-06-27T21:55:46.545659+0000) 2024-06-27T21:56:29.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:28.600+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.447579+0000 front 2024-06-27T21:55:30.447777+0000 (oldest deadline 2024-06-27T21:55:55.147561+0000) 2024-06-27T21:56:29.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:28.600+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.349249+0000 front 2024-06-27T21:55:40.349281+0000 (oldest deadline 2024-06-27T21:56:03.849025+0000) 2024-06-27T21:56:29.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:28.600+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.349786+0000 front 2024-06-27T21:55:47.349807+0000 (oldest deadline 2024-06-27T21:56:11.950211+0000) 2024-06-27T21:56:29.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:28.600+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:53.551533+0000 front 2024-06-27T21:55:53.551570+0000 (oldest deadline 2024-06-27T21:56:18.851367+0000) 2024-06-27T21:56:29.045 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:28 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8[37739]: 2024-06-27T21:56:28.600+0000 7fa9353c6640 -1 osd.8 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:01.152376+0000 front 2024-06-27T21:56:01.152469+0000 (oldest deadline 2024-06-27T21:56:25.852329+0000) 2024-06-27T21:56:29.779 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:29 smithi192 podman[194804]: 2024-06-27 21:56:29.515224538 +0000 UTC m=+5.737929521 container died 9c309558dfbaa65ae65b5af7f2d62158c6ad4de05528f0c726f76ce083f85407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD) 2024-06-27T21:56:29.779 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:29.302+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:29.302+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:29.302+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:29.302+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:29.302+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:29.302+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:29.432+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:29.432+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:29.432+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:29.432+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:29.780 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:29.432+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:30.044 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:29 smithi192 podman[194804]: 2024-06-27 21:56:29.893321374 +0000 UTC m=+6.116026356 container cleanup 9c309558dfbaa65ae65b5af7f2d62158c6ad4de05528f0c726f76ce083f85407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2) 2024-06-27T21:56:30.044 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:29 smithi192 bash[194804]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8 2024-06-27T21:56:30.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:29.816+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:30.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:29.816+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:30.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:29.816+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:30.045 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:29.816+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:30.045 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:29 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:29.816+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:30.485 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194820]: 2024-06-27 21:56:30.169679921 +0000 UTC m=+0.655983805 container remove 9c309558dfbaa65ae65b5af7f2d62158c6ad4de05528f0c726f76ce083f85407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-06-27T21:56:30.486 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:30.308+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:30.486 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:30.308+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:30.486 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:30.308+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:30.486 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:30.308+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:30.486 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:30.308+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:30.486 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:30.308+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:30.487 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:30.409+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:30.487 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:30.409+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:30.487 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:30.409+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:30.487 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:30.409+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:30.487 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:30.409+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:30.794 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194911]: 2024-06-27 21:56:30.389439539 +0000 UTC m=+0.017284077 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:30.798 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194911]: 2024-06-27 21:56:30.514776011 +0000 UTC m=+0.142620571 container create a9c0d0b58cbe196309265c4d67ac7645c90f8482fdcbb69b4bcce43f4d2bf5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux ) 2024-06-27T21:56:30.798 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194911]: 2024-06-27 21:56:30.731663377 +0000 UTC m=+0.359507899 container init a9c0d0b58cbe196309265c4d67ac7645c90f8482fdcbb69b4bcce43f4d2bf5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240625, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-27T21:56:30.798 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194911]: 2024-06-27 21:56:30.734928301 +0000 UTC m=+0.362772823 container start a9c0d0b58cbe196309265c4d67ac7645c90f8482fdcbb69b4bcce43f4d2bf5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8-deactivate, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194911]: 2024-06-27 21:56:30.814809854 +0000 UTC m=+0.442654392 container attach a9c0d0b58cbe196309265c4d67ac7645c90f8482fdcbb69b4bcce43f4d2bf5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.8.smithi192.stdout:Jun 27 21:56:30 smithi192 podman[194911]: 2024-06-27 21:56:30.976793579 +0000 UTC m=+0.604638100 container died a9c0d0b58cbe196309265c4d67ac7645c90f8482fdcbb69b4bcce43f4d2bf5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-8-deactivate, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_CLEAN=True) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:30.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:30.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:30.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:30.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:31.258 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:30 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:30.813+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:31.517 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:31.361+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:31.517 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:31.361+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:31.361+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:31.361+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:31.361+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:31.301+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:31.301+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:31.301+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:31.301+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:31.301+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:31.518 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:31.301+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:31.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.8.service' 2024-06-27T21:56:31.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:31.583 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-06-27T21:56:31.584 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-06-27T21:56:31.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.9 2024-06-27T21:56:31.791 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 systemd[1]: Stopping Ceph osd.9 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:32.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.801+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:32.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.801+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:32.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.801+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:32.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.801+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:32.045 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.801+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:32.045 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.938+0000 7f68958ef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:32.045 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.938+0000 7f68958ef640 -1 osd.9 1037 *** Got signal Terminated *** 2024-06-27T21:56:32.045 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:31 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:31.938+0000 7f68958ef640 -1 osd.9 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:32.713 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:32.324+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:32.324+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:32.324+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:32.324+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:32.324+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:32.331+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:32.331+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:32.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:32.331+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:32.715 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:32.331+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:32.715 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:32.331+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:32.715 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:32.331+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:33.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:32.760+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:33.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:32.760+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:33.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:32.760+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:33.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:32.760+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:33.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:32 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:32.760+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:33.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:33.336+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:33.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:33.336+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:33.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:33.336+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:33.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:33.336+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:33.336+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:33.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:33.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:33.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:33.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:33.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:33.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:33.284+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:34.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:33.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:34.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:33.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:34.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:33.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:34.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:33.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:34.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:33 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:33.741+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:34.715 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:34.301+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:34.715 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:34.301+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:34.301+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:34.301+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:34.301+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:34.323+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:34.323+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:34.323+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:34.323+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:34.323+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:34.716 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:34.323+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:35.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:34.715+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:35.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:34.715+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:35.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:34.715+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:35.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:34.715+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:35.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:34 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:34.715+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:35.274+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:35.274+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:35.274+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:35.274+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:35.274+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:35.274+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:35.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:35.357+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:35.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:35.357+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:35.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:35.357+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:35.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:35.357+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:35.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:35.357+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:35.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:35.357+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:36.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:35.668+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:36.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:35.668+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:36.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:35.668+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:36.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:35.668+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:36.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:35 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:35.668+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:36.236+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:36.236+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:36.236+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:36.236+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:36.236+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:36.236+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:36.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:36.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:36.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:36.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:36.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:36.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:36.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:36.333+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:37.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:36.653+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:31.336304+0000 front 2024-06-27T21:55:31.336336+0000 (oldest deadline 2024-06-27T21:55:55.436110+0000) 2024-06-27T21:56:37.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:36.653+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:40.637442+0000 front 2024-06-27T21:55:40.637506+0000 (oldest deadline 2024-06-27T21:56:01.737532+0000) 2024-06-27T21:56:37.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:36.653+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:48.739061+0000 front 2024-06-27T21:55:48.739142+0000 (oldest deadline 2024-06-27T21:56:13.438973+0000) 2024-06-27T21:56:37.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:36.653+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.940052+0000 front 2024-06-27T21:55:56.939920+0000 (oldest deadline 2024-06-27T21:56:22.740343+0000) 2024-06-27T21:56:37.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:36 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9[43577]: 2024-06-27T21:56:36.653+0000 7f6891f05640 -1 osd.9 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:04.441458+0000 front 2024-06-27T21:56:04.441415+0000 (oldest deadline 2024-06-27T21:56:26.141395+0000) 2024-06-27T21:56:37.544 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:37 smithi192 podman[195015]: 2024-06-27 21:56:37.399297439 +0000 UTC m=+5.581564729 container died c0147cb5c5df5c768e3bc5db33057ee876a79217a0110e986448de2356c908be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:37.206+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:37.206+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:37.206+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:37.206+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:37.206+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:37.206+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:37.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:37.546 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:37.546 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:37.546 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:37.546 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:37.546 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:37.546 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:37 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:37.361+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:38.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:37 smithi192 podman[195015]: 2024-06-27 21:56:37.755995425 +0000 UTC m=+5.938262709 container cleanup c0147cb5c5df5c768e3bc5db33057ee876a79217a0110e986448de2356c908be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240625, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:56:38.044 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:37 smithi192 bash[195015]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9 2024-06-27T21:56:38.369 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195041]: 2024-06-27 21:56:38.04753651 +0000 UTC m=+0.643671231 container remove c0147cb5c5df5c768e3bc5db33057ee876a79217a0110e986448de2356c908be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-27T21:56:38.369 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:38.211+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:38.369 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:38.211+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:38.369 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:38.211+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:38.369 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:38.211+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:38.370 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:38.211+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:38.370 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:38.211+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:38.691 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:38 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:38.378+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:38.692 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195132]: 2024-06-27 21:56:38.273005238 +0000 UTC m=+0.017186307 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:38.692 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195132]: 2024-06-27 21:56:38.373337732 +0000 UTC m=+0.117518799 container create ed6a7aad457734d748e2bafcfbfa8ad9f61f6177038c9af9303db45b00e95b20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-27T21:56:38.692 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195132]: 2024-06-27 21:56:38.609160251 +0000 UTC m=+0.353341323 container init ed6a7aad457734d748e2bafcfbfa8ad9f61f6177038c9af9303db45b00e95b20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9-deactivate, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-27T21:56:38.692 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195132]: 2024-06-27 21:56:38.612655067 +0000 UTC m=+0.356836136 container start ed6a7aad457734d748e2bafcfbfa8ad9f61f6177038c9af9303db45b00e95b20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, ceph=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-06-27T21:56:38.692 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195132]: 2024-06-27 21:56:38.692337915 +0000 UTC m=+0.436519001 container attach ed6a7aad457734d748e2bafcfbfa8ad9f61f6177038c9af9303db45b00e95b20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:39.043 INFO:journalctl@ceph.osd.9.smithi192.stdout:Jun 27 21:56:38 smithi192 podman[195132]: 2024-06-27 21:56:38.807994141 +0000 UTC m=+0.552175218 container died ed6a7aad457734d748e2bafcfbfa8ad9f61f6177038c9af9303db45b00e95b20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-27T21:56:39.370 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.204+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:39.371 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.204+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:39.371 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.204+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:39.371 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.204+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:39.371 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.204+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:39.371 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.204+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:39.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.9.service' 2024-06-27T21:56:39.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:39.482 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-06-27T21:56:39.482 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-06-27T21:56:39.482 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.10 2024-06-27T21:56:39.794 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 systemd[1]: Stopping Ceph osd.10 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:39.794 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:39.794 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:39.794 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:39.795 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:39.795 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:39.795 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:39.795 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:39.370+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:40.195 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.818+0000 7fe0d2f35640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:40.195 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.818+0000 7fe0d2f35640 -1 osd.10 1037 *** Got signal Terminated *** 2024-06-27T21:56:40.195 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:39 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:39.818+0000 7fe0d2f35640 -1 osd.10 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:40.195 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:40.196+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:40.195 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:40.196+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:40.195 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:40.196+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:40.196 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:40.196+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:40.196 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:40.196+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:40.196 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:40.196+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:40.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:40 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:40.375+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:41.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:41.147+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:41.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:41.147+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:41.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:41.147+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:41.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:41.147+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:41.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:41.147+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:41.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:41.147+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:41.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:41 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:41.359+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:42.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:42.342+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:42.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:42 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:42.143+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:19.568361+0000 front 2024-06-27T21:56:19.568324+0000 (oldest deadline 2024-06-27T21:56:41.868162+0000) 2024-06-27T21:56:43.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:43.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:43.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:43.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:43.113+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:19.568361+0000 front 2024-06-27T21:56:19.568324+0000 (oldest deadline 2024-06-27T21:56:41.868162+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:43.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:43 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:43.299+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:22.761250+0000 front 2024-06-27T21:55:22.761120+0000 (oldest deadline 2024-06-27T21:55:45.061148+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:30.362113+0000 front 2024-06-27T21:55:30.362175+0000 (oldest deadline 2024-06-27T21:55:55.662181+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:35.662918+0000 front 2024-06-27T21:55:35.663017+0000 (oldest deadline 2024-06-27T21:56:01.562734+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:46.863621+0000 front 2024-06-27T21:55:46.863591+0000 (oldest deadline 2024-06-27T21:56:10.963628+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:56.265053+0000 front 2024-06-27T21:55:56.265030+0000 (oldest deadline 2024-06-27T21:56:22.164952+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:12.067014+0000 front 2024-06-27T21:56:12.066996+0000 (oldest deadline 2024-06-27T21:56:34.966847+0000) 2024-06-27T21:56:44.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10[49134]: 2024-06-27T21:56:44.109+0000 7fe0cb543640 -1 osd.10 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:19.568361+0000 front 2024-06-27T21:56:19.568324+0000 (oldest deadline 2024-06-27T21:56:41.868162+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:44.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:44 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:44.297+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:45.544 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:45 smithi192 podman[195235]: 2024-06-27 21:56:45.450263154 +0000 UTC m=+5.747349561 container died d7bfbab485d0368c1c29bd7ff94c52ab93e75c2bf013049cf3485fc52a955746 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-27T21:56:45.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:45.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:45.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:45.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:45.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:45.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:45.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:45.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:45 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:45.275+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:46.043 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:45 smithi192 podman[195235]: 2024-06-27 21:56:45.798489691 +0000 UTC m=+6.095576099 container cleanup d7bfbab485d0368c1c29bd7ff94c52ab93e75c2bf013049cf3485fc52a955746 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-06-27T21:56:46.044 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:45 smithi192 bash[195235]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10 2024-06-27T21:56:46.424 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195251]: 2024-06-27 21:56:46.074838148 +0000 UTC m=+0.620947374 container remove d7bfbab485d0368c1c29bd7ff94c52ab93e75c2bf013049cf3485fc52a955746 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:46.424 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:46.424 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:46.425 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:46.425 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:46.425 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:46.425 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:46.425 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:46.425 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:46 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:46.241+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:46.693 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195349]: 2024-06-27 21:56:46.425509252 +0000 UTC m=+0.115892479 container create 3046d28c77fe60bbac29f9b5a08cf65e2d6e2c250dbb13104762a06d71457d74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0) 2024-06-27T21:56:46.693 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195349]: 2024-06-27 21:56:46.328434124 +0000 UTC m=+0.018817351 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:46.693 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195349]: 2024-06-27 21:56:46.636073145 +0000 UTC m=+0.326456379 container init 3046d28c77fe60bbac29f9b5a08cf65e2d6e2c250dbb13104762a06d71457d74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:46.693 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195349]: 2024-06-27 21:56:46.639555262 +0000 UTC m=+0.329938492 container start 3046d28c77fe60bbac29f9b5a08cf65e2d6e2c250dbb13104762a06d71457d74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.build-date=20240625, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0) 2024-06-27T21:56:46.693 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195349]: 2024-06-27 21:56:46.694499513 +0000 UTC m=+0.384882741 container attach 3046d28c77fe60bbac29f9b5a08cf65e2d6e2c250dbb13104762a06d71457d74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10-deactivate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, ceph=True) 2024-06-27T21:56:47.044 INFO:journalctl@ceph.osd.10.smithi192.stdout:Jun 27 21:56:46 smithi192 podman[195349]: 2024-06-27 21:56:46.838395294 +0000 UTC m=+0.528778536 container died 3046d28c77fe60bbac29f9b5a08cf65e2d6e2c250dbb13104762a06d71457d74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240625, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:47.414 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.207+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:47.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.10.service' 2024-06-27T21:56:47.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:47.483 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-06-27T21:56:47.484 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-06-27T21:56:47.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.11 2024-06-27T21:56:47.793 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 systemd[1]: Stopping Ceph osd.11 for 784a4e4c-34c8-11ef-bca8-c7b262605968... 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.834+0000 7fac86283640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.834+0000 7fac86283640 -1 osd.11 1037 *** Got signal Terminated *** 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:47 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:47.834+0000 7fac86283640 -1 osd.11 1037 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:48.229 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:48.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:48.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:48.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:48 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:48.229+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:49.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:49 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:49.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:50.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:50.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:50.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:50 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:50.195+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.192:6804 osd.8 since back 2024-06-27T21:56:23.643806+0000 front 2024-06-27T21:56:23.643861+0000 (oldest deadline 2024-06-27T21:56:49.543910+0000) 2024-06-27T21:56:51.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:51.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:51.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:51.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:51.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:51.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:51.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:51.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:51.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:51 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:51.199+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.192:6804 osd.8 since back 2024-06-27T21:56:23.643806+0000 front 2024-06-27T21:56:23.643861+0000 (oldest deadline 2024-06-27T21:56:49.543910+0000) 2024-06-27T21:56:52.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-06-27T21:55:19.734020+0000 front 2024-06-27T21:55:19.733937+0000 (oldest deadline 2024-06-27T21:55:49.734262+0000) 2024-06-27T21:56:52.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-06-27T21:55:32.035451+0000 front 2024-06-27T21:55:32.035376+0000 (oldest deadline 2024-06-27T21:55:53.735328+0000) 2024-06-27T21:56:52.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-06-27T21:55:38.937080+0000 front 2024-06-27T21:55:38.937174+0000 (oldest deadline 2024-06-27T21:56:02.437126+0000) 2024-06-27T21:56:52.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-06-27T21:55:47.439303+0000 front 2024-06-27T21:55:47.439236+0000 (oldest deadline 2024-06-27T21:56:12.139115+0000) 2024-06-27T21:56:52.544 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-06-27T21:55:55.640143+0000 front 2024-06-27T21:55:56.268041+0000 (oldest deadline 2024-06-27T21:56:20.940030+0000) 2024-06-27T21:56:52.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2024-06-27T21:56:00.940656+0000 front 2024-06-27T21:56:00.940708+0000 (oldest deadline 2024-06-27T21:56:26.840547+0000) 2024-06-27T21:56:52.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6820 osd.6 since back 2024-06-27T21:56:09.141648+0000 front 2024-06-27T21:56:09.141706+0000 (oldest deadline 2024-06-27T21:56:35.541947+0000) 2024-06-27T21:56:52.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.190:6828 osd.7 since back 2024-06-27T21:56:20.143396+0000 front 2024-06-27T21:56:20.143444+0000 (oldest deadline 2024-06-27T21:56:43.643368+0000) 2024-06-27T21:56:52.545 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:52 smithi192 ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11[54626]: 2024-06-27T21:56:52.159+0000 7fac7e891640 -1 osd.11 1037 heartbeat_check: no reply from 172.21.15.192:6804 osd.8 since back 2024-06-27T21:56:23.643806+0000 front 2024-06-27T21:56:23.643861+0000 (oldest deadline 2024-06-27T21:56:49.543910+0000) 2024-06-27T21:56:53.775 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:53 smithi192 podman[195458]: 2024-06-27 21:56:53.5173405 +0000 UTC m=+5.808046031 container died c02d61d821efd34ff8d6a9aca38570fbc9b5efe04dbb87abf668811dca7724b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-27T21:56:54.043 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:53 smithi192 podman[195458]: 2024-06-27 21:56:53.884019222 +0000 UTC m=+6.174724754 container cleanup c02d61d821efd34ff8d6a9aca38570fbc9b5efe04dbb87abf668811dca7724b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-06-27T21:56:54.044 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:53 smithi192 bash[195458]: ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11 2024-06-27T21:56:54.476 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195473]: 2024-06-27 21:56:54.168492135 +0000 UTC m=+0.646634483 container remove c02d61d821efd34ff8d6a9aca38570fbc9b5efe04dbb87abf668811dca7724b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240625, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, io.buildah.version=1.36.0) 2024-06-27T21:56:54.753 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195563]: 2024-06-27 21:56:54.477594184 +0000 UTC m=+0.113472228 container create a68e0b3ca197e45720a19c143a5697486bc0b42d23507493799f334c7021fd0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240625, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-27T21:56:54.753 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195563]: 2024-06-27 21:56:54.381379987 +0000 UTC m=+0.017258044 image pull 8adb0b2975c6552ca8b9f5896c86763e6883462970b60cd52e303937b47d9ecd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849 2024-06-27T21:56:54.754 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195563]: 2024-06-27 21:56:54.688544198 +0000 UTC m=+0.324422250 container init a68e0b3ca197e45720a19c143a5697486bc0b42d23507493799f334c7021fd0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11-deactivate, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1) 2024-06-27T21:56:54.754 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195563]: 2024-06-27 21:56:54.691838029 +0000 UTC m=+0.327716069 container start a68e0b3ca197e45720a19c143a5697486bc0b42d23507493799f334c7021fd0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240625, GIT_BRANCH=HEAD, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-27T21:56:54.754 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195563]: 2024-06-27 21:56:54.754932162 +0000 UTC m=+0.390810210 container attach a68e0b3ca197e45720a19c143a5697486bc0b42d23507493799f334c7021fd0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240625, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-27T21:56:55.044 INFO:journalctl@ceph.osd.11.smithi192.stdout:Jun 27 21:56:54 smithi192 podman[195563]: 2024-06-27 21:56:54.877680526 +0000 UTC m=+0.513558573 container died a68e0b3ca197e45720a19c143a5697486bc0b42d23507493799f334c7021fd0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9a0208cf1eaaadaea2238f88c55aad63973ad1d11f93ef18463ca6f02c168849, name=ceph-784a4e4c-34c8-11ef-bca8-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri12-testing-2024-06-26-0904-f51acc1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240625, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-27T21:56:55.505 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-784a4e4c-34c8-11ef-bca8-c7b262605968@osd.11.service' 2024-06-27T21:56:55.543 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-27T21:56:55.543 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-06-27T21:56:55.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --force --keep-logs 2024-06-27T21:56:55.749 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:56:57.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --force --keep-logs 2024-06-27T21:56:57.807 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:56:59.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --force --keep-logs 2024-06-27T21:56:59.895 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T21:57:01.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:57:01.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:57:01.502 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-27T21:57:01.528 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-27T21:57:01.530 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi151/crash 2024-06-27T21:57:01.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash -- . 2024-06-27T21:57:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash: Cannot open: No such file or directory 2024-06-27T21:57:01.582 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-06-27T21:57:01.584 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi190/crash 2024-06-27T21:57:01.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash -- . 2024-06-27T21:57:01.698 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash: Cannot open: No such file or directory 2024-06-27T21:57:01.698 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-06-27T21:57:01.701 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi192/crash 2024-06-27T21:57:01.702 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash -- . 2024-06-27T21:57:01.741 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/crash: Cannot open: No such file or directory 2024-06-27T21:57:01.741 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-06-27T21:57:01.742 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-27T21:57:01.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-27T21:57:01.844 INFO:teuthology.orchestra.run.smithi151.stdout:2024-06-27T21:20:00.000105+0000 mon.a (mon.0) 1522 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:57:01.845 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-27T21:57:01.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[SEC\]' /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-27T21:57:01.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]' /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-27T21:57:01.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[WRN\]' /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-06-27T21:57:02.034 INFO:teuthology.orchestra.run.smithi151.stdout:2024-06-27T21:20:00.000105+0000 mon.a (mon.0) 1522 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized 2024-06-27T21:57:02.035 INFO:tasks.cephadm:Compressing logs... 2024-06-27T21:57:02.035 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-27T21:57:02.077 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-27T21:57:02.080 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-27T21:57:02.119 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-06-27T21:57:02.119 INFO:teuthology.orchestra.run.smithi151.stderr: /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log 2024-06-27T21:57:02.121 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.2.log 2024-06-27T21:57:02.121 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log 2024-06-27T21:57:02.121 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log: /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.3.log 2024-06-27T21:57:02.121 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.1.log 2024-06-27T21:57:02.123 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mgr.y.log 2024-06-27T21:57:02.124 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log: /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.3.log: /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.0.log 2024-06-27T21:57:02.124 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.a.log 2024-06-27T21:57:02.124 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log 2024-06-27T21:57:02.124 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log 2024-06-27T21:57:02.124 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-27T21:57:02.130 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-27T21:57:02.130 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log 2024-06-27T21:57:02.130 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log 2024-06-27T21:57:02.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.c.log 2024-06-27T21:57:02.132 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.10.log 2024-06-27T21:57:02.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log: /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log 2024-06-27T21:57:02.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.11.log 2024-06-27T21:57:02.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.8.log 2024-06-27T21:57:02.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log 2024-06-27T21:57:02.134 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 85.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log.gz 2024-06-27T21:57:02.135 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.9.log 2024-06-27T21:57:02.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.8.log: /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-27T21:57:02.135 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-06-27T21:57:02.135 INFO:teuthology.orchestra.run.smithi190.stderr: -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log 2024-06-27T21:57:02.136 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log 2024-06-27T21:57:02.136 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.7.log 2024-06-27T21:57:02.136 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.5.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/objectstore_tool.2.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.4.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mgr.x.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.6.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.b.log 2024-06-27T21:57:02.138 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-27T21:57:02.193 INFO:teuthology.orchestra.run.smithi151.stderr: 82.5% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log.gz 2024-06-27T21:57:02.234 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 82.6% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log.gz 2024-06-27T21:57:02.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 82.5% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.cephadm.log.gz 2024-06-27T21:57:02.261 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log.gz 2024-06-27T21:57:02.280 INFO:teuthology.orchestra.run.smithi190.stderr: 93.5% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mgr.x.log.gz 2024-06-27T21:57:02.294 INFO:teuthology.orchestra.run.smithi190.stderr: 89.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log.gz 2024-06-27T21:57:02.295 INFO:teuthology.orchestra.run.smithi151.stderr: 89.8% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log.gz 2024-06-27T21:57:02.302 INFO:teuthology.orchestra.run.smithi190.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-27T21:57:02.303 INFO:teuthology.orchestra.run.smithi151.stderr: 95.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-27T21:57:02.361 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log.gz 2024-06-27T21:57:02.377 INFO:teuthology.orchestra.run.smithi192.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-27T21:57:02.397 INFO:teuthology.orchestra.run.smithi190.stderr: 85.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log.gz 2024-06-27T21:57:02.398 INFO:teuthology.orchestra.run.smithi192.stderr: 85.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.log.gz 2024-06-27T21:57:02.450 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph.audit.log.gz 2024-06-27T21:57:02.495 INFO:teuthology.orchestra.run.smithi151.stderr: 91.2% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-volume.log.gz 2024-06-27T21:57:03.193 INFO:teuthology.orchestra.run.smithi190.stderr: 94.4% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/objectstore_tool.2.log.gz 2024-06-27T21:57:04.253 INFO:teuthology.orchestra.run.smithi151.stderr: 91.3% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mgr.y.log.gz 2024-06-27T21:57:09.408 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.b.log.gz 2024-06-27T21:57:23.077 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.c.log.gz 2024-06-27T21:57:40.799 INFO:teuthology.orchestra.run.smithi151.stderr: 90.0% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-mon.a.log.gz 2024-06-27T22:06:05.682 INFO:teuthology.orchestra.run.smithi190.stderr: 93.3% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.6.log.gz 2024-06-27T22:16:41.731 INFO:teuthology.orchestra.run.smithi192.stderr: 92.8% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.8.log.gz 2024-06-27T22:16:57.233 INFO:teuthology.orchestra.run.smithi190.stderr: 91.7% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.4.log.gz 2024-06-27T22:18:51.460 INFO:teuthology.orchestra.run.smithi190.stderr: 92.2% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.5.log.gz 2024-06-27T22:20:09.168 INFO:teuthology.orchestra.run.smithi190.stderr: 91.6% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.7.log.gz 2024-06-27T22:20:09.170 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-06-27T22:20:09.170 INFO:teuthology.orchestra.run.smithi190.stderr:real 23m7.076s 2024-06-27T22:20:09.170 INFO:teuthology.orchestra.run.smithi190.stderr:user 8m27.288s 2024-06-27T22:20:09.170 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m24.607s 2024-06-27T22:24:45.337 INFO:teuthology.orchestra.run.smithi192.stderr: 92.0% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.10.log.gz 2024-06-27T22:25:01.413 INFO:teuthology.orchestra.run.smithi151.stderr: 91.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.1.log.gz 2024-06-27T22:25:07.053 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.9.log.gz 2024-06-27T22:25:25.184 INFO:teuthology.orchestra.run.smithi192.stderr: 92.6% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.11.log.gz 2024-06-27T22:25:25.218 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-27T22:25:25.218 INFO:teuthology.orchestra.run.smithi192.stderr:real 28m23.123s 2024-06-27T22:25:25.218 INFO:teuthology.orchestra.run.smithi192.stderr:user 9m5.219s 2024-06-27T22:25:25.219 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m27.441s 2024-06-27T22:25:25.345 INFO:teuthology.orchestra.run.smithi151.stderr: 92.3% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.2.log.gz 2024-06-27T22:27:50.757 INFO:teuthology.orchestra.run.smithi151.stderr: 91.9% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.3.log.gz 2024-06-27T22:30:44.459 INFO:teuthology.orchestra.run.smithi151.stderr: 91.1% -- replaced with /var/log/ceph/784a4e4c-34c8-11ef-bca8-c7b262605968/ceph-osd.0.log.gz 2024-06-27T22:30:44.485 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-06-27T22:30:44.485 INFO:teuthology.orchestra.run.smithi151.stderr:real 33m42.394s 2024-06-27T22:30:44.485 INFO:teuthology.orchestra.run.smithi151.stderr:user 11m53.436s 2024-06-27T22:30:44.485 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m32.930s 2024-06-27T22:30:44.486 INFO:tasks.cephadm:Archiving logs... 2024-06-27T22:30:44.487 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi151/log 2024-06-27T22:30:44.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-27T22:32:52.415 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi190/log 2024-06-27T22:32:52.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-27T22:34:20.322 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi192/log 2024-06-27T22:34:20.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-27T22:35:52.396 INFO:tasks.cephadm:Removing cluster... 2024-06-27T22:35:52.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --force 2024-06-27T22:35:53.620 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T22:35:54.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --force 2024-06-27T22:35:55.039 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T22:35:55.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 784a4e4c-34c8-11ef-bca8-c7b262605968 --force 2024-06-27T22:35:57.299 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 784a4e4c-34c8-11ef-bca8-c7b262605968 2024-06-27T22:35:58.162 INFO:tasks.cephadm:Removing cephadm ... 2024-06-27T22:35:58.163 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-27T22:35:58.182 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-27T22:35:58.293 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-27T22:35:58.310 INFO:tasks.cephadm:Teardown complete 2024-06-27T22:35:58.310 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-27T22:35:58.319 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-27T22:35:58.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-27T22:35:58.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-27T22:35:58.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-27T22:35:58.472 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-27T22:35:58.483 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-27T22:35:58.483 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T22:35:58.486 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T22:35:58.489 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-27T22:35:58.501 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-06-27T22:35:58.503 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-06-27T22:35:58.504 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-06-27T22:35:58.538 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-27T22:35:58.538 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-06-27T22:35:58.538 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 +825us[ +935us] +/- 43ms 2024-06-27T22:35:58.538 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 60 -858us[ -748us] +/- 58ms 2024-06-27T22:35:58.539 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 -194us[ -84us] +/- 84ms 2024-06-27T22:35:58.539 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T22:35:58.589 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-27T22:35:58.589 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-06-27T22:35:58.589 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +817us[ +883us] +/- 43ms 2024-06-27T22:35:58.590 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 -873us[ -807us] +/- 58ms 2024-06-27T22:35:58.590 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 56 -167us[ -167us] +/- 84ms 2024-06-27T22:35:58.590 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T22:35:58.600 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-27T22:35:58.600 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-06-27T22:35:58.601 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv01.front.sepia.ceph.com 3 7 377 62 +919us[ +966us] +/- 43ms 2024-06-27T22:35:58.601 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 126 -762us[ -714us] +/- 57ms 2024-06-27T22:35:58.601 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 67 -104us[ -57us] +/- 84ms 2024-06-27T22:35:58.601 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-06-27T22:35:58.614 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-27T22:35:58.628 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-27T22:35:58.629 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-27T22:35:58.797 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-27T22:35:59.009 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-27T22:35:59.117 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-27T22:35:59.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-27T22:35:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:35:59.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-27T22:35:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:35:59.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-27T22:35:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:35:59.351 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-27T22:35:59.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-27T22:35:59.413 INFO:teuthology.task.internal:Duration was 6032.583196 seconds 2024-06-27T22:35:59.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-27T22:35:59.465 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-27T22:35:59.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-27T22:35:59.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-27T22:35:59.471 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-27T22:35:59.523 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-27T22:35:59.526 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-27T22:35:59.557 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-27T22:35:59.896 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-27T22:35:59.896 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-06-27T22:35:59.896 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-27T22:35:59.921 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-06-27T22:35:59.922 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-27T22:35:59.954 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-06-27T22:35:59.955 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-27T22:35:59.987 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-27T22:35:59.987 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-27T22:35:59.994 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-27T22:36:00.010 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-27T22:36:00.064 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-27T22:36:00.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-27T22:36:00.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-27T22:36:00.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-27T22:36:01.745 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-27T22:36:01.755 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-27T22:36:01.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-27T22:36:01.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-27T22:36:01.836 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-27T22:36:01.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-27T22:36:01.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:36:01.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:36:01.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:36:01.938 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-06-27T22:36:01.940 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-06-27T22:36:01.963 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-06-27T22:36:01.988 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:36:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:36:02.011 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:36:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:36:02.027 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-27T22:36:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-27T22:36:02.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-27T22:36:02.055 INFO:teuthology.task.internal:Transferring archived files... 2024-06-27T22:36:02.057 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi151 2024-06-27T22:36:02.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-27T22:36:02.132 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi190 2024-06-27T22:36:02.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-27T22:36:02.178 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi/7775508/remote/smithi192 2024-06-27T22:36:02.178 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-27T22:36:02.224 INFO:teuthology.task.internal:Removing archive directory... 2024-06-27T22:36:02.224 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-27T22:36:02.226 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-27T22:36:02.229 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-27T22:36:02.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-27T22:36:02.285 INFO:teuthology.task.internal:Not uploading archives. 2024-06-27T22:36:02.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-27T22:36:02.329 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-27T22:36:02.330 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-27T22:36:02.332 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-27T22:36:02.334 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-27T22:36:02.345 INFO:teuthology.orchestra.run.smithi151.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 22:36 /home/ubuntu/cephtest 2024-06-27T22:36:02.347 INFO:teuthology.orchestra.run.smithi190.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 22:36 /home/ubuntu/cephtest 2024-06-27T22:36:02.352 INFO:teuthology.orchestra.run.smithi192.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 27 22:36 /home/ubuntu/cephtest 2024-06-27T22:36:02.353 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-27T22:36:02.412 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-27T22:36:02.549 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} duration: 6032.58319568634 failure_reason: '"2024-06-27T21:20:00.000105+0000 mon.a (mon.0) 1522 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-06-27T22:36:02.550 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-27T22:36:02.628 INFO:teuthology.run:FAIL